Candidate generation system

Abstract

This virtual space control system, which can also be called a candidate generation system, generates a candidate for an operation in a virtual space on the basis of an input inquiry, which is inquiry data input in a natural language. The virtual space control system comprises a request candidate generation unit that: extracts one or more request example records, which are combinations of an input inquiry and an operation request, on the basis of an input inquiry from a request example database that stores a plurality of request example records; uses the extracted request example record and the input inquiry to generate model input data that is text data; and generates one or more candidates for an operation request corresponding to the input inquiry by inputting the model input data to a language model.

Description

Candidate Generation System

[0001] The present invention relates to a candidate generation system.

[0002] By creating an environment that simulates the real world in a virtual space and consolidating and managing information on equipment operated and maintained in business operations, as well as business data generated daily, the efficiency of data accumulation and utilization is being improved. To achieve efficient operation in a virtual space that contains a huge amount of data, there is a need for technology that can present operation suggestions in the virtual space in response to queries from users in natural language. In other words, there is a need for a method to determine appropriate operation suggestions based on the content and context of the user's query. Patent Document 1 discloses a command processing program for causing a computer to realize a function of generating a command for executing an instruction to an operation target in a virtual space based on a user's input in natural language, the command processing program causing the computer to realize a text data acquisition function for obtaining text data based on the user's input in natural language, a syntax analysis function for extracting a command to be executed from the text data, a command analysis function for generating a primitive-type command from the command extracted by the syntax analysis function, a specific viewpoint information acquisition function for obtaining specific viewpoint information in the virtual space at least when the user performs an input operation in natural language, a command evaluation function for evaluating each option based on predetermined evaluation criteria and outputting the evaluation results of the primitive-type command generated by the command analysis function, and the command evaluation function for determining an option based on the evaluation result of the command evaluation function and determining a command, wherein the command evaluation function includes a function for evaluating each option in the primitive-type command using the specific viewpoint information acquired by the specific viewpoint information acquisition function.

[0003] Japanese Patent Application Publication No. 2019-63341

[0004] The invention described in Patent Document 1 leaves room for improvement in determining operations in virtual space.

[0005] A candidate generation system according to a first aspect of the present invention is a candidate generation system that generates operation candidates in a virtual space based on an input query, which is query data input in a natural language, and includes a request candidate generation unit that extracts one or more request example records, which are combinations of the input query and an operation request, from a request example database in which the request example records are stored, based on the input query, generates model input data, which is text data, using the extracted request example records and the input query, and inputs the model input data into a language model to generate one or more candidates for the operation request corresponding to the input query.

[0006] According to the present invention, operation candidates in a virtual space can be appropriately generated.

[0007] 1. Configuration diagram of a virtual space control system 2. Hardware configuration diagram of a virtual space control system 3. Diagram showing an example of a language model input template DB 4. Diagram showing an example of a request example DB 5. Diagram showing an example of a 3D model DB 6. Diagram showing an example of a file DB 7. Diagram showing an example of an operation DB 8. Diagram showing an example of a request candidate DB 9. Diagram showing an example of an operation history DB 10. Diagram showing an example of a user information DB 11. Flowchart showing an operation history registration process 12. Flowchart showing a user input acceptance process 13. Flowchart showing an operation history analysis process 14. Flowchart showing a document analysis process 15. Flowchart showing a request candidate generation process 16. Flowchart showing an operation determination process 17. Diagram showing an example of a user input screen 18. Diagram showing an example of input / output to a user interface

[0008] (Summary) This specification describes a system that generates operation candidates in a virtual space based on linguistic instructions from a user. By creating an environment that simulates the real world in a virtual space and consolidating and managing information on equipment operated and maintained in business operations, as well as business data generated daily, data accumulation and data utilization are made more efficient. Virtual spaces contain business data generated daily and a vast amount of 3D data used to simulate the real world, allowing users a high degree of freedom in operations. Therefore, in order to efficiently determine operations, a technology that generates operation candidates required by the user on the system side based on inquiries from the user in natural language is useful.

[0009] Furthermore, the following advantage can be obtained by estimating requirements not included in the query content based on the query content and the user's operation status at the time of the query. That is, even if the user does not provide detailed instructions or does not fully recognize the required operation, the system can complement the conditions and expand the search range for the operation content, thereby presenting a wide range of candidate operations required by the user, leading to improved efficiency. In this specification, instead of extracting candidate operations from the query sentence, the system expands the search range of the query based on case data and then narrows down the required operations, thereby generating candidate required operations taking into account even implicit requirements.

[0010] -First Embodiment- A first embodiment of a candidate generating system will be described below with reference to FIGS.

[0011] FIG. 1 is a configuration diagram of a virtual space control system 10. Since the virtual space control system 10 generates operation request candidates as described below, it can also be called a "candidate generation system." The virtual space control system 10 includes a request example registration unit 101, a request candidate generation unit 102, an operation determination unit 103, an operation history registration unit 104, a language model DB 105, a language model input template DB 106, a request example DB 107, a 3D model DB 108, a file DB 109, an operation DB 110, a request candidate DB 111, an operation history DB 112, and a user information DB 113. In this specification, databases are also referred to as "DBs."

[0012] The language model DB 105 stores at least one language model. Different language models may be used depending on the application, or only one language model may be used. The language model input template DB 106 stores multiple templates. When model input data, which is text data to be input to a language model, is created, one of the templates stored in the language model input template DB 106 is called. Appropriate values ​​are substituted for the variables in the called template to create the model input data.

[0013] The request example DB 107 stores request examples related to operations in the virtual space that correspond to combinations of user inquiries and operating conditions at the time of the inquiries. The 3D model DB 108 stores data related to 3D models registered in the virtual space. The file DB 109 stores data related to files such as business documents registered in the virtual space. The operation DB 110 stores data related to operations possible in the virtual space of the 3D viewer 115.

[0014] The request candidate DB 111 stores data of request candidates generated by the request candidate generator 102 or created in advance. The operation history DB 112 stores data representing user operations in the virtual space generated by the operation history registration unit 104 and the operation determination unit 103. The user information DB 113 stores data related to users operating the virtual space.

[0015] When the request example registration unit 101 receives an instruction for a "request example registration process" from the external system 11, it generates request example data and stores it in each database. The request example data is data indicating an example of an operation request, as will be described later. The instruction for the request example registration process is accompanied by at least one of user input data, data specifying a file registered in the virtual space control system 10, and operation history data. The user input data is data indicating a character string entered by the user and an option selected by the user. The user inputs the data using the UI 114. The operation history data is data indicating a history of operations performed by the user in the virtual space. The request example registration unit 101 uses the data attached to the request example registration process, the request candidate DB 111, the language model DB 105, the language model input template DB 106, and the user information DB 113.

[0016] The request example registration unit 101 includes an input analysis unit 1011, a history analysis unit 1012, and a document analysis unit 1013. The request example registration unit 101 activates any one of the input analysis unit 1011, the history analysis unit 1012, and the document analysis unit 1013 based on input from the external system 11. The input analysis unit 1011 analyzes user input data and registers new request examples in the request candidate DB 111. The history analysis unit 1012 analyzes operation history data and registers new request examples in the request candidate DB 111. The document analysis unit 1013 analyzes files registered in the virtual space control system 10 and registers new request examples in the request candidate DB 111.

[0017] When the request candidate generation unit 102 receives an instruction for an "operation determination process" from the external system 11, it generates "request candidate data" which is data indicating candidates for requests related to operations in the virtual space. The request candidate generation unit 102 refers to the language model DB 105, the language model input template DB 106, the request example DB 107, and the user information DB 113 stored in the language model DB 105.

[0018] The operation determining unit 103 selects an operation for each request candidate based on the request candidate data generated by the request candidate generating unit 102, aggregates the request candidate data to determine a final operation, and transmits the determined operation content to the external system 11. The operation determining unit 103 refers to the language model DB 105, the language model input template DB 106, the 3D model DB 108, the file DB 109, and the operation DB 110 in addition to the request candidate data.

[0019] When the operation history registration unit 104 receives an instruction to perform an "operation history registration process" from the external system 11, it registers the user's operation details in the virtual space as operation history data in the operation history DB 112. In other words, the operation history data is data indicating the history of operations performed by the user in the virtual space. The external system 11 includes a UI 114 having an interface with the virtual space control system 10, and a 3D viewer 115 having the function of displaying and operating the virtual space.

[0020] When text data is input, the language model included in the language model DB 105 estimates an appropriate word string to be output. The language model has the function of learning the structure and rules of natural language and interpreting and generating text data. Text data is a text written in natural language itself, or data indicating a text written in natural language. For example, when a question is given as model input data, an answer to that question is output. In recent years, deep learning technology has made remarkable progress in the field of natural language processing, and large-scale language models with an extremely large number of parameters constituting the language model have emerged, realizing advanced language processing functions close to those of humans.

[0021] Furthermore, there are known examples of large-scale language models that use text data called prompts as input. The prompts contain instructions for the task to be performed and information about the task. By inputting the prompts into the large-scale language model, the large-scale language model can be made to perform the task. The model input data mentioned above is also a prompt.

[0022] In addition, some large-scale language models, such as ChatGPT, have their functions provided externally as an API service. APIs are often used when information related to the learning of a language model, including model parameters, is kept confidential. The language processing API server 120, indicated by a dashed line in FIG. 1, is a server that processes APIs related to language models. The virtual space control system 10 may use the language processing API server 120 instead of the language model DB 105, and the results will be the same regardless of which is used. In other words, the language processing API server 120 includes a language model similar to that of the language model DB 105. However, since there is no essential difference between whether the virtual space control system 10 includes a language model or uses an API, the following description will only cover the case where the virtual space control system 10 includes a language model.

[0023] 2 is a hardware configuration diagram of the virtual space control system 10. The virtual space control system 10 includes a processor 1801, a storage device 1802, a monitor 1803, a DRAM 1804, an input device 1805, and a NIC 1806. The processor 1801 is a central processing unit. The storage device 1802 is a non-volatile data storage device, such as a hard disk drive or flash memory. The monitor 1803 is, for example, a liquid crystal display. The DRAM 1804 is a volatile data storage device capable of high-speed reading and writing. The input device 1805 is a device that accepts human input, such as a mouse or keyboard. The NIC 1806 is a network interface card that enables the virtual space control system 10 to communicate with other devices.

[0024] The processor 1801 implements the request example registration unit 101, the request candidate generation unit 102, the operation determination unit 103, and the operation history registration unit 104 by loading a program stored in the storage device 1802 into the DRAM 1804 and executing it. However, the virtual space control system 10 may be implemented by a field programmable gate array (FPGA), which is a rewritable logic circuit, or an application specific integrated circuit (ASIC), which is an application specific integrated circuit, instead of the combination of the processor 1801, the storage device 1802, and the DRAM 1804. Furthermore, the virtual space control system 10 may be implemented by a combination of different configurations, for example, a combination of the processor 1801, the storage device 1802, the DRAM 1804, and an FPGA, instead of the combination of the processor 1801, the storage device 1802, and the DRAM 1804.

[0025] The storage device 1802 stores a language model DB 105, a language model input template DB 106, a request example DB 107, a 3D model DB 108, a file DB 109, an operation DB 110, a request candidate DB 111, an operation history DB 112, and a user information DB 113. However, it is not essential that all of the above-mentioned databases be stored in the storage device 1802; the storage device 1802 may be configured to be able to access databases external to the virtual space control system 10 via the NIC 1806. In other words, it is sufficient for the virtual space control system 10 to be able to access each of the above-mentioned databases, and it does not matter whether the databases are built into the virtual space control system 10.

[0026] FIG. 3 is a diagram showing an example of the language model input template DB 106. The language model input template DB 106 stores templates used to create model input data. The language model input template DB 106 is composed of a plurality of records. Each record in the language model input template DB 106 has fields for a template ID 2021, an item 2022, and a template 2023. The template ID 2021 stores an identifier for the template. The template to be used for each process executed by the virtual space control system 10 is set in advance using the template ID 2021. The item name of the process that calls each template is stored in the item 2022.

[0027] The template 2023 stores a template of the content to be input to the language model. A template is a combination of text data in a natural language and a variable, for example, {query}. A character string that changes depending on the situation, such as the content of a user inquiry, is substituted into the variable. The programs that realize the request example registration unit 101 and the request candidate generation unit 102 pre-describe the correspondence between the variables in the program and the variables in the template. When the request example registration unit 101 and the request candidate generation unit 102 use a template stored in the language model input template DB 106, they generate model input data using this known correspondence.

[0028] 4 is a diagram showing an example of the request example DB 107. Each record constituting the request example DB 107 may be created in advance or may be created by the request example registration unit 101. The request example registration unit 101 creates the record using one of the following three methods. First, the request example registration unit 101 creates the record using data input by the user. Second, the request example registration unit 101 creates the record using operation history data in the virtual space specified by the user. Third, the request example registration unit 101 creates the record using the description of a document file specified by the user.

[0029] The request example DB 107 stores case data of operation requests, such as examples of requests related to operations in virtual space, taking into account not only the content of inquiries from users but also the operation status. The operation status when responding to an inquiry is one or more operation histories prior to the inquiry. The request example data is a collection of operation request cases, and when actually generating an operation request from a user, this collection of cases is used to generate an operation request from the content of the user's inquiry and the operation status at that time.

[0030] The request example DB 107 is made up of multiple records, and each record has fields for a request example ID 3011, a query 3012, an operation status 3013, a request 3014, a user ID 3015, and a user attribute 3016. Either the query 3012 or the operation status 3013 may be "NULL." In other words, request example data may be stored to accommodate cases where an operation request is interpreted from only the query content or from only the operation status. Hereinafter, the query 3012 and the operation status 3013 are collectively referred to as an "input query."

[0031] The request example ID 3011 stores an identifier for identifying the request example data. The query 3012 stores the query content input by the user as an operation instruction in the virtual space. However, if there is no input from the user, "NULL" is stored. The operation status 3013 stores the operation status at the time of the query written in natural language, that is, the action before the query. However, if there is no user action before the query, "NULL" is stored.

[0032] For example, if the user "started editing the inspection checklist for facility X" immediately before submitting the query, "started inspection check for facility X" or "started editing the inspection checklist for facility X" is stored in the operation status 3013. The operation status 3013 may be created based on two or more operation histories immediately before the query. If the purpose of the operations can be interpreted from multiple operation histories, the contents of those operations may be recorded in the operation status. For example, assume that the most recent operation histories are "create a report on the driver's seat brake handle" and "register image data of the driver's seat air conditioning equipment in the system." These two histories may be input into a language model in natural language, and the language model may infer and store, for example, "performing inspection work on the driver's seat equipment" in the operation status 3013.

[0033] Instead of describing a natural language in the operation status 3013, data for identifying a record in the operation history DB 112, such as an operation ID 6024 described below, may be included. If the operation ID 6024 is included in the operation status 3013, when using this operation status 3013, the 3D model DB 108, file DB 109, operation DB 110, and user information DB 113 corresponding to the described operation ID 6024 are referenced. Then, specific contents of the operation history may be acquired, and further text data of the operation status may be created using a language model as described above and then used.

[0034] The request 3014 stores natural language indicating a group of user operation requests in the virtual space corresponding to the query 3012 and the operation status 3013. For example, if the query 3012 is "I would like to carry out an inspection check of the air conditioning equipment," the request includes operation requests for the system in the virtual space, such as starting to edit an inspection checklist, checking the state of the equipment before inspection, and checking a report for defects related to the inspection target. By storing such requests in the request example DB 107 and referring to them when generating request candidates, the search range for operation requests can be widened, and a wide range of appropriate operations can be selected.

[0035] The user ID 3015 and user attribute 3016 store user information related to the corresponding request example. These correspond to the user ID 7011 and attribute 7013 in the user information DB 113. When generating a request candidate, these are used to select a record that is highly relevant to the user by comparing it with the information of the user who is the target of the operation determination process. The user ID 3015 and user attribute 3016 may be "NULL." If a value other than "NULL" is stored in the user ID 3015, the corresponding user attribute 3016 is stored based on the user information DB 113.

[0036] 5 is a diagram showing an example of the 3D model DB 108. The 3D model DB 108 stores data prepared in advance. The 3D model DB 108 stores data related to 3D models registered in the virtual space. Specifically, the 3D model DB 108 stores the names and coordinates of the 3D models in the virtual space, as well as information about parent objects. The 3D model DB 108 is made up of multiple records, and each record has fields for an object ID 4011, a name 4012, coordinates 4013, and a parent object ID 4014.

[0037] An identifier for identifying the 3D model is stored in the object ID 4011. The name 4012 stores the name of the 3D model. The object ID 4011 is a unique value, and the same value is not set for any record. The same value is set for the name 4012 for models of the same type. The coordinates 4013 store the coordinate values ​​of the 3D model in virtual space. The coordinate values ​​stored in the coordinates 4013 may be a single representative point such as the center or center of gravity of the 3D model, or may be the coordinate values ​​of all vertices of the 3D model so that the shape of the 3D model can be precisely determined. Any coordinate system may be used, such as a Cartesian coordinate system. The coordinates 4013 may further include Euler angles that indicate the orientation of the 3D model, such as (30 degrees, 60 degrees, 100 degrees).

[0038] The parent object ID 4014 stores an array of object IDs 4011 of other 3D models that are parent objects of the 3D model. For example, consider a case where there is a 3D model A (object ID = 100) that represents the first car of a certain railroad vehicle and a 3D model B that represents a seat located within the space of the first car. In this case, the parent object of 3D model B is 3D model A, and the object ID = 100 of 3D model A is stored as an element of the array of parent object IDs of 3D model B. There may be multiple parent objects, and the parent object IDs are held as a list. The parent-child relationship of 3D objects is generally defined in a game program or the like. In this embodiment, for example, the program of the 3D viewer 115 that displays the virtual space acquires data regarding the parent-child relationship of objects.

[0039] By having data on the parent-child relationship of objects, for example, if a user inquires, "I want to check the automatic door in the first car," it can be determined that the user's interest is in the automatic door "in the first car." This contributes to improving the inference accuracy when selecting operation target data from the operation request in the processing of the operation determination unit 103. If a parent object does not exist, "NULL" may be stored to indicate that it does not exist.

[0040] 6 is a diagram showing an example of the file DB 109. The file DB 109 stores data prepared in advance. The file DB 109 stores data related to files such as business documents registered in the virtual space. The file DB 109 stores information on file names, types, and related 3D model objects related to files such as text data and image data registered in the system. The file DB 109 is made up of multiple records, and each record has fields for a file ID 4021, a name 4022, a type 4023, and a related object ID 4024.

[0041] The file ID 4021 stores a file identifier. The name 4022 stores the name of the file. The type 4023 stores data indicating the type of file. The type 4023 serves as a tag used to specify data to be operated on in the target data 5013 of the operation DB 110, which will be described later. Examples of the type 4023 include "text," "checklist," and "image." The related object ID 4024 stores the object ID 4011 of the 3D model to which the file is related.

[0042] Including the related object ID 4024 in the file DB 109 has the following advantages, for example: When a user inquires, "I would like to check the report on the automatic door in the first car," it is possible to identify that the file of interest is a file of a report on "the automatic door in the first car." This contributes to improving the inference accuracy when selecting operation target data from an operation request in the processing of the operation determination unit 103. When the related object ID 4024 is not included in the file DB 109, "NULL" may be stored to indicate that no related object exists. Furthermore, when there are multiple objects related to a certain file, the identifiers of the multiple objects may be described in list format.

[0043] 7 is a diagram showing an example of the operation DB 110. The operation DB 110 stores data prepared in advance. The operation DB 110 stores data related to operations possible in the virtual space of the 3D viewer 115, specifically, the name of each operation item, the range of data to be operated, a description of the operation content, and data on whether or not the item can be selected when inquired. Whether or not the item can be selected when inquired about an operation instruction from the user. The operation DB 110 is made up of multiple records, and each record has fields for an operation ID 5011, a name 5012, target data 5013, a description 5014, and a selection target 5015.

[0044] The operation ID 5011 stores an identifier of the operation item. The name 5012 stores the name of the operation item. The target data 5013 stores the target of the operation item. For example, the "open file" operation with operation ID = 2 in Figure 7 is an operation that targets data in the file DB 109, so "file DB" is entered in the target data 5013. Furthermore, the "start editing text file" operation with operation ID = 3 in Figure 7 targets only text data in the file DB 109.

[0045] Therefore, the target data 5013 describes "file DB; type=text", clearly indicating that only data in the file DB 109 whose type 4023 is "text" is the target. As will be described later, the target data 5013 is used by the operation determining unit 103 to ensure consistency between the selected operation item and the data to be operated. However, for operation items for which no target data exists, such as "change illuminance", "NULL" is stored.

[0046] The description 5014 stores an explanation of the operation item. When the operation determining unit 103 selects an operation item from an operation request, the description 5014 is provided to the language model as information for determining whether the operation item should be selected. Based on this assumption, the example shown in Fig. 7 clearly indicates the timing for selecting an operation item. However, the description 5014 may also include other descriptions that explain the contents of the operation item.

[0047] For example, for the "move viewpoint to 3D model" operation, it is appropriate to select it when the request is "to see a 3D model," so the description 5014 would read "select when you want to see a 3D model." Here, for operation items that are not to be selected in the operation determination process, such as the "utterance reception" operation, "NULL," indicating that there is no description, may be stored. The "utterance reception" operation is an operation in which the system receives the content of what the user has said in the virtual space, regardless of whether it is a query, and records it in the operation history. This operation history is used to estimate the operation status, etc.

[0048] The selection target 5015 stores data indicating whether the corresponding operation item is a selection target in the operation determination process. An example of an item for which the operation target is set to False is the aforementioned "utterance reception" operation. This is because the "utterance reception" operation is an operation that is automatically performed by the system, and is not an operation that provides the user with the result of the operation determination.

[0049] FIG. 8 is a diagram showing an example of the request candidate DB 111. The request candidate DB 111 stores data of request candidates generated by the request candidate generator 102, as described below. However, data of request candidates created in advance may also be stored in the request candidate DB 111. The request candidate DB 111 is made up of multiple records, and each record has fields for a request candidate ID 6011, an inquiry ID 6012, a time 6013, a user ID 6014, an inquiry 6015, an operation status 6016, and a request candidate 6017. Hereinafter, the inquiry 6015 and the operation status 6016 will be collectively referred to as an "input query." Hereinafter, each record in the request candidate DB 111 will also be referred to as a "request example record."

[0050] The request candidate ID 6011 stores an identifier of the generated request candidate. The inquiry ID 6012 stores an identifier that identifies the inquiry data when the trigger for the operation determination process is a query issued by the user. As will be described later, the trigger for the operation determination process may be scheduled in advance, for example, at regular intervals. In this case, "NULL" is stored in the inquiry ID 6012. The time 6013 stores data that indicates the time when the request candidate generation process was executed.

[0051] The user ID 6014 stores an identifier that identifies the user who is the inferred target of the operation request. The user ID 6014 corresponds to the user ID 7011 in the user information DB 113. The inquiry 6015 stores inquiry data received from the user. The operation status 6016 stores data on the operation status read from the user's most recent operation history by the request candidate generation unit 102. The request candidate 6017 stores one or more request candidates generated by the request candidate generation unit 102 in list form.

[0052] 9 is a diagram showing an example of the operation history DB 112. The operation history DB 112 stores data representing user operation details in the virtual space, which data is generated by the operation history registration unit 104 and the operation determination unit 103. The operation history DB 112 is made up of a plurality of records, and each record has fields for an operation history ID 6021, a time 6022, a user ID 6023, an operation ID 6024, an object ID 6025, a file ID 6026, supplemental information 6027, and an inquiry ID 6028.

[0053] The operation history ID 6021 stores an identifier for identifying the operation history. The time 6022 stores information about the time when the operation was performed in the virtual space. The user ID 6023 stores an identifier of the user who performed the operation. The user ID 6023 corresponds to the user ID 7011 in the user information DB 113. The operation ID 6024 stores an identifier of the performed operation. The operation ID 6024 corresponds to the operation ID 5011 in the operation DB 110.

[0054] Identifiers that identify the object and file that are the target of the operation are stored in the object ID 6025 and file ID 6026. The object ID 6025 corresponds to the object ID 4011 in the 3D model DB 108, and the file ID 6026 corresponds to the file ID 4021 in the file DB 109. If the operation does not involve a corresponding 3D model or file, "NULL" is stored in the object ID 6025 and file ID 6026.

[0055] The supplemental information 6027 stores data related to operations other than the operation item and the data to be operated. For example, in order to record the content of the user's utterance for the "utterance reception" operation, "utterance = (utterance content)" or the like is stored in the supplemental information 6027. If there is no corresponding data in the supplemental information 6027, "NULL" is stored. The inquiry ID 6028 indicates a number specifying the inquiry data when the operation is performed in response to an inquiry from the user. The inquiry ID 6028 corresponds to the inquiry ID 6012 of the request candidate DB 111. If the operation is not in response to an inquiry, there is no corresponding inquiry data, so "NULL" is stored.

[0056] 10 is a diagram showing an example of the user information DB 113. The user information DB 113 stores data prepared in advance. The user information DB 113 stores data related to users who operate the virtual space. The user information DB 113 is made up of multiple records, and each record has fields for a user ID 7011, a name 7012, and an attribute 7013.

[0057] An identifier for identifying a user is stored in the user ID 7011. The user's name is stored in the name 7012. The attribute 7013 stores the user's attribute data. For example, if the virtual space control system 10 is used for business support purposes, the business position and job type are stored in the attribute 7013.

[0058] 11 is a flowchart showing the operation history registration process executed by the operation history registration unit 104. The operation history registration process is initiated when the external system 11 transmits data on the operation content when a user performs an operation in the virtual space on the 3D viewer 115, and the virtual space control system 10 receives the data on the operation content. The operation history registration process converts the data on the operation content transmitted from the external system 11 so as to associate it with the operation DB 110, the 3D model DB 108, and the file DB 109, and stores the data in the operation history DB 112.

[0059] In step S331, the operation history registration unit 104 stores the data of the operation content received from the external system 11 in the operation history DB 112, and ends the processing shown in Fig. 11 . Specifically, the operation history registration unit 104 acquires data corresponding to the time, user ID, operation ID, object ID, file ID, and supplemental information of the operation history DB 112 from the data of the operation content, and stores the data in the operation history DB 112. At this time, the user ID, operation ID, object ID, and file ID are acquired so as to correspond to the data stored in the user information DB 113, operation DB 110, 3D model DB 108, and file DB 109, respectively. If the operation content registered in the operation history registration processing is not in response to an inquiry, "NULL" is stored in the inquiry ID 6028.

[0060] 12 is a flowchart showing a user input reception process executed by the request example registration unit 101. First, in step S341, the request example registration unit 101 creates model input data using request example data input by the user and received from the external system 11 and the language model input template DB 106.

[0061] Specifically, first, the record having the template ID 2021 of "0" is read from the records in the language model input template DB 106. Then, the request example registration unit 101 creates input data by substituting the values ​​of the variables specified in the template 2023. In this process, the user's inquiry, the operation status, and text data describing the request are substituted. The input data includes instructions such as "based on the user's inquiry, the operation status, and the content of the request, delete irrelevant requests and improve vaguely expressed requests by making them more specific, and create a revised request," as well as the input user's inquiry, operation status, and request description.

[0062] Here, instructions such as "delete requests that appear irrelevant" and "specify requests that are ambiguously expressed" are described, but these are intended to correct the request example data expected by the user, and instructions with similar purposes may include other content. Furthermore, to improve the inference accuracy of the language model, sample data of user-input request examples and corrected request examples may be prepared in advance and added to the model input data. Inference accuracy can be improved by including several examples corresponding to the inference task in the model input data.

[0063] In the following step S342, the request example registration unit 101 inputs the model input data created in step S341 into the language model, generates corrected request example data, and stores it together with the user information data in the request example DB 107, thereby completing the processing shown in Fig. 12. The user information data is data corresponding to a user ID 160205 and a user attribute 160206 on the UI, which will be described later.

[0064] As described in the description of step S341, the language model generates correction data for the request description based on an instruction to correct the description of the request entered by the user. For example, the processing when an instruction to "delete requests deemed unrelated" is included will be described. Assume that the contents of the request example data entered by the user include a query such as "I would like to perform an inspection check of the air conditioning equipment" with no description of the operation status, and the operation request entered by the user includes an item such as "I would like to see a beer server." In this case, the request "I would like to see a beer server" is deleted as an incorrect input because it is unrelated to the content of the query. However, because there is a possibility that the language model may infer that a correct request is an incorrect input, the correction content may be displayed to the user on UI 114 before storing the data in request example DB 107, allowing the user to confirm whether to accept the correction content.

[0065] Furthermore, for example, the processing when an instruction to "specify the request description" is included is as follows. Suppose the contents of the request example data entered by the user include a query such as "I would like to perform an inspection check of the air conditioning equipment" with no description of the operation status, but an item such as "I would like to see related equipment" as an operation request. In this case, the expression "related equipment" is ambiguous and difficult to interpret, so the specific content can be inferred from the query content and revised to read "I would like to see the air conditioning equipment." After such a revision of the request description is made, the portion corresponding to the revised request description is extracted from the output content of the language model, and stored in the request example DB 107 together with the query content, the operation status, and user information data.

[0066] 13 is a flowchart showing the operation history analysis process executed by the history analysis unit 1012 of the request example registration unit 101. Before the operation history analysis process starts, a designated user to be analyzed is set in advance. First, in step S351, the history analysis unit 1012 reads all operation histories of the designated user from the operation history DB 112. Note that the number of designated users may be one or more. In the following step S352, the history analysis unit 1012 selects one unprocessed record from the records read in step S352 as the record to be analyzed.

[0067] In the following step S353, the history analysis unit 1012 determines whether the record to be analyzed was an operation in response to an inquiry. If it was not an operation in response to an inquiry, it means that it was an operation performed by the user himself. This determination can be made based on the value of the inquiry ID 6028 in the operation history DB 112. If the inquiry ID 6028 is not "NULL," the history analysis unit 1012 determines that it was an operation performed in response to an inquiry. If the inquiry ID 6028 is "NULL," the history analysis unit 1012 can determine that it was an operation performed by the user himself, not in response to an inquiry. If the history analysis unit 1012 determines that it was an operation in response to an inquiry, it proceeds to step S355, and if it determines that it was not an operation in response to an inquiry, it proceeds to step S354.

[0068] In step S354, the history analysis unit 1012 creates request example data using the record to be analyzed, the language model input template DB 106, and the language model, and then proceeds to step S356. A specific example of step S354 will be described later. In step S355, the history analysis unit 1012 infers a missing request, which is an omission of an operation request, based on the user's behavior after the inquiry in the record to be analyzed, i.e., the user's behavior recorded in the operation history DB 112, or a new inquiry, and then proceeds to step S356. A specific example of step S355 will be described later.

[0069] In step S356, the history analysis unit 1012 stores the data created in step S354 or step S355 in the request example DB 107. At this time, the history analysis unit 1012 stores information about the user selected for analysis in the fields of the user ID 3015 and the user attribute 3016. In the following step S357, the history analysis unit 1012 determines whether all of the records read in step S351 have been analyzed. For example, in step S352, the history analysis unit 1012 assigns a flag to the operation history DB 112 as temporary data indicating that the record has been selected as the record to be analyzed, and determines whether the flag has been assigned to all of the read records. If the history analysis unit 1012 determines that all of the read records have been processed, it ends the processing shown in FIG. 13 . If it determines that unprocessed records exist, it returns to step S352.

[0070] A specific example of step S354 will be described. The history analysis unit 1012 first creates model input data using the record to be analyzed and a template 2023 in the language model input template DB 106 whose template ID 2021 is "1". The model input data is created by substituting values ​​for variables specified in this template 2023. In this process, operation content data described from the operation item, operation target data, and supplemental information 6027 described in the selected operation history data is substituted.

[0071] The operation item described in the selected operation history data can be identified by reading the name 5012 based on the operation ID 5011 in the operation DB 110 that corresponds to the operation ID 6024 in the operation history DB 112. The data to be operated can be identified by reading the name 4012 in the 3D model DB 108 and the name 4022 in the file DB 109 based on the object ID 4011 in the 3D model DB 108 and the file ID 4021 in the file DB 109 that correspond to the object ID 6025 and file ID 6026 in the operation history DB 112, respectively.

[0072] For example, suppose an "open file" operation is performed on the file "Automatic door D1_20230715_parts repair report.txt." In this case, the operation history would include a description such as "An operation to open the file Automatic door D1_20230715_parts repair report.txt was performed." This can be achieved by setting the template to "{operation history} operation was performed on {operation target data}."

[0073] In addition, to interpret the operation status when the target operation was performed, one or more operation history data of the user performed immediately before the target operation is also read and substituted into the model input data. The operation status may include a predetermined number of immediately preceding actions, or may include actions during a predetermined immediately preceding period. For example, assume that an operation history of "started editing the checklist for automatic door D1_20230730_inspectionchecklist.json" is obtained from the immediately preceding operation history. In this case, the content "started editing the inspection checklist for automatic door D1" is read as the operation status.

[0074] Furthermore, as mentioned above, if the purpose of an operation can be inferred from multiple operation histories, the inferred purpose may be used as the operation status. For example, if the most recent operation histories are all related to the inspection work of the driver's seat of a railway vehicle, it can be interpreted as "inspecting the driver's seat equipment" and inferred into the language model. Examples of inspection work of the driver's seat of a railway vehicle include "creating a report on the brake handle of the driver's seat" and "registering image data of the air conditioning equipment of the driver's seat in the system."

[0075] Model input data is created using the operation content data created in this way and data on one or more operations performed immediately before the target operation. The model input data includes instructions such as "Describe the situation when the operation was performed and the request for the operation based on the user's operation content and the immediately preceding operation content," as well as the operation content of the target record and one or more operations performed immediately before that. To improve the inference accuracy of the language model, sample data of pairs of operation situations to be generated and one or more request examples for the target operation content and one or more operations performed immediately before the operation may be prepared in advance and added to the input data. Including several examples corresponding to the inference task in the model input data can improve inference accuracy.

[0076] Next, the aforementioned model input data is input into the language model to generate request example data. For example, if the target operation content is "An operation to open the file automatic door D1_20230715_parts_repair_report.txt" and the date of the operation, e.g., the 30th of the same month, is also used to infer a request example such as "I want to open the most recent inspection report for automatic door D1." One or more request examples may be created. Since this process does not target the history of operations corresponding to the query content, there is no corresponding query data, and "NULL" is stored for the query content. However, the created request example data may be presented to the user via UI 114 before storing the data in request example DB 107, and registered in request example DB 107 only after the user's confirmation.

[0077] Although an example in which the operation status and operation request are inferred simultaneously has been described here, it is also possible to infer either one first. For example, it is also possible to first output the text of the operation status from the immediately preceding operation history data, and then output the operation request from the target operation content and the data of the operation status that has already been output. Generally, it is difficult to infer multiple tasks simultaneously, and there is a possibility that inference accuracy will be improved by breaking down the tasks.

[0078] Furthermore, the operation status data may store the contents of the operation history data as is. In this case, only some fields in the record, specifically the operation ID 6024, object ID 6025, file ID 6026, and supplemental information 6027, may be extracted and included. If the operation status data is required to be in text format when using the data in the request example DB 107, the following approach can be considered. That is, the specific contents of the operation history are acquired from the ID information of each item registered as described above by referencing the 3D model DB 108, file DB 109, operation DB 110, and user information DB 113. Furthermore, text data of the operation status may be created from the operation history using a language model as described above before use.

[0079] In this way, by analyzing the operation status and user requests based on the past user operation history and storing them as request example data, more appropriate inferences can be made when a query is received from a user in a similar operation status in the future.

[0080] A specific example of step S355 will be described. First, the history analysis unit 1012 creates model input data using the operation history data selected in step S351 and the template 2023 having a template ID 2021 of "2" in the language model input template DB 106. Then, the history analysis unit 1012 creates input data by substituting values ​​of variables specified in the template 2023.

[0081] In this process, for the selected operation history data, the corresponding request candidate data and the operation history data immediately after the query are acquired based on the time information, based on the query ID 6028 in the operation history DB 112 that corresponds to the query ID 6012 in the request candidate DB 111. Then, the query, the operation status, the generated request candidate, and the operation content immediately after the query are substituted into the model input data.

[0082] The model input data includes instructions such as, "Describe the missing requirements based on the user's query, the operational status at the time of the query, the candidate requirements inferred by the system in response to the query, and the operations performed by the user after the query."Then, the query, operational status, candidate requirements data, and the operations performed after the query are further input into the model input data.

[0083] Furthermore, to improve the inference accuracy of the language model, sample data of a set of one or more example requests to be generated by the language model for the target query content and operation situation, and for the candidate request and the operation content immediately afterwards may be prepared in advance and added to the input data. Including several examples corresponding to the inference task in the model input data can improve the inference accuracy.

[0084] Next, the history analysis unit 1012 inputs the model input data described above into the language model and generates request example data. For example, assume that the query is "I would like to perform an inspection check of the automatic door," the operation status is "I have finished editing the inspection checklist for lamp L102," and the generated request candidate is "I would like to start editing the inspection checklist for the automatic door." If the operation content immediately after this is "I performed an operation to open the file automatic door D1_20230715_partsrepairreport.txt," the following inference can be made. That is, it can be inferred that there is a request not only to start editing the checklist when performing an inspection check, but also to check the most recent report for the equipment being inspected, for example, from the past few days.

[0085] Therefore, it is inferred that the request "I want to open the latest inspection report for the automatic door" has been omitted from the request candidates, and is added to the request example DB 107 as request example data together with the above-mentioned inquiry and operation status data. Furthermore, if there is no corresponding request example as a result of the inference, no request example data is added. Furthermore, two or more request examples may be created. However, before storing the data in the request example DB 107, the created request example data may be presented to the user using the UI 114, and registered in the request example DB 107 only after the user's confirmation.

[0086] In this way, by analyzing the missing content of the operation request inferred for the inquiry based on the history of past inquiries and storing it as request example data, it becomes possible to infer the request that was missed in the past when a similar inquiry is received in the future. This concludes the explanation of a specific example of step S355.

[0087] 14 is a flowchart showing the document analysis process executed by the document analysis unit 1013 of the request example registration unit 101. First, in step S361, the document analysis unit 1013 reads a specified file (hereinafter referred to as a "specified file") using the information specifying the file to be analyzed received from the external system 11 and the file DB 109. The number of specified files is not limited to one, and may be two or more. The specified file is a file written in a natural language, such as a text file or a checklist file.

[0088] In the following step S362, the document analysis unit 1013 extracts sentences related to operations in the virtual space (hereinafter referred to as "operation description sentences") based on the specified file read in step S361, the operation DB 110, and the language model DB 105. A specific example of step S362 will be described later. In the following step S363, the document analysis unit 1013 creates model input data using the operation description sentences extracted in step S362, and inputs this model input data into the language model DB 105 to obtain a request example output. A specific example of step S363 will be described later. In the following step S364, the document analysis unit 1013 stores the request example generated in step S363 in the request example DB 107, and ends the processing shown in FIG. 14.

[0089] A specific example of step S362 is as follows. For example, suppose the specified file is an inspection procedure manual for a certain facility X, and the description of a certain procedure includes the following sentence S, "Perform inspections according to the checklist below." This sentence S is analyzed using a language model along with the name 5012 and description 5014 of each record in the operation DB 110, to determine whether the work described in this sentence involves an operation in virtual space. In the example of the operation DB 110 shown in FIG. 7, this sentence S is determined to correspond to "Start editing the checklist," whose operation ID 5011 is "3," and this sentence S is extracted as an operation description sentence.

[0090] A second example of step S362 will be described. For example, suppose the specified file contains the following sentence T, "Refer to the equipment X manual when performing this task." In this case, analysis of the language model determines that "open a file" with operation ID 5011 of "2" is the appropriate sentence, and this sentence T is extracted as an operation description sentence. By performing a similar analysis on all sentences in the specified file, sentences involving operations in virtual space can be extracted from the specified file.

[0091] The analysis of the sentences may be performed by calculating the relevance between the content of the sentence and the operation items using a known search algorithm, and extracting documents with high relevance as operation description sentences. Alternatively, task instructions for the language model may be written as sentences and input to the language model to determine whether they should be extracted. For example, the input data may include an instruction such as "determine whether the following sentence S involves any item in the following operation list," the sentence to be analyzed, and the name 5012 and description 5014 data of each record in the operation DB 110. While the example described here extracts sentences by referencing operation items registered in the operation DB 110, it is also possible to extract sentences related to a wide range of tasks using known techniques. This concludes the description of a specific example of step S362.

[0092] A specific example of step S363 is as follows. First, the document analysis unit 1013 creates model input data based on the sentence data extracted in step S362, the file data in which this sentence is written, and the language model input template DB 106. From the language model input template DB 106, a record with a template ID 2021 of "3" is read. Then, input data is created by substituting the values ​​of the variables specified in the template 2023. The variables to be substituted are the extracted sentence data and related data from which the situation when performing the operation described in the sentence can be inferred.

[0093] This related data refers to sentences written in a specified range before and after the operation description sentence in the specified file, or sentences containing the content indicated by the demonstrative pronouns in the operation description sentence. The related data may also include sentences that are presumed to be closely related to the extracted sentence, such as sentences extracted by evaluating the degree of relevance between sentences using a known search algorithm. Furthermore, the related data may include the name of the specified file and data summarizing the contents of the specified file.

[0094] For example, as in the example presented in step S362, assume that the designated file is an inspection procedure manual for equipment X, and the operation description sentence is "Perform inspections according to the checklist below." In this case, the description in the designated file can be used to infer the general situation in which inspections for equipment X are being performed, and from the related data as described above, it can be read that "Inspection work for equipment X is being performed" is the operational situation.

[0095] The document analysis unit 1013 creates model input data using the operation description sentence and related data. By inputting this model input data into a language model, a request example is output. The model input data includes instructions such as "Describe a request for the operation you want to perform in the operation situation based on the sentence related to the operation in the virtual space and the related sentence described in the file," as well as the operation description sentence and related data. For example, an output example from the language model to which this model input data is input is an operation situation such as "Inspection work is being performed on facility X" and a request example such as "I would like to perform an inspection check on facility X."

[0096] Furthermore, if a device manual or other document contains a combination of expected questions and answers, such as the question "What should I do when xx occurs?" and the answer "Perform yy operation," it is possible to virtually extract not only the operation status but also the content of the inquiry. In this case, the language model may also output the content of the inquiry.

[0097] In order to improve the inference accuracy of the language model, the model input data may include a sample output and a sample input corresponding to this sample output. The sample input and sample output are data prepared in advance. The sample input is an example of an operation description sentence and related data. The sample output is character string data that is expected to be the output of the language model when the sample input is model input data. The inference accuracy can be improved by including several examples corresponding to the inference task in the model input data. This concludes the explanation of a specific example of step S363.

[0098] A supplement to step S364 will be described. The language model may output two or more request examples, in which case multiple request examples are stored in the request 3014 in step S364. If the model input data does not include query data, "NULL" is stored in the query 3012 of the request example DB 107 in step S364. Before storing the data in the request example DB 107, the created request example data may be displayed to the user using the UI 114, and the user may be asked whether or not to register the created data. This is a supplement to step S364. In this way, by storing combinations of operation situations and requests in the request example DB 107 based on the description of the specified file specified by the user, it is possible to suggest operations by referring to this data when receiving a query from a user in a similar operation situation.

[0099] 15 is a flowchart showing the request candidate generation process executed by the request candidate generation unit 102. When a user outputs a query to the virtual space control system 10 using the UI 114 and the virtual space control system 10 receives this query, the request candidate generation process is started by the request candidate generation unit 102. In the explanation of this flowchart, the user who inputs the query will be referred to as the "target user."

[0100] First, in step S371, the request candidate generation unit 102 reads the query content, the operation history DB 107 related to the user who issued the query, and the description of the target user in the user information DB 113. The purpose of referencing the operation history DB 107 in this step is to read one or more operation histories immediately before the query. In other words, the request candidate generation unit 102 reads data corresponding to the operation status 3013 in the request example DB 107 from the operation history DB 107. The request candidate generation unit 102 may read the most recent histories of a predetermined number of target users, or may read the histories of the target users up to a certain time ago.

[0101] In the following step S372, the request candidate generation unit 102 acquires highly relevant request example data from the request example DB 107. In this step, the request candidate generation unit 102 references the inquiry data input by the target user, the target user's operation history data, and the user information DB 113. In the following step S373, the request candidate generation unit 102 uses a language model to generate request candidate data that is a candidate for the user's operation request. In the following step S374, the request candidate generation unit 102 transmits the request candidate data generated in step S373 to the operation determining unit 103, and further stores the request candidate data in the request candidate DB 111, thereby ending the processing shown in FIG. Specific examples of steps S372 and S373 will be described.

[0102] A specific example of step S372 is as follows. First, the request candidate generation unit 102 generates operation status data, which is text data describing an operation status, from the operation history DB 107 in the same way as in step S354 of Fig. 13. Then, the request candidate generation unit 102 inputs the query data, the operation status data, the query 3012 and the operation status 3013 of each record in the request example DB 107, and the user ID 3015 and the user attribute 3016 in the user information DB 113 into a language model, thereby extracting request example data that is highly relevant to the query data. Evaluation of the relevance using the language model can be achieved by a known search algorithm.

[0103] The request candidate generator 102 evaluates the relevance of each of the query data, operation status data, and user information descriptions, and acquires highly relevant request example data based on the evaluation results, for example, by calculating a linear sum of the evaluation values ​​with appropriate weighting. To evaluate the relevance, a language model is used to vectorize character strings, and the correlation between the vectors can be used.

[0104] Regarding the user ID, the relevance may be evaluated based on whether the user ID is the same as that of the user who is the target of the operation determination process. At this time, a predetermined number of request example data items may be extracted starting from those with the highest relevance. Alternatively, request example data items with a relevance level equal to or greater than a certain value may be acquired. Furthermore, request example data items extracted by evaluating only the relevance level of the query or the relevance level of the operation status may also be included. Furthermore, the relevance level between the query or operation status and the description of the request example may also be taken into consideration. The above are specific examples of step S372.

[0105] A specific example of step S373 is as follows. The request candidate generation unit 102 first generates text data describing the operation status from the operation history data in a manner similar to that of step S354 in FIG. 13 . However, the operation status data generated in step S372 in FIG. 15 may be used as is. Next, the request candidate generation unit 102 reads a template for request candidate generation processing, i.e., template 2023 having a template ID 2021 of "4," from the language model input template DB 106. The request candidate generation unit 102 then creates input data by substituting the extracted request example data (a set of query-operation status-request), query data, and operation status data into the variables included in the template 2023. The input data includes an instruction to "generate a request for operation in the virtual space in response to the user query and operation status based on the request example data," as well as the request example data, query data, and operation status data.

[0106] A first example of generating a request candidate will be described. In this example, the query is "Start checking the inspection checklist for the automatic door," and the operation status data is "Finished editing the inspection checklist for lamp L102." The request example data is: Query: "I would like to perform an inspection check of the air conditioning equipment," operation status: no description (NULL), request: "I would like to see the air conditioning equipment, I would like to start editing the inspection checklist for the air conditioning equipment, I would like to check the latest reports on malfunctions of the air conditioning equipment." The generated request candidates in this case are: "I would like to see the automatic door, I would like to start editing the inspection checklist for the automatic door, I would like to check the latest reports on malfunctions of the automatic door." If the query data itself were interpreted, only starting editing the checklist would be selected as the required operation. However, by selecting operations based on the above-described request candidates, necessary operations that are not included in the query can be included in the candidates.

[0107] A second example of generating request candidates will be described. In this example, the inquiry is "Is there any work that needs to be done in advance?" and the operation status is "Start editing the inspection checklist for automatic door D1." The extracted request examples are: inquiry: no description (NULL), operation status "Start editing the inspection checklist for automatic door D1," and request example "I would like to open the most recent inspection report for automatic door D1." The generated request candidates in this case include "I would like to open the most recent inspection report for automatic door D1." This is the kind of request example data that can be obtained by the operation history analysis process of the request example registration unit 101. In this way, even if the user does not fully understand the operation and has not given detailed instructions, it is possible to output the necessary operation request by referring to past examples of similar operation statuses.

[0108] Here, when request example data of one or more records is given, the above process may be performed for each request example data to generate request candidates for each. Also, to prevent an increase in processing costs, an upper limit may be set on the number of request candidates to be generated. For example, based on the relevance evaluated when the request example data was extracted in step S372, request candidate generation for request examples may be repeated in descending order of relevance, and the request candidate generation process may be terminated when the number of request candidates reaches the upper limit. This concludes the description of a specific example of step S373.

[0109] 16 is a flowchart showing the operation determination process executed by the operation determination unit 103. The operation determination process starts when the operation determination unit 103 receives request candidate data from the request candidate generation unit 102. First, in step S381, the operation determination unit 103 selects an operation item and operation target data for each request candidate using the received request candidate data, the language model DB 105, the operation DB 110, the 3D model DB 108, and the file DB 109. A specific example of step S381 will be described later.

[0110] In the following step S382, the operation determining unit 103 determines the operation content to be transmitted to the external system 11 based on the processing result of step S381. However, the number of operation contents does not need to be limited to one, and may be two or more. For example, the operation determining unit 103 integrates and narrows down the operation contents. In narrowing down the operation contents, only operation contents whose relevance level is equal to or higher than a threshold may be selected, or a predetermined number of operation contents may be selected in descending order of relevance level.

[0111] In the following step S383, the operation determining unit 103 transmits data of the operation content determined in step S382 to the external system 11 and stores the data in the operation history DB 112, thereby ending the processing shown in Fig. 16. When storing the operation content in the operation history DB 112, the operation determining unit 103 also stores data of the corresponding user inquiry in the operation history DB 112. The operation determining unit 103 may also transmit the relevance of each operation content to the external system 11.

[0112] A specific example of step S381 will be described. The operation determining unit 103 first selects one request candidate from the request candidate data received from the request candidate generating unit 102. Then, for that request candidate, the operation determining unit 103 uses a language model to select an appropriate operation item from the operation DB 110 and appropriate operation target data from the 3D model DB 108 or the file DB 109, and determines the operation content, i.e., which operation to perform on which data. This process is repeated for all request candidate data, and all operation content is extracted.

[0113] The operation item is selected from records included in the operation DB 110 in which the selection target 5015 is set to "True." The operation required for a certain request candidate is selected using a language model by clarifying the correspondence between the description of the request candidate and the descriptions in the name 5012 and the description 5014. For example, using the example of the operation DB 110 shown in FIG. 7 , if the description of a certain request candidate is "I want to see facility X," the item "Move viewpoint to 3D model" with operation ID 5011 of "0" and including the description "Select when you want to see a 3D model" in the description 5014 is selected.

[0114] The method of selecting items using a language model includes a method of calculating the relevance and a method of using a language model. In the method of calculating the relevance, a known search algorithm is used to evaluate the relevance between the requirement candidate and the description of the name 5012 or the description 5014, and items with high relevance are extracted. The evaluation value of either the name 5012 or the description 5014 may be used, or the relevance may be evaluated by evaluating, for example, a linear sum of weighted evaluation values.

[0115] In the method using a language model, a task instruction sentence may be input to the language model to determine whether it should be extracted. For example, input data may be created by inputting to the language model an instruction such as "For the next user operation request, select an operation to be performed from the operation items below," followed by a description of the request candidate and data such as the name 5012 and description 5014 of each record in the operation DB 110. If there is a restriction on the length of the data input to the language model and the number of operation items is large and all descriptions cannot be included in the input data, the following countermeasure may be considered. That is, the relevance of the operation items may be evaluated using the above-mentioned method, and the operation items may be narrowed down in descending order of relevance to fit within a predetermined number of items or characters, and then entered into the model input data.

[0116] The data to be operated on is selected from records in the 3D model DB 108 and the file DB 109. The data to be selected for a given candidate request is selected by inputting a description of the candidate request, the name 4012 in the 3D model DB 108, and the name 4022 and type 4023 in the file DB 109 into a language model. For example, if the request is "I want to see facility X," data on facility X is selected from the 3D model DB. The selection of a 3D model or file using a language model may be performed using relevance, as in the selection of the operation item described above, or the language model may directly output the selection results using a text task instruction. Furthermore, attribute information and descriptions of the 3D model or file, or if a text file is used, the description or summary of the file may also be included in the 3D model DB 108 or the file DB 109 and used during selection.

[0117] When using the 3D model DB 108, the data to be selected may be narrowed down by referring to the parent object ID 4014 or the related object ID 4024. For example, for a request candidate such as "I want to see the automatic door in the first car," a 3D model corresponding to the "first car" is first searched for, and the "first car" with object ID = 100 in the example of the 3D model DB 108 shown in Figure 5 is selected. Then, based on the parent object ID 4014, a search for an "automatic door" object is performed, limited to records in which the "first car" is the parent object. This process can improve the accuracy of inferring objects in which the user is interested.

[0118] Also, consider a request such as "I want to read the maintenance procedure manual for automatic door D1." This request is input into a language model to infer the target, and it can be inferred that the target is file data, which is a manual document linked to the object "automatic door D1." Therefore, it is possible to narrow down the search to records in the file DB 109 that have a related object ID 4024 corresponding to the object ID 4011 (ID = 1) of "automatic door D1." Narrowing down the search in this way improves the accuracy of inferring files that interest the user. To achieve this, for example, objects corresponding to parent objects or related objects may be searched first, and then the data to be selected may be searched.

[0119] To determine the operation content, it is necessary to combine the operation item and the selection result of the operation target data, and the target data 5013 of the selected operation item and the operation target data are consistent. In other words, the target data 5013 and the operation target data are selected so that there is no contradiction between them. Furthermore, the order in which the operation items and the scanning target data are selected does not matter. For example, the operation items may be selected first, and then the operation target data may be selected for each operation item taking into account the constraints of the target data 5013, or vice versa, or both may be selected simultaneously.

[0120] When selecting the target data 5013 and the operation target data simultaneously, the task instruction may include a description to select an operation item and an operation target, which may then be input into a language model for inference. In this case, it is sufficient to obtain only the output pairs of operation item and target data that satisfy the constraints of the target data 5013. An example of how constraint data is taken into account is as follows: Assume that the request candidate description is "I want to see facility X," and the operation item "move viewpoint to 3D model" is selected. In this case, the target data 5013 for this operation item is "3D model DB," and the operation target data must be selected by limiting it to data stored in the 3D model DB 108. Furthermore, if there is no target data for the selected operation item, i.e., if the target data 5013 is "NULL," no target data related to the operation item is selected.

[0121] Furthermore, depending on the operation item, a separate process may be performed after the operation content is selected. For example, in the case of ID=9 "Create an answer to a question by referring to text" shown in FIG. 5, the operation target data is one or more text data. After selecting the operation content, it is necessary to create an answer to the question from the user included in the query by referring to the content of the text data of the operation target data. The answer process to this question may be handled by a function inside or outside the virtual space control system 10, and the processing result may be added to the data of the operation content. This completes the explanation of a specific example of step S381.

[0122] 17 is a diagram showing an example of a user input screen 1600 for executing the request example registration process. This diagram can be said to be a screen for setting reference data to be set before the request example registration process and for inputting the contents of each reference data, among the interface screens of the virtual space control system 10. The user input screen 1600 includes a selection window 1601, an input registration window 1602, a history registration window 1603, and a document registration window 1604.

[0123] The selection window 1601 is a setting window for reference data used in the request example registration process. The selection window 1601 includes a user input button 160101, an operation history analysis button 160102, and a document analysis button 160103. When the user presses any of the three buttons, a window corresponding to the pressed button is displayed. When the user input button 160101 is pressed, the input registration window 1602 is displayed. When the operation history analysis button 160102 is pressed, the history registration window 1603 is displayed. When the document analysis button 160103 is pressed, the document registration window 1604 is displayed. However, all windows may be displayed in advance, and the window corresponding to the button pressed by the user may become active.

[0124] The input registration window 1602 is a data input window through which the user enters data when using user input data in the request example registration process. Request example data can be newly added, edited, and deleted by operating this screen. The settings in the input registration window 1602 include a request example ID 160201, an inquiry 160202, an operation status 160203, a request 160204, a user ID 160205, and a user attribute 160206, each of which is referenced in the user input acceptance process shown in FIG. 12. One or more requests 160204 can be registered.

[0125] A text box can be added by clicking the "+" button below the text box, and an existing text box can be deleted by clicking the "x" button to the right of each text box. For user ID 160205 and user attribute 160206, a user ID 7011 and attribute 7013 can be selected from the data registered in user information DB 113. When user ID 160205 is selected, the corresponding user attribute 160206 is selected. Also, only user attribute 160206 may be selected, or neither may be selected.

[0126] When the user presses New button 160211, a numerical value that is "1" larger than the maximum value of request example ID 3011 in request example DB 107 is displayed in ID 160201, allowing a new request example to be entered. After filling in each of items 160202 to 160206, the user presses Register button 160214, which sends an instruction to execute the request example registration process and the setting content data entered in input registration window 1602 to virtual space control system 10, and the request example registration process is executed. When the user presses Call button 160212, the data corresponding to the displayed request example ID 160201 is called up from request example DB 107.

[0127] The user can edit and delete this data. The user can edit items 160202 to 160206, and when the user presses the register button 160214 after editing, the user input acceptance process shown in Fig. 12 is executed. Also, when the user presses the delete button 160213, the data corresponding to the request example ID 160201 being displayed is deleted from the request example DB 107.

[0128] It should be noted that a "forced registration" button may be added to the input registration window 1602. When the "forced registration" button is pressed, the user input acceptance process is not executed, and the request example data entered by the user is directly registered in the request example DB 107. It is also possible to allow one or more request example data to be entered at the same time and registered at one time.

[0129] The history registration window 1603 is a setting window into which the user inputs information when using operation history data in the request example registration process. By operating this window, the user can specify the user whose operation history is to be analyzed and issue an instruction to execute the analysis. The history registration window 1603 displays a user ID 160301, a name 160302, and an attribute 160303. In the user ID 160301 box, by pressing the dropdown on the right, the user can select a user ID 7011 from the data registered in the user information DB 113. The name 160302 and the attribute 160303 display data corresponding to the user ID 160301, and correspond to the name 7012 and the attribute 7013 in the user information DB 113, respectively.

[0130] When the user presses the analysis execution button 160310, the operation history data corresponding to the user ID 160301 is referenced, and the operation history analysis process shown in Fig. 13 is executed. Specifically, the data specified by the user ID 7011 = user ID 6023 among the data in the operation history DB 112 is referenced. Here, one or more users may be specified. Furthermore, instead of specifying the user ID 160301, the content of another column in the user information DB 113 may be searched and specified.

[0131] The document registration window 1604 is a setting window into which the user inputs information when using document data in the request example registration process. The user can operate this window to specify the document data to be analyzed and issue an instruction to execute the analysis. The document registration window 1604 displays a name 160401 and a file ID 160402. In the box for name 160401, the next selection can be made by pressing the drop-down menu on the right. That is, the name 4022 can be selected from file data registered in the file DB 109 whose type 4023 is written in natural language, such as "text" or "checklist."

[0132] The file ID 160402 displays data corresponding to the name 160401, and corresponds to the file ID 4021 in the file DB 109. When the user presses the analysis execution button 160410, the file data corresponding to the file ID 160402 is referenced, and the document analysis process shown in Fig. 14 is executed. The file data corresponding to the file ID 160402 is the data specified by the file ID 4021 among the data in the file DB 109.

[0133] 18 is a diagram showing an example of input and output to the user interface. This diagram can be considered as a screen for inputting a query to the virtual space control system 10, and a screen for receiving the input query and displaying the received query content and generated request candidates, among other interface screens related to the virtual space control system 10.

[0134] A user 1701 operating in the virtual space inputs an inquiry on an operation terminal 1702 and transmits the inquiry to the virtual space control system 10. Specifically, on an inquiry input screen 17021 displayed on the operation terminal 1702, the user inputs text data representing the inquiry into a text box 170211 in accordance with the guidance on the screen. When the user further presses a send button 170212, an operation decision process is initiated in the virtual space control system 10. The inquiry may be input directly from a keyboard, or the operation terminal 1702 may receive a voice input from the user and input the text using a known voice recognition method.

[0135] Screen 1704 displays the contents of the inquiry received from operation terminal 1702 and the contents of the generated request candidates. Text box 170401 shows the contents of the received inquiry, and text box 170402 displays the contents of the request candidates determined by the request candidate generation unit 102. Screen 1704 is displayed as a pop-up on 3D viewer 1703 displaying the virtual space, allowing user 1701 to determine whether the system is operating correctly.

[0136] The first embodiment described above provides the following advantageous effects. (1) The virtual space control system 10, which can also be called a candidate generation system, generates operation candidates in a virtual space based on an input query, which is query data entered in natural language. The virtual space control system 10 includes a request candidate generation unit 102 that extracts one or more request example records based on the input query from a request example DB 111, which stores multiple request example records that are combinations of the input query and an operation request. The request candidate generation unit 102 generates model input data, which is text data, using the extracted request example records and the input query, and inputs the model input data into a language model to generate one or more operation request candidates corresponding to the input query. This allows the virtual space control system 10 to appropriately generate operation candidates in a virtual space.

[0137] (2) The user's query included in the input query is a character string entered by the user in text box 170211 in FIG.

[0138] (3) As can be seen from the fact that either the query 3012 or the operation status 3013 in FIG. 4 may be "NULL," the input query is at least one of a character string based on a user's input and a character string indicating the user's most recent operation. The request candidate generation unit 102 obtains a character string indicating the user's most recent action by searching the operation history DB 112, which stores the user's past operations in the virtual space. Each record in the request example DB 111 stores a combination of at least one of a character string based on a user's input and a character string indicating the user's most recent operation, and the operation in the virtual space. Therefore, the virtual space control system 10 can generate operation request candidates based not only on the user's input but also on one or more of the user's most recent operations.

[0139] (4) The request candidate generator 102 generates model input data by substituting the extracted request example records and the input query into a pre-prepared template. Therefore, the virtual space control system 10 can use a general-purpose language model by using an appropriate template.

[0140] (5) The virtual space control system 10 includes an operation determination unit 103 that selects an operation content for each request candidate and determines an operation based on the selection result. Therefore, the virtual space control system 10 can determine the operation desired by the user.

[0141] (6) The virtual space control system 10 includes an input analysis unit 1011 that registers a request example record in the request example DB 111. The input analysis unit 1011 modifies the content of a request example record input by a user using a language model and registers the modified content in the request example DB 111. This allows the virtual space control system 10 to modify the user's input and add a new record to the request example DB 111.

[0142] (7) The virtual space control system 10 includes a history analysis unit 1012 that registers request example records in the request example DB 111. The history analysis unit 1012 inputs the history of user operations and the history of input queries into a language model to generate a request example record that is a combination of a character string indicating the user's most recent operation and the operation in the virtual space (S354 in FIG. 13), and registers the request example record in the request example DB 111. This allows the virtual space control system 10 to generate new request example records based on the user's operation history.

[0143] (8) The virtual space control system 10 includes a history analysis unit 1012 that registers request example records in the request example DB 111. The history analysis unit inputs the input query and the user's operations on the virtual space after receiving the operation request candidates generated by the request candidate generation unit 102 into a language model, infers operation requests that the request candidate generation unit 102 has not been able to generate, generates request example records (S355 in FIG. 13 ), and registers them in the request example DB 111. Therefore, the virtual space control system 10 can update the request example DB 111 so that no generated operation candidates are omitted.

[0144] (9) The virtual space control system 10 includes a document analysis unit 1013 that registers a request example record in the request example DB 111. The document analysis unit uses a language model to extract operation description sentences, which are sentences involving operations in virtual space, from text data registered in the file database, creates input data including the operation description sentences and the text data registered in the file database, inputs the input data to the language model to generate a request example record, and registers the request example record in the request example DB 111. Therefore, the virtual space control system 10 can update the request example DB 111 using documents specified by the user.

[0145] (Variation 1) The virtual space control system 10 includes a request example registration unit 101, a request candidate generation unit 102, an operation determination unit 103, and an operation history registration unit 104. However, the virtual space control system 10 only needs to include at least the request candidate generation unit 102, and may not include the request example registration unit 101, the request candidate generation unit 102, and the operation history registration unit 104.

[0146] (Variation 2) The virtual space control system 10 includes multiple databases, such as the language model DB 105. However, the virtual space control system 10 may not include one or more of these databases, or may not include any databases at all. In this case, the virtual space control system 10 only needs to be configured to be connectable to an external database via the NIC 1806.

[0147] (Variation 3) The request example registration unit 101 includes an input analysis unit 1011, a history analysis unit 1012, and a document analysis unit 1013. However, it is sufficient for the request example registration unit 101 to include at least one of the input analysis unit 1011, the history analysis unit 1012, and the document analysis unit 1013.

[0148] (Variation 4) A plurality of language models may be stored in the language model DB 105, and different language models may be used depending on the application. In this case, for example, a field for specifying the language model to be used may be added to the language model input template DB 106.

[0149] (Variation 5) As described with reference to Figure 18, the user inputs text data representing the inquiry into text box 170211. However, the user may input the inquiry by speech instead of inputting text data. In this case, a speech recognition device built into external system 11 may convert the user's speech into a character string and transmit the character string to virtual space control system 10 as in the embodiment, or speech data may be transmitted to virtual space control system 10. When external system 11 transmits speech data to virtual space control system 10, request example registration unit 101 of virtual space control system 10 has a speech recognition function, and input analysis unit 1011 analyzes the results of the speech recognition.

[0150] (Variation 6) In the above-described embodiment, the virtual space control system 10 includes the language model input template DB 106, and the request example registration unit 101, the request candidate generation unit 102, the operation determination unit 103, and the operation history registration unit 104 can use a common language model by using different templates. However, the virtual space control system 10 does not have to include the language model input template DB 106. In this case, a language model can be created specifically for each purpose.

[0151] In the above-described embodiments and modifications, the functional block configurations are merely examples. Some functional configurations shown as separate functional blocks may be integrated, or a configuration shown in a single functional block diagram may be divided into two or more functional blocks. Furthermore, some of the functions of each functional block may be provided by other functional blocks.

[0152] In each of the above-described embodiments and modifications, the virtual space control system 10 may be provided with a data input / output interface (not shown), and a program may be loaded from another device as needed via the data input / output interface and a medium usable by the virtual space control system 10. Here, the medium refers to, for example, a storage medium detachable from the input / output interface, or a communication medium, i.e., a wired, wireless, or optical network, or a carrier wave or digital signal propagating through the network. Furthermore, some or all of the functions realized by the program may be realized by a hardware circuit or FPGA.

[0153] The above-described embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention.

[0154] 10: Virtual space control system 11: External system 101: Request example registration unit 102: Request candidate generation unit 103: Operation determination unit 104: Operation history registration unit 105: Language model DB 106: Language model input template DB 107: Request example DB 109: File DB 111: Request candidate DB 112: Operation history DB 1011: Input analysis unit 1012: History analysis unit 1013: Document analysis unit

Claims

1. A candidate generation system that generates candidates for operations in a virtual space based on an input query, which is query data input in a natural language, comprising: a request candidate generation unit that extracts one or more request example records based on the input query from a request example database in which a plurality of request example records, which are combinations of the input query and an operation request, are stored, generates model input data, which is text data, using the extracted request example record and the input query, and inputs the model input data into a language model to generate one or more candidates for the operation request corresponding to the input query.

2. A candidate generation system according to claim 1, wherein the input query is a character string input by a user or a character string obtained by performing voice recognition on an utterance of the user.

3. A candidate generation system as described in claim 1, wherein the input query is at least one of a character string based on an input by a user and a character string indicating the user's most recent operation, the request candidate generation unit obtains a character string indicating the user's most recent action by searching an operation history database in which the user's past operation contents in the virtual space are stored, and the request example record stores a combination of at least one of a character string based on an input by the user and a character string indicating the user's most recent operation contents, and the operation contents in the virtual space.

4. A candidate generation system as described in claim 1, wherein the request candidate generation unit generates the model input data by substituting the extracted request example record and the input query into a pre-prepared template.

5. A candidate generating system according to claim 1, further comprising an operation determining unit which selects an operation content for each of said request candidates and determines an operation based on a selection result.

6. A candidate generation system as described in claim 1, further comprising an input analysis unit that registers the request example record in the request example database, wherein the input analysis unit modifies the content of the request example record input by a user using a language model and registers the modified content in the request example database.

7. A candidate generation system as described in claim 3, further comprising a history analysis unit that registers the request example record in the request example database, wherein the history analysis unit inputs the history of the operation content by the user and the history of the input query into a language model to generate the request example record consisting of a combination of a character string indicating the operation content immediately before the user and the operation content in the virtual space, and registers the request example record in the request example database.

8. A candidate generation system as described in claim 1, further comprising a history analysis unit that registers the request example record in the request example database, wherein the history analysis unit inputs the input query and the user's operations in the virtual space after receiving the operation request candidates generated by the request candidate generation unit into a language model, infers the operation request that the request candidate generation unit was unable to generate, and generates and registers the request example record in the request example database.

9. A candidate generation system as described in claim 1, further comprising a document analysis unit that registers the request example record in the request example database, wherein the document analysis unit uses a language model to extract an operation description sentence, which is a sentence involving an operation of the virtual space, from text data registered in a file database, creates input data including the operation description sentence and the text data registered in the file database, inputs the input data into the language model to generate the request example record, and registers the request example record in the request example database.

Images