Candidate generation system

Abstract

To enable a candidate for an operation in a virtual space to be appropriately generated.SOLUTION: A virtual space control system which can also be called a candidate generation system generates a candidate for an operation in a virtual space based on an input inquiry which is inquiry data input in a natural language. The virtual space control system includes a request candidate generation unit that extracts one or more request example records based on the input inquiry from a request example DB in which a plurality of request example records, each of which is a combination of an input inquiry and an operation request, are stored, generates model input data which is text data using the extracted request example records and the input inquiry, and generating one or more candidates for the operation request corresponding to the input inquiry by inputting the model input data to a language model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a candidate generation system.

Background Art

[0002] An environment that simulates the real world is constructed in a virtual space, and information on facilities used and maintained in operations and daily business data are aggregated and managed, thereby improving the efficiency of data accumulation and utilization. In order to realize efficient operations in a virtual space where a huge amount of data exists, there is a need for a technology that presents drafts of operations on the virtual space in response to inquiries in natural language from users. That is, a method for determining appropriate operation candidates based on the content and situation of inquiries from users is required. Patent Document 1 discloses a command processing program for causing a computer to realize a function of generating a command for executing an instruction on an operation target in a virtual space based on an input in natural language by a user. The computer includes a text data acquisition function for obtaining text data based on an input in natural language by a user, a syntax analysis function for extracting a command to be executed from the text data, a command analysis function for generating a primitive type command from the command extracted by the syntax analysis function, a specific viewpoint information acquisition function for acquiring at least specific viewpoint information in the virtual space during an input operation in natural language by the user, a command evaluation function for performing an evaluation based on a predetermined evaluation criterion for each option when the primitive type command generated by the command analysis function includes a plurality of options and outputting the evaluation, and a command determination function for determining an option and determining a command based on the evaluation result in the command evaluation function. The command evaluation function includes a function of evaluating each option in the primitive type command using the specific viewpoint information acquired by the specific viewpoint information acquisition function.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the invention described in Patent Document 1, there is room for improvement in determining operations in a virtual space.

Means for Solving the Problems

[0005] A candidate generation system according to a first aspect of the present invention is a candidate generation system that generates candidates for operations in a virtual space based on an input inquiry which is inquiry data input in natural language, and extracts one or more of the request example records from a request example database in which a plurality of request example records which are combinations of the input inquiry and operation requests are stored, based on the input inquiry, generates model input data which is text data using the extracted request example records and the input inquiry, and includes a request candidate generation unit that generates one or more candidates for the operation requests corresponding to the input inquiry by inputting the model input data into a language model.

Effects of the Invention

[0006] According to the present invention, candidates for operations in a virtual space can be appropriately generated.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Best Mode for Carrying Out the Invention

[0008] (Overview) In this specification, a system for generating candidates for operations on a virtual space based on a language instruction from a user will be described. By constructing an environment that simulates the real world on the virtual space and aggregating and managing information on facilities used in operations and maintenance in the business and business data generated daily, data accumulation and data utilization efficiency are achieved. In the virtual space, there is business data generated daily and a huge amount of 3D data for simulating the real world, providing a high degree of freedom of operation for the user. Therefore, in order to efficiently determine operations, a technology for generating candidates for operations required by the user on the system side based on an inquiry in natural language from the user is useful.

[0009] In addition, there are the following advantages in estimating requirements not included in the inquiry content based on the inquiry content and the user's operation status at the time of the inquiry. That is, even if the user does not give detailed instructions or the user himself / herself does not fully recognize the necessary operations, the system can supplement conditions and expand the search range of the operation content, so that candidates for the operations required by the user can be widely presented, leading to increased efficiency. In this specification, instead of extracting operation candidates from the inquiry sentence, the necessary operations are narrowed down after expanding the search range of the inquiry based on case data, and candidates for the necessary operations are generated taking into account implicit requirements.

[0010] - First Embodiment- Hereinafter, with reference to FIGS. 1 to 18, a first embodiment of the candidate generation system will be described.

[0011] FIG. 1 is a configuration diagram of the virtual space control system 10. Note that the virtual space control system 10 can also be called a "candidate generation system" because it generates candidates for operation requests as will be described later. The virtual space control system 10 includes a request example registration unit 101, a request candidate generation unit 102, an operation determination unit 103, an operation history registration unit 104, a language model DB 105, a language model input template DB 106, a request example DB 107, a 3D model DB 108, a file DB 109, an operation DB 110, a request candidate DB 111, an operation history DB 112, and a user information DB 113. In this specification, the database is also described as "DB".

[0012] At least one language model is stored in the language model DB 105. The language models may be used separately according to the application, or only one language model may be used. A plurality of templates are stored in the language model input template DB 106. At the timing of creating model input data, which is text data input to the language model, any one of the templates stored in the language model input template DB 106 is called. Appropriate values are substituted into the variables in the called template to create the model input data.

[0013] The requirement example DB107 stores requirement examples related to operations in a virtual space corresponding to combinations of user inquiries and operation statuses at the time of the inquiries. The 3D model DB108 stores data related to 3D models registered in the virtual space. The file DB109 stores data related to files such as business documents registered in the virtual space. The operation DB110 stores data related to operations possible on the virtual space of the 3D viewer 115.

[0014] The requirement candidate DB111 stores data of requirement candidates generated by the requirement candidate generation unit 102 or created in advance. The operation history DB112 stores data representing the content of user operations on the virtual space generated by the operation history registration unit 104 and the operation determination unit 103. The user information DB113 stores data related to the user who operates the virtual space.

[0015] When the requirement example registration unit 101 receives an instruction of "requirement example registration process" from the external system 11, it generates requirement example data and stores it in each database. The requirement example data is data indicating an example of an operation requirement, as will be described later. At least one of user input data, data specifying a file registered in the virtual space control system 10, and operation history data is attached to the instruction of the requirement example registration process. The user input data is data indicating a character string input by the user and options selected by the user. The user inputs it using the UI114. The operation history data is data indicating the history of operations by the user in the virtual space. The requirement example registration unit 101 uses the data attached to the requirement example registration process, the requirement candidate DB111, the language model DB105, the language model input template DB106, and the user information DB113.

[0016] The requirement example registration unit 101 includes an input analysis unit 1011, a history analysis unit 1012, and a document analysis unit 1013. Based on the input from the external system 11, the requirement example registration unit 101 activates any one of the input analysis unit 1011, the history analysis unit 1012, and the document analysis unit 1013. The input analysis unit 1011 analyzes user input data and registers a new requirement example in the requirement candidate DB 111. The history analysis unit 1012 analyzes operation history data and registers a new requirement example in the requirement candidate DB 111. The document analysis unit 1013 analyzes the files registered in the virtual space control system 10 and registers a new requirement example in the requirement candidate DB 111.

[0017] When the requirement candidate generation unit 102 receives an instruction of "operation decision processing" from the external system 11, it generates "requirement candidate data" which is data indicating candidates for requirements regarding operations on the virtual space. The requirement candidate generation unit 102 refers to the language model DB 105, the language model input template DB 106, the requirement example DB 107, and the user information DB 113 stored in the language model DB 105.

[0018] Based on the requirement candidate data generated by the requirement candidate generation unit 102, the operation decision unit 103 selects an operation for each requirement candidate, aggregates the requirement candidate data to determine the final operation, and transmits the operation content determined to the external system 11. The operation decision unit 103 refers not only to the requirement candidate data but also to the language model DB 105, the language model input template DB 106, the 3D model DB 108, the file DB 109, and the operation DB 110.

[0019] When the operation history registration unit 104 receives an instruction of "operation history registration processing" from the external system 11, it registers the operation content of the user on the virtual space as operation history data in the operation history DB 112. That is, the operation history data is data indicating the history of operations performed by the user in the virtual space. The external system 11 includes a UI 114 having an interface to the virtual space control system 10 and a 3D viewer 115 having a function to display and operate the virtual space.

[0020] When the language model included in the language model DB105 receives input of text data, it estimates an appropriate word sequence to output. The language model has the function of learning the structure and rules of natural language and interpreting and generating text data. The text data is the text itself in natural language or data indicating a text in natural language. For example, when giving some question text as model input data, an answer to it is output. In recent years, the development of deep learning technology in the field of natural language processing has been remarkable, and large-scale language models with a very large number of parameters constituting the language model have emerged, realizing highly advanced language processing functions close to those of humans.

[0021] In addition, in large-scale language models, an example of using text data called a prompt as input is known. The prompt describes instructions for the task to be handled and information related to the task. By inputting a prompt into the large-scale language model, the large-scale language model can be made to execute the task. The aforementioned model input data is also a prompt.

[0022] Also, some large-scale language models such as ChatGPT function as an API and provide their functions as a service externally. When information regarding the learning of the language model, including the parameters of the model, is kept confidential, the API is often used. The language processing API server 120 shown by the dashed line in FIG. 1 is a server that processes an API related to the language model. The virtual space control system 10 may use the language processing API server 120 instead of including the language model DB105, and the result is the same whichever is used. That is, the language processing API server 120 includes a language model similar to the language model DB105. However, since whether the virtual space control system 10 has a language model or uses an API is not an essential difference, only the case where the virtual space control system 10 has a language model will be described below.

[0023] Figure 2 is a hardware configuration diagram of the virtual space control system 10. The virtual space control system 10 includes a processor 1801, a storage device 1802, a monitor 1803, a DRAM 1804, an input device 1805, and a NIC 1806. The processor 1801 is a central processing unit. The storage device 1802 is a non-volatile data recording device, such as a hard disk drive or a flash memory. The monitor 1803 is, for example, a liquid crystal display. The DRAM 1804 is a volatile data recording device capable of high-speed reading and writing. The input device 1805 is a device that accepts human input, such as a mouse or a keyboard. The NIC 1806 is a network interface card that enables communication between the virtual space control system 10 and other devices.

[0024] The processor 1801 realizes a request example registration unit 101, a request candidate generation unit 102, an operation determination unit 103, and an operation history registration unit 104 by expanding and executing a program stored in the storage device 1802 in the DRAM 1804. However, the virtual space control system 10 may be realized by an FPGA (Field Programmable Gate Array), which is a rewritable logic circuit, or an ASIC (Application Specific Integrated Circuit), which is an integrated circuit for specific applications, instead of the combination of the processor 1801, the storage device 1802, and the DRAM 1804. Further, the virtual space control system 10 may be realized by a combination of different configurations, such as a combination of the processor 1801, the storage device 1802, the DRAM 1804, and the FPGA, instead of the combination of the processor 1801, the storage device 1802, and the DRAM 1804.

[0025] The memory device 1802 stores a language model DB 105, a language model input template DB 106, a request example DB 107, a 3D model DB 108, a file DB 109, an operation DB 110, a request candidate DB 111, an operation history DB 112, and a user information DB 113. However, it is not an essential configuration that all the above-mentioned databases are stored in the memory device 1802, and it may be configured to be accessible via the NIC 1806 to a database existing outside the virtual space control system 10. That is, the virtual space control system 10 only needs to be able to access each of the above-mentioned databases, regardless of whether the database is built into the virtual space control system 10 or not.

[0026] FIG. 3 is a diagram showing an example of the language model input template DB 106. The language model input template DB 106 stores templates used for creating model input data. The language model input template DB 106 is composed of a plurality of records. Each record of the language model input template DB 106 has fields of a template ID 2021, an item 2022, and a template 2023. The template ID 2021 stores an identifier of the template. For each process executed by the virtual space control system 10, which template to use is set in advance using the template ID 2021. The item 2022 stores the item name of the process for calling each template.

[0027] The template 2023 stores a template of the content to be input to the language model. The template is a combination of natural language text data and variables, such as {query}. A string that changes depending on the situation, such as the user's inquiry content, is substituted into the variable. In the programs that implement the request example registration unit 101 and the request candidate generation unit 102, the correspondence between the variables in the program and the variables in the template is described in advance. When the request example registration unit 101 and the request candidate generation unit 102 use the templates stored in the language model input template DB 106, model input data is generated using this known correspondence.

[0028] FIG. 4 is a diagram showing an example of the requirement example DB107. Each record constituting the requirement example DB107 may be created in advance or may be created by the requirement example registration unit 101. The requirement example registration unit 101 creates records by any of the following three methods. First, the requirement example registration unit 101 creates a record using the data input by the user. Second, the requirement example registration unit 101 creates a record using the operation history data on the virtual space specified by the user. Third, the requirement example registration unit 101 creates a record using the description of the document file specified by the user.

[0029] The requirement example DB107 stores case data of operation requests such as examples of requests related to operations on the virtual space considering not only the content of inquiries from the user but also the operation status. The operation status at the time of responding to an inquiry is one or more operation histories before this inquiry. The requirement example data is a collection of examples of operation requests and is used to generate operation requests from the content of the user's inquiries and operation status at each time using this collection of examples when actually generating the user's operation requests.

[0030] The requirement example DB107 is composed of a plurality of records, and each record has fields of a requirement example ID 3011, an inquiry 3012, an operation status 3013, a requirement 3014, a user ID 3015, and a user attribute 3016. Either the inquiry 3012 or the operation status 3013 may be "NULL". That is, the requirement example data may be stored in response to the case where the operation request is interpreted only from the inquiry content or the case where the operation request is interpreted only from the operation status. Hereinafter, the inquiry 3012 and the operation status 3013 are collectively referred to as "input inquiry".

[0031] In Request Example ID3011, an identifier for identifying request example data is stored. In Inquiry 3012, the content of the inquiry input by the user is stored as an operation instruction in the virtual space. However, if there is no input from the user, "NULL" is stored. In Operation Status 3013, the operation status at the time of the inquiry described in natural language, that is, the actions before the inquiry, is stored. However, if there is no previous user action before the inquiry, "NULL" is stored.

[0032] For example, if the user "started editing the inspection checklist of Facility X" immediately before making an inquiry, "started inspection check of X" or "started editing the inspection checklist of X" etc. will be stored in Operation Status 3013. Operation Status 3013 may be created based on two or more operation histories immediately before the inquiry. If the purpose of those operations can be interpreted from multiple operation histories, the content may be described in the operation status. For example, assume that the previous operation histories were "create a report on the brake handle in the driver's seat" and "register the image data of the air conditioning equipment in the driver's seat into the system". Inputting these two histories into the language model in natural language, "performing inspection work on the driver's seat equipment" etc. obtained by the inference of the language model may be stored in Operation Status 3013.

[0033] Note that instead of describing natural language in Operation Status 3013, data for specifying records in the Operation History DB112, for example, the Operation ID6024 described later, may be included. When Operation ID6024 is included in Operation Status 3013, when using this Operation Status 3013, refer to the 3D Model DB108, File DB109, Operation DB110, and User Information DB113 corresponding to the described Operation ID6024. Then, obtain the specific content of the operation history, and further create the text data of the operation status as described above using the language model before using it.

[0034] Requirement 3014 stores natural language indicating a group of operation requirements for operations on the user's virtual space corresponding to the inquiry 3012 and the operation status 3013. For example, when the inquiry 3012 is "I want to perform an inspection check on the air conditioning equipment", operation requirements for the virtual space system are included, such as wanting to start editing the inspection checklist, wanting to check the state of the equipment before the inspection, and wanting to check for defects related to the inspection target from the report. By accumulating such requirements in the requirement example DB107 and referring to them when generating requirement candidates, the search range for operation requirements can be expanded and appropriate operations can be widely selected.

[0035] The user ID 3015 and the user attribute 3016 store user information related to the corresponding requirement example. These correspond to the user ID 7011 and the attribute 7013 in the user information DB113. When generating requirement candidates, they are used to collate with the information of the user who is the target of the operation decision process and select records with a high degree of relevance for that user. The user ID 3015 and the user attribute 3016 may be "NULL". When a value other than "NULL" is stored in the user ID 3015, the corresponding user attribute 3016 is stored based on the user information DB113.

[0036] FIG. 5 is a diagram showing an example of the 3D model DB108. The 3D model DB108 stores pre-prepared data. The 3D model DB108 stores data related to the 3D models registered in the virtual space. Specifically, the 3D model DB108 stores the name, coordinates, and information of the parent object of the 3D model on the virtual space. The 3D model DB108 is composed of a plurality of records, and each record has fields of an object ID 4011, a name 4012, coordinates 4013, and a parent object ID 4014.

[0037] The object ID 4011 stores an identifier for identifying the 3D model. The name 4012 stores the name of the 3D model. The object ID 4011 is a unique value, and the same value is not set for any record. The name 4012 is set to the same value if the models are of the same type. The coordinates 4013 store the coordinate values of the 3D model in the virtual space. The coordinate values stored in the coordinates 4013 may be a representative point such as the center or centroid of the 3D model, or may be the coordinate values of all vertices of the 3D model so that the shape of the 3D model can be clearly understood. The coordinate system can be arbitrary, for example, a Cartesian coordinate system. The coordinates 4013 may further include Euler angles representing the orientation of the 3D model, such as (30 degrees, 60 degrees, 100 degrees).

[0038] The parent object ID 4014 stores an array of object IDs 4011 of another 3D model that is the parent object of the 3D model. For example, assume there is a 3D model A (object ID = 100) representing the first car of a railway vehicle and a 3D model B representing a seat existing in the space of the first car. In this case, the parent object of the 3D model B is the 3D model A, and the object ID = 100 of the 3D model A is stored as an element of the array of the parent object ID of the 3D model B. There may be multiple parent objects, and the parent object ID is held as a list. The parent-child relationship of 3D objects is generally defined in a game program or the like. In this embodiment, for example, the program of the 3D viewer 115 that displays the virtual space acquires data regarding the parent-child relationship of the objects.

[0039] By having data on the parent-child relationship of the objects, for example, when a user asks "I want to check the automatic door existing in the first car", it can be specified that the object of the user's interest is the automatic door "inside the first car". This contributes to improving the inference accuracy when selecting the operation target data from the operation request in the process of the operation determination unit 103. If there is no parent object, "NULL" may be stored to indicate its absence.

[0040] FIG. 6 is a diagram showing an example of the file DB 109. Prepared data in advance is stored in the file DB 109. In the file DB 109, data regarding files such as business documents registered in the virtual space is stored. In the file DB 109, regarding files such as text data and image data registered in the system, file name, type, and information on related 3D model objects are stored. The file DB 109 is composed of a plurality of records, and each record has fields of a file ID 4021, a name 4022, a type 4023, and a related object ID 4024.

[0041] In the file ID 4021, an identifier of the file is stored. In the name 4022, the name of the file is stored. In the type 4023, data indicating the type of the file is stored. The type 4023 serves as a tag for specifying data to be operated on in the target data 5013 of the operation DB 110 described later. The type 4023 is, for example, "text", "check list", "image", etc. In the related object ID 4024, the object ID 4011 of the 3D model related to the file is stored.

[0042] By including the related object ID 4024 in the file DB 109, there are advantages as follows, for example. When the user inquires "want to check the report of the automatic door existing in the first vehicle", it can be specified that the file of interest is the file of the report regarding the "automatic door in the first vehicle". This contributes to improving the inference accuracy when selecting the operation target data from the operation request in the process of the operation determination unit 103. When the related object ID 4024 is not included in the file DB 109, "NULL" may be stored to indicate that there is no related object. Also, when there are a plurality of objects related to a certain file, the identifiers of the plurality of objects may be described in a list format.

[0043] FIG. 7 is a diagram showing an example of the operation DB 110. The operation DB 110 stores pre-prepared data. The operation DB 110 stores data related to operations possible in the virtual space of the 3D viewer 115, specifically, the name of each operation item, the range of data to be operated on, the description of the operation content, and data on whether selection is possible at the time of inquiry. Whether selection is possible at the time of inquiry means whether it is possible to select when there is an inquiry from the user regarding an operation instruction. The operation DB 110 is composed of a plurality of records, and each record has fields of an operation ID 5011, a name 5012, target data 5013, a description 5014, and a selection target 5015.

[0044] The operation ID 5011 stores the identifier of the operation item. The name 5012 stores the name of the operation item. The target data 5013 stores the target of the operation item. For example, the "open file" operation with the operation ID = 2 in FIG. 7 is an operation targeting the data in the file DB 109, so the target data 5013 is described as "file DB". Also, the "start editing text file" operation with the operation ID = 3 in FIG. 7 targets only the text data in the file DB 109.

[0045] Therefore, the target data 5013 is described as "file DB; type = text", and it is specified that only the data with the type 4023 being "text" in the file DB 109 is the target. As will be described later, the target data 5013 is used by the operation determination unit 103 to ensure consistency between the selected operation item and the operation target data. However, for operation items without target data such as "illuminance change", "NULL" is stored.

[0046] The description 5014 stores the explanatory text of the operation item. When selecting an operation item from an operation request in the process of the operation determination unit 103, the description 5014 is given to the language model as a material for determining whether the operation item should be selected. Based on this assumption, in the example shown in FIG. 7, the timing of selecting the operation item is clearly indicated. However, the description 5014 may include other descriptions explaining the content of the operation item.

[0047] Regarding the description of the selection timing, for example, for the operation of "viewpoint movement to a 3D model", since it is appropriate to select when "want to view the 3D model" as a requirement, description 5014 states "select when want to view the 3D model". Here, for operation items that are not selection targets of the operation determination process, such as the "utterance reception" operation, "NULL" indicating no description may be stored. Note that the "utterance reception" operation is an operation in which the system receives the content spoken by the user in the virtual space and records it in the operation history regardless of whether it is an inquiry. This operation history is used for estimating the operation status and the like.

[0048] In selection target 5015, in the operation determination process, data indicating whether the corresponding operation item is a selection target is stored. Items for which the operation target is set to False include, for example, the aforementioned "utterance reception" operation. This is because the "utterance reception" operation is an operation automatically performed by the system and not an operation that provides the result of the operation determination to the user.

[0049] Figure 8 is a diagram showing an example of the requirement candidate DB 111. In the requirement candidate DB 111, data of requirement candidates generated by the requirement candidate generation unit 102 as described later is stored. However, data of requirement candidates created in advance may be stored in the requirement candidate DB 111. The requirement candidate DB 111 is composed of a plurality of records, and each record has fields of a requirement candidate ID 6011, an inquiry ID 6012, a time 6013, a user ID 6014, an inquiry 6015, an operation status 6016, and a requirement candidate 6017. Hereinafter, the inquiry 6015 and the operation status 6016 are collectively referred to as "input inquiry". Also hereinafter, each record of the requirement candidate DB 111 is also referred to as a "requirement example record".

[0050] The generated request candidate identifier is stored in the request candidate ID 6011. When the trigger for the operation decision process is the issuance of an inquiry from the user, the identifier that identifies the inquiry data is stored in the inquiry ID 6012. As will be described later, the trigger for the operation decision process may be a pre-scheduled timing, for example, every fixed period of time. In this case, "NULL" is stored in the inquiry ID 6012. Data indicating the time when the request candidate generation process was executed is stored in the time 6013.

[0051] The identifier that identifies the user who is the inference target of the operation request is stored in the user ID 6014. The user ID 6014 corresponds to the user ID 7011 in the user information DB 113. The inquiry data received from the user is stored in the inquiry 6015. The data of the operation status read from the immediate previous operation history of the user by the request candidate generation unit 102 is stored in the operation status 6016. One or more request candidates generated by the request candidate generation unit 102 are stored in the request candidate 6017 in a list format.

[0052] FIG. 9 is a diagram showing an example of the operation history DB 112. Data representing the operation content of the user in the virtual space, generated by the operation history registration unit 104 and the operation decision unit 103, is stored in the operation history DB 112. The operation history DB 112 is composed of a plurality of records, and each record has fields of an operation history ID 6021, a time 6022, a user ID 6023, an operation ID 6024, an object ID 6025, a file ID 6026, supplementary information 6027, and an inquiry ID 6028.

[0053] The identifier that identifies the operation history is stored in the operation history ID 6021. Information on the time when the operation was executed in the virtual space is stored in the time 6022. The identifier of the user who performed the operation is stored in the user ID 6023. The user ID 6023 corresponds to the user ID 7011 in the user information DB 113. The identifier of the executed operation is stored in the operation ID 6024. The operation ID 6024 corresponds to the operation ID 5011 in the operation DB 110.

[0054] The object ID 6025 and the file ID 6026 store identifiers for identifying the object and file that are the operation targets. The object ID 6025 corresponds to the object ID 4011 in the 3D model DB 108, and the file ID 6026 corresponds to the file ID 4021 in the file DB 109. In the case of an operation for which the corresponding 3D model or file does not exist, "NULL" is stored in the object ID 6025 and the file ID 6026.

[0055] The supplementary information 6027 stores data related to operations other than operation items and operation target data. For example, regarding the "utterance reception" operation, in order to record the user's utterance content, "utterance = (utterance content)" etc. is stored in the supplementary information 6027. When there is no data corresponding to the supplementary information 6027, "NULL" is stored. The inquiry ID 6028 represents a number specifying the inquiry data when the operation is performed in response to an inquiry from the user. The inquiry ID 6028 corresponds to the inquiry ID 6012 in the request candidate DB 111. When the operation is not an operation corresponding to an inquiry, since there is no corresponding inquiry data, "NULL" is stored.

[0056] FIG. 10 is a diagram showing an example of the user information DB 113. The user information DB 113 stores pre-prepared data. The user information DB 113 stores data related to the user who operates the virtual space. The user information DB 113 is composed of a plurality of records, and each record has fields of a user ID 7011, a name 7012, and an attribute 7013.

[0057] The user ID 7011 stores an identifier for identifying the user. The name 7012 stores the user's name. The attribute 7013 stores the user's attribute data. For example, when the virtual space control system 10 is used for the purpose of business support, the user's business position and job type are stored in the attribute 7013.

[0058] FIG. 11 is a flowchart showing the operation history registration process executed by the operation history registration unit 104. When the user operates in the virtual space on the 3D viewer 115, data on the operation content is transmitted from the external system 11, and when the virtual space control system 10 receives the data on the operation content, the process is started. The operation history registration process is a process of converting the data on the operation content transmitted from the external system 11 so as to associate it with the operation DB 110, the 3D model DB 108, and the file DB 109, and storing the data in the operation history DB 112.

[0059] In step S331, the operation history registration unit 104 stores the data on the operation content received from the external system 11 in the operation history DB 112 and ends the process shown in FIG. 11. Specifically, the operation history registration unit 104 obtains data corresponding to the time, user ID, operation ID, object ID, file ID, and supplementary information of the operation history DB 112 from the data on the operation content and stores it in the operation history DB 112. At this time, the user ID, operation ID, object ID, and file ID are obtained so as to correspond to the data stored in the user information DB 113, operation DB 110, 3D model DB 108, and file DB 109, respectively. When the operation content registered in the operation history registration process is not due to inquiry response, "NULL" is stored in the inquiry ID 6028.

[0060] FIG. 12 is a flowchart showing the user input reception process executed by the request example registration unit 101. First, in step S341, the request example registration unit 101 creates model input data using the request example data input by the user received from the external system 11 and the language model input template DB 106 of the language model input template DB 106.

[0061] Specifically, first, among the records in the language model input template DB 106, the record with template ID 2021 being "0" is read. Then, the request example registration unit 101 creates input data by substituting the values of the variables specified in template 2023. In this process, the user inquiry content, the operation status, and the text data describing the request are respectively substituted. The input data shall include instructions such as "Based on the user inquiry content, the operation status, and the content of the request, delete irrelevant requests, improve requests with ambiguous expressions into specific descriptions, and create the modified requests", as well as the input user inquiry content, operation status, and description of the request.

[0062] Here, instructions such as "delete requests regarded as irrelevant" and "specifically describe requests with ambiguous expressions" are described, but these are for modifying the request example data assumed by the user, and other contents may be included as instructions for the same purpose. Also, to improve the inference accuracy of the language model, sample data of the user input request example and the modified request example may be prepared in advance and appended to the model input data. By including several corresponding examples of the inference task in the model input data, the inference accuracy can be improved.

[0063] In the subsequent step S342, the request example registration unit 101 inputs the model input data created in step S341 into the language model, generates the modified request example data, stores it in the request example DB 107 together with the user information data, and ends the process shown in FIG. 12. Note that the user information data corresponds to the user ID 160205 on the UI and the user attribute 160206, which will be described later.

[0064] As described in the explanation of step S341, the language model generates correction data for the request description based on an instruction to correct the description of the request input by the user. For example, the processing in the case where an instruction such as "delete requests regarded as irrelevant" is included will be described. As the content of the request example data input by the user, the inquiry is "want to conduct an inspection of the air conditioning equipment", and the operation status is not described. Suppose that an item "want to see the beer server" is included in the operation requests input by the user. At this time, the request "want to see the beer server" is content irrelevant to the content of the inquiry, and thus is deleted as an incorrect input. However, since there is a possibility that the language model may misestimate a correct request as an incorrect input, before storing the data in the request example DB107, the corrected content may be displayed to the user on the UI114, and it may be confirmed whether the corrected content can be accepted.

[0065] Also, for example, the processing in the case where an instruction such as "specify the description of the request" is included is as follows. As the content of the request example data input by the user, the inquiry is "want to conduct an inspection of the air conditioning equipment", and the operation status is not described. Suppose that an item "want to see related equipment" is included as an operation request. At this time, since the expression "related equipment" is ambiguous and it is difficult to interpret what it refers to, the specific content is estimated from the inquiry content, and it can be corrected to the description "want to see the air conditioning equipment". After performing such a correction of the request description, a part corresponding to the corrected request description is extracted from the output content of the language model, and it is stored in the request example DB107 together with the inquiry content, the operation status, and the user information data.

[0066] FIG. 13 is a flowchart showing the operation history analysis process executed by the history analysis unit 1012 of the request example registration unit 101. Before the operation history analysis process is started, a designated user to be analyzed is set in advance. First, in step S351, the history analysis unit 1012 reads all the operation histories of the designated user from the operation history DB112. Note that the number of designated users may be one or more. In the subsequent step S352, the history analysis unit 1012 selects one of the unprocessed records as an analysis target record from the records read in step S352.

[0067] In the subsequent step S353, the history analysis unit 1012 determines whether the record to be analyzed is an operation for inquiry response. If it is not an operation for inquiry response, it means that the operation was performed by the user himself / herself. This determination can be made based on the value of the inquiry ID 6028 in the operation history DB 112. If the inquiry ID 6028 is not "NULL", the history analysis unit 1012 determines that it was an operation at the time of responding to the inquiry. If the inquiry ID 6028 is "NULL", the history analysis unit 1012 can determine that it was an operation performed by the user himself / herself regardless of the inquiry. If the history analysis unit 1012 determines that it was an operation for inquiry response, it proceeds to step S355. If it determines that it was not an operation for inquiry response, it proceeds to step S354.

[0068] In step S354, the history analysis unit 1012 creates required example data using the record to be analyzed, the language model input template DB 106, and the language model, and then proceeds to step S356. A specific example of step S354 will be described later. In step S355, the history analysis unit 1012 infers a requirement leak, which is a leak in the operation requirements, based on the user's actions after the inquiry in the record to be analyzed, that is, the user's actions recorded in the operation history DB 112, or a new inquiry, and then proceeds to step S356. A specific example of step S355 will be described later.

[0069] In step S356, the history analysis unit 1012 stores the data created in step S354 or step S355 in the request example DB107. At this time, the history analysis unit 1012 stores the information of the user to be analyzed in the fields of the user ID 3015 and the user attribute 3016. In the subsequent step S357, the history analysis unit 1012 determines whether all the records read in step S351 are to be analyzed. For example, the history analysis unit 1012 temporarily assigns a flag indicating that the record was selected as the analysis target record in step S352 to the operation history DB 112, and determines based on whether the flag is assigned to all the read records. If the history analysis unit 1012 determines that all the read records are to be processed, it ends the process shown in FIG. 13. If it determines that there are unprocessed records, it returns to step S352.

[0070] A specific example of step S354 will be described. First, the history analysis unit 1012 creates model input data using the analysis target record and the template 2023 with the template ID 2021 of "1" in the language model input template DB 106. The model input data is created by substituting values into the variables specified in this template 2023. In this process, the operation content data described from the data of the operation item, the data of the operation target, and the supplementary information 6027 described in the selected operation history data is substituted.

[0071] The operation item described in the selected operation history data can be specified by reading the name 5012 based on the operation ID 5011 of the operation DB 110 corresponding to the operation ID 6024 of the operation history DB 112. The data of the operation target can be specified by reading the name 4012 of the 3D model DB 108 and the name 4022 of the file DB 109 based on the object ID 4011 in the 3D model DB 108 and the file ID 4021 in the file DB 109 corresponding to the object ID 6025 and the file ID 6026 of the operation history DB 112, respectively.

[0072] For example, assume that an operation of "opening a file" is performed on the file "Automatic Door D1_20230715_Part Repair Report.txt". In this case, the operation history is described as "performed an operation of opening a file on Automatic Door D1_20230715_Part Repair Report.txt", etc. This can be achieved by setting the template to "{operation target data} was operated on {operation history}".

[0073] Also, in order to interpret the operation status when the target operation is executed, one or more operation history data of the user immediately before the target operation are also read and substituted into the model input data. The operation status may include a predetermined number of previous operations or operations during a predetermined previous period. For example, assume that an operation history of "started editing a checklist for Automatic Door D1_20230730_Inspection Checklist.json" is obtained from the previous operation history. In this case, the content of "started editing the inspection checklist for Automatic Door D1" is read as the operation status.

[0074] Also, as described above, when the purpose of the operation can be estimated from multiple operation histories, the estimated purpose may be used as the operation status. For example, if the previous operation histories are all related to the inspection work of the driver's cab of a railway vehicle, it can be interpreted that "performing inspection work on the driver's cab equipment", and it can be inferred by the language model. The things related to the inspection work of the driver's cab of a railway vehicle include "creating a report on the brake handle in the driver's cab", "registering the image data of the air conditioning equipment in the driver's cab into the system", etc.

[0075] Using the operation content data created in this way and the data of one or more operation contents performed immediately before the target operation, model input data is created. The model input data includes instructions such as "Describe the situation when the operation was performed and the requirements for the operation based on the user's operation content and the previous operation content respectively", the operation content of the target record, and one or more operation contents performed before it. In addition, in order to improve the inference accuracy of the language model, sample data of a set of operation situations and one or more requirement examples to be generated are prepared in advance for the target operation content and one or more operation contents performed immediately before the operation, and may be appended to the input data. By including several corresponding cases for the inference task in the model input data, the inference accuracy can be improved.

[0076] Next, the above-mentioned model input data is input into the language model to generate requirement example data. For example, when the target operation content is "Performed an operation to open a file for the automatic door D1_20230715_component repair report.txt", by inferring in combination with the time when the operation was performed, for example, the 30th of the same month, requirements such as "Want to open the latest inspection report of the automatic door D1" can be inferred. Also, one or more requirement examples may be created. Also, since this process does not target the operation history corresponding to the inquiry content, there is no corresponding inquiry data, and "NULL" is stored in the inquiry content. However, before storing the created requirement example data in the requirement example DB107, the created requirement example data may be presented to the user using the UI114 and registered in the requirement example DB107 only after the user's confirmation.

[0077] Here, an example of inferring the operation situation and operation requirements simultaneously has been described, but either one of them may be inferred first. For example, first output the text of the operation situation from the previous operation history data, and then output the operation requirements from the target operation content and the data of the operation situation already output. Generally, it is difficult to infer multiple tasks simultaneously, and decomposing the tasks may improve the inference accuracy.

[0078] In addition, the data on the operation status may store the content of the operation history data as it is. At this time, only some fields in the record, specifically, the operation ID 6024, object ID 6025, file ID 6026, supplementary information 6027, etc., may be extracted and included. When using the data of the requirement example DB 107, if it is required that the operation status data is in text format, the following countermeasures can be considered. That is, the specific content of the operation history is obtained by referring to the 3D model DB 108, file DB 109, operation DB 110, and user information DB 113 from the ID information of each item registered as described above. Furthermore, the text data of the operation status may be created from the operation history as described above using a language model and then used.

[0079] In this way, by analyzing the operation status and the user's requirements based on the past user operation history and accumulating them as requirement example data, more appropriate inferences can be made when inquiries are received from users in a similar operation status later. The above is the description of the specific example of step S354.

[0080] An example of step S355 will be described. First, the history analysis unit 1012 creates model input data using the operation history data selected in step S351 and the template 2023 in the language model input template DB 106 with the template ID 2021 being "2". Then, the history analysis unit 1012 creates input data by substituting the values of the variables specified in the template 2023.

[0081] In this process, for the selected operation history data, based on the inquiry ID 6028 of the operation history DB 112 corresponding to the inquiry ID 6012 of the requirement candidate DB 111, the corresponding requirement candidate data and the operation history data immediately after the inquiry are obtained based on the time information. Then, the inquiry, operation status, generated requirement candidates, and the operation content immediately after the inquiry are substituted into the model input data.

[0082] The model input data includes instructions such as "Based on the user's inquiry content, the operation status at the time of the inquiry, the candidate requirements inferred by the system in response to the inquiry, and the operation content performed by the user after the inquiry, describe the requirements that were missed in generation." Then, the aforementioned inquiry, operation status, candidate requirement data, and operation content performed after the inquiry are further input into the model input data.

[0083] Also, in order to improve the inference accuracy of the language model, prepare in advance sample data of one or more sets of requirement examples to be generated by the language model for the target inquiry content, operation status, candidate requirements, and the operation content immediately after, and append it to the input data. By including several corresponding examples of the inference task in the model input data, the inference accuracy can be improved.

[0084] Next, the history analysis unit 1012 inputs the aforementioned model input data into the language model to generate requirement example data. For example, assume that the inquiry is "I want to perform an inspection check on the automatic door", the operation status is "Finished editing the inspection checklist of lamp L102", and the generated candidate requirement is "I want to start editing the inspection checklist of the automatic door". If the operation content immediately after this is "Performed an operation to open the file for the automatic door D1_20230715_component repair report.txt", then the following speculation can be made. That is, it can be inferred that there is a requirement not only to start editing the checklist when performing the inspection check, but also to check the recent, for example, the reports within several days of the equipment to be inspected.

[0085] Therefore, it is inferred that the requirement "I want to open the recent inspection report of the automatic door" was missed from the candidate requirements, and it is added to the requirement example DB107 as requirement example data together with the data of the above inquiry and operation status. Also, if there is no corresponding requirement example in the result of the inference, the addition of requirement example data is not performed. Also, two or more requirement examples may be created. However, before storing the data in the requirement example DB107, the created requirement example data may be presented to the user using the UI114 and registered in the requirement example DB107 only after the user's confirmation.

[0086] In this way, based on the past inquiry response history, by analyzing the content that was insufficient as an operation requirement inferred for the inquiry and accumulating it as requirement example data, when a similar inquiry is received later, it becomes possible to infer the requirements that were missed in the past. The above is the explanation of the specific example of step S355.

[0087] FIG. 14 is a flowchart showing the document analysis process executed by the document analysis unit 1013 of the requirement example registration unit 101. First, in step S361, the document analysis unit 1013 uses the designation information of the file to be analyzed received from the external system 11 and the file DB 109 of the file DB 109 to read the designated file (hereinafter referred to as the "designated file"). The number of designated files is not limited to 1 and may be 2 or more. Also, the designated file is a file described in natural language, such as a text file or a checklist file.

[0088] In the subsequent step S362, the document analysis unit 1013 extracts a sentence related to the operation in the virtual space (hereinafter referred to as the "operation description sentence") based on the designated file read in step S361, the operation DB 110, and the language model DB 105. A specific example of step S362 will be described later. In the subsequent step S363, the document analysis unit 1013 creates model input data using the operation description sentence extracted in step S362, and inputs this model input data into the language model DB 105 to obtain an output of the requirement example. A specific example of step S363 will be described later. In the subsequent step S364, the document analysis unit 1013 stores the requirement example generated in step S363 in the requirement example DB 107 and ends the process shown in FIG. 14.

[0089] A specific example of step S362 is as follows. For example, assume that the specified file is an inspection procedure manual for a certain facility X, and the following sentence S, "Perform the inspections in the following checklist," is included in the description of a certain procedure. This sentence S is analyzed using a language model together with the names 5012 and descriptions 5014 of each record in the operation DB110 to determine whether the operations described in this sentence involve operations in the virtual space. In the example of the operation DB110 shown in FIG. 7, this sentence S is determined to correspond to the "start editing checklist" where the operation ID 5011 is "3," and this sentence S is extracted as an operation description sentence.

[0090] A second example of step S362 will be described. For example, assume that the following sentence T, "Refer to the manual of facility X when performing this operation," is included in the specified file. In this case, through the analysis of the language model, it is determined that the "open file" where the operation ID 5011 is "2" corresponds, and this sentence T is extracted as an operation description sentence. By performing a similar analysis on all sentences in the specified file, sentences involving operations in the virtual space can be extracted from the specified file.

[0091] For the analysis of the sentence, a known search algorithm may be used to calculate the degree of relevance between the content of the sentence and the operation items, and documents with a high degree of relevance may be extracted as operation description sentences, or a task instruction to the language model may be formulated as a sentence and input into the language model to determine whether it should be extracted. For example, an instruction such as "Determine whether the following sentence S involves any item in the following operation list," the sentence to be analyzed, and the data of the names 5012 and descriptions 5014 of each record in the operation DB110 may be combined as input data. Also, here, an example of extracting a sentence by referring to the operation items registered in the operation DB110 has been described, but publicly known techniques may be used to widely extract sentences related to any operation. The above is the description of the specific example of step S362.

[0092] A specific example of step S363 is as follows. First, the document analysis unit 1013 creates model input data based on the sentence data extracted in step S362, the file data in which this sentence is described, and the language model input template DB 106 of the language model input template DB 106. From the language model input template DB 106, a record with template ID 2021 being "3" is read. Then, the input data is created by substituting the values of the variables specified in template 2023. The variables to be substituted are the extracted sentence data and related data from which the situation at the time of performing the operation described in the sentence can be inferred.

[0093] This related data is the text described in a predetermined range before and after the operation description sentence in the specified file, and the text including the content pointed to by the directive in the operation description sentence. Further, the related data may include a sentence estimated to be strongly related to the extracted sentence, for example, a sentence extracted by evaluating the degree of relevance between sentences using a known search algorithm. Furthermore, the related data may include the name of the specified file and data summarizing the content of the specified file.

[0094] For example, as in the example presented in step S362, assume that the inspection procedure manual for equipment X is the specified file and the operation description sentence is "Perform the following checklist inspection." In this case, from the description of the specified file, it can generally be inferred that the situation of inspecting equipment X is occurring, and from the related data as described above, it can be read that the operation situation is "performing the inspection work of equipment X."

[0095] The document analysis unit 1013 creates model input data using the operation description text and related data. By inputting this model input data into a language model, required examples are output. The model input data includes instructions such as "Based on the text regarding operations in the virtual space and the related text described in the file, describe the requirements for the operations to be performed on the operation situation", the operation description text, and the related data. Output examples from the language model into which this model input data is input are, for example, the operation situation is "Performing inspection work on equipment X", and the required example is "Want to perform an inspection check on equipment X", etc.

[0096] Also, if in the equipment manual document, etc., as a combination of assumed questions and answers, there is a question text "What should I do at xx" and an answer text "Perform operation yy" described, not only the operation situation but also the inquiry content can be extracted virtually. In this case, the inquiry content may also be output to the language model.

[0097] Note that in order to improve the inference accuracy of the language model, the model input data may include sample outputs and sample inputs corresponding to these sample outputs. The sample inputs and sample outputs are pre-prepared data. The sample input is an example of the operation description text and related data. The sample output is string data expected as the output of the language model when the sample input is the model input data. By including several corresponding examples for the inference task in the model input data, the inference accuracy can be improved. The above is the description of a specific example of step S363.

[0098] Describe the supplement of step S364. The language model may output two or more request examples. In this case, a plurality of request examples are stored in request 3014 in step S364. Also, when the inquiry data is not included in the model input data, "NULL" is stored in inquiry 3012 of the request example DB107 in step S364. Also, before storing data in the request example DB107, the created request example data may be displayed to the user using the UI114, and the user may be asked whether the created data can be registered. The above is the supplement of step S364. In this way, based on the description of the specified file specified by the user, by accumulating the combination of the operation status and the request in the request example DB107, when an inquiry is received from a user in a similar operation situation, this data can be referred to and an operation can be proposed.

[0099] FIG. 15 is a flowchart showing the request candidate generation process executed by the request candidate generation unit 102. When the user outputs an inquiry to the virtual space control system 10 using the UI114 and the virtual space control system 10 receives this inquiry, the request candidate generation process by the request candidate generation unit 102 is started. In the description of this flowchart, the user who inputs the inquiry is called the "target user".

[0100] First, in step S371, the request candidate generation unit 102 reads the description of the inquiry content, the operation history DB107, and the target user in the user information DB113 regarding the user who output the inquiry. The reference to the operation history DB107 in this step is for the purpose of reading one or more operation histories immediately before the inquiry. In other words, the request candidate generation unit 102 reads data corresponding to the operation status 3013 of the request example DB107 from the operation history DB107. The request candidate generation unit 102 may read the history immediately before a predetermined number of target users, or may read the history of the target users until a certain time ago.

[0101] In the subsequent step S372, the request candidate generation unit 102 acquires request example data with a high degree of relevance from the request example DB 107. In this step, the request candidate generation unit 102 refers to the inquiry data input by the target user, the operation history data of the target user, and the user information DB 113. In the subsequent step S373, the request candidate generation unit 102 uses a language model to generate request candidate data that is a candidate for the user's operation request. In the subsequent step S374, the request candidate generation unit 102 transmits the request candidate data generated in step S373 to the operation determination unit 103, and further stores the request candidate data in the request candidate DB 111 to end the process shown in FIG. 15. Specific examples of step S372 and step S373 will be described.

[0102] A specific example of step S372 is as follows. First, the request candidate generation unit 102 generates operation status data, which is text data describing the operation status, from the operation history DB 107 in the same manner as step S354 in FIG. 13. Then, the request candidate generation unit 102 inputs the inquiry data, the operation status data, the descriptions of the inquiry 3012 and the operation status 3013 of each record in the request example DB 107, and the user ID 3015 and the user attribute 3016 in the user information DB 113 into the language model to extract request example data with a high degree of relevance to the inquiry data. The evaluation of the degree of relevance by the language model can be realized by a known search algorithm.

[0103] The request candidate generation unit 102 evaluates the degree of relevance for the inquiry data, the operation status data, and the description of the user information respectively, and based on these evaluation results, for example, takes a linear sum of the evaluation values with appropriate weighting to obtain request example data with a high degree of relevance. For the evaluation of the degree of relevance, the language model can be used to vectorize the character strings and the correlation between the vectors can be used.

[0104] Regarding the user ID, the relevance may be evaluated based on whether it is the same user ID as the user targeted by the operation determination process. At this time, a predetermined number of request example data may be extracted from those with high relevance. Alternatively, request example data with a relevance equal to or higher than a certain value may be acquired. Also, the request example data extracted by evaluating only the relevance of the inquiry or only the relevance of the operation status may be included. Furthermore, the relevance between the inquiry and the operation status and the description of the request example may also be considered. The above is a specific example of step S372.

[0105] A specific example of step S373 is as follows. The request candidate generation unit 102 first generates text data describing the operation status from the operation history data in the same manner as in step S354 of FIG. 13. However, the operation status data generated in step S372 of FIG. 15 may be used as it is. Next, the request candidate generation unit 102 reads a template for request candidate generation processing, that is, template 2023 with template ID 2021 being "4", from the language model input template DB 106. Then, the request candidate generation unit 102 substitutes the extracted request example data (inquiry - operation status - request set), inquiry data, and operation status data into the variables included in template 2023 to create input data. The input data includes an instruction to the effect of "Based on the request example data, generate a request for an operation in the virtual space for the user inquiry and the operation status", the request example data, the inquiry data, and the operation status data.

[0106] Describe the first generation example of a requirement candidate. In this example, the inquiry is "Start checking the checklist for the automatic door", and the operation status data is "Finished editing the checklist for lamp L102". Also, the requirement example data is: inquiry: "Want to perform a check on the air conditioning equipment", operation status: not described (NULL), requirement: "Want to view the air conditioning equipment, want to start editing the checklist for the air conditioning equipment, want to check the latest report on the malfunction of the air conditioning equipment". In this case, the generated requirement candidate is "Want to view the automatic door, want to start editing the checklist for the automatic door, want to check the latest report on the malfunction of the automatic door". If we interpret the inquiry data itself, only starting to edit the checklist is selected as the required operation. However, by selecting an operation based on the above-mentioned requirement candidate, operations that are not included in the inquiry but are necessary can be included in the candidate.

[0107] Describe the second generation example of a requirement candidate. In this example, the inquiry is "Is there any work required in advance?", and the work status is "Start editing the checklist for the automatic door D1". Also, the extracted requirement example is: inquiry: not described (NULL), operation status "Start editing the checklist for the automatic door D1", requirement example "Want to open the latest inspection report for the automatic door D1". In this case, the generated requirement candidate includes "Want to open the latest inspection report for the automatic door D1". This is requirement example data obtained through the operation history analysis process of the requirement example registration unit 101. In this way, even if the user does not fully understand the operations and no detailed instructions are given, by referring to past similar operation status cases, the necessary operation requirements can be output.

[0108] When request example data for one or more records is given here, the above processing may be performed for each request example data, and request candidates may be generated respectively. Further, in order to prevent an increase in processing cost, an upper limit on the number of request candidates to be generated may be provided. For example, when extracting request example data in step S372, based on the relevance evaluated at that time, generation of request candidates for the request examples may be repeated in descending order of relevance, and the process of generating request candidates may be terminated when the number of request candidates reaches the upper limit. The above is the description of a specific example of step S373.

[0109] FIG. 16 is a flowchart showing the operation determination process executed by the operation determination unit 103. The operation determination process starts when the operation determination unit 103 receives request candidate data from the request candidate generation unit 102. First, in step S381, the operation determination unit 103 selects an operation item and operation target data for each request candidate using the received request candidate data, the language model DB 105, the operation DB 110, the 3D model DB 108, and the file DB 109. A specific example of step S381 will be described later.

[0110] In the subsequent step S382, the operation determination unit 103 determines the operation content to be transmitted to the external system 11 based on the processing result of step S381. However, the operation content does not necessarily have to be limited to one, and two or more may be used. For example, the operation determination unit 103 performs integration and narrowing down of the operation content. For narrowing down the operation content, only operation contents with a relevance higher than a threshold value may be selected, or a predetermined number may be selected in descending order of relevance.

[0111] In the subsequent step S383, the operation determination unit 103 transmits the data of the operation content determined in step S382 to the external system 11, stores it in the operation history DB 112, and ends the process shown in FIG. 16. When storing the operation content in the operation history DB 112, the operation determination unit 103 also stores the data of the corresponding user inquiry in the operation history DB 112. Further, the operation determination unit 103 may also transmit the relevance of each operation content to the external system 11.

[0112] A specific example of step S381 will be described. The operation determination unit 103 first selects one request candidate from the request candidate data received from the request candidate generation unit 102. Then, for that request candidate, using the language model, an appropriate operation item is selected from the operation DB 110, and appropriate operation target data is selected from the 3D model DB 108 and the file DB 109, and the operation content, that is, which operation is to be performed on which data, is determined. This is repeated for all the request candidate data to extract all the operation contents.

[0113] The selection of the operation item is made from the records included in the operation DB 110 in which the selection target 5015 is set to "True". For a certain request candidate, which operation is required is selected using the language model by clarifying the correspondence between the description of the request candidate and the descriptions of the name 5012 and the description 5014. For example, using the example of the operation DB 110 shown in FIG. 7, when the description of a certain request candidate is "I want to see equipment X", the item of "viewpoint movement to 3D model" with the operation ID 5011 being "0", which includes the description "select when you want to see a 3D model" in the description 5014, is selected.

[0114] The method for selecting items using the language model includes a method for calculating the relevance and a method for using the language model. In the method for calculating the relevance, a known search algorithm is used to evaluate the relevance between the request candidate and the descriptions of the name 5012 and the description 5014, and items with high relevance are extracted. Either the evaluation value of either the name 5012 or the description 5014 may be used, or for example, the relevance may be evaluated by evaluating the linear sum of the weighted evaluation values.

[0115] In the method of using a language model, it may be determined whether to input and extract the sentence of the task instruction into the language model. For example, following an instruction to the language model such as "select the operation to be executed from the following operation items for the next user operation request", the description of the request candidate and the data of the name 5012 and description 5014 of each record in the operation DB 110 may be combined and described to create input data. If there are restrictions on the length of the data input to the language model and the number of operation items is large and all descriptions cannot be included in the input data, the following countermeasures can be considered. That is, the relevance to the operation items may be evaluated by the above-described method, and after narrowing down the operation items so that they fit within a predetermined number of items or characters in descending order of relevance, they may be described in the above model input data.

[0116] The operation target data is selected from the records of the 3D model DB 108 and the file DB 109. For a certain request candidate, which data should be selected is determined by inputting the description of the request candidate and the descriptions of the name 4012 of the 3D model DB 108, the name 4022 and type 4023 of the file DB 109 into the language model for selection. For example, if it is "want to see equipment X", the data of equipment X is selected from the 3D model DB. The selection of 3D models and files using the language model may use relevance in the same way as the selection of the above-described operation items, or the selection result may be directly output to the language model by the task instruction in the text. In addition, for the attribute information, description text of 3D models and files, and when using a text file, the description in the file and its summary may also be included in the 3D model DB 108 and the file DB 109 and used at the time of selection.

[0117] When using the 3D model DB108, the data to be selected may be narrowed down by referring to the parent object ID 4014 and the related object ID 4024. For example, for a candidate requirement of "want to see the automatic door on the first car body", first, a 3D model corresponding to "the first car body" is searched, and in the example of the 3D model DB108 shown in FIG. 5, the "first car body vehicle" with object ID = 100 is selected. Then, based on the parent object ID 4014, the object of "automatic door" is searched by limiting the records whose parent object is the "first car body vehicle". By this process, the inference accuracy of the object that the user is interested in can be improved.

[0118] Also, assume a requirement of "want to read the maintenance procedure manual for the automatic door D1". By inputting this requirement into the language model to infer the target, it can be inferred that the target is the file data of the manual document associated with the object of "automatic door D1". Therefore, it is possible to narrow down to the records in the file DB109 that have the related object ID 4024 corresponding to the object ID 4011 (ID = 1) of the "automatic door D1". By narrowing down the target in this way, the inference accuracy of the file that the user is interested in can be improved. To achieve this, for example, you may first search for objects corresponding to the parent object and related objects, and then search for the data to be selected.

[0119] To determine the operation content, it is necessary to combine the operation item and the selection result of the operation target data, and the target data 5013 of the selected operation item is consistent with the operation target data. In other words, both are selected so that there is no contradiction between the target data 5013 and the operation target data. Also, the selection order of the operation item and the scanning target data does not matter. For example, the operation item may be selected first, and the operation target data may be selected in consideration of the constraints of the target data 5013 for each operation item, or vice versa, or both may be selected simultaneously.

[0120] When simultaneously selecting the target data 5013 and the operation target data, the task instruction text may describe to select the operation item and the operation target, and input it into the language model for inference. At this time, for the output combination of the operation item and the target data, only those that satisfy the constraints of the target data 5013 need to be acquired. Examples where constraint data is taken into account are as follows. Assume that for the description of the requirement candidate "want to see facility X", the operation item of "viewpoint movement to 3D model" is selected. In this case, the target data 5013 of this operation item is "3D model DB", and it is necessary to select the operation target data limited to the data stored in the 3D model DB108. Also, when the target data of the selected operation item does not exist, that is, when the target data 5013 is "NULL", the target data related to the operation item is not selected.

[0121] Also, according to the operation item, separate processing may be performed after selecting the operation content. For example, in the case of ID = 9 "create an answer to a question by referring to text" shown in FIG. 5, the operation target data is one or more text data. After selecting the operation content, it is necessary to create an answer to the question from the user included in the inquiry by referring to the content of the text data of the operation target data. The answer processing for this question may be handled by functions inside and outside the virtual space control system 10, and the processing result may be added to the data of the operation content. The above is the description of a specific example of step S381.

[0122] FIG. 17 is a diagram showing an example of a user input screen 1600 for executing the requirement example registration process. This figure can also be said to be an input screen for the reference data setting set before the requirement example registration process and the content of each reference data among the interface screens of the virtual space control system 10. The user input screen 1600 includes a selection window 1601, an input registration window 1602, a history registration window 1603, and a document registration window 1604.

[0123] The selection window 1601 is a setting window for reference data used in the claim example registration process. The selection window 1601 includes a user input button 160101, an operation history analysis button 160102, and a document analysis button 160103. When any one of the three buttons is pressed by the user, the window corresponding to the pressed button is displayed. When the user input button 160101 is pressed, the input registration window 1602 is displayed. When the operation history analysis button 160102 is pressed, the history registration window 1603 is displayed. When the document analysis button 160103 is pressed, the document registration window 1604 is displayed. However, all windows may be displayed in advance, and the window corresponding to the button pressed by the user may become active.

[0124] The input registration window 1602 is a data input window for the user to input when using user input data in the claim example registration process. Through the operations on this screen, new addition, editing, and deletion of claim example data can be performed. The settings of the input registration window 1602 include a claim example ID 160201, an inquiry 160202, an operation status 160203, a requirement 160204, a user ID 160205, and a user attribute 160206, which are respectively referred to in the user input reception process shown in FIG. 12. One or more requirements 160204 can be registered.

[0125] The text box can be added with the "+" button below the text box, and the existing text box can be deleted with the "×" button on the right side of each text box. The user ID 160205 and the user attribute 160206 can select the user ID 7011 and the attribute 7013 from the data registered in the user information DB 113. When the user ID 160205 is selected, the corresponding user attribute 160206 is selected. Also, only the user attribute 160206 may be selected, or neither may be selected.

[0126] When the user presses the new button 160211, a numerical value that is "1" greater than the maximum value of the requirement example ID 3011 in the requirement example DB 107 is displayed in ID 160201, and a new requirement example can be input. After each item from 160202 to 160206 is described, when the user presses the registration button 160214, an execution instruction for the requirement example registration process and the setting content data described in the input registration window 1602 are sent to the virtual space control system 10, and the requirement example registration process is executed. When the user presses the call button 160212, the data corresponding to the currently displayed requirement example ID 160201 is called from the requirement example DB 107.

[0127] The user can edit and delete this data. The user can edit the items from 160202 to 160206. When the user presses the registration button 160214 after the edit, the user input reception process shown in FIG. 12 is executed. Also, when the user presses the delete button 160213, the data corresponding to the currently displayed requirement example ID 160201 is deleted from the requirement example DB 107.

[0128] Note that a "forced registration" button may be added to the input registration window 1602. When the "forced registration" button is pressed, the user input reception process is not executed, and the requirement example data input by the user is directly registered in the requirement example DB 107. Also, it may be possible to input one or more requirement example data at the same time and register them all at once.

[0129] The history registration window 1603 is a setting window for the user to input when using operation history data in the requirement example registration process. Through the operations of this window, the user can specify the user to be analyzed for the operation history and issue an instruction to execute the analysis. In the history registration window 1603, the user ID 160301, name 160302, and attribute 160303 are displayed. In the box of the user ID 160301, by pressing the dropdown on the right, the user ID 7011 can be selected from the data registered in the user information DB 113. The name 160302 and attribute 160303 display the corresponding data for the user ID 160301, which correspond to the name 7012 and attribute 7013 in the user information DB 113 respectively.

[0130] When the user presses the analysis execution button 160310, the operation history data corresponding to the user ID 160301 is referenced, and the operation history analysis process shown in Figure 13 is executed. Specifically, among the data in the operation history DB 112, the data specified by user ID 7011 = user ID 6023 is referenced. Here, more than one user can be specified. Also, instead of specifying with the user ID 160301, it may be searched and specified by the content of other columns in the user information DB 113.

[0131] The document registration window 1604 is a setting window for the user to input when using document data in the requirement example registration process. The user can operate this window to specify the document data to be analyzed and issue an instruction to execute the analysis. In the document registration window 1604, the name 160401 and file ID 160402 are displayed. In the box of the name 160401, the following selection can be made by pressing the dropdown on the right. That is, from the data registered in the file DB 109, among the file data in which the type 4023 is "text", "checklist", etc. and natural language is described, the name 4022 can be selected.

[0132] In File ID 160402, data corresponding to Name 160401 is displayed, each corresponding to File ID 4021 in File DB 109. When the user presses the analysis execution button 160410, the file data corresponding to File ID 160402 is referenced, and the document analysis process shown in FIG. 14 is executed. The file data corresponding to File ID 160402 is the data specified by File ID 4021 among the data in File DB 109.

[0133] FIG. 18 is a diagram showing an input / output example to the user interface. This figure can also be said to be a screen for inputting an inquiry to the system and a screen for displaying the received inquiry content and the generated request candidates among the interface screens related to the virtual space control system 10.

[0134] The user 1701 who operates in the virtual space inputs an inquiry using the operation terminal 1702 and transmits it to the virtual space control system 10. Specifically, on the inquiry input screen 17021 displayed on the operation terminal 1702, text data representing the inquiry content is input into the text box 170211 according to the guidance within the screen. Further, when the user presses the send button 170212, the operation determination process is started in the virtual space control system 10. The input of the inquiry content may be directly input using the keyboard, or the operation terminal 1702 may receive voice input from the user and input the content textified by a known voice recognition method.

[0135] Screen 1704 displays the inquiry content received from the operation terminal 1702 and the content of the generated request candidates. The text box 170401 represents the received inquiry content, and the text box 170402 displays the content of the request candidates determined by the request candidate generation unit 102. Screen 1704 is displayed as a pop-up on the 3D viewer 1703 on which the virtual space is displayed, allowing the user 1701 to determine whether the system is operating correctly.

[0136] According to the above-described first embodiment, the following operational effects can be obtained. (1) A virtual space control system 10, which can also be called a candidate generation system, generates candidates for operations in a virtual space based on an input query, which is query data input in natural language. The virtual space control system 10 extracts one or more request example records from a request example DB 111 in which a plurality of request example records, which are combinations of input queries and operation requests, are stored, based on the input query, generates model input data, which is text data, using the extracted request example records and the input query, and inputs the model input data into a language model to generate one or more candidates for operation requests corresponding to the input query, i.e., a request candidate generation unit 102. Therefore, the virtual space control system 10 can appropriately generate candidates for operations in the virtual space.

[0137] (2) The query from the user included in the input query is a character string input by the user and input into the text box 170211 in FIG. 18.

[0138] (3) As can also be seen from the fact that one of the query 3012 and the operation status 3013 may be "NULL" in FIG. 4, the input query is at least one of a character string based on the user's input and a character string indicating the user's previous operation content. The request candidate generation unit 102 retrieves a character string indicating the user's previous action by searching an operation history DB 112 in which the user's past operation content in the virtual space is stored. Each record in the request example DB 111 stores a combination of at least one of a character string based on the user's input and a character string indicating the user's previous operation content, and the operation content in the virtual space. Therefore, the virtual space control system 10 can generate candidates for operation requests based not only on the user's input but also on one or more of the user's previous operations.

[0139] (4) The request candidate generation unit 102 generates model input data by substituting the extracted request example records and the input query into a prepared template. Therefore, the virtual space control system 10 can utilize a general-purpose language model by using an appropriate template.

[0140] (5) The virtual space control system 10 includes an operation determination unit 103 that selects operation contents for each request candidate and determines an operation based on the selection result. Therefore, the virtual space control system 10 can determine the operation desired by the user.

[0141] (6) The virtual space control system 10 includes an input analysis unit 1011 that registers a request example record in the request example DB 111. The input analysis unit 1011 corrects the content of the request example record input from the user using a language model and registers it in the request example DB 111. Therefore, the virtual space control system 10 can correct the user's input and add a new record to the request example DB 111.

[0142] (7) The virtual space control system 10 includes a history analysis unit 1012 that registers a request example record in the request example DB 111. The history analysis unit 1012 inputs the history of the operation content by the user and the history of the input inquiry into the language model, generates a request example record consisting of a combination of the character string indicating the operation content immediately before the user's operation and the operation content in the virtual space (S354 in FIG. 13), and registers the request example record in the request example DB 111. Therefore, the virtual space control system 10 can generate a new request example record based on the history of the user's operations.

[0143] (8) It includes a history analysis unit 1012 that registers a request example record in the request example DB 111. After receiving the input inquiry and the candidates for the operation requests generated by the request candidate generation unit 102, the history analysis unit inputs the operation of the user to the virtual space into the language model, infers the operation requests that the request candidate generation unit 102 could not generate, generates a request example record (S355 in FIG. 13), and registers it in the request example DB 111. Therefore, the virtual space control system 10 can update the request example DB 111 so that there is no omission in the generated operation candidates.

[0144] (9) The document analysis unit 1013 for registering a requirement example record in the requirement example DB111 is provided. The document analysis unit extracts an operation description sentence, which is a sentence involving operations in the virtual space, from the text data registered in the file database using a language model, creates input data including the operation description sentence and the text data registered in the file database, inputs the input data into the language model to generate a requirement example record, and registers the requirement example record in the requirement example DB111. Therefore, the virtual space control system 10 can update the requirement example DB111 using the document specified by the user.

[0145] (Modification Example 1) The virtual space control system 10 includes a requirement example registration unit 101, a requirement candidate generation unit 102, an operation determination unit 103, and an operation history registration unit 104. However, the virtual space control system 10 only needs to include at least the requirement candidate generation unit 102, and does not necessarily need to include the requirement example registration unit 101, the requirement candidate generation unit 102, and the operation history registration unit 104.

[0146] (Modification Example 2) The virtual space control system 10 includes a plurality of databases such as the language model DB105. However, the virtual space control system 10 does not necessarily need to include one or more of these databases, and may not include any databases. In this case, the virtual space control system 10 only needs to be configured to be connectable to a database existing externally via the NIC 1806.

[0147] (Modification Example 3) The requirement example registration unit 101 includes an input analysis unit 1011, a history analysis unit 1012, and a document analysis unit 1013. However, the requirement example registration unit 101 only needs to include at least one of the input analysis unit 1011, the history analysis unit 1012, and the document analysis unit 1013.

[0148] (Modification Example 4) A plurality of language models are stored in the language model DB 105, and the language models may be selectively used according to the application. In this case, for example, a field for specifying the language model to be used is added to the language model input template DB 106.

[0149] (Modification Example 5) As described with reference to FIG. 18, the user input text data representing the inquiry content into the text box 170211. However, instead of inputting text data, the user may input by speech. In this case, the speech recognition device incorporated in the external system 11 may convert the user's speech into a character string and transmit the character string to the virtual space control system 10 in the same manner as in the embodiment, or may transmit the speech data to the virtual space control system 10. When the external system 11 transmits the speech data to the virtual space control system 10, the request example registration unit 101 of the virtual space control system 10 has a speech recognition function, and the input analysis unit 1011 analyzes the result of the speech recognition.

[0150] (Modification Example 6) In the above-described embodiment, the virtual space control system 10 includes the language model input template DB 106, and the request example registration unit 101, the request candidate generation unit 102, the operation determination unit 103, and the operation history registration unit 104 can use a common language model by selectively using the templates. However, the virtual space control system 10 may not include the language model input template DB 106. In this case, it can be handled by creating dedicated language models for each application.

[0151] In each of the above-described embodiments and modification examples, the configuration of the functional blocks is merely an example. Some of the functional configurations shown as separate functional blocks may be integrally configured, or the configuration represented by one functional block diagram may be divided into two or more functional blocks. Also, a configuration may be adopted in which a part of the functions of each functional block is provided by other functional blocks.

[0152] In each of the above-described embodiments and modifications, the virtual space control system 10 may include a data input / output interface (not shown), and when necessary, a program may be read from another device via the data input / output interface and a medium that can be used by the virtual space control system 10. Here, the medium refers to, for example, a storage medium detachable from the input / output interface, or a communication medium, that is, a network such as wired, wireless, or optical, or a carrier wave or digital signal propagating through the network. Also, part or all of the functions realized by the program may be realized by a hardware circuit or FPGA.

[0153] Each of the above-described embodiments and modifications may be combined. Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

Explanation of Reference Numerals

[0154] 10: Virtual space control system 11: External system 101: Request example registration unit 102: Request candidate generation unit 103: Operation decision unit 104: Operation history registration unit 105: Language model DB 106: Language model input template DB 107: Request example DB 109: File DB 111: Request candidate DB 112: Operation history DB 1011: Input analysis unit 1012: History analysis unit 1013: Document analysis unit

Claims

1. A candidate generation system that generates candidates for operations in a virtual space based on an input inquiry which is inquiry data input in natural language, comprising: a request candidate generation unit that extracts one or more of the request example records based on the input inquiry from a request example database in which a plurality of request example records, which are combinations of the input inquiry and operation requests, are stored, generates model input data which is text data using the extracted request example records and the input inquiry, and generates one or more candidates for the operation requests corresponding to the input inquiry by inputting the model input data into a language model.

2. The candidate generation system according to claim 1, wherein the input inquiry is a character string input by a user or a character string obtained by voice recognition of the user's speech.

3. The candidate generation system according to claim 1, wherein the input inquiry is at least one of a character string based on an input by the user and a character string indicating the content of the user's previous operation, the request candidate generation unit obtains a character string indicating the user's previous action by searching an operation history database in which the content of the user's past operations in the virtual space is stored, and the request example records store combinations of at least one of a character string based on an input by the user and a character string indicating the content of the user's previous operation and the operation content in the virtual space.

4. The candidate generation system according to claim 1, wherein the request candidate generation unit generates the model input data by substituting the extracted request example records and the input inquiry into a prepared template.

5. The candidate generation system according to claim 1, further comprising an operation determination unit that selects an operation content for each request candidate and determines an operation based on the selection result.

6. The candidate generation system according to claim 1, further comprising an input analysis unit that registers the request example records in the request example database, wherein the input analysis unit corrects the content of the request example records input by the user using a language model and registers the corrected records in the request example database.

7. The candidate generation system according to claim 3, wherein The candidate generation system further includes a history analysis unit that registers the request example record in the request example database. The history analysis unit generates the request example record, which is a combination of a character string indicating the operation content of the user immediately before and the operation content in the virtual space, by inputting the history of the operation content by the user and the history of the input inquiry into a language model, and registers the request example record in the request example database.

8. The candidate generation system according to claim 1, further comprising a history analysis unit that registers the request example record in the request example database. After receiving the input inquiry and the candidate of the operation request generated by the request candidate generation unit, the history analysis unit inputs the operation of the user in the virtual space into a language model, infers the operation request that the request candidate generation unit could not generate, generates the request example record, and registers the request example record in the request example database.

9. The candidate generation system according to claim 1, further comprising a document analysis unit that registers the request example record in the request example database. The document analysis unit extracts an operation description sentence, which is a sentence involving an operation in the virtual space, from the text data registered in the file database using a language model, creates input data including the operation description sentence and the text data registered in the file database, inputs the input data into a language model to generate the request example record, and registers the request example record in the request example database.

Images