Virtual space generation device and virtual space generation method

Abstract

A virtual space generation device (100) comprises: an asset acquisition unit that, on the basis of a language expression of the content of work to be performed in a virtual space, uses a language model (121) to acquire a keyword of an asset pertaining to the content of the work, and on the basis of the acquired keyword, acquires an asset to be subjected to the work by referring to an asset database (130) in which the keyword and the asset are stored in association with each other; an asset arrangement unit (113) that arranges the asset in the virtual space; and a display control unit (116) that outputs the asset to the display device.

Description

Virtual space generation device and virtual space generation method

[0001] The present invention relates to a virtual space generation device and a virtual space generation method for generating a virtual space (metaverse).

[0002] Advances in digital technology have made it possible to digitally recreate real-world objects such as buildings and facilities as high-resolution virtual objects in virtual spaces. Virtual spaces are also known as metaverse spaces. For example, in the case of railway maintenance, the facilities and equipment to be maintained, such as tracks, trains, electrical systems, signal systems, and stations, can be digitized and recreated in the metaverse. However, digitizing the maintenance work itself is difficult. One example of digitizing maintenance work itself is remote maintenance. Furthermore, the transfer of maintenance skills due to population decline has become an issue. If maintenance work training and simulations were possible in virtual spaces, it would be possible to pass on maintenance techniques and know-how.

[0003] When conducting work training in a virtual space, it is desirable to conduct training under a variety of circumstances and scenarios. For example, when it comes to maintenance and repair work on trains, tracks, signal systems, etc., training is required for a variety of circumstances, scenarios, and work targets, depending on the needs of the maintenance personnel users. For this reason, it is necessary to build multiple virtual spaces / metaverses that combine different types of train and track structures, signal systems, platforms, work time periods, work seasons, work locations, etc. Trains and tracks include, for example, rails, sleepers, roadbeds, bridges, and tunnels. Work locations can also refer to, for example, urban areas, rural areas, and suburban areas.

[0004] One technology related to virtual spaces is described in Patent Document 1. Patent Document 1 discloses "a method for visualizing an interaction between an embodied artificial agent and digital content on an end-user display device of an electronic computing device, the method comprising the steps of creating an agent virtual environment having virtual environment coordinates, simulating the digital content in the agent virtual environment, simulating an embodied artificial agent in the agent virtual environment, enabling the embodied artificial agent to interact with the simulated digital content, and displaying the interaction between the embodied artificial agent and the digital content on the end-user display device."

[0005] Special Publication 2021-531603

[0006] The technology described in Patent Document 1 improves the interaction between objects in a metaverse and users. However, Patent Document 1 does not describe a method for building multiple metaverses. Building multiple virtual spaces / metaverses is not only necessary for training in railway maintenance work, but is also required for work and training in the development, design, production, and maintenance of other facilities, equipment, and devices. The present invention has been made in light of this background, and aims to provide a virtual space generation device and a virtual space generation method that enable the generation of virtual spaces according to user requests.

[0007] In order to solve the above-mentioned problems, the virtual space generation device of the present invention includes an asset acquisition unit that uses a language model to acquire keywords for assets related to the content of work to be performed in a virtual space based on the linguistic expression of the content of the work, and refers to an asset database in which the keywords and assets are stored in association with each other to acquire the assets that are the subject of the work based on the acquired keywords; an asset placement unit that places the assets in the virtual space; and a display control unit that outputs the assets to a display device.

[0008] According to the present invention, it is possible to provide a virtual space generation device and a virtual space generation method that enable the generation of a virtual space according to the user's requests. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.

[0009] FIG. 1 is a functional block diagram of a virtual space generation device according to the present embodiment. FIG. 2 is a diagram for explaining a language model according to the present embodiment. FIG. 3 is a diagram for explaining a language model according to the present embodiment. FIG. 4 is a data configuration diagram of an asset database according to the present embodiment. FIG. 5 is a diagram for explaining a virtual space generation process according to the present embodiment. FIG. 6 is a diagram for explaining an asset arrangement according to the present embodiment. FIG. 7 is a diagram for explaining a three-dimensional model generation process according to the present embodiment. FIG. 8 is a diagram for explaining a sound diffusion model according to the present embodiment. FIG. 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the virtual space generation device according to the above-mentioned embodiment.

[0010] A virtual space generating device according to an embodiment of the present invention will be described below. In this embodiment, railway maintenance work will be described as an example. The virtual space generating device of this embodiment generates a virtual space for railway maintenance personnel to train in maintenance work. To explain in more detail, first, a maintenance personnel, who is a user of the virtual space generating device, inputs details of the maintenance work to be trained and details about the facilities and equipment to be maintained into the virtual space generating device. Details about the facilities and equipment include, for example, names and functions. The virtual space generating device then uses a language model based on the input details to acquire keywords for the facilities and equipment (also referred to as assets) that will be the target of the maintenance work. Next, the virtual space generating device acquires 3D models of the facilities and equipment from an asset database based on the keywords and places them in the virtual space. Furthermore, in response to instructions from the maintenance personnel, the virtual space generating device rearranges the facilities and equipment (3D models).

[0011] By using such a virtual space generator, maintenance personnel can train in the virtual space of their choice. Maintenance personnel can generate a variety of virtual spaces by simply changing the work content entered into the virtual space generator, allowing them to train in maintenance work in a variety of situations with different work environments such as weather, seasons, and time of day, as well as different types, locations, and conditions of facilities and equipment.

[0012] <Configuration of Virtual Space Generator> Fig. 1 is a functional block diagram of a virtual space generator 100 according to this embodiment. The virtual space generator 100 is a computer, and includes a control unit 110, a storage unit 120, and an input / output unit 180. User interface devices such as a display, keyboard, mouse, and microphone are connected to the input / output unit 180. The input / output unit 180 may include a communication device, enabling data transmission and reception with a head-mounted display.

[0013] <Virtual Space Generation Device: Storage Unit> The storage unit 120 is configured to include storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). The storage unit 120 stores a language model 121, an image diffusion model 122, an image generation model 123, a sound diffusion model 124, an asset database 130, a term library 140, asset placement data 150, and a program 128. Note that the various storage contents of the storage unit 120 may be stored in an external storage device such as a cloud server and read as needed.

[0014] <Storage Unit: Language Model> The language model 121 in this embodiment is a language model related to railway maintenance work, and is a machine learning model that has already learned text related to railway maintenance work.

[0015] 2 is a diagram illustrating a language model 121 according to this embodiment. The language model 121 is generated by training using learning data such as specifications (asset specifications) for facilities and equipment (assets) such as trains, tracks, and signal systems that are the targets of railway maintenance work, and a maintenance work training manual 211.

[0016] 3 is a diagram illustrating the language model 121 according to this embodiment. The language model 121 may be generated by transfer learning or fine-tuning the pre-trained language model 221 using training data such as an asset specification or a training manual 222. The image diffusion model 122, the image generation model 123, and the sound diffusion model 124 will be described together with the model generation unit 115 described later.

[0017] <Storage Unit: Asset Database> Returning to Figure 1, we will continue to explain the storage unit 120. The asset database 130 is a database that stores information on assets, which are facilities and equipment related to railways. Examples of assets include trains, tracks, electrical systems, signal systems, platforms, and station buildings.

[0018] FIG. 4 is a data configuration diagram of the asset database 130 according to this embodiment. The asset database 130 is, for example, data in a table format, with each row (record) representing an asset. A record includes columns (attributes (attribute name and attribute value)) of identification information (denoted as "ID" in FIG. 4), name, category, type, use, function, and model. Here, a model is a three-dimensional model placed in a virtual space. The model is not only displayed in the virtual space, but also operates in response to operations by a maintenance worker. For example, the display of a model may change in response to button operations, or a failure state may be restored to a normal state by replacing a part.

[0019] The asset database 130 may include other attributes. The asset database 130 may include, for example, location relationships between assets. Examples of location relationships include whether a train is on a track, whether a platform is located along a track, and the distance between a platform and a track. The asset database 130 shown in FIG. 4 is in a table format, but is not limited to this. The asset database 130 may also be a non-relational database, such as a key-value type, document type, or tree-structure type.

[0020] <Storage Unit: Term Library, Asset Location Data, Program> Returning to FIG. 1, the description of the storage unit 120 will continue. The term library 140 includes meanings and related words found in dictionaries and encyclopedias, meanings and related words in ontologies, and meanings and related words defined by users (maintenance personnel). The term library 140 is referenced by the asset acquisition unit 112, which will be described later. The asset location data 150 stores location information for assets (facilities and equipment) located in virtual space. The program 128 includes descriptions of the processing of functional units provided in the control unit 110, which will be described later.

[0021] <Virtual Space Generating Device: Control Unit> The control unit 110 is configured to include a CPU (Central Processing Unit) and is equipped with a model driving unit 111, an asset acquisition unit 112, an asset placement unit 113, a change instruction receiving unit 114, a model generation unit 115, and a display control unit 116. The control unit 110 may be configured to include a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc.

[0022] <<Control Unit: Virtual Space Generation Processing>> An overview of the virtual space generation processing for arranging assets in an empty virtual space will be described before describing each of the functional units included in the control unit 110. Fig. 5 is a diagram for explaining the virtual space generation processing according to this embodiment.

[0023] The asset acquisition unit 112 acquires text 231 including the details of the maintenance work and the work target (asset) from the maintenance worker. Next, the model driving unit 111 uses the language model 121 to acquire keywords 232 related to the text 231.

[0024] Next, the asset acquisition unit 112 uses the term library 140 to add further keywords related to the keyword 232 to acquire the keyword 233. Next, the asset acquisition unit 112 refers to the asset database 130 to acquire the asset 234 related to the keyword 233. Next, the asset placement unit 113 places the asset 234 in the virtual space 235.

[0025] <<Control Unit: Model Driving Unit>> The model driving unit 111 acquires asset keywords 232 related to the content of input text 231, based on the input text 231, by using the language model 121. For example, the model driving unit 111 generates a prompt including the text "Please provide keywords for facilities and equipment related to the text shown below," and the input text 231, and inputs this into the language model 121 to acquire asset keywords 232 related to the content of the text 231.

[0026] <<Control Unit: Asset Acquisition Unit>> The asset acquisition unit 112 acquires an asset 234 related to text input by a maintenance technician. More specifically, the asset acquisition unit 112 outputs text 231 input by the maintenance technician to the model driving unit 111 and acquires asset keywords 232 related to the content of the text 231. Here, the asset acquisition unit 112 may refer to the term library 140 and add related words or words having the same meaning as the acquired keywords 232 to create new keywords 233.

[0027] Next, the asset acquisition unit 112 refers to the asset database 130 and acquires assets 234 related to the keywords 233. For example, the asset acquisition unit 112 acquires assets that include the keywords in the name, category, type, use, or function in the asset database 130 shown in Fig. 4. If the asset database 130 is a key-value type, assets that include the keywords in the value may be acquired.

[0028] Keyword matching does not have to be an exact match; partial matches are also possible. For example, assets containing keywords close to the keywords obtained using the k-nearest neighbor method may be acquired. Keywords may also be in the form of an attribute name and attribute value, or a key and value. In such cases, the asset acquisition unit 112 matches the value of the attribute name or key in the asset database 130 with the attribute value or value of the keyword. The key and value may also be considered as the attribute name and attribute value.

[0029] The asset acquisition unit 112 may acquire an asset when matching of multiple keywords is successful. For example, if the name is not included, the asset acquisition unit 112 may acquire an asset when matching of type and purpose or type and function is successful. Attribute names or keys may be weighted, and the asset acquisition unit 112 may acquire an asset when the sum of the weights of successfully matched attribute names or keys is equal to or greater than a predetermined value.

[0030] The text input by the maintenance technician is not necessarily input from a keyboard connected to the input / output unit 180. For example, the text may be text (language, linguistic expression) converted from voice input using a microphone connected to the input / output unit 180.

[0031] As described above, the virtual space generating device 100 includes an asset acquisition unit 112 that acquires asset keywords 232 related to the content of work to be performed in the virtual space based on the linguistic expression of the content of the work (see text 231) by using the language model 121. The asset acquisition unit 112 references an asset database 130 in which keywords and assets are stored in association with each other, and acquires an asset 234 that is the target of the work based on the acquired keywords 232.

[0032] The asset database 130 includes attribute values ​​related to assets. The asset acquisition unit 112 acquires assets 234 whose attribute values ​​match successfully with the acquired keywords.

[0033] In addition to the first keyword (see keyword 232) that is the acquired keyword, the asset acquisition unit 112 references a term library 140 that stores related words or meanings of keywords to acquire a second keyword (see keyword 233) that is a related word of the first keyword or a keyword with the same meaning. The asset acquisition unit 112 acquires assets 234 of attribute values ​​that have successfully matched with the first keyword or the second keyword.

[0034] <<Control Unit: Asset Placement Unit / Change Instruction Receiving Unit>> The asset placement unit 113 places the assets acquired by the asset acquisition unit 112 in the virtual space. The asset placement unit 113 references the positional relationships between assets, for example, by referring to the asset database 130, and places the assets so as to satisfy those positional relationships. The asset placement unit 113 stores information on the positions and orientations of the placed assets in asset placement data 150.

[0035] The change instruction receiving unit 114 receives an instruction from a worker to change the placement of an asset. Generally, there can be multiple placements for the same asset. The change instruction from the worker may include the position and orientation of the asset after the change, or may be an instruction to change it randomly. Upon receiving the change instruction from the maintenance worker, the change instruction receiving unit 114 instructs the asset placement unit 113 to make the change. The asset placement unit 113 re-places the asset in accordance with the change instruction.

[0036] 6 is a diagram for explaining the placement of assets according to this embodiment. Trains, traffic lights, and platforms are placed as assets in virtual spaces 281, 282, and 283, but at different positions. The asset placement unit 113 places the same asset at different positions.

[0037] As described above, the virtual space generating device 100 includes the asset placement unit 113 that places the assets 234 in the virtual space 235. The asset database 130 stores the positional relationships between the assets 234. The asset placement unit 113 places the assets 234 in accordance with the positional relationships.

[0038] The virtual space generating device 100 includes a change instruction receiving unit 114 that receives an instruction to change the placement of the asset 234. The asset placement unit 113 places the asset 234 in accordance with the placement change instruction.

[0039] <<Control Unit: Model Generation Unit>> Returning to Fig. 1 , the description of the control unit 110 continues. The model generation unit 115 generates a three-dimensional model of an asset (see the "model" attribute of the asset database 130 shown in Fig. 4 ). If the asset database 130 does not contain an asset corresponding to a keyword 233 (see Fig. 5 ), the asset acquisition unit 112 instructs the model generation unit 115 to generate a model corresponding to the keyword 233, and acquires the asset. The model generation unit 115 generates a three-dimensional model of the asset corresponding to the keyword 233 and stores it in the asset database 130.

[0040] 7 is a diagram illustrating the 3D model generation process according to this embodiment. The model generation unit 115 generates an asset image 242 corresponding to the keywords 232 and 233 using the image diffusion model 122 based on the keywords 232 and 233. The image 242 is, for example, an image of the front or side of the asset. The image diffusion model 122 is a diffusion model that has learned images of railway-related assets, and is used to generate an image of an asset related to a keyword based on the keyword related to the asset.

[0041] Next, the model generation unit 115 uses the image generation model 123 based on the image 242 to generate an image 243 of the asset viewed from a different viewpoint. The image generation model 123 is a machine learning model that learns the angle of the viewpoint relative to the object and the image of the object viewed from that viewpoint. The image generation model 123 is consistent with the color and texture of the object depicted in the input image and is used to generate an image of the object viewed from a different viewpoint. The image generation model 123 may be an image generation model obtained by fine-tuning such an image generation model to a railway asset using Low-Rank Adaptation (LoRA). Next, the model generation unit 115 uses the image 243 as input and executes a 3D conversion process 244 to generate a 3D model 245.

[0042] The model generation unit 115 generates sounds related to the asset in addition to the image. More specifically, the model generation unit 115 generates sounds corresponding to the keyword 233 using the sound diffusion model 124. Examples of sounds include an alarm sound, a sound during tapping, and a sound during asset operation.

[0043] 8 is a diagram illustrating the sound diffusion model 124 according to this embodiment. The sound diffusion model 124 is a diffusion model that has learned the correspondence between a keyword 251 and a spectrum 253 of a sound 252 corresponding to the keyword 251. The model generation unit 115 acquires a spectrum corresponding to the keywords 232 and 233 using the sound diffusion model 124, and generates a sound corresponding to the keywords 232 and 233 by converting the spectrum into a sound.

[0044] 1, the description of the control unit 110 will continue. The display control unit 116 refers to the asset placement data 150, and outputs and displays the assets 234 placed in the virtual space 235 on a display connected to the input / output unit 180. The display control unit 116 may also output and display the assets 234 on a head-mounted display.

[0045] As described above, the virtual space generating device 100 includes a model generation unit 115 that acquires a first image (see image 242), which is an image of an asset, from the asset's keywords 232, 233 using the image diffusion model 122. Based on the first image, the model generation unit 115 generates a second image (see image 243), which is an image of the asset viewed from a different direction from the first image, using the image generation model 123. The model generation unit 115 performs three-dimensional conversion processing 244 based on the first image and the second image to generate a three-dimensional model 245 of the asset.

[0046] When an asset corresponding to a keyword 232, 233 does not exist in the asset database 130, the asset acquisition unit 112 instructs the model generation unit 115 to generate and acquire a three-dimensional model 245 related to the keyword. The model generation unit 115 generates sound for the asset based on the asset's keyword using a sound diffusion model 124. The virtual space generation device 100 includes a display control unit 116 that outputs the asset to a display device (e.g., a monitor).

[0047] <Features of the Virtual Space Generator> The virtual space generator 100 provides a virtual space 235 for maintenance work training, in which facilities and equipment (assets) desired by the user, who is a maintenance worker, are placed. First, the maintenance worker inputs the training content and the facilities and equipment in a language (linguistic expression) such as text or voice. The virtual space generator then acquires keywords 232 and 233 using the language model 121 and terminology library 140 based on the input language (see text 231). Next, the virtual space generator 100 references the asset database 130 to acquire assets 234 and places them in the virtual space 235. The maintenance worker performs maintenance work training in the virtual space 235.

[0048] With this virtual space generating device 100, maintenance personnel can generate a virtual space and train without the need to go to the trouble of searching for and arranging the facilities and equipment they are training on. Furthermore, maintenance personnel can rearrange the facilities and equipment by issuing an instruction to change the asset layout to the virtual space generating device 100. Because maintenance personnel can train while changing the combination and layout of facilities and equipment, it is expected that they will be able to acquire knowledge and skills efficiently.

[0049] Although several embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the technical scope of the present invention. The virtual space generation device 100 in the above-described embodiments is intended for training in railway maintenance work, but is not limited to this. By constructing the language model 121, image diffusion model 122, image generation model 123, asset database 130, etc. according to the work, the virtual space generation device 100 can create virtual spaces for work or work training related to the development, design, production, and maintenance of other facilities, equipment, and devices.

[0050] The present invention can take on various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and modifications are included in the scope and spirit of the invention described in this specification, etc., and are also included in the invention described in the claims and their equivalents.

[0051] <Hardware Configuration> The virtual space generation device 100 according to the above-described embodiment is realized by, for example, a computer 900 configured as shown in Fig. 9. Fig. 9 is a hardware configuration diagram showing an example of a computer 900 that realizes the functions of the virtual space generation device 100 according to the above-described embodiment. The computer 900 includes a CPU 901, a ROM 902, a RAM 903, an SSD 904, an input / output interface 905 (referred to as an input / output I / F (Interface) in Fig. 9), a communication interface 906 (referred to as a communication I / F in Fig. 9), and a media interface 907 (referred to as a media I / F in Fig. 9). The computer 900 may include a hard disk drive (HDD) instead of the SSD 904, or may include an HDD in addition to the SSD 904.

[0052] The CPU 901 operates based on a program stored in the ROM 902 or the SSD 904, and performs control by the control unit 110 in Fig. 9. The ROM 902 stores a boot program executed by the CPU 901 when the computer 900 is started, programs related to the hardware of the computer 900, and the like.

[0053] The CPU 901 controls input devices 910 such as a mouse, keyboard, and microphone, and output devices 911 such as a display and printer, via an input / output interface 905. The CPU 901 acquires data from the input device 910 via the input / output interface 905, and outputs generated data to the output device 911.

[0054] The SSD 904 stores programs executed by the CPU 901 and data used by the programs. The communication interface 906 receives data from other devices (not shown) via a communication network and outputs the data to the CPU 901, and also transmits data generated by the CPU 901 to other devices via the communication network.

[0055] The media interface 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 is an optical recording medium such as a DVD (Digital Versatile Disk), a magneto-optical recording medium such as an MO (Magneto Optical Disk), a magnetic recording medium, a conductive memory tape medium, a semiconductor memory, or the like.

[0056] For example, when the computer 900 functions as the virtual space generating device 100 according to the embodiment described above, the CPU 901 of the computer 900 executes a program 128 (see FIG. 1 ) loaded onto the RAM 903, thereby realizing the functions of the virtual space generating device 100. The CPU 901 reads and executes the program from a recording medium 912. Alternatively, the CPU 901 may read the program from another device via a communication network, or may install and execute the program 128 from the recording medium 912 onto the SSD 904.

[0057] REFERENCE SIGNS LIST 100 Virtual space generation device 111 Model driving unit 112 Asset acquisition unit 113 Asset placement unit 114 Change instruction reception unit 115 Model generation unit 116 Display control unit 121 Language model 122 Image diffusion model 123 Image generation model 124 Sound diffusion model 128 Program 130 Asset database 140 Terminology library 150 Asset placement data

Claims

1. A virtual space generation device comprising: an asset acquisition unit that uses a language model to acquire keywords for assets related to the content of work to be performed in a virtual space based on the linguistic expression of the content of the work; an asset placement unit that places the assets in the virtual space; and a display control unit that outputs the assets to a display device, by referring to an asset database in which the keywords and assets are stored in association with each other.

2. A virtual space generating device as described in claim 1, wherein the asset database includes attribute values ​​related to the assets, and the asset acquisition unit acquires assets whose attribute values ​​successfully match the acquired keywords.

3. The virtual space generation device according to claim 2, wherein the asset acquisition unit acquires, in addition to the acquired first keyword, a second keyword that is a related word of the first keyword or a keyword with the same meaning by referring to a term library that stores related words or meanings of the keyword, and acquires assets of attribute values ​​that successfully match the first keyword or the second keyword.

4. A virtual space generation device as described in claim 1, comprising: a model generation unit that acquires a first image, which is an image of the asset, from a keyword of the asset using an image diffusion model; generates a second image, which is an image of the asset viewed from a different direction from the first image, using an image generation model based on the first image; and performs a three-dimensional conversion process based on the first image and the second image to generate a three-dimensional model of the asset, wherein the asset acquisition unit instructs the model generation unit to generate and acquire a three-dimensional model related to the keyword when an asset corresponding to the keyword does not exist in the asset database.

5. The virtual space generating device according to claim 4, wherein the model generating unit generates the sound of the asset using a sound diffusion model based on the asset's keywords.

6. The virtual space generating device according to claim 1, wherein the asset database further stores positional relationships between the assets, and the asset placement unit places the assets in accordance with the positional relationships.

7. The virtual space generating device according to claim 1, further comprising a change instruction receiving unit that receives an instruction to change the placement of the asset, wherein the asset placement unit places the asset in accordance with the placement change instruction.

8. A virtual space generation method in which a virtual space generation device executes the steps of: acquiring keywords for assets related to the content of work to be performed in the virtual space using a language model based on the linguistic expression of the content of the work; referencing an asset database in which the keywords and assets are stored in association with each other to acquire assets that are the subject of the work based on the acquired keywords; arranging the assets in the virtual space; and outputting the assets to a display device.

Images