Simulation device, simulation method, and program

Abstract

To allow pilot training of a control target or remote control of a control target to be carried out while maintaining access restrictions to the control target.SOLUTION: An acquisition unit acquires data on piloting operations performed by a pilot. A motion unit drives an avatar associated with the pilot in a virtual world according to the acquired data on piloting operations. A simulation unit simulates control of the control target in the real world through actions of the avatar associated with the pilot.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a simulation device, a simulation method, and a program, and more particularly to a simulation device, a simulation method, and a program for simulating the control of a control target. [Background technology]

[0002] Related technologies acquire sensing data from objects such as people and objects, extract features from the sensing data, and generate avatars or icons from the extracted features and predetermined parameters (e.g., object position, object attributes, time, environment).

[0003] For example, Patent Document 1 describes a specific example of a simulation system that generates an image seen from a virtual camera in a virtual space (virtual world). One example of this system realizes virtual reality in which an image seen from the virtual camera is displayed on an HMD (Head Mount Display). Another example provides a racing game in which a user can enjoy driving a race car in a virtual space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-000171 Summary of the Invention [Problem to be solved by the invention]

[0005] There are many cases where beginners are required to train on controlled objects or where operators are required to remotely control controlled objects. However, depending on the controlled object, due to confidentiality requirements or other reasons, only authorized personnel with administrative authority are permitted to directly control the controlled object.

[0006] In the above cases, the related technology cannot provide the pilot with an opportunity to practice piloting the controlled object or to remotely control the controlled object.

[0007] The present disclosure has been made in consideration of the above-mentioned problems, and its purpose is to enable pilot training of a controlled object or remote operation of a controlled object while maintaining access restrictions to the controlled object. [Means for solving the problem]

[0008] A simulation device according to one aspect of the present disclosure includes an acquisition means for acquiring data on control operations performed by a pilot, an operation means for operating an avatar associated with the pilot in a virtual world based on the acquired data on the control operations, and an imitation means for simulating control of a control object in the real world through the operation of the avatar associated with the pilot.

[0009] In a simulation method according to one aspect of the present disclosure, a computer acquires data on control operations performed by a pilot, and based on the acquired data on control operations, moves an avatar associated with the pilot in a virtual world, simulating control of a control target in the real world via the movement of the avatar associated with the pilot.

[0010] A program according to one aspect of the present disclosure causes a computer to perform the following processes: acquiring data on control operations performed by a pilot; operating an avatar associated with the pilot in a virtual world based on the acquired data on the control operations; and simulating control of a control target in the real world through the operation of the avatar associated with the pilot. [Effects of the Invention]

[0011] According to one aspect of the present disclosure, it is possible to perform pilot training or remote control of a controlled object while maintaining access restrictions on the controlled object. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of a system including a simulation device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating a flow of a simulation of control of a control target performed by a simulation device according to an embodiment. [Figure 3] 1 is a block diagram showing a configuration of a simulation device according to an embodiment; [Figure 4] 1 is a flowchart illustrating an operation of a simulation device according to an embodiment. [Figure 5] 1 is a block diagram showing a configuration of a simulation device according to an embodiment; [Figure 6] 1 is a flowchart illustrating an operation of a simulation device according to an embodiment. [Figure 7] FIG. 10 is a diagram illustrating a process performed by a confirmation unit of the simulation device according to an embodiment. [Figure 8] 1 is a block diagram showing a configuration of a simulation device according to an embodiment; [Figure 9] 1 is a flowchart illustrating an operation of a simulation device according to an embodiment. [Figure 10] FIG. 10 is a block diagram showing a configuration of a simulation device according to a modified example of an embodiment. [Figure 11] 10 is a flowchart illustrating an operation of a simulation device according to a modified example of an embodiment. [Figure 12] FIG. 1 illustrates an example of a hardware configuration of a simulation apparatus according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] (An example of System 1) A system 1 including any one of simulation devices 10, 20, 30, and 30A according to embodiments 1 to 3 described below will be described with reference to Fig. 1. Fig. 1 is a diagram schematically illustrating an example of the system 1.

[0014] As shown in Fig. 1, the system 1 includes a simulation device 10 (20, 30, 30A), a head mounted display (HMD) 100, a control device 200, and a controlled object 300. Note that there may be one or more of each of the head mounted display (HMD) 100, the control device 200, and the controlled object 300. In the system 1, an avatar used by the operator and information used to verify the operator's identity (e.g., biometric information) are linked to the operator's identification information and registered in advance in a database 400 shown in Fig. 1.

[0015] Here, "simulation device 10 (20, 30, 30A)" means "any of the simulation devices 10, 20, 30, 30A."

[0016] The HMD 100 is a display device worn on the head of an operator. In one example, the HMD 100 is equipped with a monitor and a sensor. The monitor of the HMD 100 displays a VR (Virtual Reality) image generated by the simulation device 10 (20, 30, 30A). The sensor of the HMD 100 detects sensing data including biometric information and feature quantities (e.g., iris image, voice data, acceleration data) of the operator.

[0017] The HMD 100 transmits sensing data to the simulation device 10 (20, 30, 30A) via an arbitrary communication network NW (FIG. 1). The communication network NW is, for example, any combination of a local area network (LAN), a wide area network (WAN), a mobile network, and the Internet.

[0018] The control device 200 is used by the operator to perform control operations while viewing the VR image displayed on the monitor of the HMD 100. For example, the control device 200 is at least one of a control stick, a side stick, a control rudder, and pedals. The control device 200 converts the operator's input (control operations) into signal data.

[0019] The control device 200 transmits signal data to the simulation device 10 (20, 30, 30A) via any communication network NW (FIG. 1).

[0020] The control target 300 is, for example, a special machine or special mobility that requires operational training, or an industrial robot. The control target 300 receives a control signal from a simulation device 10 (20, 30, 30A) via an arbitrary communication network NW (FIG. 1).

[0021] The simulation device 10 (20, 30, 30A) acquires data on the control operation performed by the operator from the control device 200. The simulation device 10 (20, 30, 30A) moves an avatar associated with the operator in the virtual world based on the acquired data on the control operation. The simulation device 10 (20, 30, 30A) simulates the control of the control target 300 via the movement of the avatar associated with the operator.

[0022] Furthermore, the simulation device 10 (20, 30, 30A) may actually control the control object 300 in the real world based on the result of simulating the control of the control object 300. In this case, the simulation device 10 (20, 30, 30A) generates a control signal for controlling the control object 300 based on the result of simulating the control of the control object 300. Then, the simulation device 10 (20, 30, 30A) transmits the control signal to the control object 300 via an arbitrary communication network NW (FIG. 1).

[0023] Furthermore, the simulation device 10 (20, 30, 30A) may generate VR video data that represents how the control target 300 is controlled by a steering operation based on the results of simulating the control of the control target 300, and transmit the VR video data to the HMD 100 via an arbitrary communication network NW (FIG. 1). For example, the VR video data represents the movement of the virtual control target 300 in the virtual world.

[0024] (Flow of simulation of control target control) The flow of a simulation of the control of the control target 300 by the simulation device 10 (20, 30, 30A) will be described with reference to Fig. 2. The flow of the simulation described here is merely an example.

[0025] As shown in FIG. 2, first, a pilot wearing the HMD 100 uses a control device 200 such as a joystick or a handle to perform a control operation to move an avatar. The control device 200 transmits input (control operation) data by the pilot to the simulation device 10 (20, 30, 30A). Note that the pilot is not permitted to directly control the control target 300. Therefore, the control device 200 cannot transmit the input (control operation) data by the pilot to the control target 300.

[0026] The simulation device 10 (20, 30, 30A) moves an avatar associated with a controller based on input data received from the controller 200. In the virtual world, the avatar associated with the controller imitates the controller's control operations. The avatar performs control operations on a virtual control target 300.

[0027] Furthermore, the simulation device 10 (20, 30, 30A) executes a simulation of the control of the control target 300 based on input (operational control) data from the operator. For example, the simulation device 10 (20, 30, 30A) reproduces, in a virtual world, how the control target 300 behaves in response to the operation of the operator (or avatar) through physical simulation.

[0028] The simulation device 10 (20, 30, 30A) generates VR video data to be displayed on the monitor of the HMD 100 based on the results of simulating the control of the control target 300. The simulation device 10 (20, 30, 30A) transmits the generated VR video data to the HMD 100 via an arbitrary communication network NW (FIG. 1).

[0029] The simulation device 10 (20, 30, 30A) may actually control the control object 300 in the real world based on the results of simulating the control of the control object 300. In this case, the simulation device 10 (20, 30, 30A) generates a control signal for controlling the control object 300 based on the operation of the avatar. Then, the simulation device 10 (20, 30, 30A) transmits the generated control signal to the control object 300 via an arbitrary communication network NW (FIG. 1).

[0030] Alternatively, the simulation device 10 (20, 30, 30A) may determine an abnormality in the control operation performed by the operator based on the result of simulating the control of the control target 300. An abnormality here means a control operation that differs from a predetermined procedure or a control operation that makes the operation of the control target 300 unstable.

[0031] If it is determined that there is an abnormality in the control operation performed by the pilot, the simulation device 10 (20, 30, 30A) may correct the abnormality and then actually control the control target 300 in the real world. Alternatively, the simulation device 10 (20, 30, 30A) may stop actually controlling the control target 300 based on the result of simulating the control of the control target 300.

[0032] When a pilot is training to operate the control target 300, the simulation device 10 (20, 30, 30A) may notify the pilot or a training instructor that there is an abnormality in the pilot operation performed by the pilot. In one example, the simulation device 10 (20, 30, 30A) displays an alert on the HMD 100 indicating that there is an abnormality in the pilot operation performed by the pilot. Furthermore, the simulation device 10 (20, 30, 30A) may display an explanation of the reason why the pilot operation was determined to be abnormal on the monitor of the HMD 100.

[0033] In this way, system 1 can allow an operator to train or remotely control controlled object 300 while restricting unauthorized operators from directly accessing controlled object 300.

[0034] The configuration and operation of the simulation device 10 (20, 30, 30A) will be described in the following first to third embodiments.

[0035] [Embodiment 1] The first embodiment will be described with reference to FIGS.

[0036] (Configuration of simulation device 10) 3 is a block diagram showing the configuration of the simulation device 10 according to the present embodiment 1. As shown in FIG.

[0037] The acquisition unit 11 acquires data on the operation performed by the operator. The acquisition unit 11 is an example of an acquisition means.

[0038] For example, the acquisition unit 11 receives input (steering operation) data by the operator from the control device 200 (FIG. 1). The acquisition unit 11 outputs the input data received from the control device 200 to the operation unit 12.

[0039] The operation unit 12 moves the avatar associated with the operator in the virtual world based on the acquired operation data. The operation unit 12 is an example of an operation means.

[0040] For example, the operation unit 12 receives input data indicating a control operation performed by the operator from the acquisition unit 11. The operation unit 12 converts the received input data into information indicating the movement of the avatar by referring to a table (not shown) or the like that associates the control operation performed by the operator with the movement of the avatar.

[0041] Then, the operation unit 12 operates the avatar in the virtual world based on the obtained information. The avatar's operation at this time corresponds to the control operation performed by the operator. For example, when the operator performs a control operation by moving a joystick (an example of the control device 200), the avatar also performs an operation in the virtual world that corresponds to the same control operation as the operator.

[0042] The movement unit 12 outputs information indicating the movement of the avatar to the simulation unit 13 .

[0043] The simulator 13 simulates the control of the control target 300 in the real world through the movement of an avatar associated with the operator. The simulator 13 is an example of a simulator.

[0044] For example, the simulator 13 outputs information indicating the movement of the avatar associated with the operator from the operation unit 12. The simulator 13 calculates how the control target 300 will be controlled when the avatar moves in that way, according to a simulation program prepared in advance. The simulator 13 then outputs the calculation result, i.e., the result of simulating the control of the control target 300 in the real world.

[0045] In one example, the simulator 13 generates VR video data that expresses how the control target 300 is controlled by a steering operation, based on the results of simulating the control of the control target 300. Then, the simulator 13 transmits the generated VR video data to the HMD 100 via an arbitrary communication network NW (FIG. 1).

[0046] (Operation of the simulation device 10) 4 is a flowchart showing the operation of the simulation device 10 according to the present embodiment 1. As shown in FIG. 4, the acquisition unit 11 acquires data on the control operation performed by the operator from the control device 200 (FIG. 1) (S101). The acquisition unit 11 outputs the input data received from the control device 200 to the operation unit 12.

[0047] Next, the operation unit 12 causes the avatar associated with the operator to move in the virtual world based on the acquired control operation data (S102). The operation unit 12 outputs information indicating the movement of the avatar to the simulation unit 13.

[0048] The simulator 13 simulates the control of the control target 300 in the real world through the movement of the avatar associated with the operator (S103).

[0049] Thereafter, the simulator 13 outputs the result of simulating the control of the control target 300 to an external device, a storage device, or another device (not shown). For example, the simulator 13 displays an image representing the movement of the control target 300 on the HMD 100 based on the result of simulating the control of the control target 300.

[0050] This completes the operation of the simulation device 10 according to the first embodiment.

[0051] (Effects of this embodiment) According to the configuration of this embodiment, the acquisition unit 11 acquires data on the control operation performed by the operator. The operation unit 12 operates an avatar associated with the operator in the virtual world based on the acquired data on the control operation. The simulation unit 13 simulates the control of the control target 300 in the real world through the operation of the avatar associated with the operator.

[0052] Therefore, based on the control operation by the operator, the avatar associated with the operator moves, and the avatar controls the virtual control target 300. The virtual control target 300 is controlled based on the results of simulating the control of the control target 300. The operator does not need to directly access the control target 300 in the real world.

[0053] This makes it possible to perform training on the control target 300 or remotely control the control target 300 while maintaining access restrictions on the control target 300.

[0054] [Embodiment 2] A second embodiment will be described with reference to Fig. 5 to Fig. 7. In the second embodiment, a configuration will be described in which the identity of the operator is verified periodically or irregularly while data on the operation performed by the operator is being acquired.

[0055] In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

[0056] (Configuration of simulation device 20) Fig. 5 is a block diagram showing the configuration of a simulation device 20 according to the second embodiment. As shown in Fig. 5, the simulation device 20 includes an acquisition unit 11, an operation unit 12, and a simulation unit 13. The simulation device 20 further includes a confirmation unit 24.

[0057] The confirmation unit 24 verifies the identity of the pilot while the acquisition unit 11 is acquiring data of the pilot's operation (input data). The confirmation unit 24 is an example of a confirmation means. Furthermore, the confirmation unit 24 may verify the identity of the pilot before the acquisition unit 11 acquires the data of the pilot's operation (input data).

[0058] For example, the confirmation unit 24 acquires an iris image from the HMD 100 (FIG. 1) obtained by capturing an image of the pilot's iris with the in-camera of the HMD 100. The confirmation unit 24 then uses the iris image of the pilot to verify the pilot's identity. In one example, while the pilot is performing a piloting operation in accordance with a piloting training scenario provided by the simulation device 20, the confirmation unit 24 may repeatedly acquire the pilot's biometric information as part of the piloting training scenario and verify the pilot's identity each time the biometric information is acquired. This has the advantage that the pilot's identity can be verified multiple times while the piloting training is being performed, thereby preventing anyone other than the pilot from performing the piloting training, and further, the immersive feeling of piloting is not impaired.

[0059] The verification unit 24 may also verify the identity of the operator using biometric information other than an iris image (for example, the operator's face, fingerprint, veins, or ear acoustic features). For example, the verification unit 24 may verify the identity of the operator using video (an example of biometric information) obtained by capturing the operator's gestures. The above-described system 1 (FIG. 1) may also include a spatial recognition device (for example, a camera) for capturing the operator's gestures. Alternatively, the verification unit 24 may verify the identity of the operator using a voiceprint (an example of biometric information) obtained by recording the operator's speech.

[0060] Furthermore, the verification unit 24 authenticates the authenticity of the avatar using the result of the identity verification of the pilot. Specifically, if the identity verification of the pilot is successful, the verification unit 24 updates the expiration date of the trust authentication of the avatar associated with the pilot.

[0061] On the other hand, if the verification unit 24 fails to verify the identity of the operator, it invalidates the trust authentication of the avatar associated with the operator. The operator and avatar are associated only before the expiration date of the avatar's trust authentication. When the expiration date of the avatar's trust authentication expires, the association between the operator and avatar is lost. The operator will no longer be able to control the avatar.

[0062] The verification unit 24 outputs the result of verifying the identity of the operator to the operation unit 12.

[0063] The operation unit 12 receives input data indicating the operation performed by the operator from the acquisition unit 11. Furthermore, the operation unit 12 receives from the verification unit 24 the result of verification of the operator's identity.

[0064] When the trust authentication of the avatar is updated, the operation unit 12 operates the avatar associated with the operator in the virtual world based on the input data indicating the operation received from the acquisition unit 11.

[0065] On the other hand, if the trust authentication of the avatar has expired, the operation unit 12 will not allow the operator to operate the avatar in the virtual world. In this case, the verification unit 24 may retry verifying the identity of the operator using biometric information or other information of the operator.

[0066] In one example, if identity verification based on the iris image of the pilot fails, the verification unit 24 requests the pilot to open his / her eyes widely. Thereafter, the verification unit 24 reacquires an iris image obtained by photographing the pilot's iris from the HMD 100. The verification unit 24 then retries verifying the pilot's identity using the recaptured iris image. Note that a request to the pilot to reacquire the pilot's biometric information may also be implemented in the piloting training scenario provided by the simulation device 20.

[0067] In another example, the confirmation unit 24 may acquire characteristics related to the manner of movement by having the operator perform a gesture such as a pointing movement, and use the acquired characteristics related to the manner of movement for identity verification. In yet another example, if identity verification based on the iris image of the operator fails, the confirmation unit 24 may retry identity verification using biometric information other than the iris image (for example, a voiceprint).

[0068] (Operation of the simulation device 20) The operation of the simulation device 20 according to the second embodiment will be described with reference to Figures 6 and 7. Figure 6 is a flowchart showing the operation of the simulation device 20 according to the second embodiment.

[0069] 6, the acquisition unit 11 acquires data on the control operation performed by the operator from the control device 200 (FIG. 1) (S201). The acquisition unit 11 outputs the input data received from the control device 200 to the operation unit 12.

[0070] Next, the verification unit 24 verifies the identity of the operator while acquiring data on the operation performed by the operator (S202).

[0071] As shown in FIG. 7, in step S204, the confirmation unit 24 acquires the operator's biometric information (S2021).

[0072] Next, the verification unit 24 collates the biometric information of the operator with the biometric information registered in the database 400 (FIG. 1) (S2022).

[0073] The verification unit 24 determines whether the identity verification of the operator has been successful (S2023). Furthermore, the verification unit 24 authenticates the authenticity of the avatar using the result of the identity verification of the operator.

[0074] If the verification unit 24 has successfully verified the identity of the operator (Yes in S2023), the verification unit 24 updates the expiration date of the trust authentication of the avatar associated with the operator (S2024A).

[0075] On the other hand, if the verification unit 24 fails to verify the identity of the operator (No in S2023), the trust authentication of the avatar associated with the operator is invalidated (S2024B). When the trust authentication of the avatar is invalidated, the association between the operator and the avatar is lost. As a result, the operator is no longer able to control the avatar.

[0076] The verification unit 24 outputs the result of verifying the identity of the operator to the operation unit 12. The flow proceeds to step S203 in FIG.

[0077] In step S202, if the trust authentication of the avatar associated with the operator is updated (Yes in S203), the operation unit 12 causes the avatar associated with the operator to move in the virtual world based on the acquired operation data (S204). The operation unit 12 outputs information indicating the movement of the avatar to the simulation unit 13.

[0078] The simulator 13 simulates the control of the control target 300 in the real world through the movement of the avatar associated with the operator (S205).

[0079] Thereafter, the simulator 13 outputs the results of simulating the control of the controlled object 300 to an external device or storage device (not shown), or other device.

[0080] On the other hand, if the trust authentication of the avatar associated with the operator has expired in step S202 (No in S203), the operation unit 12 prevents the operator from operating the avatar in the virtual world, and the flow then ends.

[0081] This completes the operation of the simulation device 20 according to the second embodiment.

[0082] (Effects of this embodiment) According to the configuration of this embodiment, the acquisition unit 11 acquires data on the control operation performed by the operator. The operation unit 12 operates an avatar associated with the operator in the virtual world based on the acquired data on the control operation. The simulation unit 13 simulates the control of the control target 300 in the real world through the operation of the avatar associated with the operator.

[0083] Therefore, based on the control operation by the operator, the avatar associated with the operator moves, and the avatar controls the virtual control target 300. The virtual control target 300 is controlled based on the results of simulating the control of the control target 300. The operator does not need to directly access the control target 300 in the real world.

[0084] This makes it possible to perform training on the control target 300 or remotely control the control target 300 while maintaining access restrictions on the control target 300.

[0085] Furthermore, according to the configuration of this embodiment, the confirmation unit 24 verifies the identity of the pilot while acquiring data on the pilot's operation. If the confirmation unit 24 successfully verifies the identity of the pilot, the operation unit 12 operates an avatar associated with the pilot in the virtual world based on the acquired pilot operation data. This prevents a malicious third party from impersonating the pilot and moving the avatar. Furthermore, according to the configuration of this embodiment, the identity of the pilot is confirmed as part of the piloting training scenario while the pilot is performing the piloting operation, which has the advantage that the immersive feeling of the piloting operation is not impaired due to the pilot's identity confirmation.

[0086] [Embodiment 3] A third embodiment will be described with reference to Figures 8 to 11. In the third embodiment, a configuration for controlling the controlled object 300 (Figure 1) will be described based on the results of simulating the control of the controlled object 300.

[0087] In the third embodiment, the same components as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

[0088] (Configuration of simulation device 30) Fig. 8 is a block diagram showing the configuration of a simulation device 30 according to the third embodiment. As shown in Fig. 8, the simulation device 30 includes an acquisition unit 11, an operation unit 12, and a simulation unit 13. The simulation device 30 further includes a control unit 34.

[0089] The control unit 34 controls the control target 300 (FIG. 1) in the real world based on the results of simulating the control of the control target 300. The control unit 34 is an example of a control means.

[0090] For example, the control unit 34 receives the result of simulating the control of the control target 300 from the simulator 13. The control unit 34 actually controls the control target 300 based on the result of the simulation by the simulator 13.

[0091] In one example, the control unit 34 controls the control object 300 in the real world so as to trace the control of the simulated control object 300. The control here means causing the control object 300 to perform a function or operating a drive unit provided in the control object 300.

[0092] The control unit 34 generates a control signal for controlling the control target 300 based on the result of simulating the control of the control target 300. Then, the control unit 34 transmits the control signal to the control target 300 via an arbitrary communication network NW (FIG. 1).

[0093] (Operation of the simulation device 30) 9 is a flowchart showing the operation of the simulation device 30 according to the third embodiment. As shown in FIG. 9, the acquisition unit 11 acquires data on the control operation performed by the operator from the control device 200 (FIG. 1) (S301). The acquisition unit 11 outputs the input data received from the control device 200 to the operation unit 12.

[0094] Next, the operation unit 12 causes the avatar associated with the operator to move in the virtual world based on the acquired control operation data (S302). The operation unit 12 outputs information indicating the movement of the avatar to the simulation unit 13.

[0095] The simulator 13 simulates the control of the control target 300 in the real world through the movement of the avatar associated with the operator (S303).

[0096] Thereafter, the simulator 13 outputs the results of simulating the control of the controlled object 300 to an external device or storage device (not shown), or other device.

[0097] Moreover, the simulator 13 outputs the results of simulating the control of the controlled object 300 to the controller .

[0098] The control unit 34 controls the control target 300 in the real world based on the result of simulating the control of the control target 300 (S304).

[0099] This completes the operation of the simulation device 30 according to the third embodiment.

[0100] [Modification] A modification of the third embodiment will now be described.

[0101] (Configuration of Simulation Device 30A) Fig. 10 is a block diagram showing the configuration of a simulation device 30A according to a modified example of the present embodiment 3. As shown in Fig. 10, the simulation device 30A includes an acquisition unit 11, an operation unit 12, and a simulation unit 13. The simulation device 30A further includes a control unit 34 and a determination unit 35.

[0102] The determination unit 35 determines whether there is an abnormality in the control operation performed by the operator based on the results of simulating the control of the control target 300 (FIG. 1). The determination unit 35 is an example of a determination means. An abnormality here means a control operation that differs from a predetermined procedure or a control operation that makes the operation of the control target 300 unstable.

[0103] For example, the determination unit 35 receives the results of a simulation of the control of the control target 300 from the simulator 13. The determination unit 35 determines whether the operation of the control target 300 will become unstable if the control target 300 is actually controlled in the real world according to the results of the simulation. For example, if the control target 300 is an aircraft and there is a possibility that the altitude of the control target 300 will suddenly decrease, the determination unit 35 determines that there is an abnormality in the control operation performed by the pilot, since the operation of the control target 300 will become unstable.

[0104] If it is determined that there is an abnormality in the control operation performed by the pilot, the determination unit 35 may correct the abnormality. Alternatively, the determination unit 35 may stop actually controlling the control target 300 based on the result of simulating the control of the control target 300.

[0105] When the pilot is training to control the controlled object 300, the determination unit 35 may notify the pilot or the instructor that there is an abnormality in the pilot operation performed by the pilot. In one example, the determination unit 35 displays an alert indicating that there is an abnormality in the pilot operation performed by the pilot on the HMD 100. Furthermore, the determination unit 35 may display an explanation of the reason why the pilot operation is determined to be abnormal on the HMD 100.

[0106] (Operation of simulation device 30A) 10 is a flowchart showing the operation of simulation device 30A according to a modified example of Embodiment 3. As shown in FIG. 10, acquisition unit 11 acquires data on the control operation performed by the pilot from control device 200 (FIG. 1) (S301′). Acquisition unit 11 outputs the input data received from control device 200 to operation unit 12.

[0107] Next, the operation unit 12 causes the avatar associated with the operator to move in the virtual world based on the acquired control operation data (S302'). The operation unit 12 outputs information indicating the movement of the avatar to the simulation unit 13.

[0108] The simulator 13 simulates the control of the control target 300 in the real world through the movement of the avatar associated with the operator (S303').

[0109] The determination unit 35 detects an abnormality in the control operation performed by the operator based on the result of simulating the control of the controlled object 300 (S304').

[0110] If there is an abnormality in the control operation performed by the pilot (Yes in S305'), the flow ends.

[0111] On the other hand, if there is no abnormality in the control operation performed by the pilot (No in S305'), the simulation unit 13 outputs the results of simulating the control of the controlled object 300 to an external device or storage device not shown, or other device.

[0112] Moreover, the simulator 13 outputs the results of simulating the control of the controlled object 300 to the controller .

[0113] The control unit 34 controls the control target 300 in the real world based on the result of simulating the control of the control target 300 (S306').

[0114] This completes the operation of the simulation device 30A according to the modified example of the third embodiment.

[0115] (Effects of this embodiment) According to the configuration of this embodiment, the acquisition unit 11 acquires data on the control operation performed by the operator. The operation unit 12 operates an avatar associated with the operator in the virtual world based on the acquired data on the control operation. The simulation unit 13 simulates the control of the control target 300 in the real world through the operation of the avatar associated with the operator.

[0116] Therefore, based on the control operation by the operator, the avatar associated with the operator moves, and the avatar controls the virtual control target 300. The virtual control target 300 is controlled based on the results of simulating the control of the control target 300. The operator does not need to directly access the control target 300 in the real world.

[0117] This makes it possible to perform training on the control target 300 or remotely control the control target 300 while maintaining access restrictions on the control target 300.

[0118] Furthermore, according to the configuration of this embodiment, the control unit 34 controls the control target 300 in the real world based on the results of simulating the control of the control target 300. This makes it possible to perform training in operation of the control target 300 or remotely control the control target 300 while maintaining access restrictions on the control target 300.

[0119] (Example of hardware configuration) Each of the components of the simulation devices 10, 20, 30, and 30A described in the first to third embodiments represents a functional block. Some or all of these components are realized by an information processing device such as that shown in Fig. 11. Fig. 11 is a block diagram showing an example of the hardware configuration of the information processing device.

[0120] 11, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. Note that the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111.

[0121] The CPU 111 loads the programs (codes) of this embodiment stored in the storage device 113 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). The programs of this embodiment are provided in a state stored in a computer-readable recording medium 120. The programs of this embodiment may be distributed over the Internet connected via the communication interface 117.

[0122] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0123] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0124] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0125] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0126] (Appendix 1) an acquisition means for acquiring data on the operation performed by the pilot; an operation means for operating an avatar associated with the operator in a virtual world based on the acquired operation data; a simulation means for simulating the control of a control object in the real world through the movement of an avatar associated with the operator; A simulation device comprising:

[0127] (Appendix 2) The system further includes a verification means for verifying the identity of the operator while acquiring the data of the operation performed by the operator. 2. The simulation device according to claim 1,

[0128] (Appendix 3) While the data of the operation performed by the operator is being acquired, the verification means repeatedly verifies the identity of the operator using biometric information of the operator. 3. The simulation device according to claim 2,

[0129] (Appendix 4) The verification means verifies the authenticity of the avatar using the result of the identity verification of the operator. 4. The simulation device according to claim 2 or 3.

[0130] (Appendix 5) The system further includes a control unit that controls the control object in the real world based on the result of simulating the control of the control object. 2. The simulation device according to claim 1,

[0131] (Appendix 6) The control system further includes a determination unit for determining whether the control operation performed by the operator is abnormal based on the results of a simulation of the control of the controlled object. 6. The simulation device according to claim 5,

[0132] (Appendix 7) The computer Acquire data on the pilot's operations, operating an avatar associated with the operator in a virtual world based on the acquired data of the operation; Control of a control object is simulated through the movement of an avatar associated with the operator. Simulation method.

[0133] (Appendix 8) The computer While acquiring data on the operation performed by the operator, the operator's identity is verified. 8. The simulation method according to claim 7,

[0134] (Appendix 9) While the computer is acquiring data on the operation performed by the operator, the computer repeatedly verifies the identity of the operator using biometric information of the operator. 9. The simulation method according to claim 8,

[0135] (Appendix 10) The computer authenticates the authenticity of the avatar using the results of the pilot's identity verification. 10. The simulation method according to claim 8 or 9,

[0136] (Appendix 11) The computer controls the control object in the real world based on the result of simulating the control of the control object. 8. The simulation method according to claim 7,

[0137] (Appendix 12) The computer determines an abnormality in the control operation performed by the operator based on a result of simulating the control of the controlled object. 12. The simulation method according to claim 11,

[0138] (Appendix 13) A process of acquiring data on the operation performed by the pilot; a process of operating an avatar associated with the operator in a virtual world based on the acquired data of the operation; a process of simulating control of a control object through the movement of an avatar associated with the operator; A program that causes a computer to execute the following.

[0139] (Appendix 14) While acquiring data on the operation performed by the operator, the operator's identity is verified, and the avatar's trust is authenticated using the result of the verification of the operator's identity. 14. The program according to claim 13, which causes the computer to execute the above.

[0140] (Appendix 15) A process of repeatedly verifying the identity of the operator using biometric information of the operator while acquiring data on the operation performed by the operator. 15. The program according to claim 14, which causes the computer to execute the above.

[0141] (Appendix 16) A process of authenticating the authenticity of the avatar using the result of the pilot's identity verification. 16. The program according to claim 14 or 15, which causes the computer to execute the above.

[0142] (Appendix 17) A process of controlling the control object in the real world based on the result of simulating the control of the control object. 14. The program according to claim 13, which causes the computer to execute the above.

[0143] (Appendix 18) A process for determining whether there is an abnormality in the control operation performed by the operator based on the result of simulating the control of the controlled object. 18. The program according to claim 17, which causes the computer to execute the above.

[0144] The present disclosure has been described above with reference to several embodiments. However, the present disclosure is not limited to the above embodiments. Each embodiment can be combined with other embodiments as appropriate. Furthermore, various modifications that can be understood by those skilled in the art can be made to the configurations and details of the above embodiments within the scope of the present disclosure. [Industrial Applicability]

[0145] The present disclosure can be used, for example, to simulate the control of a control target. [Explanation of symbols]

[0146] 10 Simulation equipment 11 Acquisition Department 12 Operating unit 13 Simulation Department 20 Simulation Device 24 Verification Department 30 Simulation Device 34 Control Unit 35 Judgment section 100 HMD 200 Controls 300 Control Target

Claims

1. an acquisition means for acquiring data on the operation performed by the pilot; a verification means for verifying the identity of the operator while acquiring data on the operation performed by the operator; an operation means for operating an avatar associated with the operator in a virtual world based on the acquired operation data; a simulation means for simulating control of a control target in the real world through a movement of an avatar associated with the operator; A simulation device comprising:

2. While the data of the operation performed by the operator is being acquired, the verification means repeatedly verifies the identity of the operator using biometric information of the operator.

2. The simulation device according to claim 1.

3. The verification means verifies the authenticity of the avatar using the result of the identity verification of the operator.

3. The simulation device according to claim 1, wherein:

4. An acquisition means for acquiring data on the operation of a vehicle performed by a pilot; an operation means for operating an avatar associated with the operator in a virtual world based on the acquired operation data; a simulation means for simulating control of a control target in the real world through a movement of an avatar associated with the operator; a control means for controlling the control object in the real world based on a result of simulating control of the control object; A simulation device comprising:

5. The control system further includes a determination unit for determining whether the control operation performed by the operator is abnormal based on the results of a simulation of the control of the controlled object.

5. The simulation device according to claim 4.

6. The computer Acquire data on the pilot's operations, While acquiring data on the operation performed by the operator, the operator is identified; operating an avatar associated with the operator in a virtual world based on the acquired data of the operation; A control of a control object in the real world is simulated through the movement of an avatar associated with the operator. Simulation method.

7. A computer comprising: Acquire data on the pilot's operations, operating an avatar associated with the operator in a virtual world based on the acquired data of the operation; simulating control of a control object in the real world through the movement of an avatar associated with the operator; controlling the control object in the real world based on a result of simulating control of the control object; Simulation method.

8. A process of acquiring data on the operation performed by the pilot; A process of verifying the identity of the operator while acquiring data of the operation performed by the operator; a process of operating an avatar associated with the operator in a virtual world based on the acquired data of the operation; A process of simulating control of a control object in the real world through the movement of an avatar associated with the operator; A program that causes a computer to execute the following.

9. A process for acquiring data on the operation of a vehicle performed by a pilot; a process of operating an avatar associated with the operator in a virtual world based on the acquired data of the operation; a process of simulating control of a control target in the real world through a movement of an avatar associated with the operator; a process of controlling the control object in the real world based on a result of simulating control of the control object; A program that causes a computer to execute the following.

Images