Simulation device, simulation method, and program

Abstract

To allow pilot training of a control target or remote control of a control target to be carried out while maintaining access restrictions to the control target.SOLUTION: An acquisition unit acquires data on piloting operations performed by a pilot. A motion unit drives an avatar associated with the pilot in a virtual world according to the acquired data on piloting operations. A simulation unit simulates control of the control target in the real world through actions of the avatar associated with the pilot.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a simulation device, a simulation method, and a program, and more particularly to a simulation device, a simulation method, and a program for simulating control of a control target. [Background technology]

[0002] Related technology involves obtaining sensing data from objects such as people and objects, extracting features from the sensing data, and generating an avatar or icon from the extracted features and specified parameters (e.g., the object's position, object attributes, time, and environment).

[0003] For example, Patent Document 1 describes a specific example of a simulation system that generates an image seen from a virtual camera in a virtual space (virtual world). One example of the system realizes a virtual reality in which an image seen from a virtual camera is displayed on an HMD (Head Mount Display). Another example provides a racing game in which a user can enjoy driving a race car in a virtual space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-000171 A Summary of the Invention [Problem to be solved by the invention]

[0005] There are many cases where beginners need to train on controlled objects, or where operators need to remotely control controlled objects. However, depending on the controlled object, due to confidentiality requirements or the like, only authorized persons with administrative authority are permitted to directly control the controlled object.

[0006] In the above-mentioned cases, the related art cannot provide the pilot with an opportunity to train in piloting the controlled object or to remotely control the controlled object.

[0007] The present disclosure has been made in consideration of the above-mentioned problems, and its purpose is to enable pilot training of a controlled object or remote operation of the controlled object while maintaining access restrictions to the controlled object. [Means for solving the problem]

[0008] A simulation device according to one aspect of the present disclosure includes an acquisition means for acquiring data on a control operation performed by a pilot, an operation means for operating an avatar associated with the pilot in a virtual world based on the acquired data on the control operation, and an emulation means for simulating control of a control object in the real world via the operation of the avatar associated with the pilot.

[0009] In a simulation method according to one aspect of the present disclosure, a computer acquires data of a control operation performed by a pilot, and based on the acquired data of the control operation, moves an avatar associated with the pilot in a virtual world, and simulates control of a control object in the real world via the movement of the avatar associated with the pilot.

[0010] A program according to one embodiment of the present disclosure causes a computer to execute the following processes: acquiring data on steering operations performed by a pilot; moving an avatar associated with the pilot in a virtual world based on the acquired data on the steering operations; and simulating control of a control object in the real world through the movement of the avatar associated with the pilot. Effect of the Invention

[0011] According to one aspect of the present disclosure, it is possible to perform pilot training or remote operation of a controlled object while maintaining access restrictions to the controlled object. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a system including a simulation device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a flow of a simulation of control of a control target performed by a simulation device according to an embodiment. [Diagram 3] 1 is a block diagram showing a configuration of a simulation device according to an embodiment; [Figure 4] 4 is a flowchart showing an operation of the simulation device according to an embodiment. [Diagram 5] 1 is a block diagram showing a configuration of a simulation device according to an embodiment; [Figure 6] 4 is a flowchart showing an operation of the simulation device according to an embodiment. [Figure 7] FIG. 11 is a diagram illustrating a process performed by a confirmation unit of the simulation device according to an embodiment. [Figure 8] 1 is a block diagram showing a configuration of a simulation device according to an embodiment; [Figure 9] 4 is a flowchart showing an operation of the simulation device according to an embodiment. [Figure 10] FIG. 13 is a block diagram showing a configuration of a simulation device according to a modified example of an embodiment. [Figure 11] 11 is a flowchart showing an operation of a simulation device according to a modified example of an embodiment. [Figure 12] FIG. 2 is a diagram illustrating an example of a hardware configuration of a simulation apparatus according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (An example of System 1) A system 1 including any one of simulation devices 10, 20, 30, and 30A according to embodiments 1 to 3 described below will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the system 1 in schematic form.

[0014] As shown in Fig. 1, the system 1 includes a simulation device 10 (20, 30, 30A), a head mounted display (HMD) 100, a control device 200, and a control target 300. There may be one or more of each of the head mounted display (HMD) 100, the control device 200, and the control target 300. In the system 1, an avatar used by the pilot and information used to verify the identity of the pilot (e.g., biometric information) are linked to the identification information of the pilot and are registered in advance in a database 400 shown in Fig. 1.

[0015] Here, "simulation device 10 (20, 30, 30A)" refers to "any of the simulation devices 10, 20, 30, 30A."

[0016] The HMD 100 is a display device worn on the head of an operator. In one example, the HMD 100 is equipped with a monitor and a sensor. A VR (Virtual Reality) image generated by the simulation device 10 (20, 30, 30A) is displayed on the monitor of the HMD 100. The sensor of the HMD 100 detects sensing data including biometric information and feature amounts (e.g., iris image, voice data, acceleration data) of the operator.

[0017] The HMD 100 transmits sensing data to the simulation device 10 (20, 30, 30A) via an arbitrary communication network NW (FIG. 1). The communication network NW is, for example, an arbitrary combination of a local area network (LAN), a wide area network (WAN), a mobile network, and the Internet.

[0018] The control device 200 is used by the operator to perform control operations while viewing the VR image displayed on the monitor of the HMD 100. For example, the control device 200 is at least one of a control stick, a side stick, a control rudder, and a pedal. The control device 200 converts the input (control operation) of the operator into signal data.

[0019] The control device 200 transmits signal data to the simulation device 10 (20, 30, 30A) via an arbitrary communication network NW (FIG. 1).

[0020] The control target 300 is, for example, a special machine or special mobility that requires operation training, or an industrial robot. The control target 300 receives a control signal from a simulation device 10 (20, 30, 30A) via an arbitrary communication network NW (FIG. 1).

[0021] The simulation device 10 (20, 30, 30A) acquires data of the control operation performed by the pilot from the control device 200. The simulation device 10 (20, 30, 30A) moves an avatar associated with the pilot in the virtual world based on the acquired control operation data. The simulation device 10 (20, 30, 30A) simulates the control of the control target 300 via the movement of the avatar associated with the pilot.

[0022] Furthermore, the simulation device 10 (20, 30, 30A) may actually control the control object 300 in the real world based on the result of simulating the control of the control object 300. In this case, the simulation device 10 (20, 30, 30A) generates a control signal for controlling the control object 300 based on the result of simulating the control of the control object 300. Then, the simulation device 10 (20, 30, 30A) transmits the control signal to the control object 300 via an arbitrary communication network NW (FIG. 1).

[0023] Furthermore, the simulation device 10 (20, 30, 30A) may generate VR video data expressing how the control target 300 is controlled by a steering operation based on the result of simulating the control of the control target 300, and transmit the VR video data to the HMD 100 via an arbitrary communication network NW (FIG. 1). For example, the VR video data expresses the movement of the virtual control target 300 in the virtual world.

[0024] (Flow of simulation of control of controlled object) The flow of a simulation of the control of a control target 300 by the simulation device 10 (20, 30, 30A) will be described with reference to Fig. 2. The flow of the simulation described here is merely an example.

[0025] As shown in Fig. 2, first, a pilot wearing an HMD 100 uses a control device 200 such as a joystick or a handle to perform a control operation to move an avatar. The control device 200 transmits input (control operation) data by the pilot to the simulation device 10 (20, 30, 30A). Note that the pilot is not permitted to directly control the control target 300. Therefore, the control device 200 cannot transmit the input (control operation) data by the pilot to the control target 300.

[0026] The simulation device 10 (20, 30, 30A) moves an avatar associated with a pilot based on input data received from a control device 200. In the virtual world, the avatar associated with the pilot imitates the pilot's piloting operations. The avatar performs piloting operations on a virtual control target 300.

[0027] Furthermore, the simulation device 10 (20, 30, 30A) executes a simulation of the control of the control target 300 based on input (operational operation) data by the operator. For example, the simulation device 10 (20, 30, 30A) reproduces, in a virtual world, how the control target 300 behaves in response to the operation of the operator (or an avatar) through physical simulation.

[0028] The simulation device 10 (20, 30, 30A) generates VR image data to be displayed on the monitor of the HMD 100 based on the results of simulating the control of the control target 300. The simulation device 10 (20, 30, 30A) transmits the generated VR image data to the HMD 100 via an arbitrary communication network NW (FIG. 1).

[0029] The simulation device 10 (20, 30, 30A) may actually control the control object 300 in the real world based on the result of simulating the control of the control object 300. In this case, the simulation device 10 (20, 30, 30A) generates a control signal for controlling the control object 300 based on the operation of an avatar. Then, the simulation device 10 (20, 30, 30A) transmits the generated control signal to the control object 300 via an arbitrary communication network NW (FIG. 1).

[0030] Alternatively, the simulation device 10 (20, 30, 30A) may determine an abnormality in the control operation performed by the operator based on the result of simulating the control of the control target 300. The abnormality here means a control operation that differs from a predetermined procedure or a control operation that makes the operation of the control target 300 unstable.

[0031] When it is determined that there is an abnormality in the control operation performed by the pilot, the simulation device 10 (20, 30, 30A) may correct the abnormality and then actually control the control target 300 in the real world. Alternatively, the simulation device 10 (20, 30, 30A) may stop actually controlling the control target 300 based on the result of simulating the control of the control target 300.

[0032] When a pilot is training to control the control target 300, the simulation device 10 (20, 30, 30A) may notify the pilot or a training instructor that there is an abnormality in the pilot's control operation. In one example, the simulation device 10 (20, 30, 30A) displays an alert on the HMD 100 indicating that there is an abnormality in the pilot's control operation. Furthermore, the simulation device 10 (20, 30, 30A) may display an explanation of the reason why the pilot's control operation was determined to be abnormal on the monitor of the HMD 100.

[0033] In this manner, system 1 can enable an operator to train on or remotely control controlled object 300 while restricting unauthorized operators from having direct access to controlled object 300.

[0034] The configuration and operation of the simulation device 10 (20, 30, 30A) will be described in the following first to third embodiments.

[0035] [Embodiment 1] The first embodiment will be described with reference to FIG. 3 and FIG.

[0036] (Configuration of Simulation Device 10) 3 is a block diagram showing the configuration of the simulation device 10 according to the embodiment 1. As shown in FIG.

[0037] The acquisition unit 11 acquires data on the operation of the pilot. The acquisition unit 11 is an example of an acquisition means.

[0038] For example, the acquisition unit 11 receives input (control operation) data by a pilot from the control device 200 (FIG. 1). The acquisition unit 11 outputs the input data received from the control device 200 to the operation unit 12.

[0039] The operation unit 12 moves the avatar associated with the operator in the virtual world based on the acquired operation data. The operation unit 12 is an example of an operation means.

[0040] For example, operation unit 12 receives input data indicating a control operation performed by the operator from acquisition unit 11. Operation unit 12 converts the received input data into information indicating the movement of the avatar by referring to a table (not shown) or the like that associates the control operation of the operator with the movement of the avatar.

[0041] Then, the operation unit 12 operates the avatar in the virtual world based on the obtained information. The operation of the avatar at this time corresponds to the control operation performed by the operator. For example, when the operator performs a control operation of moving a joystick (one example of the control device 200), the avatar also performs an operation in the virtual world that corresponds to the same control operation as the operator.

[0042] The movement unit 12 outputs to the simulation unit 13 information indicating the movement of the avatar.

[0043] The simulator 13 simulates the control of the control target 300 in the real world through the movement of an avatar associated with the operator. The simulator 13 is an example of a simulator.

[0044] For example, the simulator 13 outputs information indicating the movement of the avatar associated with the operator from the operation unit 12. The simulator 13 calculates how the control target 300 will be controlled when the avatar moves in that way according to a simulation program prepared in advance. The simulator 13 then outputs the calculation result, i.e., the result of simulating the control of the control target 300 in the real world.

[0045] In one example, the simulator 13 generates VR video data expressing how the control target 300 is controlled by a steering operation based on the result of simulating the control of the control target 300. Then, the simulator 13 transmits the generated VR video data to the HMD 100 via an arbitrary communication network NW (FIG. 1).

[0046] (Operation of Simulation Device 10) Fig. 4 is a flowchart showing the operation of the simulation device 10 according to the present embodiment 1. As shown in Fig. 4, the acquisition unit 11 acquires data of the control operation performed by the pilot from the control device 200 (Fig. 1) (S101). The acquisition unit 11 outputs the input data received from the control device 200 to the operation unit 12.

[0047] Next, the operation unit 12 moves the avatar associated with the operator in the virtual world based on the acquired control operation data (S102). The operation unit 12 outputs information indicating the operation of the avatar to the simulation unit 13.

[0048] The simulator 13 simulates the control of the control target 300 in the real world through the movement of the avatar associated with the operator (S103).

[0049] Thereafter, the simulator 13 outputs the result of simulating the control of the control target 300 to an external device or storage device (not shown), or to another device. For example, the simulator 13 displays an image showing the movement of the control target 300 on the HMD 100 based on the result of simulating the control of the control target 300.

[0050] This is the end of the operation of the simulation device 10 according to the first embodiment.

[0051] (Effects of this embodiment) According to the configuration of this embodiment, the acquisition unit 11 acquires data of the control operation performed by the operator. The operation unit 12 operates an avatar associated with the operator in the virtual world based on the acquired data of the control operation. The simulation unit 13 simulates the control of the control target 300 in the real world through the operation of the avatar associated with the operator.

[0052] Therefore, based on the control operation by the operator, the avatar associated with the operator moves, and the avatar controls the virtual control target 300. The virtual control target 300 is controlled based on the result of simulating the control of the control target 300. The operator does not need to directly access the control target 300 in the real world.

[0053] This makes it possible to perform training on the control target 300 or remotely control the control target 300 while maintaining access restrictions to the control target 300.

[0054] [Embodiment 2] A second embodiment will be described with reference to Fig. 5 to Fig. 7. In the second embodiment, a configuration will be described in which the identity of the pilot is verified periodically or irregularly while data on the pilot's operation is being acquired.

[0055] In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

[0056] (Configuration of Simulation Device 20) Fig. 5 is a block diagram showing a configuration of a simulation device 20 according to the present embodiment 2. As shown in Fig. 5, the simulation device 20 includes an acquisition unit 11, an operation unit 12, and a simulation unit 13. The simulation device 20 further includes a confirmation unit 24.

[0057] The confirmation unit 24 performs identity verification of the pilot while the acquisition unit 11 is acquiring data of the pilot's operation (input data). The confirmation unit 24 is an example of a verification means. Furthermore, the confirmation unit 24 may perform identity verification of the pilot before the acquisition unit 11 acquires the data of the pilot's operation (input data).

[0058] For example, the confirmation unit 24 acquires an iris image obtained by photographing the iris of the pilot with the in-camera of the HMD 100 from the HMD 100 (FIG. 1). The confirmation unit 24 then uses the iris image of the pilot to confirm the identity of the pilot. In one example, while the pilot is performing a piloting operation according to a piloting training scenario provided by the simulation device 20, the confirmation unit 24 may repeatedly acquire biometric information of the pilot as part of the piloting training scenario, and confirm the identity of the pilot each time the biometric information is acquired. This has the advantage that the pilot's identity can be prevented from being performed by anyone other than the pilot by repeatedly confirming the identity of the pilot while the piloting training is being performed, and the immersive feeling of piloting is not impaired.

[0059] The verification unit 24 can also verify the identity of the operator using biometric information other than the iris image (for example, the operator's face, fingerprint, veins, or ear acoustic features). For example, the verification unit 24 may verify the identity of the operator using an image (one example of biometric information) obtained by photographing the operator's gestures. The above-mentioned system 1 (FIG. 1) may be equipped with a spatial recognition device (for example, a camera) for photographing the operator's gestures. Alternatively, the verification unit 24 may verify the identity of the operator using a voiceprint (one example of biometric information) obtained by recording the operator's speech.

[0060] Furthermore, the verification unit 24 authenticates the authenticity of the avatar using the result of the identity verification of the pilot. Specifically, when the identity verification of the pilot is successful, the verification unit 24 updates the expiration date of the trust authentication of the avatar associated with the pilot.

[0061] On the other hand, if the verification unit 24 fails to verify the identity of the pilot, it invalidates the trust authentication of the avatar associated with the pilot. The pilot and avatar are associated only before the expiration date of the trust authentication of the avatar. When the expiration date of the trust authentication of the avatar expires, the association between the pilot and the avatar is lost. The pilot will no longer be able to move the avatar.

[0062] The verification unit 24 outputs the result of verifying the identity of the pilot to the operation unit 12.

[0063] The operation unit 12 receives input data indicating the control operation performed by the operator from the acquisition unit 11. Furthermore, the operation unit 12 receives the result of identity verification of the operator from the verification unit 24.

[0064] When the trust authentication of the avatar is updated, the operation unit 12 operates the avatar associated with the operator in the virtual world based on the input data indicating the operation received from the acquisition unit 11.

[0065] On the other hand, if the trust authentication of the avatar is revoked, the operation unit 12 does not allow the operator to operate the avatar in the virtual world. In this case, the confirmation unit 24 may retry the identity confirmation of the operator by using the operator's biometric information or other information.

[0066] In one example, if identity verification based on the iris image of the pilot fails, the verification unit 24 requests the pilot to open his / her eyes wide. After that, the verification unit 24 reacquires an iris image obtained by photographing the iris of the pilot from the HMD 100. Then, the verification unit 24 retries identity verification of the pilot using the retaken iris image. Note that a request to the pilot to reacquire the pilot's biometric information may also be implemented in the piloting training scenario provided by the simulation device 20.

[0067] In another example, the confirmation unit 24 may obtain features related to the manner of the motion by having the operator perform a gesture such as a pointing motion, and may use the obtained features related to the manner of the motion for identity verification. In yet another example, if identity verification based on the iris image of the operator fails, the confirmation unit 24 may retry identity verification using biometric information other than the iris image (for example, a voiceprint).

[0068] (Operation of simulation device 20) The operation of the simulation device 20 according to the second embodiment will be described with reference to Fig. 6 to Fig. 7. Fig. 6 is a flowchart showing the operation of the simulation device 20 according to the second embodiment.

[0069] 6, acquisition unit 11 acquires data of a control operation performed by a controller from controller 200 (FIG. 1) (S201). Acquisition unit 11 outputs the input data received from controller 200 to operation unit 12.

[0070] Next, the verification unit 24 performs identity verification of the pilot while acquiring data on the pilot's operation (S202).

[0071] As shown in FIG. 7, in step S204, the confirmation unit 24 acquires biological information of the operator (S2021).

[0072] Next, the confirmation unit 24 compares the operator's biometric information with the biometric information registered in the database 400 (FIG. 1) (S2022).

[0073] The verification unit 24 determines whether the identity verification of the operator has been successful (S2023). Furthermore, the verification unit 24 authenticates the authenticity of the avatar using the result of the identity verification of the operator.

[0074] If the confirmation unit 24 has successfully confirmed the identity of the pilot (Yes in S2023), the confirmation unit 24 updates the expiration date of the trust authentication of the avatar associated with the pilot (S2024A).

[0075] On the other hand, if the verification unit 24 fails to verify the identity of the operator (No in S2023), the trust authentication of the avatar associated with the operator is invalidated (S2024B). When the trust authentication of the avatar is invalidated, the association between the operator and the avatar is lost. As a result, the operator cannot control the avatar.

[0076] The verification unit 24 outputs the result of the identity verification of the operator to the operation unit 12. The flow proceeds to step S203 in FIG.

[0077] In step S202, if the trust authentication of the avatar associated with the operator is updated (Yes in S203), the operation unit 12 operates the avatar associated with the operator in the virtual world based on the acquired operation data (S204). The operation unit 12 outputs information indicating the operation of the avatar to the simulation unit 13.

[0078] The simulator 13 simulates the control of the control target 300 in the real world through the movement of the avatar associated with the operator (S205).

[0079] Thereafter, the simulator 13 outputs the result of simulating the control of the controlled object 300 to an external device or storage device (not shown), or to other devices.

[0080] On the other hand, if the trust authentication of the avatar associated with the operator is invalidated in step S202 (No in S203), the operation unit 12 prevents the operator from operating the avatar in the virtual world, and the flow ends.

[0081] This is the end of the operation of the simulation device 20 according to the second embodiment.

[0082] (Effects of this embodiment) According to the configuration of this embodiment, the acquisition unit 11 acquires data of the control operation performed by the operator. The operation unit 12 operates an avatar associated with the operator in the virtual world based on the acquired data of the control operation. The simulation unit 13 simulates the control of the control target 300 in the real world through the operation of the avatar associated with the operator.

[0083] Therefore, based on the control operation by the operator, the avatar associated with the operator moves, and the avatar controls the virtual control target 300. The virtual control target 300 is controlled based on the result of simulating the control of the control target 300. The operator does not need to directly access the control target 300 in the real world.

[0084] This makes it possible to perform training on the control target 300 or remotely control the control target 300 while maintaining access restrictions to the control target 300.

[0085] Furthermore, according to the configuration of this embodiment, the confirmation unit 24 performs identity verification of the pilot while acquiring data of the pilot's operation. If the confirmation unit 24 succeeds in identifying the pilot, the operation unit 12 operates an avatar associated with the pilot in the virtual world based on the acquired data of the pilot operation. This makes it possible to prevent a malicious third party from impersonating the pilot and moving the avatar. Furthermore, according to the configuration of this embodiment, the identity of the pilot is verified as part of the piloting training scenario while the pilot is performing the piloting operation, which has the advantage that the immersive feeling of the piloting operation is not impaired due to the verification of the pilot's identity.

[0086] [Embodiment 3] A third embodiment will be described with reference to Fig. 8 to Fig. 11. In the third embodiment, a configuration for controlling a control target 300 (Fig. 1) based on a result of simulating the control of the control target 300 will be described.

[0087] In the third embodiment, the same components as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

[0088] (Configuration of Simulation Device 30) Fig. 8 is a block diagram showing a configuration of a simulation device 30 according to the present embodiment 3. As shown in Fig. 8, the simulation device 30 includes an acquisition unit 11, an operation unit 12, and a simulation unit 13. The simulation device 30 further includes a control unit 34.

[0089] The control unit 34 controls the control target 300 (FIG. 1) in the real world based on the result of simulating the control of the control target 300. The control unit 34 is an example of a control means.

[0090] For example, the control unit 34 receives a result of simulating the control of the control target 300 from the simulation unit 13. The control unit 34 actually controls the control target 300 based on the result of the simulation by the simulation unit 13.

[0091] In one example, the control unit 34 controls the control object 300 in the real world so as to trace the control of the simulated control object 300. The control here means making the control object 300 perform a function or operating a drive unit provided in the control object 300.

[0092] The control unit 34 generates a control signal for controlling the control target 300 based on the result of simulating the control of the control target 300. Then, the control unit 34 transmits the control signal to the control target 300 via an arbitrary communication network NW (FIG. 1).

[0093] (Operation of simulation device 30) Fig. 9 is a flowchart showing the operation of simulation device 30 according to embodiment 3. As shown in Fig. 9, acquisition unit 11 acquires data of a control operation performed by a pilot from control device 200 (Fig. 1) (S301). Acquisition unit 11 outputs the input data received from control device 200 to operation unit 12.

[0094] Next, the operation unit 12 moves the avatar associated with the operator in the virtual world based on the acquired control operation data (S302). The operation unit 12 outputs information indicating the operation of the avatar to the simulation unit 13.

[0095] The simulator 13 simulates the control of the control target 300 in the real world through the movement of the avatar associated with the operator (S303).

[0096] Thereafter, the simulator 13 outputs the result of simulating the control of the controlled object 300 to an external device or storage device (not shown), or to other devices.

[0097] Moreover, the simulator 13 outputs the result of simulating the control of the controlled object 300 to the controller .

[0098] The control unit 34 controls the control target 300 in the real world based on the result of simulating the control of the control target 300 (S304).

[0099] This is the end of the operation of the simulation device 30 according to the third embodiment.

[0100] [Modifications] A modification of the third embodiment will be described below.

[0101] (Configuration of Simulation Device 30A) Fig. 10 is a block diagram showing a configuration of a simulation device 30A according to a modified example of the present embodiment 3. As shown in Fig. 10, the simulation device 30A includes an acquisition unit 11, an operation unit 12, and a simulation unit 13. The simulation device 30A further includes a control unit 34 and a determination unit 35.

[0102] The determination unit 35 determines whether there is an abnormality in the control operation performed by the operator based on the result of simulating the control of the control target 300 (FIG. 1). The determination unit 35 is an example of a determination means. An abnormality here means a control operation that differs from a predetermined procedure or a control operation that makes the operation of the control target 300 unstable.

[0103] For example, the determination unit 35 receives a result of simulating the control of the control target 300 from the simulation unit 13. The determination unit 35 determines whether the operation of the control target 300 becomes unstable when the control target 300 is actually controlled in the real world according to the result of the simulation. For example, if the control target 300 is an aircraft and there is a possibility that the altitude of the control target 300 will suddenly decrease, the determination unit 35 determines that there is an abnormality in the pilot's operation since the operation of the control target 300 becomes unstable.

[0104] When it is determined that there is an abnormality in the control operation performed by the pilot, the determination unit 35 may correct the abnormality. Alternatively, the determination unit 35 may stop actually controlling the control target 300 based on the result of simulating the control of the control target 300.

[0105] When the pilot is undergoing training to control the control target 300, the determination unit 35 may notify the pilot or the instructor that there is an abnormality in the pilot's control operation. In one example, the determination unit 35 displays an alert indicating that there is an abnormality in the pilot's control operation on the HMD 100. Furthermore, the determination unit 35 may display an explanation of the reason why the pilot's control operation was determined to be abnormal on the HMD 100.

[0106] (Operation of simulation device 30A) Fig. 10 is a flowchart showing the operation of simulation device 30A according to one variation of embodiment 3. As shown in Fig. 10, acquisition unit 11 acquires data of a control operation performed by a pilot from control device 200 (Fig. 1) (S301'). Acquisition unit 11 outputs the input data received from control device 200 to operation unit 12.

[0107] Next, the operation unit 12 moves the avatar associated with the operator in the virtual world based on the acquired control operation data (S302'). The operation unit 12 outputs information indicating the operation of the avatar to the simulation unit 13.

[0108] The simulator 13 simulates the control of the control target 300 in the real world through the movement of the avatar associated with the operator (S303').

[0109] The determination unit 35 detects an abnormality in the control operation performed by the operator based on the result of simulating the control of the control target 300 (S304').

[0110] If there is an abnormality in the control operation performed by the pilot (Yes in S305'), the flow ends.

[0111] On the other hand, if there is no abnormality in the control operation performed by the pilot (No in S305'), the simulator 13 outputs the results of simulating the control of the control target 300 to an external device or storage device (not shown), or other device.

[0112] Moreover, the simulator 13 outputs the result of simulating the control of the controlled object 300 to the controller .

[0113] The control unit 34 controls the control target 300 in the real world based on the result of simulating the control of the control target 300 (S306').

[0114] This is the end of the operation of the simulation device 30A according to the modified example of the third embodiment.

[0115] (Effects of this embodiment) According to the configuration of this embodiment, the acquisition unit 11 acquires data of the control operation performed by the operator. The operation unit 12 operates an avatar associated with the operator in the virtual world based on the acquired data of the control operation. The simulation unit 13 simulates the control of the control target 300 in the real world through the operation of the avatar associated with the operator.

[0116] Therefore, based on the control operation by the operator, the avatar associated with the operator moves, and the avatar controls the virtual control target 300. The virtual control target 300 is controlled based on the result of simulating the control of the control target 300. The operator does not need to directly access the control target 300 in the real world.

[0117] This makes it possible to perform training on the control target 300 or remotely control the control target 300 while maintaining access restrictions to the control target 300.

[0118] Furthermore, according to the configuration of this embodiment, the control unit 34 controls the control target 300 in the real world based on the result of simulating the control of the control target 300. This makes it possible to perform piloting training for the control target 300 or remotely operate the control target 300 while maintaining access restrictions to the control target 300.

[0119] (Example of hardware configuration) Each of the components of the simulation devices 10, 20, 30, and 30A described in the first to third embodiments is represented by a functional block. Some or all of these components are realized by an information processing device as shown in Fig. 11. Fig. 11 is a block diagram showing an example of the hardware configuration of the information processing device.

[0120] 11, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. Note that the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to the CPU 111 or instead of the CPU 111.

[0121] The CPU 111 loads the programs (codes) in this embodiment stored in the storage device 113 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a dynamic random access memory (DRAM). The programs in this embodiment are provided in a state stored in a computer-readable recording medium 120. The programs in this embodiment may be distributed over the Internet connected via a communication interface 117.

[0122] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0123] Data reader / writer 116 mediates data transmission between CPU 111 and recording medium 120, reads programs from recording medium 120, and writes processing results in computer 110 to recording medium 120. Communication interface 117 mediates data transmission between CPU 111 and other computers.

[0124] Specific examples of recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or optical recording media such as a CD-ROM (Compact Disk Read Only Memory).

[0125] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0126] (Appendix 1) An acquisition means for acquiring data on a control operation performed by a pilot; an operation means for operating an avatar associated with the operator in a virtual world based on the acquired operation data; A simulation means for simulating the control of a control target in the real world through the movement of an avatar associated with the operator; A simulation device comprising:

[0127] (Appendix 2) The vehicle further includes a verification unit for verifying the identity of the operator while the data of the operation performed by the operator is being acquired. 2. A simulation device according to claim 1.

[0128] (Appendix 3) While the data of the operation performed by the operator is being acquired, the verification means repeatedly verifies the identity of the operator by using biometric information of the operator. 3. A simulation device according to claim 2.

[0129] (Appendix 4) The verification means verifies the authenticity of the avatar using the result of the identity verification of the operator. 4. The simulation device according to claim 2 or 3.

[0130] (Appendix 5) The control means further includes a control unit for controlling the control object in the real world based on a result of simulating the control of the control object. 2. A simulation device according to claim 1.

[0131] (Appendix 6) The control system further includes a determination unit for determining an abnormality in the control operation performed by the operator based on a result of simulating the control of the controlled object. 6. A simulation device according to claim 5.

[0132] (Appendix 7) The computer Acquire data on the pilot's operations, operating an avatar associated with the operator in a virtual world based on the acquired data of the operation; A control of a control object is simulated through a movement of an avatar associated with the operator. Simulation method.

[0133] (Appendix 8) The computer includes: While acquiring data on the operation performed by the operator, the operator's identity is verified. 8. The simulation method according to claim 7,

[0134] (Appendix 9) While the computer is acquiring data on the operation performed by the operator, the computer repeatedly verifies the identity of the operator using biometric information of the operator. 9. The simulation method according to claim 8,

[0135] (Appendix 10) The computer authenticates the authenticity of the avatar using the results of the operator's identity verification. 10. The simulation method according to claim 8 or 9,

[0136] (Appendix 11) The computer controls the control target in the real world based on a result of simulating the control of the control target. 8. The simulation method according to claim 7,

[0137] (Appendix 12) The computer determines an abnormality in the control operation performed by the operator based on a result of simulating the control of the controlled object. 12. The simulation method according to claim 11,

[0138] (Appendix 13) A process of acquiring data on the operation performed by the pilot; A process of operating an avatar associated with the operator in a virtual world based on the acquired data of the operation; A process of simulating control of a control object through a movement of an avatar associated with the operator; A program for causing a computer to execute the following.

[0139] (Appendix 14) While acquiring data on the operation performed by the operator, the operator is identified, and the result of the identification is used to perform trust authentication of the avatar. The program described in Appendix 13, which causes the computer to execute the above steps.

[0140] (Appendix 15) A process of repeatedly verifying the identity of the operator using biometric information of the operator while acquiring data on the operation performed by the operator. The program described in Appendix 14, which causes the computer to execute the above steps.

[0141] (Appendix 16) A process of authenticating the authenticity of the avatar using the result of the pilot's identity verification. 16. The program according to claim 14 or 15, which causes the computer to execute the above steps.

[0142] (Appendix 17) A process of controlling the control target in the real world based on a result of simulating the control of the control target. The program described in Appendix 13, which causes the computer to execute the above steps.

[0143] (Appendix 18) A process for determining an abnormality in the control operation performed by the operator based on a result of simulating the control of the controlled object. The program according to claim 17, which causes the computer to execute the above steps.

[0144] The present disclosure has been described above with reference to several embodiments. However, the present disclosure is not limited to the above embodiments. Each embodiment can be appropriately combined with other embodiments. In addition, various modifications that can be understood by a person skilled in the art can be made to the configurations and details of the above embodiments within the scope of the present disclosure. [Industrial Applicability]

[0145] The present disclosure can be used, for example, to simulate the control of a control target. [Explanation of symbols]

[0146] 10 Simulation device 11 Acquisition Department 12 Operating unit 13 Simulation Department 20 Simulation Device 24 Verification Department 30 Simulation Device 34 Control section 35 Judgment section 100 HMD 200 Flight Controls 300 Control Target

Claims

1. An acquisition means for acquiring data on a control operation performed by a pilot; an operation means for operating an avatar associated with the operator in a virtual world based on the acquired operation data; A simulation means for simulating the control of a control target in the real world through the movement of an avatar associated with the operator; A simulation device comprising:

2. The vehicle further includes a verification unit for verifying the identity of the operator while the data of the operation performed by the operator is being acquired.

2. The simulation device according to claim 1.

3. While the data of the operation performed by the operator is being acquired, the verification means repeatedly verifies the identity of the operator by using biometric information of the operator.

3. The simulation device according to claim 2.

4. The verification means verifies the authenticity of the avatar using the result of the identity verification of the operator.

4. The simulation device according to claim 2 or 3.

5. The control means further includes a control unit for controlling the control object in the real world based on a result of simulating the control of the control object.

2. The simulation device according to claim 1.

6. The control system further includes a determination unit for determining an abnormality in the control operation performed by the operator based on a result of simulating the control of the controlled object.

6. The simulation device according to claim 5.

7. The computer Acquire data on the pilot's operations, operating an avatar associated with the operator in a virtual world based on the acquired data of the operation; A control of a control target in the real world is simulated through a movement of an avatar associated with the operator. Simulation method.

8. The computer includes: While acquiring data on the operation performed by the operator, the operator is identified. The simulation method according to claim 7 .

9. A process of acquiring data on the operation performed by the pilot; A process of operating an avatar associated with the operator in a virtual world based on the acquired data of the operation; A process of simulating control of a control target in the real world through a movement of an avatar associated with the operator; A program for causing a computer to execute the following.

10. A process for verifying the identity of the pilot while acquiring data on the pilot's operation. The program according to claim 9, which causes the computer to execute the steps of:

Images