Flight simulation system

Abstract

To provide a flight simulation system with low costs and excellent general applicability.SOLUTION: A flight simulation system includes a head-mounted display for displaying a virtual reality image, a haptic device disposed in a real space, and a control device. The haptic device has a touch sensor for detecting a touch location of a user on the haptic device, and a vibrator for vibrating the touch location. The control device has a processor, and a memory that stores coordinate data indicating coordinate locations of a plurality of operation portions installed in a cockpit of each of models regarding a plurality of models. The processor executes processing including causing the vibrator to vibrate the touch location when the user touches the haptic device, and identifying one operation portion corresponding to the touch location among the plurality of operation portions of a selected model on the basis of the coordinate data and the touch location.SELECTED DRAWING: Figure 2

Description

Technical Field

[0004] , , , ,

[0005] , , ,

[0003] , , ,

[0001] The present invention relates to a flight simulation system.

Background Art

[0002] Conventional flight simulation systems include a flight simulation system with a dome-shaped screen. The flight simulation system with a dome-shaped screen can reproduce the cockpit of an aircraft inside the dome and accurately simulate the experience of the actual flight of the aircraft. On the other hand, the flight simulation system with a dome-shaped screen requires a space of a certain size for installation, and the system itself is expensive. Therefore, there is a technology that is small, inexpensive, and can simulate the flight of an actual aircraft. For example, as in Patent Document 1, it is a flight simulation system using a head-mounted display.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The conventional flight simulation system using the head-mounted display described in Patent Document 1 above includes a physical input unit of HMI (Human Machine Interface) such as a control wheel, a rudder pedal, and a switch. However, for aircraft, the arrangement of operation units such as instrument panels and various switches may vary depending on the aircraft model. Therefore, when performing flight simulations of different aircraft models, it is necessary to prepare an HMI for each aircraft model, and there is a limit to supporting multiple aircraft models with one flight simulation system.

[0005] Therefore, the present invention aims to provide an inexpensive and highly versatile flight simulation system. [Means for solving the problem]

[0006] To solve the above problems, a flight simulation system according to one embodiment of the present invention is provided. A flight simulation system configured to perform flight simulations for multiple aircraft types, A head-mounted display that can be worn by the user and displays a virtual reality image representing the cockpit of an aircraft in a virtual space, In real space, a haptic device is positioned at locations corresponding to multiple control units installed in the cockpit of the virtual reality image, A control device for controlling the flight simulation using the head-mounted display and the haptic device, Equipped with, The head-mounted display has an imaging unit that captures the surrounding environment of the head-mounted display in the real space, The aforementioned tactile device is A contact sensor for detecting the user's contact position with the tactile device, A vibrator that vibrates the aforementioned contact position, A display unit that displays identification information for the plurality of operating units at positions corresponding to the plurality of operating units in the real space, It has, The control device is Processor and With respect to the aforementioned multiple aircraft models, a memory is provided for storing coordinate data representing the coordinate positions of the multiple operating units installed in the cockpit of each aircraft model. It has, The aforementioned processor, The process involves performing a flight simulation of one aircraft selected by the user from among the aforementioned multiple aircraft, Displaying the virtual reality image, in which the multiple operating units of the selected model are depicted, on the head-mounted display, The operation unit corresponding to the identification information captured by the imaging unit is superimposed and displayed on the virtual reality image displayed on the head-mounted display, When the user touches the haptic device, the contact point is vibrated by the vibrator. Based on the coordinate data and contact position of the selected model, one of the multiple operating parts of the selected model that corresponds to the contact position is identified. To reflect the operation of the identified control unit in the flight simulation of the selected aircraft model, Execute the process that includes this. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an inexpensive and highly versatile flight simulation system. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing the overall configuration of the flight simulation system according to the first embodiment. [Figure 2] Figure 2 is a perspective view showing an example of how the flight simulation system according to the first embodiment is used. [Figure 3] Figure 3 is a schematic diagram showing an example of a virtual reality image displayed in the flight simulation system according to the first embodiment. [Figure 4] Figure 4 is a block diagram showing an example of the functional configuration of a flight simulation system according to the first embodiment. [Figure 5] Figure 5 is a flowchart showing the flight simulation control process by the control device according to the first embodiment. [Figure 6] Figure 6 is a block diagram showing an example of the functional configuration of a flight simulation system according to the second embodiment. [Figure 7] Figure 7 is a perspective view showing an example of how the flight simulation system according to the second embodiment is used. [Figure 8]FIG. 8 is a flowchart showing flight simulation control processing by the control device according to the second embodiment. [Figure 9] FIG. 9 is a flowchart showing coordinate position correction processing by the control device according to the second embodiment.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The specific dimensions, materials, numerical values, etc. shown in such embodiments are merely examples for facilitating the understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same functions and configurations are denoted by the same reference numerals to omit redundant explanations, and elements not directly related to the present invention are not shown.

[0010] [1. Overall Configuration of Flight Simulation System According to the First Embodiment] First, referring to FIGS. 1, 2, and 3, the overall configuration of a flight simulation system 1 according to the first embodiment of the present invention will be described. FIG. 1 is a schematic diagram showing the overall configuration of the flight simulation system 1 according to the first embodiment. FIG. 2 is a perspective view showing an example of the usage mode of the flight simulation system 1 according to the first embodiment. FIG. 3 is a schematic diagram showing an example of a virtual reality image 400 displayed on the flight simulation system 1 according to the first embodiment.

[0011] The flight simulation system 1 is configured to be capable of executing flight simulations for a plurality of aircraft models. As shown in FIGS. 1 and 2, the flight simulation system 1 includes a control device 10, a head-mounted display 20, a tactile device 30, a steering wheel 40, a rudder pedal 50, and a cockpit 60.

[0012] The control device 10 may consist of, for example, a personal computer, a tablet computer, a smartphone, etc. The control device 10 is connected to the head-mounted display 20, a haptic device 30, a control wheel 40, and rudder pedals 50 so as to be able to communicate with each other by wire or wireless. The control device 10 controls the flight simulation using the head-mounted display 20 and the haptic device 30. Flight simulation is the simulation of flight conditions for purposes such as pilot training or flight experience of aircraft. In the flight simulation according to this embodiment, for example, the flight conditions of a specific aircraft are reproduced in a virtual space, and a simulated flight of the aircraft 402 is performed in the virtual space in response to the operation of user 2 in the real space. The real space is the space where the flight simulation system 1 is actually installed. The virtual space is an artificial space on a computer created by calculations of a computer, etc. The control device 10 controls the simulated flight of the aircraft 402 reproduced in the virtual space in response to the operation of user 2 in the real space. The control device 10 has a processor 12 and a memory 14 connected to the processor 12.

[0013] The processor 12 is an arithmetic processing unit installed in the computer. The processor 12 is composed of, for example, a CPU (Central Processing Unit), but may also be composed of other microprocessors. Furthermore, the processor 12 may be composed of one or more processors. The processor 12 executes programs stored in the memory 14 or other storage medium, thereby executing various processes in the control unit 10.

[0014] Memory 14 is a storage medium for storing programs and various other data. Memory 14 includes, for example, RAM (Random Access Memory) and ROM (Read Only Memory). ROM is a non-volatile memory that stores programs used by the processor 12, and data necessary to run those programs. RAM is a volatile memory that temporarily stores data such as variables, arithmetic parameters, and calculation results used in processes executed by the processor 12. Programs stored in ROM are read into RAM and executed by the processor 12, such as the CPU.

[0015] The head-mounted display 20 is a display device that can be worn on the head of user 2. As shown in Figure 2, for example, the head-mounted display 20 is an opaque display device that can be worn so as to completely cover the user 2's field of vision. The head-mounted display 20 is communicatively connected to the control device 10, receives information from the control device 10, and displays images related to the flight simulation to user 2. For example, the head-mounted display 20 displays a virtual reality image 400. As shown in Figure 3, the virtual reality image 400 is an image representing the cockpit 404 of the aircraft 402 in a virtual space. For example, the virtual reality image 400 includes images representing multiple control units 411, 412, 413, 414, 415 (hereinafter collectively referred to simply as "control units 410") installed inside the cockpit 404, and an image representing the instrument panel 420.

[0016] The control unit 410 consists of various operating devices located in the cockpit 404 of the aircraft 402. Specifically, the control unit 410 consists of various operating tools and devices used for flight control, piloting, or other controls related to the operation of the aircraft 402. The control unit 410 may be, for example, physical operating tools such as switches, levers, knobs, or buttons used for piloting the aircraft 402, or it may be a touch panel or the like that simulates these physical operating tools. For example, the control unit 410 may be a push-button switch, snap switch, rocker switch, rotary switch, slide switch, etc. The instrument panel 420 is a display device that visualizes values ​​measured by various sensors and the like provided in the aircraft 402.

[0017] Furthermore, the virtual reality image 400 includes, for example, an image representing the interior of the cockpit 404, as well as an image representing the external environment 406 of the aircraft 402 as seen from the cockpit 404 (represented by hatching in Figure 2). The external environment 406 is the environment outside the aircraft 402 in virtual space, and is, for example, the scenery above or on the ground that is visible from the cockpit 404.

[0018] Returning to Figure 1, the head-mounted display 20 includes an external camera 22 and an attitude sensor 24. The external camera 22 is an example of an imaging unit and is composed of, for example, a visible light camera. The external camera 22 is not limited to a visible light camera and may be composed of, for example, a near-infrared camera. The external camera 22 is installed, for example, on the outside in front of the head-mounted display 20 and captures the surrounding environment of the head-mounted display 20 in real space. The external camera 22 captures the surrounding environment of the user 2's cockpit 60. For example, the external camera 22 mainly captures the environment in front of the user 2.

[0019] The posture sensor 24 is, for example, a gyroscope, an angular velocity sensor, or an accelerometer. The posture sensor 24 is installed inside the head-mounted display 20. The posture sensor 24 detects various movements, postures, and orientations of the head of the user 2 wearing the head-mounted display 20. For example, if the user 2 wearing the head-mounted display 20 turns their face to the right, the posture sensor 24 detects this posture and transmits posture information to the control device 10.

[0020] The haptic device 30 is a device capable of providing tactile feedback to the user 2 through sensations such as vibration, force, movement, heat, or static electricity. The haptic device 30 is composed of, for example, a rectangular touch panel. One or more haptic devices 30 are arranged around the cockpit 60 where the user 2 is seated. For example, in the example in Figure 2, a total of three haptic devices 30 are arranged: one in front of the cockpit 60 and two to the right of the cockpit 60, but this is not limited to this. Two or fewer, or four or more, haptic devices 30 may be arranged. Also, the haptic devices 30 may be arranged, for example, above and in front of the cockpit 60, or to the left of the cockpit 60. In real space, the haptic devices 30 are positioned in locations corresponding to a plurality of control units 410 installed in the cockpit 404 of the virtual reality image 400.

[0021] For example, as shown in Figure 2, the haptic device 30 is positioned opposite the cockpit 60 or to the right of the cockpit 60. For example, the haptic device 30 is installed such that the distance and direction from the position of the cockpit where user 2 is seated in the virtual space to the coordinate positions of the control units 411, 412, 413, 414, and 415 shown in Figure 3 are the same as the distance and direction from the position of the cockpit 60 where user 2 is seated in the real space to the control unit setting positions 311, 312, 313, 314, and 315 (hereinafter collectively referred to simply as "control unit setting positions 310") shown in Figure 2. The control unit setting positions 310 are coordinate positions in the real space that represent specific positions on the surface of the haptic device 30. Each of the multiple control unit setting positions 310 set on the surface of the haptic device 30 corresponds to the coordinate positions of the multiple control units 410 in the virtual space.

[0022] Here, the coordinate positions will be explained. In the flight simulation system 1 according to this embodiment, the three-dimensional coordinate positions (virtual space) of each of the multiple control units 410 in the cockpit 404 of the aircraft 402, which are drawn in the virtual reality image 400, are set. These three-dimensional coordinate positions (virtual space) may be XYZ axis coordinates with a specific first reference position as the origin. Each of these three-dimensional coordinate positions (virtual space) of the multiple control units 410 in the virtual reality image 400 is set to a position corresponding to the three-dimensional coordinate positions (real space) of the multiple control units actually placed in the cockpit of the aircraft in real space. Here, the specific first reference position may be, for example, the center of the cockpit seat 60, or the center of the cockpit 404 of the aircraft 402.

[0023] Then, corresponding to each of the three-dimensional coordinate positions (virtual space) of the multiple operation units 410 in the virtual reality image 400, an operation unit setting position 310 (three-dimensional coordinate position in real space) is set on the surface of the haptic device 30 in real space. The operation unit setting position 310 (three-dimensional coordinate position in real space) may be an XYZ axis coordinate with a specific second reference position on the surface of the haptic device 30 as the origin. The specific second reference position may be, for example, the center of the surface of the haptic device 30, or one of the four corners of the haptic device 30.

[0024] Returning to Figure 1, the tactile device 30 includes a contact sensor 32, a vibrator 34, and a liquid crystal display 36. The contact sensor 32 is installed on the surface of the tactile device 30. The contact sensor 32 detects the contact position, which is the position where user 2 makes contact with the tactile device 30. For example, when user 2's fingers touch the surface of the tactile device 30, the contact sensor 32 detects the contact position and transmits the contact position information to the control device 10.

[0025] One or more vibrators 34 are placed inside the tactile device 30. For example, multiple vibrators 34 may be arranged vertically and horizontally at predetermined intervals on the surface of the tactile device 30. The vibrators 34 vibrate at the contact position detected by the contact sensor 32. For example, when the user 2's fingers touch the tactile device 30, the vibrators 34 at the contact position detected by the contact sensor 32, or near the contact position, vibrate.

[0026] The liquid crystal display 36 is an example of a display unit. The liquid crystal display 36 is installed on the surface of the tactile device 30. The liquid crystal display 36 receives information transmitted from the control device 10 and displays an image related to that information.

[0027] The control wheel 40 is a control device used by user 2 to control aircraft 402 in a flight simulation. The control wheel 40 is located in front of the cockpit 60. For example, the control wheel 40 is provided with a gripping section so that user 2 can grasp it with both hands. The control wheel 40 is communicated with the control device 10. When user 2 operates the control wheel 40, information representing the operation of the control wheel 40 is transmitted to the control device 10, and the control device 10 reflects the operation of the control wheel 40 in the flight simulation. By pushing or pulling the control wheel 40, user 2 can raise or lower the aircraft 402 flying in the flight simulation. Also, by rotating the control wheel 40, user 2 can turn the aircraft 402 flying in the flight simulation. Although the control device for controlling aircraft 402 is described as a control wheel here, it is not limited to this and may be a control stick.

[0028] The rudder pedals 50 are controls used by user 2 to operate aircraft 402 in flight simulation. The rudder pedals 50 are located in front of and below the cockpit 60. For example, the rudder pedals 50 are equipped with pedals so that user 2 can operate them by pressing them with their feet. The rudder pedals 50 are communicated with the control device 10. When user 2 operates the rudder pedals 50, information representing the operation of the rudder pedals 50 is transmitted to the control device 10, and the control device 10 reflects the operation of the rudder pedals 50 in the flight simulation. By pressing the rudder pedals 50, user 2 can change the direction of the nose of aircraft 402 flying in the flight simulation.

[0029] The cockpit 60 is a chair in which user 2 sits while the flight simulation is running. The cockpit 60 may be, for example, a chair specifically designed for flight simulation, or it may be a general-purpose chair. The cockpit 60 is positioned adjacent to the haptic device 30, the control wheel 40, and the rudder pedals 50, facing them. This allows user 2, seated in the cockpit 60, to operate the haptic device 30, the control wheel 40, and the rudder pedals 50 using their hands and feet.

[0030] [2. Functional configuration of the flight simulation system according to the first embodiment] Next, with reference to Figure 4, the functional configuration of the flight simulation system 1 according to the first embodiment will be described. Figure 4 is a block diagram showing an example of the functional configuration of the flight simulation system 1 according to the first embodiment.

[0031] As shown in Figure 4, the control device 10 includes a control unit 100, a display control unit 102, a contact position acquisition unit 104, a determination unit 106, a vibration control unit 108, and a coordinate position database 110.

[0032] The coordinate position database 110 is pre-stored in memory 14. The coordinate position database 110 associates first coordinate data, second coordinate data, and identification information of the control unit 410. The processor 12 can uniquely derive other information associated with one of the pieces of information—first coordinate data, second coordinate data, or identification information of the control unit 410—by searching the coordinate position database 110. The coordinate position database 110 has a table for each aircraft type in which the first coordinate data, second coordinate data, and identification information of the control unit 410 are associated.

[0033] The first coordinate data represents the coordinate positions (in virtual space) of multiple control units 410 installed in the cockpit 404 of aircraft 402 in the virtual space for each of several aircraft models. For example, the first coordinate data according to this embodiment represents the coordinate positions of each of the multiple control units 410 included in the virtual reality image 400 shown in Figure 3.

[0034] The second coordinate data represents the coordinate position (in real space) on the surface of the haptic device 30 in real space, and is coordinate data that represents the coordinate position (in real space) corresponding to the coordinate position (in real space) of the plurality of operation units 410 (in virtual space). More specifically, the second coordinate data according to this embodiment represents the coordinate position (in real space) of the operation unit setting position 310 on the surface of the haptic device 30 shown in Figure 2. The plurality of operation unit setting positions 311, 312, 313, 314, and 315 (in real space) correspond to the coordinate positions (in virtual space) of the plurality of operation units 411, 412, 413, 414, and 415 in the virtual reality image 400, respectively.

[0035] The identification information for the control unit 410 is identification information used to identify each of the multiple control units 410. The identification information for the control unit 410 may be an identification number, identification symbol, etc., generated for each of the multiple control units 410. Since the types and number of multiple control units 410 installed in the cockpit 404 differ depending on the aircraft model, the identification information for the control unit 410 is set for each aircraft model. This identification information makes it possible to uniquely identify each of the multiple control units 410 installed in the cockpit 404 of each aircraft model 402.

[0036] The control unit 100 controls the flight simulation. For example, the control unit 100 executes a flight simulation of one aircraft model selected by the user 2 from among several aircraft models. For example, the control unit 100 reproduces and controls the simulated flight of the aircraft 402 of the aircraft model selected by the user 2 in a virtual space. For example, the control unit 100 reflects the operation of the haptic device 30, the control wheel 40, the rudder pedals 50, etc., by the user 2 in the flight movements of the flight simulation that is being executed. The control unit 100 also controls the external environment other than the aircraft 402 in the virtual space. The external environment consists of environmental conditions that may affect the flight of the aircraft 402, such as weather, temperature, wind speed, etc.

[0037] The display control unit 102 displays images related to the flight simulation on the head-mounted display 20. For example, the display control unit 102 displays the virtual reality image 400 shown in Figure 2 on the head-mounted display 20. For example, the display control unit 102 displays the virtual reality image 400 on the head-mounted display 20, which depicts multiple control units 410 located in the cockpit 404 of the aircraft model selected by the user 2.

[0038] Furthermore, the display control unit 102 changes the display mode of the cockpit 404 of the virtual reality image 400 based on, for example, the detection results from the attitude sensor 24. Specifically, the display control unit 102 changes the display mode of the cockpit 404 of the virtual reality image 400 displayed on the head-mounted display 20 in conjunction with various movements, postures, and orientations of the head of the user 2 wearing the head-mounted display 20. For example, when the user 2 wearing the head-mounted display 20 is seated in the cockpit 60 and facing forward, the display control unit 102 displays the virtual reality image 400 on the head-mounted display 20 that represents the area of ​​the cockpit 404 in the virtual space that is visible directly in front of the cockpit. Then, when the user 2 wearing the head-mounted display 20 is seated in the cockpit 60 and turns to the right, the display control unit 102 displays the virtual reality image 400 on the head-mounted display 20 that represents the area of ​​the cockpit 404 in the virtual space that is visible to the right of the cockpit. Similarly, if User 2 turns to the left, rear, up, or down, the virtual reality image 400 representing the area visible to the left, rear, up, or down of the cockpit 404 is displayed. In this way, the display control unit 102 can display a 360-degree omnidirectional virtual reality image 400 of the cockpit 404 of the aircraft 402 in the virtual space, in conjunction with various movements, postures, and orientations of User 2's head detected by the attitude sensor 24. This improves User 2's immersion in the flight simulation.

[0039] The contact position acquisition unit 104 acquires second coordinate data from the contact sensor 32. For example, when the user 2's finger touches an operation unit setting position 310 on the surface of the tactile device 30, the contact position acquisition unit 104 acquires the second coordinate data of the operation unit setting position 310 corresponding to the contact position from the contact sensor 32. For example, when the user 2's finger touches a specific operation unit setting position 311 on the surface of the tactile device 30, the contact sensor 32 detects the coordinate position of the operation unit setting position 311, which is the contact position, and transmits the second coordinate data of the operation unit setting position 311 to the control device 10. The contact position acquisition unit 104 then acquires the second coordinate data of the operation unit setting position 311 transmitted by the contact sensor 32. By acquiring this second coordinate data, it is possible to identify which operation unit 410 the user 2 operated during the flight simulation.

[0040] The determination unit 106 determines, for example, whether user 2's fingers have come into contact with the tactile device 30, based on the detection result of the contact sensor 32 of the tactile device 30. For example, the determination unit 106 determines whether user 2's fingers have come into contact with one of the multiple operation unit setting positions 310 set on the surface of the tactile device 30. For example, if user 2's fingers come into contact with a specific operation unit setting position 311 on the surface of the tactile device 30, the contact sensor 32 detects the coordinate position of the operation unit setting position 311, which is the contact position, and transmits the second coordinate data of the operation unit setting position 311 to the control device 10. Then, if the determination unit 106 receives the second coordinate data of the operation unit setting position 311 from the contact sensor 32, it determines that user 2's fingers have come into contact with the operation unit setting position 311.

[0041] Furthermore, the determination unit 106 identifies the control unit 410 and the coordinate position of the control unit 410 corresponding to the contact position, based on the coordinate position database 110 and the contact position (control unit setting position 310). In detail, the determination unit 106 uses the second coordinate data corresponding to the contact position acquired by the contact position acquisition unit 104 to search the coordinate position database 110 of one aircraft model selected by the user 2, detects the identification information and first coordinate data of one control unit 410 associated with the second coordinate data, and identifies the one control unit 410 and the coordinate position of that one control unit 410. Here, one control unit 410 is one of several control units 410 located in the cockpit 404 of one aircraft model selected by the user 2 that corresponds to the contact position (control unit setting position 310).

[0042] For example, when user 2 touches the operating unit setting position 311 of the haptic device 30 shown in Figure 2, the determination unit 106 identifies the operating unit 411 of the virtual reality image 400 shown in Figure 3 and the coordinate position of the operating unit 411 as the operating unit 410 corresponding to the contact position. More specifically, when user 2 touches the operating unit setting position 311, the contact position acquisition unit 104 acquires second coordinate data of the operating unit setting position 311, which is the contact position in real space. Then, the determination unit 106 searches the coordinate position database 110 using the second coordinate data and acquires the identification information and first coordinate data of the operating unit 411 associated with the second coordinate data. Then, the determination unit 106 identifies the operating unit 411 and the coordinate position of the operating unit 411 based on the identification information and first coordinate data of the operating unit 411.

[0043] Furthermore, the determination unit 106 may, for example, use the second coordinate data acquired by the contact position acquisition unit 104 to search the coordinate position database 110 and determine whether the identification information of the operation unit 410 and the first coordinate data corresponding to the second coordinate data are registered in the coordinate position database 110. As a result, if the first coordinate data is not registered, it can be determined that the user 2's contact with the tactile device 30 is not an operation of the operation unit 410.

[0044] The vibration control unit 108 controls the vibration of the vibrator 34. When the user 2 touches the surface of the tactile device 30, the vibration control unit 108 vibrates the contact point on the surface of the tactile device 30 using the vibrator 34. For example, when a single operating part 410 corresponding to the contact point of the user 2 on the tactile device 30 is identified, the vibration control unit 108 vibrates that contact point using the vibrator 34.

[0045] Furthermore, when a user 2 touches the tactile device 30, the vibration control unit 108 may change the vibration pattern of the vibrator 34 according to the type of one operation unit 410 in the virtual space identified from the contact position. For example, if the one operation unit 410 identified from the contact position is a push-button switch, the vibration control unit 108 will vibrate the vibrator 34 twice, once when the button is pressed and again when it is released. Alternatively, if the one operation unit 410 identified from the contact position is a snap switch, the vibration control unit 108 will vibrate the vibrator 34 once, once when the snap switch is tilted.

[0046] [3. Processing flow of the flight simulation system according to the first embodiment] Next, with reference to Figure 5, the processing flow by the control device 10 of the flight simulation system 1 according to the first embodiment will be described. Figure 5 is a flowchart showing the flight simulation control processing by the control device 10 according to the first embodiment.

[0047] As shown in Figure 5, User 2 selects an aircraft model to be used for flight simulation from among several models (step S100). The control unit 100 then sets the one model selected by User 2 as the model for which flight simulation will be performed.

[0048] Next, the display control unit 102 retrieves a virtual reality image 400 from the memory 14, which depicts multiple control units 410 located in the cockpit 404 for one model selected by the user 2. Then, the display control unit 102 displays the retrieved virtual reality image 400 on the head-mounted display 20 (step S102).

[0049] Subsequently, the control unit 100 starts a flight simulation using the selected aircraft (step S104). During the execution of this flight simulation, the control unit 100 and the display control unit 102 change the content of the virtual reality image 400 displayed on the head-mounted display 20 according to the simulation content and the operations performed by the user 2.

[0050] When the flight simulation starts, the determination unit 106 determines whether or not user 2 has made contact with the control unit setting position 310 on the surface of the tactile device 30 (step S106).

[0051] As a result, if it is determined that user 2 has not made contact with the tactile device 30 (NO in step S106), the process moves to step S116. On the other hand, if it is determined that user 2 has made contact with the tactile device 30 (YES in step S106), the contact position acquisition unit 104 acquires second coordinate data of the operation unit setting position 310 corresponding to the contact position from the contact sensor 32 (step S108). For example, if the contact position made by user 2 is the operation unit setting position 311, the contact position acquisition unit 104 acquires second coordinate data of the operation unit setting position 311, which is the contact position, from the contact sensor 32.

[0052] Subsequently, the determination unit 106 identifies the operation unit 410 and the coordinate position of the operation unit 410 corresponding to the contact position, based on the second coordinate data of the operation unit setting position 310 corresponding to the contact position acquired in step S108 and the coordinate position database 110 (step S110).

[0053] Subsequently, the control unit 100 reflects the operation performed on the identified single control unit 410 in the flight simulation of the selected aircraft model (step S112).

[0054] For example, if the user 2's contact point is the control unit setting position 310, the control unit 100 determines that the control unit 410 corresponding to the control unit setting position 310 has been operated, and reflects the operation performed on the control unit 410 in the flight movements of the ongoing flight simulation. For example, if the control unit 411 is a switch that performs a specific action of the aircraft 402, when the user 2 touches the control unit setting position 311 in the real world, the control unit 100 determines that the control unit 411 has been operated, and reflects that specific action in the flight movements of the ongoing flight simulation. For example, if the control unit 411 in the virtual space is a switch related to the engine starter, and the control unit 411 is in the off state, when the user 2 touches the control unit setting position 311 in the real world, the control unit 100 switches the control unit 411 from off to on. The control unit 100 then determines that the control unit 411 has been operated to on, and reflects this in the flight movements of the flight simulation to start the engine of the aircraft 402 in the virtual space. Furthermore, the display control unit 102 reflects the operation of a specific operation unit 410 in the virtual reality image 400 and displays it on the head-mounted display 20. For example, if the operation unit 411 in the virtual space is off, when the user 2 touches the operation unit setting position 311 in the real space, the display control unit 102 displays a virtual reality image 400 on the head-mounted display 20 in which the operation unit 411 in the virtual space has switched from off to on.

[0055] Subsequently, the vibration control unit 108 vibrates the contact position of the tactile device 30 in a vibration mode of the vibrator 34 corresponding to the type of the identified operating unit 410 (step S114).

[0056] If, in step S106, it is determined that user 2 is not in contact with the tactile device 30 (NO in step S106), or if the tactile device 30 is vibrated in step S114, the control unit 100 determines whether the flight simulation has ended or not (step S116). If it is determined that the flight simulation has not ended (NO in step S116), the process is repeated from step S106 and the flight simulation continues. On the other hand, if it is determined that the flight simulation has ended (YES in step S116), the flight simulation control process is terminated.

[0057] As described above, according to the flight simulation system 1 of the first embodiment, first coordinate data representing the coordinate positions of multiple control units 410 installed in the cockpit 404 of the aircraft 402 in the virtual space for each of the multiple aircraft 402 models is stored in advance. When the user 2 touches the haptic device 30 while a flight simulation of the aircraft selected by the user 2 is being performed, the control device 10 can identify the control unit 410 operated by the user 2 based on the coordinate position database 110, in which the first coordinate data and the second coordinate data are pre-associated, and the second coordinate data representing the actual contact position. As a result, even when performing flight simulations of aircraft models with different arrangements of control units 410 in the same flight simulation system 1, it is possible to simulate multiple control units 410 of different aircraft models using the same haptic device 30. In this way, when performing a flight simulation of one of multiple aircraft models, it is not necessary to prepare hardware corresponding to that aircraft model, thus providing an inexpensive and highly versatile flight simulation system.

[0058] Furthermore, according to the flight simulation system 1 of the first embodiment, when user 2 touches the surface of the tactile device 30, the vibration control unit 108 vibrates the contact point with the vibrator 34. As a result, when user 2 operates the operation unit 410 displayed in the virtual reality image 400, the vibration can inform user 2 that the operation of the operation unit 410 has been performed, enabling user 2 to reliably operate the operation unit 410 in the virtual space.

[0059] Furthermore, according to the flight simulation system 1 of the first embodiment, when the user 2 touches the tactile device 30, the vibration control unit 108 changes the vibration pattern of the vibrator 34 according to the type of one of the operating parts 410 in the virtual space, which is identified from the contact position. This allows the user 2 to easily understand which operating part 410 in the virtual space has been operated based on the vibration pattern, thereby improving the operability of the tactile device 30 and the realism of the operating parts 410 in the flight simulation.

[0060] [4. Functional configuration of the flight simulation system according to the second embodiment] Next, with reference to Figures 6 and 7, the flight simulation system 1 according to the second embodiment of the present invention will be described in detail. The second embodiment is a modified version of the first embodiment, and the differences from the first embodiment will be described below, while detailed descriptions of the same configuration and functions as the first embodiment will be omitted.

[0061] In the flight simulation system 1 according to the first embodiment described above, the control unit 410 and its coordinate position are identified in the virtual space based on the coordinate position database 110 stored in the memory 14 and the contact position of the haptic device 30 that the user 2 has touched. However, the flight simulation system 1 according to the first embodiment assumes that the haptic device 30 is installed at an appropriate coordinate position (real space) corresponding to the coordinate position (virtual space) of the control unit 410.

[0062] Therefore, if the haptic device 30 is installed in a position that is deviated from the correct position (real space), the operating unit setting position 310 of the haptic device 30 in real space will not match the coordinate position of the operating unit 410 represented by the first coordinate data stored in the coordinate position database 110.

[0063] Therefore, in the flight simulation system 1 according to the second embodiment, two-dimensional barcodes 511, 512, 513, 514, and 515 (hereinafter collectively referred to simply as "two-dimensional barcodes 510") representing the operation unit setting positions 310 are displayed on the liquid crystal display 36 of the tactile device 30 (see Figure 7), and the external camera 22 of the head-mounted display 20 captures images of the two-dimensional barcodes 510. As a result, the flight simulation system 1 according to the second embodiment can correct the coordinate position of the operation unit 410 in the virtual space based on the display position (i.e., the operation unit setting position 310), which is the position in the real space where the two-dimensional barcodes 510 are displayed on the liquid crystal display 36, and can accurately match the coordinate position of the operation unit 410 in the virtual space with the display position of the two-dimensional barcodes 510.

[0064] Figure 6 is a block diagram showing an example of the functional configuration of the flight simulation system 1 according to the second embodiment. Figure 7 is a perspective view showing an example of how the flight simulation system 1 according to the second embodiment is used. As shown in Figure 6, the control device 10 of the flight simulation system 1 according to the second embodiment includes, in addition to the control unit 100, display control unit 102, contact position acquisition unit 104, determination unit 106, vibration control unit 108, and coordinate position database 110 which are components of the control device 10 of the flight simulation system 1 according to the first embodiment, an identification information display control unit 200, an identification information acquisition unit 202, and a coordinate position correction unit 204.

[0065] The identification information display control unit 200 causes one or more two-dimensional barcodes 510 to be displayed on the liquid crystal display 36 of the tactile device 30. The two-dimensional barcodes 510 are an example of identification information. The two-dimensional barcodes 510 are identification information generated for each of the multiple operation units 410 in order to identify each of the multiple operation units 410 in the virtual space. Note that the two-dimensional barcodes 510 of the operation units 410 are not limited to two-dimensional barcodes, but may be QR codes (registered trademarks), one-dimensional barcodes, text information such as identification information for the operation unit 410 or identification symbols, various patterns, or any information that can represent the operation unit 410 in an identifiable way, such as the emission pattern of an infrared lamp.

[0066] The identification information display control unit 200, for example, refers to a coordinate position database 110 pre-stored in the memory 14 and identifies the coordinate position and two-dimensional barcode 510 on the surface of the haptic device 30 in the real space that corresponds to one of the multiple operation units 410 in the virtual space. The identification information display control unit 200 then displays the two-dimensional barcode 510 corresponding to that operation unit 410 at that coordinate position on the haptic device 30. The identification information display control unit 200 displays the two-dimensional barcodes 511, 512, 513, 514, and 515 on the liquid crystal display 36, for example, as shown in Figure 7. The two-dimensional barcodes 511, 512, 513, 514, and 515 are identification information corresponding to the operation units 411, 412, 413, 414, and 415 in the virtual reality image 400 shown in Figure 3, respectively. For example, the identification information display control unit 200 uses the first coordinate data of the operation unit 411 in the virtual reality image 400 shown in Figure 3 to search the coordinate position database 110 and derive the second coordinate data corresponding to the first coordinate data. Then, the identification information display control unit 200 displays a two-dimensional barcode 511 at the coordinate position on the surface of the haptic device 30 in real space corresponding to the second coordinate data, as shown in Figure 7.

[0067] Furthermore, the coordinate position database 110 according to this embodiment stores the first coordinate data, the second coordinate data, the identification information of the operation unit 410, and the two-dimensional barcode 510 in association with each other.

[0068] The identification information acquisition unit 202 acquires, for example, a two-dimensional barcode 510 captured by the external camera 22 and the imaging position, which is the position where the two-dimensional barcode 510 was captured. The identification information acquisition unit 202 acquires, for example, the distance and direction from the position of the cockpit 60 where the user 2 is seated to the captured two-dimensional barcode 510 as the imaging position.

[0069] The coordinate position correction unit 204 corrects the coordinate position of the operation unit 410 in the virtual space. For example, the coordinate position correction unit 204 determines whether the imaging position of the two-dimensional barcode 510 acquired by the identification information acquisition unit 202 matches the coordinate position of the operation unit 410 in the virtual space that corresponds to the two-dimensional barcode 510. For example, the coordinate position correction unit 204 determines whether the distance and direction from the position of the cockpit 60 where user 2 is seated to the imaged two-dimensional barcode 510 in the real space matches the distance and direction from the position of the cockpit where user 2 is seated to the coordinate position of the operation unit 410 that corresponds to the two-dimensional barcode 510 in the virtual space.

[0070] The coordinate position correction unit 204 then corrects the coordinate position of the operation unit 410 in the virtual space if, for example, the imaging position of the two-dimensional barcode 510 does not match the coordinate position of the operation unit 410 in the virtual space. The coordinate position correction unit 204 corrects the coordinate position of the operation unit 410 in the virtual space corresponding to the two-dimensional barcode 510 based on the imaging position of the two-dimensional barcode 510 captured by the external camera 22. In detail, the coordinate position correction unit 204 replaces the distance and direction from the position of the cockpit 60 where the user 2 is seated to the captured two-dimensional barcode 510 in the virtual space, and corrects the coordinate position in the virtual space that is at the same distance and direction from the position of the cockpit where the user 2 is seated as the new coordinate position of the operation unit 410 corresponding to the two-dimensional barcode 510.

[0071] The display control unit 102 overlays an image representing the operation unit 410 in the virtual space onto the virtual reality image 400. The virtual reality image 400 displays images in the following order, overlaid: an image representing the external environment 406, an image representing the cockpit 404, an image representing the instrument panel 420, and an image representing the operation unit 410. The display control unit 102 overlays an image representing the operation unit 410 corresponding to the two-dimensional barcode 510 captured by the external camera 22 onto the virtual reality image 400 displayed on the head-mounted display 20. Furthermore, if, for example, the coordinate position of the operation unit 410 in the virtual space is corrected by the coordinate position correction unit 204, the display control unit 102 changes the image representing the operation unit 410 from its pre-correction coordinate position to its new corrected coordinate position and overlays it onto the virtual reality image 400.

[0072] [5. Processing flow of the flight simulation system according to the second embodiment] Next, with reference to Figures 8 and 9, the processing flow by the control device 10 of the flight simulation system 1 according to the second embodiment will be described. Figure 8 is a flowchart showing the flight simulation control processing by the control device 10 according to the second embodiment. Note that the processing shown in Figure 8 is the same as the flight simulation control processing by the control device 10 according to the first embodiment shown in Figure 5, except for step S200, and the explanation of steps other than S200 will be omitted.

[0073] As shown in Figure 8, after displaying the virtual reality image 400 on the head-mounted display 20 in step S102, the control device 10 performs the coordinate position correction process shown in Figure 9 (step S200).

[0074] In step S200, the control device 10 displays a two-dimensional barcode 510 on the liquid crystal display 36 of the tactile device 30. The control device 10 then captures the two-dimensional barcode 510 with the external camera 22 and obtains the capture position of the two-dimensional barcode 510 from the external camera 22. The control device 10 then determines whether the capture position of the two-dimensional barcode 510 matches the coordinate position of the operation unit 410 corresponding to the two-dimensional barcode 510. If the capture position of the two-dimensional barcode 510 does not match the coordinate position of the operation unit 410 corresponding to the two-dimensional barcode 510, the control device 10 corrects the coordinate position of the operation unit 410 corresponding to the two-dimensional barcode 510. The control device 10 then displays the operation unit 410 on the head-mounted display 20 based on the corrected coordinate position.

[0075] Subsequently, the process moves to step S104 shown in Figure 8. The process from step S104 to step S116 shown in Figure 8 is the same as the process from step S104 to step S116 of the flight simulation control process by the control device 10 according to the first embodiment shown in Figure 5, and therefore the explanation is omitted.

[0076] Here, the coordinate position correction process (step S200) shown in Figure 8 will be explained in more detail with reference to Figure 9. Figure 9 is a flowchart of the above coordinate position correction process (step S200) by the control device 10 according to the second embodiment.

[0077] As shown in Figure 9, the identification information display control unit 200 displays all of the two-dimensional barcodes 510 representing the multiple control units 410 located in the cockpit 404 of a single model selected by the user 2 on the liquid crystal display 36 of the tactile device 30 (step S202). The liquid crystal display 36 of the tactile device 30 receives information from the identification information display control unit 200 and displays the two-dimensional barcodes 510 of the multiple control units 410 at the positions corresponding to the multiple control units 410 in the real world.

[0078] Subsequently, the external camera 22 of the head-mounted display 20 detects the two-dimensional barcode 510 displayed on the liquid crystal display 36 of the haptic device 30. Next, the identification information acquisition unit 202 acquires the detected two-dimensional barcode 510 and the imaging position of the two-dimensional barcode 510 from the external camera 22 (step S204).

[0079] Subsequently, the coordinate position correction unit 204 determines whether the imaging position of the two-dimensional barcode 510 acquired by the identification information acquisition unit 202 matches the coordinate position of the operation unit 410 in the virtual space corresponding to the two-dimensional barcode 510 (step S206).

[0080] As a result, if it is determined that the imaging position of the two-dimensional barcode 510 and the coordinate position of the operation unit 410 match (YES in step S206), the coordinate position correction process is terminated. On the other hand, if it is determined that the imaging position of the two-dimensional barcode 510 and the coordinate position of the operation unit 410 do not match (NO in step S206), the coordinate position correction unit 204 corrects the coordinate position of the operation unit 410 in the virtual space corresponding to the two-dimensional barcode 510 based on the imaging position of the two-dimensional barcode 510 acquired by the identification information acquisition unit 202 (step S208).

[0081] Subsequently, the display control unit 102 changes the image representing the operation unit 410 in the virtual space corresponding to the two-dimensional barcode 510 from its original coordinate position to the corrected new coordinate position and displays it on the head-mounted display 20 (step S210), thus ending the coordinate position correction process. The display control unit 102 superimposes the image representing the operation unit 410 corresponding to the two-dimensional barcode 510 onto the virtual reality image 400 displayed on the head-mounted display 20 and displays it at the corrected coordinate position.

[0082] As described above, according to the flight simulation system 1 of the second embodiment, a two-dimensional barcode 510 is displayed on the liquid crystal display 36, and the external camera 22 captures an image of the two-dimensional barcode 510. As a result, the coordinate position correction unit 204 can correct the coordinate position of the operation unit 410 in the virtual space corresponding to the two-dimensional barcode 510 based on the image position of the two-dimensional barcode 510. The display control unit 102 then displays an image of the operation unit 410 at the corrected coordinate position of the operation unit 410 so as to be superimposed on the virtual reality image 400. As a result, even if the haptic device 30 is not positioned appropriately, the display position of the two-dimensional barcode 510 displayed on the liquid crystal display 36 in the real space can be matched with the coordinate position of the operation unit 410 in the virtual space corresponding to the two-dimensional barcode 510, enabling the user 2 to accurately operate the operation unit 410 in the virtual space.

[0083] Embodiments of the present invention have been described above with reference to the attached drawings, but it goes without saying that the present invention is not limited to these embodiments. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of the present invention.

[0084] For example, although the tactile device 30 was described as being rectangular in shape in the above embodiment, it is not limited to this example. The tactile device 30 may be, for example, polygonal in shape or curved in shape.

[0085] The series of processes performed by the flight simulation system 1 according to the above embodiment may be implemented using software, hardware, or a combination of software and hardware. The program constituting the software is pre-stored in, for example, a non-transitory storage medium provided inside or outside each device. The program is then read from, for example, the non-transitory storage medium (e.g., ROM) to a temporary storage medium (e.g., RAM) and executed by a processor such as a CPU.

[0086] Furthermore, according to the embodiments described above, a program for executing the processing of each function of the flight simulation system 1 can be provided. In addition, a computer-readable non-temporary recording medium on which the program is stored can also be provided. The non-temporary recording medium may be a disk-type recording medium such as an optical disk, magnetic disk, or magneto-optical disk, or it may be a semiconductor memory such as a flash memory or USB memory. [Explanation of Symbols]

[0087] 1. Flight Simulation System 2 users 10 Control device 12 processors 14 memory 20 Head-mounted displays 22 External camera (imaging unit) 24. Attitude sensors 30 Haptic Devices 32 Contact Sensors 34 Vibrators 36. Liquid crystal display (display unit) 100 Control Unit 102 Display Control Unit 106 Judgment section 108 Vibration Control Unit 204 Coordinate position correction section 400 virtual reality images 402 Aircraft 404 Cockpit 410 Operation section 510 Two-dimensional barcode

Claims

1. A flight simulation system configured to perform flight simulations for multiple aircraft types, A head-mounted display that can be worn by the user and displays a virtual reality image representing the cockpit of an aircraft in a virtual space, In real space, a haptic device is positioned at locations corresponding to multiple control units installed in the cockpit of the virtual reality image, A control device for controlling the flight simulation using the head-mounted display and the haptic device, Equipped with, The head-mounted display has an imaging unit that captures the surrounding environment of the head-mounted display in the real space, The aforementioned tactile device is A contact sensor for detecting the user's contact position with the tactile device, A vibrator that vibrates the aforementioned contact position, A display unit that displays identification information for the plurality of operating units at positions corresponding to the plurality of operating units in the real space, It has, The control device is Processor and With respect to the aforementioned multiple aircraft models, a memory is provided for storing coordinate data representing the coordinate positions of the multiple operating units installed in the cockpit of each aircraft model. It has, The aforementioned processor, The process involves performing a flight simulation of one aircraft selected by the user from among the aforementioned multiple aircraft, The process involves displaying the virtual reality image, in which the multiple operating units of the selected model are depicted, on the head-mounted display. The operation unit corresponding to the identification information captured by the imaging unit is superimposed and displayed on the virtual reality image displayed on the head-mounted display, When the user touches the haptic device, the contact point is vibrated by the vibrator. Based on the coordinate data and contact position of the selected model, one of the multiple operating parts of the selected model that corresponds to the contact position is identified. To reflect the operation of the identified control unit in the flight simulation of the selected aircraft model, A flight simulation system that performs processes including those mentioned above.

2. The flight simulation system according to claim 1, wherein the processor changes the vibration mode of the vibrator according to the type of one identified operating part when the user touches the haptic device.

3. The flight simulation system according to claim 1 or 2, wherein the coordinate position of the operation unit superimposed on the virtual reality image is corrected based on the position of the identification information captured by the imaging unit.

4. The head-mounted display has a posture sensor that detects the movement of the user's head while wearing the head-mounted display. The aforementioned processor, The flight simulation system according to claim 1 or 2, wherein, based on the detection results from the attitude sensor, the display mode of the cockpit of the virtual reality image displayed on the head-mounted display is changed in conjunction with the movement of the head of the user wearing the head-mounted display.

Images