Programs and simulators

Abstract

To provide a program and a simulator which enable a user to experience maneuvering a construction machine without requiring a real construction machine, remote controller, or simulation device.SOLUTION: A program 32 causes a computer 30 to perform: first setting processing of setting a construction machine model 81 in a first virtual space 80; first generation processing of generating a two-dimensional texture obtained by viewing the construction machine model 81 from a position of a first view point 88 within the first virtual space 80; second setting processing of setting a display device model 92 in a second virtual space 90; second generation processing of generating a video obtained by viewing the display device model 92 from a position of a second view point 98 within the second virtual space 90; and mapping processing of mapping the two-dimensional texture generated by the first generation processing to the display device model 92 in the video generated by the second generation processing.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a program and a simulator.

Background Art

[0002] Patent Document 1 discloses a system for remotely operating a construction machine. The operator's cab is located at a place separated from the construction machine at the work site, and a remote controller and a display are installed in the operator's cab. A camera at the work site photographs the construction machine, and the photographed image is displayed on the display. The operator remotely operates the construction machine using the remote controller while looking at the display.

[0003] However, the video signal and the operation signal are not transmitted immediately, and it takes time to transmit those signals. Therefore, the photographed image is displayed on the display with a delay from the photographing time at the work site, and the construction machine operates with a delay from the operation time in the operator's cab. In order to solve such problems, the technique disclosed in Patent Document 2 has been proposed.

[0004] In addition, the video signal deteriorates during transmission, and the image displayed on the display is distorted. In order to solve such problems, the technique disclosed in Patent Document 3 has been proposed.

[0005] In addition, since the operator is so focused on operating the construction machine through the remote controller, the operator may not notice an abnormal inclination of the construction machine and accidentally cause the construction machine to tip over. In order to solve such problems, the technique disclosed in Patent Document 4 has been disclosed.

[0006] However, even due to the existence of Patent Documents 1 to 4, in the remote operation of a construction machine, characteristics in the operating feeling, such as the amount of information received by the operator being less and the responsiveness being different compared to the operation in actual machine boarding, are not eliminated. Therefore, in order to engage in the remote operation of a construction machine, it is essential to practice newly acquiring the said characteristics.

[0007] Ideally, the aforementioned practice should include three elements: the same remote control system used in the actual project, the same construction machinery used in the actual project being remotely controlled by this system, and a space where the construction machinery can safely perform practice maneuvers and where some mistakes are tolerated. However, while having the same construction machinery used in the actual project being remotely controlled is feasible, it is economically difficult to constantly provide a space where the construction machinery can safely perform practice maneuvers. At best, after receiving a contract for the actual construction and arriving on-site, limited and restrictive practice sessions were conducted under the guise of test construction, with the client's consent.

[0008] On the other hand, a simulator is used as a means to secure a free practice space for operating construction machinery. The simulator is a replica of the control room and control levers of construction machinery, and is designed to reproduce the operation of a virtual construction machinery model displayed on a screen under the same conditions as operating actual construction machinery. The configuration of this simulator is also applicable to remote control systems. Specifically, the remote control room and control levers remain those of the simulator, but the target of operation displayed on the screen is replaced from the construction machinery model to an actual machine, thereby realizing a remote control system. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2016-072727 [Patent Document 2] Japanese Patent Publication No. 2017-092908 [Patent Document 3] Japanese Patent Publication No. 2018-021395 [Patent Document 4] Japanese Patent Publication No. 2018-142123 [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] The technologies disclosed in Patent Documents 1 to 4 require the preparation of construction machinery and remote control devices, which incur significant costs, time, and effort. Even technologies using simulation devices require significant costs, time, and effort to prepare the simulation devices. The remote control device referred to here comprises a remote control room, a remote controller, construction machinery, multiple cameras installed around the work area of ​​the construction machinery, a display installed in the remote control room that displays the images captured by the cameras, and communication equipment for transmitting images to the display. The simulation device referred to here comprises a simulated control room, a simulated remote controller, a virtual reality practice space constructed by a computer, a construction machinery model in the practice space, and a display that displays the construction machinery model and a virtual reality landscape from the viewpoint of a virtual camera placed in the practice space. Therefore, the object of the present invention is to provide a program and simulator that allows users to practice remote control of construction machinery even in cases where neither actual construction machinery and remote control devices, nor simulated devices that mimic remote control, are available. [Means for solving the problem]

[0011] According to a first aspect of the present invention for solving the above problems, a program causes a computer to perform virtual reality processing that displays a three-dimensional display device model in a second virtual space, on which a three-dimensional construction machine model in a first virtual space, which operates based on the output signal of an input device, is drawn, on a display device.

[0012] According to a second aspect of the present invention, the virtual reality processing includes: a first setting process for setting the construction machine model in the first virtual space; a first generation process for generating a two-dimensional texture obtained by viewing the construction machine model from a first viewpoint in the first virtual space; a second setting process for setting the display device model in the second virtual space; a second generation process for generating an image obtained by viewing the display device model from a second viewpoint in the second virtual space; a mapping process for mapping the two-dimensional texture generated by the first generation process onto the display device model in the image generated by the second generation process; and a display process for displaying the image onto which the two-dimensional texture has been mapped by the mapping process on the display device.

[0013] According to a third aspect of the present invention, the first setup process includes a process that causes the construction machine model to operate in the first virtual space based on a command or information entered by the user through the input device.

[0014] According to a fourth aspect of the present invention, the first setting process includes a process of changing the position of the first viewpoint or the direction of its line of sight based on a command or information entered by the user through the input device.

[0015] According to a fifth aspect of the present invention, the second setting process includes a process for setting the position of the second viewpoint and the direction of the line of sight based on the tilt angle measured by a posture measurement unit integrated with the display device.

[0016] According to a sixth aspect of the present invention, the second setting process includes a process of setting the cockpit model together with the display device model in the second virtual space, and the second generation process includes a process of generating the image obtained by viewing the cockpit model together with the display device model from the position of a second viewpoint.

[0017] According to the seventh aspect of the present invention, the second setting process includes a process of setting a control lever model together with the display device model in the second virtual space, and a process of tilting the control lever model in the second virtual space based on a command or information input by a user through the input device. The second generation process includes a process of generating the video obtained by viewing the control lever model together with the display device model from the position of the second viewpoint.

[0018] According to the eighth aspect of the present invention, the first setting process includes a process of setting a construction target model together with the construction machine model in the first virtual space. The first generation process includes a process of generating the two-dimensional texture obtained by viewing the construction target model together with the construction machine model from the position of the first viewpoint.

[0019] According to the ninth aspect of the present invention, the simulator includes the computer.

Advantages of the Invention

[0020] According to the present invention, a program and a simulator capable of implementing remote operation training of a construction machine for a user can be provided even in a case where neither a real construction machine and a remote control device nor a simulation device simulating remote operation can be prepared. As an effect of the provision, the feasibility and economy of arbitrary implementation of remote operation training are enhanced, and the training of remote operation operators can be implemented efficiently and economically.

Brief Description of the Drawings

[0021] [Figure 1] FIG. 1 shows a real space. [Figure 2] FIG. 2 shows a three-dimensional first virtual space. [Figure 3] FIG. 3 shows a three-dimensional second virtual space. [Figure 4] FIG. 4 shows a simulator.

Embodiments for Carrying Out the Invention

[0022] Embodiments will be described below with reference to the drawings. However, the scope of the present invention is not limited to the embodiments disclosed below. The drawings are provided for illustrative purposes only, and therefore the scope of the present invention is not limited to the examples shown in the drawings.

[0023] [1. Overview] Figure 1 is a diagram showing the real space 5. Figure 2 is a diagram showing the 3D first virtual space 80. Figure 3 is a diagram showing the 3D second virtual space 90. Figure 4 is a block diagram of the real simulator 1.

[0024] As shown in Figure 1, user 8 and simulator 1 are present in real space 5. As shown in Figures 1 and 4, simulator 1 includes a display device 10, an input device 20, and a computer 30.

[0025] As shown in Figure 2, the construction machinery model 81 and the construction target model 82 exist in the first virtual space 80. The construction machinery model 81 and the construction target model 82 are virtual and simulated objects created and set up within the first virtual space 80 through computational processing by the computer 30. The construction machinery model 81 is a model of construction machinery, particularly an excavator. The construction target model 82 is a model of a tunnel as the target of construction. The construction machinery model 81 and the construction target model 82 are polygon models, wireframe models, solid models, or point cloud models, or a combination of two or more of these models.

[0026] The construction machine model 81 includes a body model 81a, a boom model 81b, and a boring bit model 81c. The body model 81a models the body and running gear of the construction machine. The boom model 81b models the boom of the construction machine. The boring bit model 81c models the boring bit of the construction machine. The boom model 81b is connected to the front end of the body model 81a, and the boring bit model 81c is connected to the tip of the boom model 81b.

[0027] The construction target model 82 is a model of a tunnel as the target of construction. The construction target model 82 has an inner wall / floor section 82a and a face section 82b. The inner wall / floor section 82a models the inner wall and floor of the tunnel. The face section 82b models the face at the end of the tunnel. The face section 82b is connected to the front end of the inner wall / floor section 82a.

[0028] As shown in Figure 3, a cockpit model 91, multiple display device models 92, and multiple control lever models 93 exist in the second virtual space 90. The display device models 92, control lever models 93, and cockpit model 91 in the second virtual space 90 are virtual and simulated, set up and created through calculations performed by the computer 30. The display device model 92 models a flat panel display device. The control lever models 93 model control levers installed in the cockpit of a construction machine. The cockpit model 91 models the interior of the cockpit of a construction machine. The model containing the cockpit model 91, display device models 92, and control lever models 93 models a construction machine simulation device and a remote control device.

[0029] The display device model 92, the control lever model 93, and the cockpit model 91 in the second virtual space 90 are rendered on the display device 10 using 3D computer graphics technology with the computer 30. When the display device model 92 is rendered on the display device 10, the construction machine model 81 and the construction target model 82 in the first virtual space 80 are mapped onto the 2D display device model 92, as the display device model 92 is 2D.

[0030] User 8 operates a construction machine model 81 in a first virtual space 80 and a control lever model 93 in a second virtual space 90 using an input device 20 while viewing the display on the display device 10 in the real space 5. The computer 30 then operates the construction machine model 81 in the first virtual space 80 and the control lever model 93 in the second virtual space 90 in its calculations, according to the signals transmitted from the input device 20. User 8 can virtually experience operating a simulated construction machine through the display device 10, and can also virtually experience operating a construction machine through the display device 10 and the display device model 92. In particular, User 8 can virtually experience remote control of a construction machine.

[0031] The operation of the construction machine Model 81 refers to the following (1) to (3). Any of the following operations (1) to (3) may be performed individually, or two or more of the following operations (1) to (3) may be performed in combination. (1) The construction machine model 81 moves as a whole within the first virtual space 80. (2) The boom model 81b tilts relative to the main body model 81a within the first virtual space 80, and the boring bit model 81c follows the tip of the boom model 81b. (3) The boring bit model 81c rotates relative to the boom model 81b within the first virtual space 80.

[0032] The operation of the control lever model 93 refers to the tilting of the control lever model 93 within the second virtual space 90.

[0033] The coordinate system that represents position and vectors in the first virtual space 80 is called the first world coordinate system, and the coordinate system that represents position and vectors in the second virtual space 90 is called the second world coordinate system. These world coordinate systems are also called absolute coordinate systems. The first world coordinate system is defined by mutually orthogonal X1, Y1, and Z1 axes, and position and vectors in the first virtual space 80 are represented by the X1, Y1, and Z1 coordinates. The second world coordinate system is defined by mutually orthogonal X2, Y2, and Z2 axes, and position and vectors in the second virtual space 90 are represented by the X2, Y2, and Z2 coordinates.

[0034] [2. A simulator of reality] As described above, simulator 1 includes a display device 10, an input device 20, and a computer 30. These will be described in detail below.

[0035] [2-1. Display device] The display device 10 is a head-mounted display device worn on the head of the user 8. The display device 10 includes a head mount, an interface circuit 11, a posture measurement unit 12, and one or two display devices 13.

[0036] The head device is a device that is attached to the user 8's head so as to be positioned in front of the user 8's eyes. The head device includes, for example, goggles, eyeglasses, sunglasses, eyeglass frames, sunglass frames, eye masks, and eye patches. The interface circuit 11, posture measurement unit 12, and display device 13 are attached to the head device. Thus, the interface circuit 11, posture measurement unit 12, display device 13, and head device are integrated into a single unit.

[0037] The attitude measurement unit 12 measures the attitude of the display device 10, i.e., the roll angle, pitch angle, and yaw angle, and transmits the measurement signal, including the measured values ​​of the roll angle, pitch angle, and yaw angle, to the computer 30 via the interface circuit 11. The roll angle refers to the tilt angle around the longitudinal axis. The pitch angle refers to the tilt angle around the lateral axis. The yaw angle refers to the tilt angle around the vertical axis. The attitude measurement unit 12 includes sensors such as a 3D tilt sensor and a 3D gyroscope.

[0038] If there is one display device 13, this display device 13 is for both eyes. If there are two display devices 13, one display device 13 is for the left eye and the other display device 13 is for the right eye. The display device 13 displays an image according to the video signal transferred from the computer 30 to the display device 13 through the interface circuit 11.

[0039] The interface circuit 11 receives measurement signals from the attitude measurement unit 12 and outputs those measurement signals to the computer 30 according to a predetermined protocol. The interface circuit 11 also receives video signals from the computer 30 and outputs those video signals to the display device 13 according to a predetermined protocol.

[0040] The display device 10 may be a liquid crystal display, organic EL display, plasma display, or projector that is stationary and not attached to the head. In this case, the display device 10 does not need to have a posture measurement unit 12, and the display device 10 has one display device 13.

[0041] [2-2. Input Devices] The input device 20 includes one or more devices such as buttons, keys, joysticks, gamepads, mice, trackballs, switches, touch panels, touchpads, and keyboards. The input device 20 is operated by the user 8 and inputs output signals corresponding to that operation to the computer 30. This allows the user 8 to input commands and information to the computer 30 through the input device 20.

[0042] [2-3. Computers] Computer 30 includes a CPU, GPU, RAM, ROM, interface devices, and a storage device 31. The storage device 31 is either semiconductor memory or a hard disk drive. The operating system (hereinafter referred to as OS) is installed in computer 30 by being stored in the storage device 31. Computer 30 operates the OS by having the CPU read the OS from the storage device 31 and executing the OS using RAM as a storage area or working area. While the OS is running, computer 30 manages and operates the GPU, RAM, ROM, interface devices, storage device 31, and display device 10 on the OS via the CPU.

[0043] Program 32 is installed on the computer 30 by being stored in the storage device 31. The computer 30 then executes program 32 using the CPU and GPU on the operating system.

[0044] Construction machine model data 33 is stored in the storage device 31. The construction machine model data 33 is data that can be read by the CPU and GPU. The construction machine model data 33 is three-dimensional shape data for setting up the construction machine model 81. The construction machine model data 33 includes main body model data 33a, boom model data 33b, and boring bit model data 33c.

[0045] The main model data 33a contains three-dimensional shape data that represents multiple points, lines, planes, and vectors constituting the main model 81a in a three-dimensional local coordinate system, for the purpose of setting up the main model 81a. A local coordinate system refers to the coordinate system that each individual model possesses. A local coordinate system is also called a relative coordinate system, individual coordinate system, or object coordinate system. A local coordinate system is defined by mutually orthogonal x, y, and z axes, and the position in the local coordinate system is represented by the x, y, and z coordinates.

[0046] The boom model data 33b has three-dimensional shape data that represents multiple points, lines, surfaces, and vectors constituting the boom model 81b in a three-dimensional local coordinate system for setting up the boom model 81b. Furthermore, the boom model data 33b has node data that represents the nodes for connecting the boom model 81b to the main body model data 33a in the local coordinate system of the main body model 81a.

[0047] The boring bit model data 33c has three-dimensional shape data that represents multiple points, lines, planes, and vectors constituting the boring bit model 81c in a three-dimensional local coordinate system for setting up the boring bit model 81c. Furthermore, the boom model data 33b has nodal data that represents the nodes for connecting the boring bit model 81c to the boom model 81b in the local coordinate system of the boom model 81b.

[0048] The construction target model data 34 is stored in the storage device 31. The construction target model data 34 is data that can be read by the CPU and GPU. The construction target model data 34 has three-dimensional shape data that represents multiple points, lines, surfaces, and vectors constituting the construction target model 82 in a local coordinate system for setting up the construction target model 82, that is, the interior wall / floor portion 82a and the face portion 82b. The construction target model data 34 has a coordinate transformation matrix that transforms the coordinate system of the cockpit model 91 from the local coordinate system to the second world coordinate system in order to determine the position and orientation of the construction target model 82 in the first virtual space 80 and to place the cockpit model 91 in the second virtual space 90.

[0049] The cockpit model data 35 is stored in the storage device 31. The cockpit model data 35 is data that can be read by the CPU and GPU. The cockpit model data 35 has three-dimensional shape data that represents multiple points, lines, planes, and vectors that constitute the cockpit model 91 in the local coordinate system for setting up the cockpit model 91. The cockpit model data 35 has a coordinate transformation matrix that transforms the coordinate system of the cockpit model 91 from the local coordinate system to the second world coordinate system in order to determine the position and orientation of the cockpit model 91 in the second virtual space 90 and place the cockpit model 91 in the second virtual space 90.

[0050] The display device model data 36 is stored in the storage device 31. The display device model data 36 is data that can be read by the CPU and GPU. The display device model data 36 has three-dimensional shape data that represents multiple points, lines, planes, and vectors constituting each display device model 92 in the local coordinate system in order to configure each display device model 92. The display device model data 36 has a coordinate transformation matrix that transforms the coordinate system of the display device model 92 from the local coordinate system to the second world coordinate system in order to determine the position and orientation of the display device model 92 in the second virtual space 90 and to place the display device model 92 in the second virtual space 90.

[0051] The control lever model data 37 is stored in the storage device 31. The control lever model data 37 is data that can be read by the CPU and GPU. The control lever model data 37 has three-dimensional shape data that represents multiple points, lines, planes, and vectors constituting each control lever model 93 in the local coordinate system for setting up the control lever model 93. The control lever model data 37 has node data that represents the nodes for connecting the control lever model 93 to the cockpit model 91 in the second world coordinate system.

[0052] [2-4. Computer processing steps according to the program] Computer 30 performs the following processes using its CPU and GPU according to program 32.

[0053] (1) Setting of construction machinery models and construction target models in the first virtual space The computer 30 sets the construction target model 82 in the first virtual space 80 according to the construction target model data 34.

[0054] The computer 30 sets the main body model 81a, boom model 81b, and boring bit model 81c in the 3D local space of their respective local coordinate systems, according to the 3D shape data of the main body model data 33a, boom model data 33b, and boring bit model data 33c.

[0055] Computer 30 uses a coordinate transformation matrix based on the nodal data of the boring bit model data 33c and the output signal of the input device 20 (i.e., the command or information input by user 8 through the input device 20) to transform the coordinate system of the boring bit model 81c from the local coordinate system of the boring bit model 81c to the local coordinate system of the boom model 81b. This sets the boring bit model 81c and the boom model 81b in the same three-dimensional local space, and connects the boring bit model 81c to the boom model 81b via its nodes. Here, the coordinate transformation matrix for transforming the coordinate system of the boring bit model 81c from the local coordinate system of the boring bit model 81c to the local coordinate system of the boom model 81b is calculated by computer 30 based on the output signal of the input device 20, so that the relative orientation of the boring bit model 81c with respect to the boom model 81b is determined based on the output signal of the input device 20. Therefore, when user 8 operates the input device 20 to input a command to the computer 30 to rotate the boring bit model 81c, the orientation of the boring bit model 81c changes and the boring bit model 81c rotates.

[0056] Computer 30 uses a coordinate transformation matrix based on the nodal data of the boom model data 33b and the output signal of the input device 20 to transform the coordinate systems of the boom model 81b and the boring bit model 81c from the local coordinate system of the boom model 81b to the local coordinate system of the main body model 81a. As a result, the boring bit model 81c, the boom model 81b, and the main body model 81a are set up in the same three-dimensional local space, and the boom model 81b is coupled to the main body model 81a. Here, the coordinate transformation matrix for transforming the coordinate systems of the boom model 81b and the boring bit model 81c from the local coordinate system of the boom model 81b to the local coordinate system of the main body model 81a is calculated by computer 30 based on the output signal of the input device 20, so that the relative attitude of the boom model 81b with respect to the main body model 81a is determined based on the output signal of the input device 20.Therefore, if user 8 inputs a command to tilt the boom model 81b to computer 30 by operating the input device 20, the attitude of the boom model 81b changes and the boom model 81b tilts. The boring bit model 81c follows the tip of the boom model 81b in accordance with the tilting of the boom model 81b.

[0057] Computer 30 uses a coordinate transformation matrix based on the output signal of input device 20 to transform the coordinate systems of the main body model 81a, boom model 81b, and boring bit model 81c from the local coordinate system of the main body model 81a to the first world coordinate system. This sets the main body model 81a, boring bit model 81c, and boom model 81b in the first virtual space 80. Here, the coordinate transformation matrix for transforming the coordinate systems of the main body model 81a, boom model 81b, and boring bit model 81c from the local coordinate system to the first world coordinate system is calculated by computer 30 based on the output signal of input device 20, so that the position and orientation of the main body model 81a in the first virtual space 80 are determined based on the output signal of input device 20. Therefore, when user 8 inputs a command to move the main body model 81a to computer 30 by operating input device 20, the main body model 81a is displaced and moves. The boom model 81b follows the main body model 81a as it moves. The boring bit model 81c follows the tip of the boom model 81b as the main body model 81a moves.

[0058] As described above, by setting the construction target model 82 and the construction machine model 81 in the first virtual space 80, the construction machine model 81 is positioned inside the inner wall and floor portion 82a of the construction target model 82. Furthermore, when user 8 inputs commands for the operation of the construction machine model 81 to the computer 30 by operating the input device 20, the construction machine model 81 operates inside the inner wall and floor portion 82a of the construction target model 82. Here, the operation of the construction machine model 81 refers to the rotation of the boring bit model 81c, the tilting of the boom model 81b, and the movement of the main body model 81a, as described above.

[0059] (2) Generation of 2D textures Computer 30 generates multiple 2D textures obtained by viewing the construction target model 82 and the construction machine model 81 in the first virtual space 80 from different positions. The construction target model 82 and the construction machine model 81 are depicted in these multiple 2D textures. The generation of 2D textures will be described in detail below.

[0060] The computer 30 sets up multiple viewpoints 88 in the first virtual space 80, each corresponding to one of the multiple display device models 92. Specifically, the computer 30 sets the position, line of sight vector, and upward vector of each viewpoint 88 in the first virtual space 80, thereby setting up a first view transformation matrix for each viewpoint 88, which is determined by the position, line of sight vector, and upward vector of each viewpoint 88.

[0061] The computer 30 uses a first view transformation matrix for each viewpoint 88 to transform the coordinate systems of the construction target model 82 and the construction machine model 81 from the first world coordinate system to the first view coordinate system for each viewpoint 88. With respect to the first view coordinate system, the origin of the first view coordinate system is the position of viewpoint 88, and the planes parallel to the X3 axis and Y3 axis of the three orthogonal axes that define the first view coordinate system are orthogonal to the line of sight vector of viewpoint 88, the Y3 axis is parallel to the upward vector of viewpoint 88, and the Z3 axis is parallel to the line of sight vector of viewpoint 88.

[0062] The computer 30 transforms the coordinate systems of the construction target model 82 and the construction machine model 81 from a three-dimensional first view coordinate system to a two-dimensional first Cartesian coordinate system for each viewpoint 88, so that they are projected in perspective onto a first plane perpendicular to the line-of-sight vector of each viewpoint 88. During perspective projection, the computer 30 performs hidden line processing.

[0063] Computer 30 rasterizes the construction target model 82 and the construction machine model 81 by rendering them in a two-dimensional first Cartesian coordinate system. This generates a two-dimensional texture for each viewpoint 88 obtained by rasterizing the construction target model 82 and the construction machine model 81. The two-dimensional texture is associated with each viewpoint 88, and thereby associated with each display device model 92. The construction target model 82 and the construction machine model 81 are drawn on the two-dimensional texture.

[0064] Furthermore, when user 8 operates the input device 20 to input commands for the operation of the construction machine model 81 to the computer 30, the construction machine model 81 operates within the second virtual space 90. Therefore, as the 2D texture is updated over time, the construction machine model 81 drawn on the 2D texture operates over time.

[0065] When user 8 operates the input device 20 to input changes to the position, line of sight vector, or upward vector of viewpoint 88 to computer 30, computer 30 changes the position, line of sight vector, or upward vector of viewpoint 88. Therefore, when the 2D texture is updated over time, an effect equivalent to changing the position and orientation of a real camera that photographs real construction machinery can be obtained.

[0066] The computer 30 may also synthesize guidance for operating the construction machine model 81 into a two-dimensional texture. Guidance refers to a representation of the completed state of the construction object using drawings, symbols, or numerical values, or drawings, symbols, letters, or numerical values ​​for guiding or advising the construction machine model 81 on its destination.

[0067] (3) Setting of the cockpit model, display device model and control lever model in the second virtual space The computer 30 sets the cockpit model 91 in the second virtual space 90 according to the cockpit model data 35.

[0068] The computer 30 sets the display device model 92 in the 3D local space of its local coordinate system according to the 3D shape data of the display device model data 36. The computer 30 transforms the coordinates of the display device model 92 from its local coordinate system to the second world coordinate system based on the coordinate transformation matrix of the display device model data 36. As a result, the computer 30 sets the display device model 92 in the second virtual space 90.

[0069] The computer 30 sets the control lever model 93 in the 3D local space of its local coordinate system according to the 3D shape data of the control lever model data 37. The computer 30 uses a coordinate transformation matrix based on the node data of the control lever model data 37 and the output signal of the input device 20 to transform the coordinates of the control lever model 93 from its local coordinate system to the second world coordinate system. In this way, the computer 30 sets the control lever model 93 in the second virtual space 90. The coordinate transformation matrix for transforming the control lever model 93 from its local coordinate system to the second world coordinate system is calculated by the computer 30 based on the output signal of the input device 20, so that the attitude of the control lever model 93 in the second virtual space 90 is determined based on the output signal of the input device 20. Therefore, when user 8 inputs a command to tilt the control lever model 93 to the computer 30 by operating the input device 20, the attitude of the control lever model 93 changes and the control lever model 93 tilts. Furthermore, the tilting of the control lever model 93 is linked to the operation of the construction machine model 81.

[0070] (4) Video generation The computer 30 generates images of the cockpit model 91, display device model 92, and control lever model 93 in the second virtual space 90, as viewed from a predetermined position. If there are two display devices 13, the computer 30 generates two images. The images depict the cockpit model 91, display device model 92, and control lever model 93. The generation of images will be described in detail below.

[0071] The computer 30 sets the same number of viewpoints 98 for the display device 13 based on the measurement signals from the attitude measurement unit 12. Specifically, the computer 30 sets the position, line of sight vector, and upward vector of each viewpoint 98 in the second virtual space 90, thereby setting the second view transformation matrix determined by the position, line of sight vector, and upward vector of each viewpoint 98. Whether the number of viewpoints 98 is one or two, the position of each viewpoint 98 is in front of the display device model 92 in the second virtual space 90. When there are two viewpoints 98, the computer 30 sets the position, line of sight vector, and upward vector for each viewpoint 98, so that the positions of the two viewpoints 98 are far apart, the line of sight vectors of the two viewpoints 98 are parallel to each other, and the upward vectors of the two viewpoints 98 are parallel to each other. Therefore, parallax occurs between the two viewpoints 98.

[0072] Here, the computer 30 sets the line-of-sight vector and upward vector of the viewpoint 98 based on the measurement signals from the attitude measurement unit 12. Therefore, when the user 8 moves their head, the measured values ​​of the roll angle, pitch angle, and yaw angle measured by the attitude measurement unit 12 change, and thus the direction of the line-of-sight vector and upward vector of the viewpoint 98 changes.

[0073] Computer 30 uses a second view transformation matrix to transform the cockpit model 91, the display device model 92, and the control lever model 93 from the second world coordinate system to the second view coordinate system. The position of viewpoint 98 is the origin of the second view coordinate system, and the planes parallel to the X4 axis and Y4 axis of the three orthogonal axes that define the view coordinate system are orthogonal to the line of sight vector of viewpoint 98, the Y4 axis is parallel to the upward vector of viewpoint 98, and the Z4 axis is parallel to the line of sight vector of viewpoint 98.

[0074] Computer 30 converts the coordinate systems of the cockpit model 91, display device model 92, and control lever model 93 from a three-dimensional second view coordinate system to a two-dimensional second Cartesian coordinate system so that the cockpit model 91, display device model 92, and control lever model 93 are projected in perspective onto a second plane perpendicular to the line of sight vector of the viewpoint 98. During perspective projection, computer 30 performs hidden line processing.

[0075] Computer 30 rasterizes the cockpit model 91, display model 92, and control lever model 93 by rendering them in a second Cartesian coordinate system. This generates an image obtained by rasterizing the cockpit model 91, display model 92, and control lever model 93. The image depicts the cockpit model 91, display model 92, and control lever model 93.

[0076] (5) Mapping The computer 30 maps the 2D texture, on which the construction target model 82 and the construction machine model 81 are drawn, from a first Cartesian coordinate system to a second Cartesian coordinate system of the video, for each of the multiple display device models 92 corresponding to each of the multiple viewpoints 88. Specifically, the computer 30 maps the 2D texture to the display device model 92 in the video by performing an affine transformation or a projection transformation on the 2D texture. This composites the 2D texture into the video.

[0077] If the number of viewpoints (98) and the number of display devices (13) are 2, a 2D texture will be superimposed on both of the two images.

[0078] The computer 30 may also synthesize guidance for operating the construction machine model 81 into the video. Guidance refers to a representation of the completed state of the construction object using drawings, symbols, or numerical values, or drawings, symbols, letters, or numerical values ​​for guiding or advising on the destination of the construction machine model 81.

[0079] (6) Display The computer 30 converts the image with the composite 2D textures into a video signal and outputs the video signal to the display device 13 of the display device 10. If the number of viewpoints 98 and the number of display devices 13 are 2, the computer 30 generates a video signal for each image with the composite 2D textures and outputs the video signal for each image to the display device 13.

[0080] [3. Advantageous technical effects] (1) Even without actual simulation equipment, remote control devices, and construction machinery, user 8 can virtually experience operating a construction machinery simulation device, operating a construction machinery remote control device, and operating construction machinery, especially remotely. In other words, simulator 1 and program 32 enhance the voluntary implementation of remote control practice and improve the efficiency of training remote control operators.

[0081] (2) Since it does not require real-world simulation equipment, remote control devices, and construction machinery, a low-cost simulator 1 can be provided. In other words, simulator 1 and program 32 have the effect of enabling economical remote control practice.

[0082] (3) By operating the input device 20, the user 8 can input changes to the position of the viewpoint 88, the line of sight vector, or the upward vector to the computer 30, thereby achieving the same effect as changing the position and orientation of a real camera that photographs real construction machinery. Thus, it is possible to virtually experience how real construction machinery looks when photographed with a real camera at its actual position and orientation. This experience can be utilized in the design and development of remote control devices for real construction machinery.

[0083] (4) Simulator 1 can be used for training and rehearsal of operation of construction machinery. Simulator 1 can also be used to educate inexperienced personnel who are not accustomed to operating construction machinery. In other words, Simulator 1 and Program 32 have the effect of enabling the economical training of remote operators.

[0084] (5) The images showing the cockpit model 91, the display device model 92, and the control lever model 93 are two-dimensional. The textures projected onto these images are also two-dimensional. Therefore, the processing load on the computer 30 is light.

[0085] (6) When the number of viewpoints 98 and the number of display devices 13 are 2, parallax occurs in viewpoint 98, so that the cockpit model 91, display device model 92 and control lever model 93 displayed on the display device 10 appear three-dimensional. Thus, the user 8 can experience the operation of a construction machine simulation device, the operation of a construction machine remote control device, and the remote operation of a construction machine in a virtual reality with a sense of immersion.

[0086] [4. Variant] However, the present invention is not limited to the embodiments described above. The above embodiments may be modified, for example, as follows.

[0087] (1) The computer 30 may generate an image obtained by viewing the construction target model 82 and the construction machine model 81 in the first virtual space 80 from the position of the viewpoint 88, convert the image into a video signal, and output the video signal to the display device 13 of the display device 10.

[0088] (2) Computer 30 may download programs 32 and data 33-37 from another computer and record programs 32 and data 33-37 in storage device 31.

[0089] (3) Program 32 and data 33-37 may be stored in the storage device of the cloud computer, and the cloud computer may execute program 32. In this case, computer 30 may connect to the cloud computer using remote desktop technology, and the cloud computer may transfer video to computer 30 by executing program 32, and computer 30 may display the video on the display device 13.

[0090] (4) In the above embodiment, after rasterizing the construction target model 82, construction machine model 81, cockpit model 91, display device model 92 and control lever model 93, the computer 30 maps the 2D texture onto the image. Alternatively, the computer 30 may rasterize the construction target model 82, construction machine model 81, cockpit model 91, display device model 92 and control lever model 93 in the image after mapping the 2D texture onto the image. [Explanation of symbols]

[0091] 10 Display device 12. Posture Measurement Unit 13 Display Devices 20 Input Devices 30 Computers 32 Programs 80. First Virtual Space 81 Construction Machinery Models 82 Models to be constructed 88 First Perspective 90 Second Virtual Space 91 Cockpit Model 92 Display Models 93 Control Lever Model 98 Second Perspective

Claims

1. A program that causes a computer to perform virtual reality processing, which displays on a display device a three-dimensional display device model in a second virtual space in which a three-dimensional construction machine model in a first virtual space, which operates by an output signal of an input device, is drawn. The aforementioned virtual reality processing, A first setup process for setting up the construction machine model in the first virtual space, A first generation process that generates a two-dimensional texture obtained by viewing the construction machine model from the position of a first viewpoint in the first virtual space, A second setting process for setting the display device model in the second virtual space, A second generation process that generates an image obtained by viewing the display device model from the position of a second viewpoint in the second virtual space, A mapping process that maps the two-dimensional texture generated by the first generation process onto the display device model in the image generated by the second generation process, A display process that displays the image onto the display device, which is obtained by the mapping process, A program that includes this.

2. The first setting process is, A process that operates the construction machine model in the first virtual space based on commands or information entered by the user through the input device. The program according to claim 1, including the program described in claim 1.

3. The first setting process is, A process to change the position of the first viewpoint or the direction of its line of sight based on a command or information entered by the user through the input device. A program according to claim 1 or 2, including the program described in claim 1 or 2.

4. The second setting process described above is A process to set the position and direction of the line of sight of the second viewpoint based on the tilt angle measured by the attitude measurement unit integrated with the display device. A program according to claim 1 or 2, including the program described in claim 1 or 2.

5. The second setting process described above is Process of setting the cockpit model together with the display device model in the second virtual space. Includes, The second generation process is, Process to generate the image obtained by viewing the cockpit model from the position of the second viewpoint together with the display device model. A program according to claim 1 or 2, including the program described in claim 1 or 2.

6. The second setting process described above is The process of setting the control lever model together with the display device model in the second virtual space, A process of tilting the control lever model in the second virtual space based on a command or information entered by the user through the input device, Includes, The second generation process is, Process to generate the image obtained by viewing the control lever model together with the display device model from the position of the second viewpoint. A program according to claim 1 or 2, including the program described in claim 1 or 2.

7. The first setting process is, Process of setting the construction target model together with the construction machine model in the first virtual space. Includes, The first generation process is, Process to generate the two-dimensional texture obtained by viewing the construction target model together with the construction machine model from the position of the first viewpoint. A program according to claim 1 or 2, including the program described in claim 1 or 2.

8. A simulator comprising a computer that performs virtual reality processing, which displays on a display device a three-dimensional display device model in a second virtual space in which a three-dimensional construction machine model in a first virtual space, which operates by an output signal of an input device, is drawn, The aforementioned virtual reality processing, A first setup process for setting up the construction machine model in the first virtual space, A first generation process that generates a two-dimensional texture obtained by viewing the construction machine model from the position of a first viewpoint in the first virtual space, A second setting process for setting the display device model in the second virtual space, A second generation process that generates an image obtained by viewing the display device model from the position of a second viewpoint in the second virtual space, A mapping process that maps the two-dimensional texture generated by the first generation process onto the display device model in the image generated by the second generation process, A display process that displays the image onto the display device, which is obtained by the mapping process, A simulator that includes this.

Images