Simulation device, control method, and control program

Abstract

To improve convenience.SOLUTION: A simulation device comprises: a simulation operating device 3 that has a plurality of simulation operating parts 31; a photographing device 41 that acquires a real image according to a user's filed of vision; a display device 4 that can display the real image acquired by the photographing device 41 and an image of a first virtual model corresponding to a drilling machine; and a control device 6 that causes the display device 4 to display the real image and the image of the first virtual model. The control device 6 includes a display control unit 637 that executes any display processing of first display processing of causing the display device 4 to display the image of the first virtual model while preventing the display of the real image according to the direction of the line of sight of the user, and second display processing of causing the display device 4 to display both the real image and the image of the first virtual model.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a simulation device, a control method, and a control program.

Background Art

[0002] Conventionally, in quarries, limestone mines, etc., drilling machines have been used to drill blast holes in rock formations (see, for example, Patent Document 1). In such a drilling machine, a tool attached to the tip of a rock drill is pressed against a crushing target by a feeding mechanism, and the impact force generated by a striking mechanism and the rotational force generated by a rotational mechanism are transmitted to the crushing target through the tool to drill a blast hole. And in such a drilling operation, positioning of the boom and the guide shell, operation of the feeding mechanism, the striking mechanism, and the rotational mechanism, operation of a centering device and a rod changer, etc. are performed. That is, in the drilling machine, the operation targets are diverse, and many operation parts for operating each operation target are provided.

[0003] And since many operation parts are provided in the drilling machine as described above, it is not easy to acquire the skill of operating the drilling machine. Therefore, it is conceivable to use a simulation device for acquiring the skill of operating the drilling machine without using the actual drilling machine. For example, the simulation device allows a trainer to visually recognize an image of a virtual model of the drilling machine (hereinafter referred to as a model image) through a display device such as a head-mounted display worn by the trainer. Also, the simulation device, in response to an operation of the trainer on any of a plurality of pseudo-operation parts that pseudo-represent a plurality of operation parts provided in the drilling machine, pseudo-acts on the display screen of the display device a part such as a boom corresponding to the pseudo-operation part among the model images. Conventionally, as such a simulation device, a simulation device for a work vehicle such as a forklift has been proposed (see, for example, Patent Document 2).

Prior Art Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2000-303768 [Patent Document 2] Japanese Patent Publication No. 2004-252024 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, if only model images are displayed on a display device such as a head-mounted display, the trainee wearing the device cannot perceive the real space. In other words, the trainee cannot see the simulated control parts in the real space. In particular, in a drilling machine simulation device, since the drilling machine has many control parts, the number of simulated control parts corresponding to those control parts is also very large. As a result, the trainee wearing the display device cannot perceive the real space and will have to operate the multiple simulated control parts in the real space by touch. Therefore, there is a need for technology that can improve convenience by eliminating the need for trainees to operate multiple simulated control units by touch.

[0006] The present invention has been made in view of the above, and aims to provide a simulation device, a control method, and a control program that can improve convenience. [Means for solving the problem]

[0007] To solve the above-mentioned problems and achieve the objective, the simulation device according to the present invention comprises: a simulation operation device provided on a drilling machine and having a plurality of simulated operation units that simulate a plurality of operation units used to operate the drilling machine; a shooting device worn by a user to acquire a real image corresponding to the user's field of view; a display device worn by the user that can display the real image and an image of a first virtual model corresponding to the drilling machine; and a control device that displays the real image and the image of the first virtual model on the display device, and simulates the operation of the first virtual model in response to the operation of the simulated operation unit operated by the user, wherein the control device executes either a first display process that displays the image of the first virtual model on the display device while obstructing the display of the real image on the display device, or a second display process that displays both the real image and the image of the first virtual model on the display device, depending on the user's line of sight.

[0008] Furthermore, in the simulation apparatus according to the present invention, the display control unit projects the real image onto a second virtual model, which is placed in a virtual space on which the first virtual model is located and functions as a screen onto which an image is projected, thereby displaying the real image on the display device. In the first display process, the transparency of the second virtual model is set to a first transparency to prevent the display of the real image on the display device. In the second display process, the transparency of the second virtual model is set to a second transparency, which is smaller than the first transparency, thereby allowing the display device to display the real image.

[0009] Furthermore, in the simulation apparatus according to the present invention, the control device further comprises a reference line creation unit that creates a virtual reference line extending linearly from a specific position of the display device according to the user's line of sight, a calculation unit that calculates the shortest distance between a preset switching reference position and the reference line, and a determination unit that determines whether the shortest distance is less than or equal to a specific threshold. The display control unit executes the first display process if the determination unit determines that the shortest distance exceeds the specific threshold, and executes the second display process if the determination unit determines that the shortest distance is less than or equal to the specific threshold.

[0010] Furthermore, in the simulation device according to the present invention, the switching reference position is set to a position corresponding to the placement position of the simulated operating device.

[0011] Furthermore, in the simulation device according to the present invention, multiple reference positions for switching are set.

[0012] Furthermore, the control method according to the present invention is a control method executed by a control device of a drilling machine simulation device, and the control device executes either a first display processing step, which is performed according to the user's line of sight, a first display processing step, which is performed to cause an image of a first virtual model corresponding to the drilling machine to be displayed on the display device while preventing the display of a real image on the display device, or a second display processing step, which is performed to cause both the real image and the image of the first virtual model to be displayed on the display device.

[0013] Furthermore, the control program according to the present invention is a control program that causes a computer to execute either a first display processing step, which is performed to display an image of a first virtual model corresponding to a drilling machine on a display device while preventing the display of a real image on the display device, or a second display processing step, which is performed to display both the real image and the image of the first virtual model on the display device, according to the user's line of sight. [Effects of the Invention]

[0014] According to the simulation device, control method, and control program according to the present invention, convenience can be improved.

Brief Description of the Drawings

[0015] [Figure 1] FIG. 1 is a diagram showing a punching machine that is a target for acquiring operation skills by the simulation device according to the embodiment. [Figure 2] FIG. 2 is a diagram showing the configuration of the simulation device according to the embodiment. [Figure 3] FIG. 3 is a block diagram showing the configuration of the control device. [Figure 4] FIG. 4 is a flowchart showing the control method. [Figure 5] FIG. 5 is a diagram for explaining steps S1 to S3. [Figure 6] FIG. 6 is a diagram showing a part of the virtual space. [Figure 7] FIG. 7 is a diagram for explaining step S4. [Figure 8] FIG. 8 is a diagram for explaining step S5.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments for carrying out the present invention (hereinafter referred to as embodiments) will be described with reference to the drawings. Note that the present invention is not limited by the embodiments described below. Further, in the description of the drawings, the same parts are denoted by the same reference numerals.

[0017] 〔Schematic Configuration of Punching Machine〕 FIG. 1 is a diagram showing a punching machine 100 that is a target for acquiring operation skills by a simulation device 1 according to the embodiment. First, before explaining the configuration of the simulation device 1 according to the present embodiment, the configuration of the punching machine 100 that is a target for acquiring operation skills by the simulation device 1 will be explained. In describing the drilling machine 100, the term "front side" below refers to the right side as shown in Figure 1.

[0018] The drilling machine 100 drills blast holes in the rock mass according to the operator's instructions. As shown in Figure 1, the drilling machine 100 comprises a traveling carriage 110 having a pair of left and right tracks 111 (only one track 111 is shown in Figure 1), and a boom 120, a rock drilling unit 130, and a control room 140 mounted on the traveling carriage 110, respectively.

[0019] As shown in Figure 1, the boom 120 comprises a boom body 121, a boom pedestal 122, a boom slewing cylinder (not shown), and a boom lift cylinder 123. The boom pedestal 122 is mounted on a base 112 (Figure 1) located on the right front side of the traveling carriage 110, so as to be able to rotate (rotate) around a pivot axis 124 (Figure 1) that is aligned vertically. When the boom slewing cylinder (not shown) is driven, the boom 120 rotates. The boom body 121 is mounted on the boom pedestal 122 so as to be able to rotate (raise and lower) around a luffing axis 125 (Figure 1) that is aligned horizontally (forward and backward). When the boom lift cylinder 123 is driven, the boom 120 raises and lowers.

[0020] As shown in Figure 1, the rock-drilling unit 130 is attached to the tip of the boom 120. This rock-drilling unit 130 comprises a guide shell 131 attached to the tip of the boom 120, a rock-drilling machine 132 that is provided to move forward and backward along the longitudinal direction of the guide shell 131, and a tool 133 attached to the tip of the rock-drilling machine 132. Here, the rock drill 132 moves forward and backward on the guide shell 131 by a feed mechanism (not shown) provided on the guide shell 131, and is equipped with a known striking mechanism (not shown) and a rotation mechanism (not shown). Although not specifically shown, the tool 133 consists of a shank rod, sleeve, rod, and bit connected in this order from the base end. Then, in the drilling machine 100, the tool 133 is pressed against the object to be crushed by a feeding mechanism (not shown), and the impact force generated by the striking mechanism (not shown) and the rotational force generated by the rotation mechanism (not shown) are transmitted to the object to be crushed via the tool 133 to drill a blast hole.

[0021] As shown in Figure 1, the control room 140 is located on the left-front side of the traveling chassis 110. Inside the control room 140, as shown in Figure 1, there is a driver's seat 141 where an operator who operates the drilling machine 100 sits. On both sides of the driver's seat 141, although not shown in detail, there are several operating parts used to operate the drilling machine 100.

[0022] [Outline configuration of the simulation device] Next, we will describe the configuration of the simulation device 1. Figure 2 shows the configuration of the simulation device 1 according to the embodiment. As shown in Figure 2, the simulation device 1 comprises a training seat 2, a simulated operation device 3, a head-mounted display 4, a base station 5, and a control device 6.

[0023] Training seat 2 is the area where a trainee, who is a user of the simulation device 1, sits to acquire skills in operating the drilling machine 100. As shown in Figure 2, the simulated operation device 3 is provided on both sides of the training seat 2 and simulates the operation device (not shown) provided on the drilling machine 100. More specifically, the simulated operation device 3 has a plurality of simulated operation units 31, each of which simulates a plurality of operation units (not shown) provided on the said operation device. These plurality of simulated operation units 31 are each connected to the control device 6 wirelessly or by wire, and each outputs a signal to the control device 6 in response to user operation by a trainee or the like.

[0024] For example, among the multiple simulated operation units 31, the simulated operation unit 311 is located to the right of the training seat 2. This simulated operation unit 311 is a tiltable lever that simulates the operation units for the feed mechanism (not shown), the impact mechanism (not shown), and the rotation mechanism (not shown) of the drilling machine 100.

[0025] The head-mounted display 4 corresponds to the display device according to the present invention and is connected to the control device 6 wirelessly or via a wired connection for communication. The head-mounted display 4 has the appearance of eyeglasses and is worn by the trainee. The head-mounted display 4 is capable of displaying both a real image and a predetermined image corresponding to the field of view of the trainee wearing the head-mounted display 4. The head-mounted display 4 comprises a shooting unit 41, a display unit 42, and a plurality of infrared output units 43 (see Figure 3).

[0026] The imaging unit 41 corresponds to the imaging device according to the present invention and is a camera that includes an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) that receives incident light and converts it into an electrical signal. The imaging unit 41 also generates an image by capturing an area corresponding to the field of view of the trainee wearing the head-mounted display 4 under the control of the control device 6. The imaging unit 41 then outputs the data of the generated image to the control device 6.

[0027] The display unit 42 is composed of a display using liquid crystal or organic EL (Electro-Luminescence), and is positioned opposite at least one of the left or right eyes of the trainee wearing the head-mounted display 4. The display unit 42 then displays various images under the control of the control device 6.

[0028] Multiple infrared output units 43 are used to determine the position (3D position) and orientation (the gaze direction of the trainee wearing the head-mounted display 4) of the head-mounted display 4. These multiple infrared output units 43 are each positioned at different locations and each emits (irradiates) infrared light.

[0029] The base station 5 is used to detect the position (3D position) and orientation (the gaze direction of the trainee wearing the head-mounted display 4) of the head-mounted display 4. The base station 5 is also connected to the control device 6 wirelessly or via a wired connection. The base station 5 consists of two infrared cameras 51 (Figure 2) that detect infrared light emitted from multiple infrared output units 43. Note that the number of infrared cameras 51 is not limited to two; any other number may be provided. The two infrared cameras 51 then output the infrared image data generated by the capture to the control device 6.

[0030] Figure 3 is a block diagram showing the configuration of the control device 6. For the sake of explanation, only one of each of the simulated operation unit 31, infrared output unit 43, and infrared camera 51 is shown in Figure 3. The control device 6 controls the operation of the entire simulation device 1. As shown in Figure 3, the control device 6 comprises an input unit 61, a storage unit 62, and a control unit 63. The input unit 61 consists of buttons, switches, touch panels, etc., that accept user operations by trainers, etc., and outputs signals corresponding to such user operations to the control unit 63.

[0031] The storage unit 62 stores various programs executed by the control unit 63 (including the control program according to the present invention), as well as data necessary when the control unit 63 performs processing. Examples of data required when the control unit 63 performs processing include data for the first 3D model (first virtual model), data for the second 3D model (second virtual model), data indicating a reference position for switching, and data indicating a specific threshold. The first 3D model data is the 3D model data corresponding to the drilling machine 100. The second 3D model data is data for a 3D model that functions as a screen onto which the captured image generated by the imaging unit 41 is projected. These first and second 3D models are 3D models generated by, for example, CAD (Computer-Aided Design) software, and the position and angle of each part of the first and second 3D models are associated in a virtual spatial coordinate system that is aligned with the real spatial coordinate system on which the simulation device 1 is installed.

[0032] The data indicating the reference position for switching is data indicating the reference position (position in real space coordinate system (3D position) or position in virtual space coordinate system (3D position)) used when switching between the first and second display processes, which will be described later, executed by the control unit 63. The data indicating a specific threshold is the data used to determine whether to switch between the first and second display processes.

[0033] The control unit 63 is implemented by a controller such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs stored in the memory unit 62, and controls the operation of the entire simulation device 1. The control unit 63 is not limited to a CPU or MPU, but may also be composed of integrated circuits such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array). As shown in Figure 3, the control unit 63 comprises a first image acquisition unit 631, a second image acquisition unit 632, a first calculation unit 633, a reference line creation unit 634, a second calculation unit 635, a determination unit 636, and a display control unit 637.

[0034] The first image acquisition unit 631 acquires the data of the captured image generated by the imaging unit 41. The second image acquisition unit 632 acquires the data of the infrared image generated by each of the two infrared cameras 51.

[0035] The first calculation unit 633 calculates the position and orientation of the head-mounted display 4 (the direction of the trainee's gaze while wearing the head-mounted display 4). Specifically, the first calculation unit 633 recognizes the arrival time and angle of infrared light that reached each infrared camera 51 from multiple infrared output units 43 based on the data of each infrared image acquired by the second image acquisition unit 632. Then, the first calculation unit 633 calculates the position and orientation of the head-mounted display 4 (the direction of the trainee's gaze while wearing the head-mounted display 4) from the arrival time and angle.

[0036] The reference line creation unit 634 creates a virtual reference line that extends in a straight line from a specific position on the head-mounted display 4 according to the trainee's line of sight. The second calculation unit 635 corresponds to the calculation unit according to the present invention and calculates the shortest distance between the switching reference position and the reference line created by the reference line creation unit 634 based on data indicating the switching reference position stored in the storage unit 62. The determination unit 636 refers to data indicating a specific threshold stored in the storage unit 62 and determines whether the shortest distance calculated by the second calculation unit 635 is less than or equal to the said specific threshold. Further details regarding the aforementioned reference line, switching reference position, and shortest distance will be explained in the "Control Method" section below.

[0037] The display control unit 637 generates a display image to be displayed on the display unit 42. Specifically, the display control unit 637 recognizes the position and orientation of the head-mounted display 4 (the direction of the trainee's gaze while wearing the head-mounted display 4) calculated by the first calculation unit 633. Then, based on the data of the first 3D model stored in the memory unit 62, the display control unit 637 generates an image of the first 3D model (hereinafter referred to as the model image) that is recognized from the position of the head-mounted display 4 in the direction of the trainee's gaze while wearing the head-mounted display 4. Furthermore, the display control unit 637 simulates the operation of the part of the first 3D model corresponding to the simulated operation unit 31 operated by the trainee wearing the head-mounted display 4 in response to the operation of the simulated operation unit 31. Furthermore, the display control unit 637 executes either a first display process that displays a model image while obstructing the display of the captured image (real image) generated by the shooting unit 41 on the display unit 42, based on the determination result of the determination unit 636 (depending on the trainee's gaze direction), or a second display process that displays the captured image on the display unit 42 using the data of a second 3D model together with the model image. Details of the first and second display processes executed by the display control unit 637 will be explained later in the "Control Method" section.

[0038] [Control method] Next, the control method executed by the control device 6 will be described. Figure 4 is a flowchart of the control method. Figure 5 is a diagram illustrating steps S1 to S3. In Figure 5, the space VS, indicated by the dashed line, is a virtual space where the first and second 3D models are placed. The symbol "TR" represents a trainee wearing the head-mounted display 4 and seated in the training seat 2. For the sake of explanation, only the head of trainee TR is shown in Figure 5. The line indicated by the dashed line is the reference line RL created by the reference line creation unit 634. The symbol "SC" represents the screen, which is the second 3D model. The symbol "RP" represents the switching reference position. The symbol "DS" represents the shortest distance between the switching reference position RP and the reference line RL. The lower side of Figure 5 is the downward side along the vertical axis. That is, Figure 5 shows the trainee TR's line of sight directed diagonally downward.

[0039] First, the reference line creation unit 634 creates a virtual reference line RL that extends linearly from a specific position on the head-mounted display 4 according to the trainee TR's line of sight (step S1). Specifically, the reference line creation unit 634 identifies a position corresponding to the midpoint between the left and right eyes of the trainee TR on the head-mounted display 4, for example, based on the position of the head-mounted display 4 calculated by the first calculation unit 633. Then, the reference line creation unit 634 creates a virtual line called the reference line RL that extends linearly from the identified position along the line of sight of the trainee TR, which is the posture of the head-mounted display 4 calculated by the first calculation unit 633. The reference line RL is expressed by a mathematical formula that represents a straight line in a real-space coordinate system or a virtual-space coordinate system.

[0040] After step S1, the second calculation unit 635 calculates the shortest distance DS between the switching reference position RP and the reference line RL based on the data indicating the switching reference position RP stored in the storage unit 62 (step S2). After step S2, the determination unit 636 refers to data indicating a specific threshold stored in the storage unit 62 and determines whether the shortest distance DS calculated in step S2 is less than or equal to the specific threshold (step S3).

[0041] Then, if the display control unit 637 determines that the shortest distance DS exceeds a specific threshold (step S3: No), it executes the first display process as shown below (step S4: first display process step). After this, the control unit 63 returns to step S1. Figure 6 shows a part of the virtual space VS. In Figure 6, the symbol "120M" is the boom model (first 3D model) corresponding to boom 120. The symbol "121M" is the boom body model (first 3D model) corresponding to boom body 121. Furthermore, the symbol "122M" is the boom pedestal model (first 3D model) corresponding to boom pedestal 122. Furthermore, the symbol "123M" is the boom lift cylinder model (first 3D model) corresponding to boom lift cylinder 123. Furthermore, the symbol "124M" is the slewing axis model (first 3D model) corresponding to slewing axis 124. Furthermore, the symbol "125M" is the luffing axis model (first 3D model) corresponding to luffing axis 125. Furthermore, the symbol "112M" is the base model (first 3D model) corresponding to base 112. Furthermore, the symbol "3M" is an operating device model (first 3D model) corresponding to an operating device (not shown) provided on the drilling machine 100 and having the same configuration as the simulated operating device 3. Figure 7 is a diagram illustrating step S4. Specifically, Figure 7 is a diagram showing the display image F1 displayed on the display unit 42 when the field of view of trainee TR in the virtual space VS is the region Ar1 shown in Figure 6.

[0042] Here, multiple reference positions RP for switching are set at positions corresponding to the placement of the simulated operating device 3 (the placement of the simulated operating device 3 in the real space coordinate system, and the position of the operating device model 3M in the virtual space coordinate system). That is, as shown in Figure 5, the reference positions RP for switching are each located below the trainee TR, who is wearing the head-mounted display 4 and seated in the training seat 2.

[0043] In describing the first display process below, we will assume that the trainee TR has raised their head higher than in the state shown in Figure 5, and that the trainee TR's field of view in the virtual space VS is in a relatively upper region, which is region Ar1 shown in Figure 6. In this case, the reference line RL is located above the switching reference position RP, and the shortest distance DS calculated in step S2 is a relatively large distance. As a result, it is determined that the shortest distance DS exceeds a certain threshold (step S3: No), and the display control unit 637 executes the first display process (step S4).

[0044] Specifically, in step S4, the display control unit 637 generates model images of the boom model 120M, base model 112M, and operating device model 3M, which are recognized in region Ar1 corresponding to the trainee TR's field of view in the virtual space VS, based on the position and orientation (trainee TR's gaze direction) of the head-mounted display 4 calculated by the first calculation unit 633 and the data of the first 3D model stored in the storage unit 62. The display control unit 637 then generates a display image F1 (Figure 7) in which the generated model images are arranged. As a result, the display image F1 is displayed on the display unit 42.

[0045] In the first display process, when the display control unit 637 generates the display image F1, it places a screen SC onto which the captured image generated by the shooting unit 41 is projected in a specific area of ​​the display image F1 (the central area in the left-right direction on the lower side). The display control unit 637 also sets the transparency of the screen SC to a first transparency level (in this embodiment, transparency level: 100, which means it is completely transparent). In Figure 7, the screen SC set to the first transparency level is represented by a dashed line. Therefore, in the first display process, as shown in Figure 7, only the model images of the boom model 120M, the base model 112M, and the operating device model 3M recognized in area Ar1 are displayed in the display image F1. In other words, the captured image is not displayed in the display image F1. Furthermore, the first transparency is not limited to a transparency of 100 that makes the screen SC transparent; any other transparency is acceptable as long as it prevents the display of the captured image generated by the imaging unit 41 on the display unit 42.

[0046] On the other hand, if the display control unit 637 determines that the shortest distance DS is below a certain threshold (step S3: Yes), it executes a second display process as shown below (step S5: second display process step). After this, the control unit 63 returns to step S1. Figure 8 illustrates step S5. Specifically, Figure 8 shows the display image F2 shown on the display unit 42 when the trainee TR's field of view in the virtual space VS is the region Ar2 shown in Figure 6. In Figure 8, the symbol "CI" is the captured image generated by the capture unit 41. The symbol "HA" is the trainee TR's hand captured in the captured image CI. In Figure 8, dots are added to the captured image CI to distinguish it from the model images of the boom model 120M, the base model 112M, and the operating device model 3M.

[0047] In describing the second display process below, we will assume that the trainee TR lowers their head as shown in Figure 5, and that the trainee TR's field of view in the virtual space VS is in a relatively lower area, which is region Ar2 as shown in Figure 6. In this case, the reference line RL is close to the switching reference position RP, and the shortest distance DS calculated in step S2 is a relatively small distance. As a result, it is determined that the shortest distance DS is below a certain threshold (step S3: Yes), and the display control unit 637 executes the second display process (step S5).

[0048] Specifically, in step S5, the display control unit 637 generates model images of the boom model 120M, base model 112M, and operating device model 3M, which are recognized in the region Ar2 corresponding to the trainee TR's field of view in the virtual space VS, based on the position and orientation (trainee TR's gaze direction) of the head-mounted display 4 calculated by the first calculation unit 633 and the data of the first 3D model stored in the storage unit 62. The display control unit 637 then generates a display image F2 (Figure 8) in which the generated model images are arranged. As a result, the display image F2 is displayed on the display unit 42.

[0049] In the second display process, when the display control unit 637 generates the display image F2, it places a screen SC onto which the captured image CI generated by the shooting unit 41 is projected, in a specific area of ​​the display image F2 (the central area in the left-right direction on the lower side), similar to the first display process. The display control unit 637 also sets the transparency of the screen SC to a second level, which is less than the first level of transparency. In Figure 8, the screen SC set to the second level of transparency is represented by a solid line. Therefore, in the second display process, as shown in Figure 8, the captured image CI is displayed on the display image F2, and parts of the model images of the boom model 120M, base model 112M, and operating device model 3M recognized in area Ar2 are hidden by the screen SC (captured image CI).

[0050] According to the embodiment described above, the following effects are achieved. In the simulation apparatus 1 according to this embodiment, the control device 6 executes either a first or second display process depending on the trainee TR's line of sight direction. Here, the first display process is a display process that displays a model image of the first 3D model recognized in an area corresponding to the trainee TR's field of view in the virtual space VS, without displaying the captured image CI on the display unit 42. On the other hand, the second display process is a display process that displays both the captured image CI and the model image of the first 3D model recognized in an area corresponding to the trainee TR's field of view in the virtual space VS on the display unit 42. Therefore, once the second display process is executed, the trainee TR can visually identify the simulated operation unit 31 in real space from the captured image CI displayed on the display image F2 shown on the display unit 42. For example, the captured image CI shown in Figure 8 includes the simulated operation unit 31, and the trainee TR can find the desired simulated operation unit 31 from this captured image CI. Thus, according to the simulation device 1 of this embodiment, the trainee TR does not have to operate multiple simulated operation units 31 by touch, thus improving convenience.

[0051] In particular, the first and second display processes are switched according to the trainee's (TR) gaze direction. Therefore, by having the second display process run continuously, compared to a configuration where the captured image CI is always displayed on the display unit 42, the captured image CI can be displayed on the display unit 42 only when, for example, the trainee TR is searching for the desired simulated operation unit 31. In other words, for example, when the trainee TR has found the desired simulated operation unit 31 and is operating it, the first display process is run, allowing the trainee TR to see only the model image of the first 3D model that operates in a simulated manner in response to the operation of the simulated operation unit 31 from the display image F1. Thus, convenience can be further improved by not displaying the captured image CI, which is unnecessary when the simulated operation unit 31 is being operated, on the display unit 42.

[0052] Furthermore, in the simulation apparatus 1 according to this embodiment, the control device 6 places a second 3D model, which is a screen SC, on the virtual space VS, and projects the captured image CI onto the screen SC, thereby displaying the captured image CI on the display unit 42. Here, in the first display process, the control device 6 sets the screen SC to a first transparency, and in the second display process, it sets the screen SC to a second transparency. Therefore, a first display process that does not display the captured image CI on the display unit 42, and a second display process that displays the captured image CI on the display unit 42, can be easily performed with simple processing.

[0053] Furthermore, in the simulation device 1 according to this embodiment, the control device 6 creates a reference line RL corresponding to the trainee TR's line of sight, and calculates the shortest distance DS between a preset switching reference position RP and the reference line RL. Then, based on the determination result of whether the shortest distance DS is below a certain threshold, the control device 6 executes one of the first or second display processes. Therefore, the conditions for the trainee TR's line of sight when the second display process is executed can be precisely set. Consequently, the effect described above, "for example, the captured image CI can be displayed on the display unit 42 only when the trainee TR is searching for the desired simulated operation unit 31," can be suitably realized.

[0054] In particular, multiple reference positions RP for switching are provided, corresponding to the placement positions of the simulated operating device 3. Therefore, the above-mentioned effect, "for example, the captured image CI can be displayed on the display unit 42 only when the trainee TR is searching for the desired simulated operation unit 31," can be more reliably achieved.

[0055] (Other embodiments) While embodiments for carrying out the present invention have been described so far, the present invention should not be limited to the embodiments described above. In the embodiments described above, the condition for executing the first display process is not limited to the condition that the shortest distance DS exceeds a specific threshold. Similarly, the condition for executing the second display process is not limited to the condition that the shortest distance DS is less than or equal to a specific threshold. Other conditions may be adopted as long as either the first or second display process is executed according to the trainee TR's line of sight direction. For example, the second display process may be executed when the trainee TR's line of sight is directed downwards relative to the horizontal plane. In other words, the first display process may be executed when the trainee TR's line of sight is directed on the horizontal plane or upwards relative to the horizontal plane.

[0056] In the embodiment described above, an example was explained in which multiple switching reference positions RP are set, but it is also acceptable for only one to be set. In the above-described embodiment, some functions of the control unit 63 may be provided outside the control device 6. For example, the functions of the first calculation unit 633, which calculates the position and orientation of the head-mounted display 4, may be provided on the base station 5. In this case, the control device 6 acquires information indicating the position and orientation of the head-mounted display 4 from the base station 5. [Explanation of Symbols]

[0057] 1. Simulation device 2 Training seat 3 Simulated operating device 3M Control Device Model 4. Head-mounted display 5 Base Stations 6 Control device 31,311 Simulation operation section 41 Photography Department 42 Display section 43 Infrared output section 51 Infrared Camera 61 Input section 62 Memory section 63 Control Unit 100 Drilling Machine 110 Bogie 111 Trucks 112 base 112M Base Model 120 Boom 120M Boom Model 121 Boom body 121M Boom Unit Model 122 Boom Pedestal 122M Boom Pedestal Model 123 Boom Lift Cylinder 123M Boom Lift Cylinder Model 124 Swivel axis 124M Swivel Axis Model 125 Relief axis 125M elevation axis model 130 rock-cutting units 131 Guide Shell 132 Rock drilling machine 133 Tools 140 Cockpit 141 Driver's seat 631 First image acquisition unit 632 Second image acquisition unit 633 First Calculation Unit 634 Reference Line Creation Section 635 Second calculation unit 636 Judgment section 637 Display Control Unit Ar1,Ar2 area CI (Civil Identity) Photographed Images DS shortest distance F1, F2 display images HA hand RL reference line RP switching reference position SC Screen TR Trainer VS Virtual Space

Claims

1. A simulated operating device having multiple simulated operating parts that are provided on a drilling machine and each simulated reproduces one of the multiple operating parts used to operate the drilling machine, A shooting device worn by a user that acquires a real image corresponding to the user's field of vision, A display device attached to the user, capable of displaying the actual image and an image of a first virtual model corresponding to the drilling machine, The device displays the real image and the image of the first virtual model on the display device, and the device controls the part of the image of the first virtual model that corresponds to the simulated operation unit operated by the user, in accordance with the operation on the simulated operation unit. The control device is A reference line creation unit creates a virtual reference line that extends linearly from a specific position on the display device according to the user's line of sight, A calculation unit that calculates the shortest distance between a pre-set switching reference position and the reference line, A determination unit that determines whether the shortest distance is less than or equal to a specific threshold, The display control unit performs either a first display process, which prevents the display of the actual image on the display device while displaying an image of the first virtual model on the display device, or a second display process, which displays both the actual image and an image of the first virtual model on the display device, depending on the user's line of sight. The display control unit, A simulation device that executes the first display process when the determination unit determines that the shortest distance exceeds the specific threshold, and executes the second display process when the determination unit determines that the shortest distance is less than or equal to the specific threshold.

2. The display control unit, The real image is projected onto a second virtual model, which is placed in the virtual space on which the first virtual model is located and functions as a screen onto which an image is projected, thereby displaying the real image on the display device. In the first display process described above, By setting the transparency of the second virtual model to the first transparency, the display of the actual image on the display device is prevented. In the second display process described above, The simulation apparatus according to claim 1, wherein the transparency of the second virtual model is set to a second transparency which is smaller than the first transparency, thereby displaying the actual image on the display device.

3. The aforementioned reference position for switching is, The simulation device according to claim 1, which is set at a position corresponding to the placement position of the aforementioned simulated operation device.

4. The aforementioned reference position for switching is, A simulation apparatus according to claim 1, wherein multiple settings are configured.

5. A control method performed by the control device of a drilling machine simulation device, A reference line creation step involves creating a virtual reference line that extends linearly from a specific position on the display device according to the user's line of sight, and A calculation step to calculate the shortest distance between a pre-set switching reference position and the reference line, A determination step of determining whether the shortest distance is less than or equal to a specific threshold, If the determination step determines that the shortest distance exceeds a specific threshold, a first display processing step is performed to cause an image of a first virtual model corresponding to the drilling machine to be displayed on the display device while preventing the display of a real image on the display device; A control method comprising: a second display processing step, which, if the determination step determines that the shortest distance is less than or equal to a specific threshold, performs a second display processing step to display both the real image and the image of the first virtual model on the display device.

6. A reference line creation step of creating a virtual reference line that extends linearly from a specific position on the display device according to the user's line of sight, A calculation step to calculate the shortest distance between a pre-set switching reference position and the reference line, A determination step of determining whether the shortest distance is less than or equal to a specific threshold, If the determination step determines that the shortest distance exceeds the specific threshold, a first display processing step is performed to display an image of a first virtual model corresponding to the drilling machine on the display device while preventing the display of the actual image on the display device. A control program that causes a computer to execute a second display processing step, which, if the determination step determines that the shortest distance is less than or equal to a specific threshold, executes a second display processing step that causes the display device to display both the real image and the image of the first virtual model.

Images