Simulation device, control method, and control program

Abstract

To allow a trainee to easily learn the skill of operating a drilling machine.SOLUTION: A simulation device comprises: a simulation operating device 3 that has a plurality of simulation operating parts 31; a display device 4 that displays an image of a virtual model corresponding to a drilling machine; and a control device 6 that causes the display device 4 to display the image of the virtual model. The control device 6 includes a determination unit 634 that determines, of the plurality of simulation operating parts 31, a simulation operating part that a user is to operate as a target operating part, and a display control unit 635 that causes the display device 4 to display a portion of the virtual model corresponding to the target operating part in such a manner that the portion can be distinguished from the other portions.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a simulation device, a control method, and a control program.

Background Art

[0002] Conventionally, in a quarry, a limestone mine, etc., a drilling machine has been used to drill blasting holes in a rock mass (for example, see Patent Document 1). In such a drilling machine, a tool attached to the tip of a rock cutter is pressed against a crushing target by a feed mechanism, and a blasting hole is drilled by transmitting the impact force generated by a striking mechanism and the rotational force generated by a rotating mechanism to the crushing target through the tool. In such a drilling operation, positioning of a boom and a guide shell, operations of a feed mechanism, a striking mechanism, and a rotating mechanism, operations of a centering device and a rod changer, etc. are performed. That is, in the drilling machine, there are a wide variety of operation targets, and many operation parts for operating each operation target are provided.

[0003] And, since many operation parts are provided in the drilling machine as described above, it is not easy to acquire the skill of operating the drilling machine. Therefore, it is conceivable to use a simulation device for acquiring the skill of operating the drilling machine without using the actual drilling machine. For example, the simulation device causes a trainer to visually recognize an image of a virtual model of the drilling machine (hereinafter referred to as a model image) through a display device such as a head-mounted display worn by the trainer. Further, the simulation device causes a part such as a boom corresponding to the pseudo-operation part to operate virtually on a display screen of the display device in accordance with an operation of the trainer on any one of a plurality of pseudo-operation parts that pseudo-reproduce a plurality of operation parts provided in the drilling machine. Conventionally, as such a simulation device, a simulation device for a work vehicle such as a forklift has been proposed (for example, see Patent Document 2).

Prior Art Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2000-303768 [Patent Document 2] Japanese Patent Publication No. 2004-252024 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, even in drilling machine simulation devices, the number of simulated operating parts is very large, just like the operating parts on the actual drilling machine. Therefore, it is not easy for trainees to grasp the relationship between these simulated operating parts and the corresponding operating parts on the drilling machine. In other words, it is difficult for trainees to acquire the skills to operate drilling machines.

[0006] The present invention has been made in view of the above, and aims to provide a simulation device, a control method, and a control program that enable trainees to easily acquire skills in operating a drilling machine. [Means for solving the problem]

[0007] To solve the above-mentioned problems and achieve the objective, the simulation device according to the present invention comprises: a simulation operation device provided in a drilling machine and having a plurality of simulated operation units that simulate a plurality of operation units used to operate the drilling machine; a display device that displays an image of a virtual model corresponding to the drilling machine; and a control device that displays the image of the virtual model on the display device and simulates the operation of the part of the virtual model corresponding to the simulated operation unit operated by the user in response to the operation of the simulated operation unit, wherein the control device comprises a determination unit that determines which of the plurality of simulated operation units the user intends to operate as the target operation unit, and a display control unit that displays the part of the virtual model corresponding to the target operation unit on the display device in a way that allows it to be identified from other parts.

[0008] Furthermore, in the simulation apparatus according to the present invention, each of the plurality of simulated operation units outputs an operation signal to the control device in accordance with the operation by the user, and the determination unit determines that the simulated operation unit that is outputting the operation signal is the target operation unit among the plurality of simulated operation units.

[0009] Furthermore, in the simulation apparatus according to the present invention, when the display control unit determines that a simulated operation unit that accepts multiple operations from among the multiple simulated operation units is the target operation unit, it displays the part of the virtual model that is operated in response to the operation on the target operation unit on the display device in a manner that allows it to be identified from other parts.

[0010] Furthermore, in the simulation device according to the present invention, the determination unit determines that, among the plurality of simulated operation units, the simulated operation unit whose positional relationship with the position of the user's hand is a specific positional relationship is the target operation unit.

[0011] Furthermore, in the simulation device according to the present invention, the determination unit measures the position coordinates of the user's hand, analyzes the positional relationship between the position coordinates of the user's hand and the position coordinates of the preset plurality of simulated operation units, and determines that the simulated operation unit that has the specific positional relationship is the target operation unit based on the results of the analysis.

[0012] Furthermore, the simulation apparatus according to the present invention further comprises a shooting device that photographs an area including the plurality of simulated operation units and outputs captured image data, the control device further comprises an image acquisition unit that acquires the captured image data, and the determination unit measures the position coordinates of the user's hand position based on the captured image, analyzes the positional relationship between the position coordinates of the user's hand position and the pre-set position coordinates of the plurality of simulated operation units, and determines that the simulated operation unit that has the specific positional relationship is the target operation unit based on the results of the analysis.

[0013] Furthermore, in the simulation device according to the present invention, the determination unit recognizes the position of the user's hand by detecting the skeletal structure of the user's hand based on the captured image.

[0014] Furthermore, in the simulation device according to the present invention, the determination unit determines that the simulated operation unit among the plurality of simulated operation units is the target operation unit if the distance between it and the position coordinates of the user's hand is below a certain threshold and the position coordinates of the specific positional relationship are the shortest.

[0015] Furthermore, in the simulation apparatus according to the present invention, the plurality of simulated operation units each output an operation signal to the control device in response to the user's operation, and the display control unit each executes a first identification display process and a second identification display process, the first identification display process being a process that displays on the display device the part of the virtual model corresponding to the target operation unit in a manner that allows it to be identified from other parts, and the second identification display process being executed after the first identification display process when there is an operation on the target operation unit, and is a process that displays on the display device an identification state that identifies the part of the virtual model corresponding to the target operation unit from other parts in a manner different from when the first identification display process is executed.

[0016] Furthermore, the control method according to the present invention is a control method executed by a control device of a simulation device, and includes a determination step of determining which of a plurality of simulated operation units, each of which simulated operation units used to operate a drilling machine, is a target operation unit that the user intends to operate, and a display processing step of displaying an image of a virtual model corresponding to the drilling machine on a display device, and displaying on the display device the part of the virtual model corresponding to the target operation unit in a manner that can be identified from other parts.

[0017] Further, the control program according to the present invention is a control program for causing a computer to execute a determination step of determining, as a target operation unit, a simulated operation unit that virtually reproduces each of a plurality of operation units used to operate a drilling machine, and a display processing step of causing a display device to display an image of a virtual model corresponding to the drilling machine and causing the display device to display a part of the virtual model corresponding to the target operation unit in a distinguishable manner from other parts.

Effect of the Invention

[0018] According to the simulation device, control method, and control program of the present invention, the operation skills of a drilling machine can be easily acquired by a trainee.

Brief Description of the Drawings

[0019] [Figure 1] FIG. 1 is a diagram showing a drilling machine that is a target for acquiring operation skills by the simulation device according to Embodiment 1. [Figure 2] FIG. 2 is a diagram showing the configuration of the simulation device according to Embodiment 1. <>< [Figure 3] FIG. 3 is a block diagram showing the configuration of the control device. [Figure 4] FIG. 4 is a flowchart showing the control method. [Figure 5] FIG. 5 is a diagram showing a display image generated before the identification display process (step S1C) is executed. [Figure 6] FIG. 6 is a diagram showing a display image generated after the identification display process (step S1C) is executed. [Figure 7] FIG. 7 is a block diagram showing the configuration of the simulation device according to Embodiment 2. [Figure 8] FIG. 8 is a diagram showing the arrangement position of the imaging device. [Figure 9] FIG. 9 is a diagram for explaining the related information stored in the related information DB. [Figure 10]Figure 10 is a flowchart showing the control method. [Figure 11] Figure 11 is a diagram illustrating the determination process (step S2C) and the first identification display process (step S2E). [Figure 12] Figure 12 illustrates the second identification display process (step S2G). [Modes for carrying out the invention]

[0020] The embodiments for carrying out the present invention (hereinafter referred to as "embodiments") will be described below with reference to the drawings. However, the present invention is not limited to the embodiments described below. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0021] (Embodiment 1) [Outline configuration of a drilling machine] Figure 1 shows a drilling machine 100 that is the target of skill acquisition for operation using the simulation device 1 according to Embodiment 1. First, before describing the configuration of the simulation device 1 according to this embodiment 1, we will describe the configuration of the drilling machine 100, which is the target of skill acquisition for operation using the simulation device 1. In describing the drilling machine 100, the term "front side" below refers to the right side as shown in Figure 1.

[0022] The drilling machine 100 drills blast holes in the rock mass according to the operator's instructions. As shown in Figure 1, the drilling machine 100 comprises a traveling carriage 110 having a pair of left and right tracks 111 (only one track 111 is shown in Figure 1), and a boom 120, a rock drilling unit 130, and a control room 140 mounted on the traveling carriage 110, respectively.

[0023] As shown in Figure 1, the boom 120 comprises a boom body 121, a boom pedestal 122, a boom slewing cylinder (not shown), and a boom lift cylinder 123. The boom pedestal 122 is mounted on a base 112 (Figure 1) located on the right front side of the traveling carriage 110, so as to be able to rotate (rotate) around a pivot axis 124 (Figure 1) that is aligned vertically. When the boom slewing cylinder (not shown) is driven, the boom 120 rotates. The boom body 121 is mounted on the boom pedestal 122 so as to be able to rotate (raise and lower) around a luffing axis 125 (Figure 1) that is aligned horizontally (forward and backward). When the boom lift cylinder 123 is driven, the boom 120 raises and lowers.

[0024] As shown in Figure 1, the rock-drilling unit 130 is attached to the tip of the boom 120. This rock-drilling unit 130 comprises a guide shell 131 attached to the tip of the boom 120, a rock-drilling machine 132 that is provided to move forward and backward along the longitudinal direction of the guide shell 131, and a tool 133 attached to the tip of the rock-drilling machine 132. Here, the rock drill 132 moves forward and backward on the guide shell 131 by a feed mechanism (not shown) provided on the guide shell 131, and is equipped with a known striking mechanism (not shown) and a rotation mechanism (not shown). Although not specifically shown, the tool 133 consists of a shank rod, sleeve, rod, and bit connected in this order from the base end. Then, in the drilling machine 100, the tool 133 is pressed against the object to be crushed by a feeding mechanism (not shown), and the impact force generated by the striking mechanism (not shown) and the rotational force generated by the rotation mechanism (not shown) are transmitted to the object to be crushed via the tool 133 to drill a blast hole.

[0025] As shown in Figure 1, the control room 140 is located on the left-front side of the traveling chassis 110. Inside the control room 140, as shown in Figure 1, there is a driver's seat 141 where an operator who operates the drilling machine 100 sits. On both sides of the driver's seat 141, although not shown in detail, there are several operating parts used to operate the drilling machine 100.

[0026] [Outline configuration of the simulation device] Next, we will describe the configuration of the simulation device 1. Figure 2 shows the configuration of the simulation device 1 according to Embodiment 1. As shown in Figure 2, the simulation device 1 comprises a training seat 2, a simulated operation device 3, a head-mounted display 4, a base station 5, and a control device 6.

[0027] Training seat 2 is the area where a trainee, who is a user of the simulation device 1, sits to acquire skills in operating the drilling machine 100. As shown in Figure 2, the simulated operation device 3 is provided on both sides of the training seat 2 and simulates the operation device (not shown) provided on the drilling machine 100. More specifically, the simulated operation device 3 has a plurality of simulated operation units 31, each of which simulates a plurality of operation units (not shown) provided on the said operation device. These plurality of simulated operation units 31 are each connected to the control device 6 wirelessly or by wire, and each outputs an operation signal to the control device 6 in response to user operation by a trainee or the like.

[0028] For example, among the multiple simulated operation units 31, the first simulated operation unit 31a is a simulated operation unit that simulates a drilling operation unit, which is a multi-functional operation unit that accepts multiple operations for operating the feed mechanism (not shown), the impact mechanism (not shown), and the rotation mechanism (not shown) of the drilling machine 100, respectively. Furthermore, for example, among the multiple simulated operation units 31, the second simulated operation unit 31b (see Figures 5 and 6) is a simulated operation unit that simulates a boom operation unit, which is a multi-functional operation unit that accepts rotation operations for executing the rotational movement of the boom 120 and luffing operations for executing the luffing movement of the boom 120.

[0029] The head-mounted display 4 corresponds to the display device according to the present invention and is connected to the control device 6 wirelessly or via a wired connection for communication. The head-mounted display 4 has the appearance of eyeglasses and is worn by the trainee. The head-mounted display 4 is capable of displaying both a real image and a predetermined image corresponding to the field of view of the trainee wearing the head-mounted display 4. The head-mounted display 4 comprises a shooting unit 41, a display unit 42, and a plurality of infrared output units 43 (see Figure 3).

[0030] The imaging unit 41 is a camera that includes an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) that receives incident light and converts it into an electrical signal. The imaging unit 41 also captures an area corresponding to the field of view of the trainee wearing the head-mounted display 4 under the control of the control device 6 and generates an image (hereinafter referred to as the first image). The imaging unit 41 then outputs the data of the generated first image to the control device 6.

[0031] The display unit 42 is composed of a display using liquid crystal or organic EL (Electro-Luminescence), and is positioned opposite at least one of the left or right eyes of the trainee wearing the head-mounted display 4. The display unit 42 then displays various images under the control of the control device 6.

[0032] Multiple infrared output units 43 are used to determine the position (3D position) and orientation (the gaze direction of the trainee wearing the head-mounted display 4) of the head-mounted display 4. These multiple infrared output units 43 are each positioned at different locations and each emits (irradiates) infrared light.

[0033] The base station 5 is used to detect the position (3D position) and orientation (the gaze direction of the trainee wearing the head-mounted display 4) of the head-mounted display 4. The base station 5 is also connected to the control device 6 wirelessly or via a wired connection. The base station 5 consists of two infrared cameras 51 (Figure 2) that detect infrared light emitted from multiple infrared output units 43. Note that the number of infrared cameras 51 is not limited to two; any other number may be provided. The two infrared cameras 51 then output the infrared image data generated by the capture to the control device 6.

[0034] Figure 3 is a block diagram showing the configuration of the control device 6. For the sake of explanation, only one of each of the simulated operation unit 31, infrared output unit 43, and infrared camera 51 is shown in Figure 3. The control device 6 controls the operation of the entire simulation device 1. As shown in Figure 3, the control device 6 comprises an input unit 61, a storage unit 62, and a control unit 63. The input unit 61 consists of buttons, switches, touch panels, etc., that accept user operations by trainers, etc., and outputs signals corresponding to such user operations to the control unit 63.

[0035] The storage unit 62 stores various programs executed by the control unit 63 (including the control program according to the present invention), as well as data necessary when the control unit 63 performs processing. Here, examples of data required when the control unit 63 performs processing include data for the first 3D model (virtual model) and data for the second 3D model. The first 3D model data is the 3D model data corresponding to the drilling machine 100. The data for the second 3D model is the data for a 3D model that functions as a screen onto which the first captured image, generated by the imaging unit 41, is projected. These first and second 3D models are 3D models generated by, for example, CAD (Computer-Aided Design) software, and the position and angle of each part of the first and second 3D models are associated in a virtual spatial coordinate system that is aligned with the real spatial coordinate system on which the simulation device 1 is installed.

[0036] The control unit 63 is implemented by a controller such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs stored in the memory unit 62, and controls the operation of the entire simulation device 1. The control unit 63 is not limited to a CPU or MPU; it may also be composed of integrated circuits such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array). As shown in Figure 3, the control unit 63 comprises a first image acquisition unit 631, a second image acquisition unit 632, a calculation unit 633, a determination unit 634, and a display control unit 635.

[0037] The first image acquisition unit 631 acquires the data of the first captured image generated by the imaging unit 41. The second image acquisition unit 632 acquires infrared image data generated by each of the two infrared cameras 51.

[0038] The calculation unit 633 calculates the position and orientation of the head-mounted display 4 (the direction of the trainee's gaze while wearing the head-mounted display 4). Specifically, the calculation unit 633 recognizes the arrival time and angle of infrared light that reached each infrared camera 51 from multiple infrared output units 43 based on the data of each infrared image acquired by the second image acquisition unit 632. Then, the calculation unit 633 calculates the position and orientation of the head-mounted display 4 (the direction of the trainee's gaze while wearing the head-mounted display 4) from the arrival time and angle.

[0039] The determination unit 634 performs a determination process to determine which of the multiple simulated operation units 31 the trainee intends to operate is the target operation unit. Further details regarding the judgment process will be explained in the "Control Method" section below.

[0040] The display control unit 635 generates a display image to be displayed on the display unit 42. Specifically, the display control unit 635 recognizes the position and orientation of the head-mounted display 4 (the direction of the trainee's gaze while wearing the head-mounted display 4) calculated by the calculation unit 633. Then, based on the data of the first 3D model stored in the storage unit 62, the display control unit 635 generates an image of the first 3D model (hereinafter referred to as the model image) that is recognized from the position of the head-mounted display 4 in the direction of the trainee's gaze while wearing the head-mounted display 4. In addition, the display control unit 635 recognizes the simulated operation unit 31 operated by the trainee wearing the head-mounted display 4 by inputting an operation signal. Then, the display control unit 635 simulates the operation of the part of the first 3D model corresponding to the simulated operation unit 31 in accordance with the operation of the simulated operation unit 31. Furthermore, based on the result of the determination process by the determination unit 634, the display control unit 635 performs an identification display process to display the part of the first 3D model corresponding to the target operation unit determined by the determination process on the display unit 42 in a way that makes it distinguishable from other parts. Further details regarding the identification and display process will be explained in the "Control Method" section below.

[0041] [Control method] Next, the control method executed by the control device 6 will be described. Figure 4 is a flowchart showing the control method. First, the determination unit 634 performs a determination process (step S1A: determination step). Specifically, in step S1A, the determination unit 634 determines that the simulated operation unit that is outputting an operation signal is the target operation unit among the multiple simulated operation units 31. In other words, the determination unit 634 determines that the simulated operation unit being operated by the trainee is the target operation unit among the multiple simulated operation units 31.

[0042] If the result of the determination process (step S1A) is that there is no simulated operating unit that is the target operating unit (there is no simulated operating unit being operated by the trainee) (step S1B: No), the determination unit 634 continues to step S1A.

[0043] Here, the display image generated by the display control unit 635 before executing the identification display process (step S1C) will be explained with reference to Figure 5. Figure 5 shows the display image F1 generated before the identification display process (step S1C) is executed. In Figure 5, the code "120M" is the boom model (first 3D model) corresponding to boom 120. The code "121M" is the boom body model (first 3D model) corresponding to boom body 121. Furthermore, the code "122M" is the boom pedestal model (first 3D model) corresponding to boom pedestal 122. Furthermore, the code "123M" is the boom lift cylinder model (first 3D model) corresponding to boom lift cylinder 123. Furthermore, the code "124M" is the slewing axis model (first 3D model) corresponding to slewing axis 124. Furthermore, the code "125M" is the luffing axis model (first 3D model) corresponding to luffing axis 125. Furthermore, the code "112M" is the base model (first 3D model) corresponding to base 112. Furthermore, the symbol "3M" represents an operating device model (first 3D model) corresponding to an operating device (not shown) provided on the drilling machine 100 and having a configuration similar to that of the simulated operating device 3. In addition, the symbol "SC" represents a screen, which is a second 3D model. The symbol "CI1" represents the first captured image generated by the imaging unit 41. In Figure 5, dots are added to the first captured image CI1 to distinguish it from the model images of the boom model 120M, the base model 112M, and the operating device model 3M.

[0044] The display control unit 635 generates model images of each of the first 3D models to be recognized in an area corresponding to the trainee's field of view in the virtual space, based on the position and orientation of the head-mounted display 4 (the trainee's line of sight while wearing the head-mounted display 4) calculated by the calculation unit 633 and the data of the first 3D model stored in the memory unit 62. In the example in Figure 5, the first 3D models recognized in that area are the boom model 120M (boom body model 121M, boom pedestal model 122M, boom lift cylinder model 123M, slewing axis model 124M, luffing axis model 125M), the base model 112M, and the operating device model 3M. The display control unit 635 then generates a display image F1 (Figure 5) in which the generated model images are arranged. As a result, the display image F1 is displayed on the display unit 42.

[0045] Here, when the display control unit 635 generates the display image F1, it places a screen SC onto which the first captured image CI1 generated by the imaging unit 41 is projected in a specific area of ​​the display image F1 (the central area in the left-right direction on the lower side). The display control unit 635 also changes the transparency of the screen SC based on the results of the comparison process shown below. The comparison process involves comparing the shortest distance between a reference line and a pre-set switching reference position with a specific threshold. The reference line is a virtual line created based on the position and orientation of the head-mounted display 4 calculated by the calculation unit 633, and extends linearly from a specific position on the head-mounted display 4 according to the trainee's line of sight. One or more switching reference positions are set at positions corresponding to the placement of the simulated operating device 3 (the placement of the simulated operating device 3 in the real space coordinate system, and the position of the operating device model 3M in the virtual space coordinate system).

[0046] Furthermore, the display control unit 635 makes the screen SC transparent if, as a result of the comparison process, the shortest distance exceeds a certain threshold (hereinafter referred to as the first case). An example of this first case is when the area corresponding to the trainee's field of view in the virtual space is relatively high up, and the operating device model 3M is not recognized in that area. In the first case, as a result of making the screen SC transparent, the display image shows only the model images of each first 3D model that are recognized in the area corresponding to the trainee's field of view in the virtual space. In other words, the first captured image is not displayed in the display image.

[0047] On the other hand, the display control unit 635 reduces the transparency of the screen SC if, as a result of the comparison process, the shortest distance is below a certain threshold (hereinafter referred to as the second case). An example of this second case is when the area corresponding to the trainee's field of view in the virtual space is relatively low, and the operating device model 3M is recognized in that area. In the second case, as a result of reducing the transparency of the screen SC, the display image shows the model images of each first 3D model recognized in the area corresponding to the trainee's field of view in the virtual space, as well as the first captured image. Note that the display image F1 shown in Figure 5 is the display image in the second case. Therefore, the first captured image CI1 is displayed in the display image F1.

[0048] If, as a result of the determination process (step S1A), there is a simulated operation unit that is the target operation unit (there is a simulated operation unit being operated by a trainee) (step S1B: Yes), the display control unit 635 executes identification display processing (step S1C: display processing step). Figure 6 shows the display image F2 generated after the identification display process (step S1C) is performed. Specifically, Figure 6 corresponds to Figure 5 and shows the display image in the second case. In Figure 6, the code "HA" represents the trainee's hand captured in the first captured image CI1. Figure 6 illustrates a case in which the trainee performs an elevation operation on the second simulated operation unit 31b, and in the determination process (step S1A), the second simulated operation unit 31b is determined to be the target operation unit.

[0049] In describing the identification display process (step S1C) below, we will assume that in the determination process (step S1A), the second simulated operation unit 31b is determined to be the target operation unit when the trainee performs an elevation operation on the second simulated operation unit 31b. In this case, in step S1C, the display control unit 635 performs an identification display process to display on the display unit 42 the part of the boom model 120M corresponding to the second simulated operation unit 31b so that it can be identified from other first 3D models. More specifically, the display control unit 635 displays on the display unit 42 the boom body model 121M, boom lift cylinder model 123M, and luffing axis model 125M, which operate in response to luffing operations on the second simulated operation unit 31b, so that they can be identified from other first 3D models. Here, an example of the identification display process (step S1C) is a process to make the boom body model 121M, boom lift cylinder model 123M, and luffing axis model 125M different in color from other first 3D models (hereinafter referred to as the color change process). For example, in display image F1 before the identification display process is performed, the entire boom model 120M, the base model 112M, and the operating device model 3M are blue. However, in display image F2 after the identification display process is performed, only the boom body model 121M, the boom lift cylinder model 123M, and the luffing axis model 125M are changed to red. In Figure 6, diagonal lines are drawn only on the boom body model 121M, the boom lift cylinder model 123M, and the luffing axis model 125M to indicate that they are different colors. The control method executed by the control device 6 is completed by the above steps S1A to S1C.

[0050] According to the first embodiment described above, the following effects are achieved. In the simulation device 1 according to this embodiment 1, the control device 6 determines which of the multiple simulated operation units 31 the trainee intends to operate as the target operation unit. The control device 6 then displays the part of the first 3D model corresponding to the target operation unit on the display unit 42 so that it can be identified from other parts. Therefore, trainees can easily grasp the relationship between the simulated operation unit 31 they intend to operate and the corresponding target area for operation from the display image shown on the display unit 42. Accordingly, the simulation device 1 according to this embodiment 1 makes it possible to easily enable trainees to acquire the skills to operate the drilling machine 100.

[0051] Furthermore, in the simulation device 1 according to this embodiment 1, the control device 6 determines that the simulated operation unit that outputs an operation signal is the target operation unit among the multiple simulated operation units 31. Therefore, the target operating unit can be easily identified, and the processing load on the control device 6 can be reduced when determining the target operating unit.

[0052] Here, the multiple simulated operation units 31 include, for example, a second simulated operation unit 31b which is a multi-functional operation unit that accepts multiple operations. In the simulation device 1 according to this embodiment 1, if the control device 6 determines, for example, that the second simulated operation unit 31b is the target operation unit, it displays on the display unit 42 the part of the first 3D model that is operated in response to, for example, an elevation operation on the second simulated operation unit 31b, so that it can be identified from other parts. Therefore, when operating the multi-function control unit, trainees can easily understand from the display image on the display unit 42 which operation will activate which part of the unit, among the multiple operations that the multi-function control unit accepts. Consequently, trainees can acquire the skills to operate the drilling machine 100 more easily.

[0053] (Embodiment 2) Next, we will describe Embodiment 2. In the following description, components similar to those in Embodiment 1 described above are denoted by the same reference numerals, and their detailed descriptions are omitted or simplified. Figure 7 is a block diagram showing the configuration of the simulation device 1A according to Embodiment 2. Specifically, Figure 7 corresponds to Figure 3. The simulation device 1A according to this second embodiment executes different judgment processing and identification display processing (first and second identification display processing) than the judgment processing and identification display processing described in the first embodiment described above. Furthermore, the simulation device 1A has an additional imaging device 7 (Figure 7) compared to the simulation device 1 described in the first embodiment described above, and the configuration of the control device 6 has been changed. Hereinafter, the control device according to this second embodiment will be referred to as control device 6A (Figure 7).

[0054] [Configuration of the imaging device] Figure 8 shows the arrangement of the imaging device 7. The imaging device 7 is a camera that includes an image sensor such as a CCD or CMOS that receives incident light and converts it into an electrical signal, and captures a specific area to generate an image (hereinafter referred to as the second image). As shown in Figure 8, the imaging device 7 is positioned above the simulated operation device 3 and, under the control of the control device 6A, captures an area including the first to 18th simulated operation units 31a to 31r among the multiple simulated operation units 31 to generate the second image. The imaging device 7 then outputs the data of the generated second image to the control device 6A.

[0055] [Control device configuration] In control device 6A, the configuration of the storage unit 62 and the function of the control unit 63 are modified compared to the control device 6 described in Embodiment 1 above. Hereinafter, the storage unit and control unit according to Embodiment 2 will be referred to as storage unit 62A (Figure 7) and control unit 63A (Figure 7), respectively.

[0056] The storage unit 62A stores various programs executed by the control unit 63A (including the control program according to the present invention), as well as data necessary when the control unit 63A performs processing. As shown in Figure 7, the storage unit 62A includes a 3D model DB (Data Base) 621, a related information DB (Data Base) 622, and a learning model DB (Data Base) 623. The 3D model DB621 stores the data for the first 3D model and the data for the second 3D model described in Embodiment 1 above.

[0057] Figure 9 is a diagram illustrating the related information stored in the related information DB622. Specifically, Figure 9 shows the main part of the second captured image CI2 generated by the imaging device 7. Here, as shown in Figure 9, the position coordinates Pa to Pr of the first to 18th simulated operation units 31a to 31r, which are included as subjects in the second captured image CI2, are predetermined position coordinates. In this embodiment 2, these position coordinates are the pixel positions (x,y) in the second captured image CI2, and are two-dimensional coordinates in the horizontal plane. The related information DB622 stores related information, which associates the position coordinates Pa to Pr of the first to 18th units with the first to 18th simulated operation units 31a to 31r located at these position coordinates Pa to Pr. For example, the second simulated operation unit 31b is associated with the second position coordinate Pb.

[0058] The learning model DB623 stores the learning model used in the decision processing according to this embodiment 2. This learning model is, for example, a model generated by machine learning using artificial intelligence (AI). Further details about the learning model will be explained in the "Control Method" section below.

[0059] In the control unit 63A, the function of the third image acquisition unit 636 (Figure 7) is added to the control unit 63 described in the first embodiment above, and the functions of the determination unit 634 and the display control unit 635 are modified. Hereinafter, the determination unit and the display control unit according to this second embodiment will be referred to as the determination unit 634A (Figure 7) and the display control unit 635A (Figure 7), respectively. The third image acquisition unit 636 acquires data from the second captured image generated by the imaging device 7.

[0060] The determination unit 634A differs from the determination unit 634 described in the above-described embodiment 1 in that it performs a different determination process. The display control unit 635A differs from the display control unit 635 described in Embodiment 1 above in that it performs different identification display processing (first and second identification display processing). Details of the determination process and identification display process (first and second identification display processes) related to this second embodiment will be explained in the "Control Method" section below.

[0061] [Control method] Next, the control method executed by the control device 6A will be described. Figure 10 is a flowchart showing the control method. First, the third image acquisition unit 636 acquires the data of the second captured image generated by the imaging device 7 (step S2A).

[0062] After step S2A, the determination unit 634A uses the learning model stored in the learning model DB623 to perform image recognition (so-called skeleton detection) and detects the skeleton of the trainee's hand included as a subject in the second captured image acquired in step S2A, thereby recognizing the position coordinates of the hand (pixel position (x,y) in the second captured image) (step S2B). In this embodiment 2, the determination unit 634A recognizes the position coordinates of the position between the thumb and index finger of the hand as the position coordinates of the hand. The learning model stored in the learning model DB623 is a model obtained by using a second image of a human hand, on which the positions of the joint points of the hand have been pre-labeled, as training data, and then using machine learning (e.g., deep learning) to determine the positions of the joint points based on this training data.

[0063] After step S2B, the determination unit 634A executes a determination process (step S2C: determination step). Specifically, the determination unit 634A analyzes the positional relationship between the position coordinates of the trainee's hand recognized in step S2B and the first to 18th position coordinates Pa to Pr included in the related information stored in the related information DB 622. The determination unit 634A also extracts the position coordinates of the first to 18th simulated operation units 31a to 31r that have the shortest distance from the position coordinates of the trainee's hand recognized in step S2B and that is below a certain threshold. Then, the determination unit 634A refers to the related information stored in the related information DB 622 and determines that the simulated operation unit associated with the extracted position coordinates is the target operation unit that the trainee intends to operate.

[0064] Figure 11 is a diagram illustrating the determination process (step S2C) and the first identification display process (step S2E). Specifically, Figure 11 corresponds to Figure 6 and shows the display image F3 generated after the first identification display process (step S2E) is executed in the second case. Figure 11 illustrates the case where the trainee is attempting to operate the second simulated operation unit 31b. For example, when a trainee is about to operate the second simulated operation unit 31b, the trainee's hand HA will be in close proximity to the second simulated operation unit 31b, as shown in Figure 11. The second position coordinate Pb of the second simulated operation unit 31b will have a specific positional relationship with the position coordinate of the trainee's hand HA. For this reason, the second simulated operation unit 31b is determined to be the target operation unit in the determination process (step S2C).

[0065] If the result of the determination process (step S2C) is that there is no simulated operating unit that is the target operating unit (there is no simulated operating unit that the trainee is trying to operate) (step S2D: No), the determination unit 634A continues to step S2C. On the other hand, if the result of the determination process (step S2C) indicates that there is a simulated operation unit that is the target operation unit (i.e., there is a simulated operation unit that the trainee is trying to operate) (step S2D: Yes), the display control unit 635A executes the first identification display process (step S2E: display processing step). The first identification display process is a process that displays the part of the first 3D model corresponding to the target operation unit determined by the determination process (step S2C) on the display unit 42 in a way that makes it identifiable from other parts.

[0066] In describing the first identification display process (step S2E) in detail below, we will assume that in the determination process (step S2C), the second simulated operation unit 31b is determined to be the target operation unit. Furthermore, for the sake of explanation, we will use as an example the case in which a color change process is adopted as the first identification display process (step S2E). In this case, in step S2E, the display control unit 635A displays the entire boom model 120M corresponding to the second simulated operation unit 31b and the other first 3D models on the display unit 42 in different colors. For example, in the display image before the first identification display processing is executed, the entire boom model 120M, the base model 112M, and the operation device model 3M were blue, but in the display image F3 after the first identification display processing is executed, only the entire boom model 120M is changed to red. Note that in Figure 11, diagonal lines are drawn only on the entire boom model 120M to indicate that they are different colors.

[0067] After step S2E, the determination unit 634A determines whether or not an operation signal was output from the simulated operation unit that was determined to be the target operation unit among the multiple simulated operation units 31, that is, whether or not the target operation unit was operated (step S2F). If it is determined that the target operating unit is not being operated (step S2F: No), the control unit 63A returns to step S2C. On the other hand, if it is determined that the target operation unit has been operated (step S2F: Yes), the display control unit 635A executes a second identification display process (step S2G: display processing step). The second identification display process is a process that displays on the display unit 42, in a manner different from when the first identification display process is performed, the identification status that distinguishes the part of the first 3D model corresponding to the target operation unit determined by the determination process (step S2C) from other parts.

[0068] Figure 12 is a diagram illustrating the second identification display process (step S2G). Specifically, Figure 12 corresponds to Figure 11 and shows the display image F4 generated after the second identification display process (step S2G) is executed in the second case. In Figure 12, an example is shown in which, in the determination process (step S2C), the second simulated operation unit 31b is determined to be the target operation unit, and the trainee is performing an elevation operation on the second simulated operation unit 31b.

[0069] In describing the second identification display process (step S2G) in detail below, we will assume that in the determination process (step S2C), the second simulated operation unit 31b is determined to be the target operation unit, and the trainee is performing an elevation operation on the second simulated operation unit 31b. Furthermore, for the sake of explanation, we will illustrate the case where a color change process is adopted as the second identification display process (step S2G). In this case, in step S2G, the display control unit 635A displays the boom body model 121M, boom lift cylinder model 123M, and luffing axis model 125M, which are operated in accordance with the luffing operation, on the display unit 42 in a state where they are different colors from the other first 3D models. For example, in the display image F3 before the second identification display processing is executed, the entire boom model 120M was red and the other first 3D models were blue, but in the display image F4 after the second identification display processing is executed, only the boom body model 121M, boom lift cylinder model 123M, and luffing axis model 125M are changed to red. Furthermore, the display control unit 635A makes the color intensity of the boom body model 121M, boom lift cylinder model 123M, and luffing axis model 125M darker than before the second identification display processing was executed. In Figure 12, diagonal lines are used only on the boom body model 121M, boom lift cylinder model 123M, and luffing axis model 125M to indicate that they are different colors. Also, as can be seen by comparing Figures 11 and 12, the intensity of the color is represented by the spacing (density) of the lines in the diagonal lines. The control method executed by the control device 6A is completed by the above steps S2A to S2G.

[0070] According to this embodiment 2 described above, in addition to the same effects as in embodiment 1 described above, the following effects are achieved. In the simulation device 1A according to this second embodiment, the control device 6A determines that the simulated operation unit among the first to 18th simulated operation units 31a to 31r whose positional relationship with the position of the trainee's hand is a specific positional relationship is the target operation unit. Here, the simulated operation unit that has a specific positional relationship is the simulated operation unit among the first to 18th simulated operation units 31a to 31r whose positional coordinates are the shortest and whose distance from the positional coordinates of the trainee's hand is below a specific threshold. In other words, the control device 6A determines the simulated operation unit 31 that the trainee intends to operate as the target operation unit before the trainee actually operates the simulated operation unit 31. Therefore, the trainee can easily grasp the relationship between the simulated operation unit 31 and the corresponding target operation part from the display image shown on the display unit 42 before actually operating the simulated operation unit 31.

[0071] In particular, the control device 6A recognizes the position coordinates of the trainee's hand through image recognition. Therefore, the hardware configuration of the control device 6A can be made more complex, while still allowing for accurate recognition of the trainee's hand position coordinates. Furthermore, the control device 6A recognizes the position coordinates of the trainee's hand by so-called skeletal detection. Therefore, there is no need to perform the cumbersome task of attaching markers or the like to the trainee's hand, and the position coordinates of the trainee's hand can be recognized accurately.

[0072] Furthermore, in the simulation device 1A according to this second embodiment, the control device 6A executes the first identification display process described above when it determines that any of the multiple simulated operation units 31 is the target operation unit. After the first identification display process, the control device 6A executes the second identification display process described above when an operation is performed on the target operation unit. Here, the second identification display process is a process that displays on the display unit 42 that the identification state for distinguishing the part of the first 3D model corresponding to the target operation unit from other parts is different from the state when the first identification display process is executed. Therefore, trainees can clearly understand from the display image on the display unit 42 whether or not they have actually operated the target operating unit, and which target part operates in response to that operation. Consequently, trainees can acquire the skills to operate the drilling machine 100 more easily.

[0073] (Other embodiments) While embodiments for carrying out the present invention have been described so far, the present invention should not be limited to the embodiments described above. In the embodiments 1 and 2 described above, a color change process was given as an example of the identification display process (first and second identification display processes) according to the present invention, but it is not limited to this. As the identification display process (first and second identification display processes) according to the present invention, for example, a process that changes the brightness between a part of the virtual model corresponding to the target operation unit and other parts, or a process that changes the transparency between a part of the virtual model corresponding to the target operation unit and other parts may be adopted.

[0074] In the embodiments 1 and 2 described above, the first captured image CI1 was displayed on the display unit 42 in the second case so that the trainee could operate multiple simulated operation units 31 while wearing the head-mounted display 4, but the invention is not limited to this. For example, an infrared light-emitting unit is attached to the trainee's hand. Also, data of a 3D model corresponding to a human hand is stored in the memory unit 62 (62A). Here, the control unit 63 (63A) calculates the position and orientation of the trainee's hand based on the infrared images generated by the two infrared cameras 51, similar to when calculating the position and orientation of the head-mounted display 4. Then, based on the calculated position and orientation of the trainee's hand, the control unit 63 (63A) displays a model image of a 3D model corresponding to a human hand on the display unit 42.

[0075] In the embodiments 1 and 2 described above, some functions of the control unit 63(63A) may be provided outside the control device 6(6A). For example, the functions of the calculation unit 633, which calculates the position and orientation of the head-mounted display 4, may be provided on the base station 5. In this case, the control device 6(6A) acquires information indicating the position and orientation of the head-mounted display 4 from the base station 5.

[0076] In the above-described embodiment 2, the imaging device 7 was provided only above the first to 18th simulated operation units 31a to 31r located to the right of the training seat 2, and one of the first to 18th simulated operation units 31a to 31r was determined to be the target operation unit. However, the system is not limited to this. For example, the imaging device may also be provided above the simulated operation unit 31 located to the left of the training seat 2, and the system may be configured to also determine that the simulated operation unit 31 located to the left of the training seat 2 is the target operation unit.

[0077] In the above-described embodiment 2, the position coordinates Pa to Pr of the first to 18 simulated operation units 31a to 31r, and the position coordinates of the trainee's hand were configured using two-dimensional coordinates in the horizontal plane, but the invention is not limited to this. For example, the imaging device 7 is configured with a stereo camera. Then, by simultaneously capturing images from different viewpoints using the stereo camera and using the relative displacement in the images of the same subject, a stereo measurement technique is used to calculate the three-dimensional position of the subject based on the principle of triangulation. The three-dimensional coordinates, including the vertical height, may be used as the 1st to 18th position coordinates Pa to Pr, and also as the position coordinates of the trainee's hand. Furthermore, for example, the imaging device 7 may be configured using a distance image sensor such as a TOF (Time of Flight) sensor. The three-dimensional coordinates, including the vertical height, may also be used as the 1st to 18th position coordinates Pa to Pr, and the position coordinates of the trainee's hand.

[0078] In the above-described embodiment 2, the determination unit 634A recognized the position coordinates of the trainee's hand position by image recognition using a learning model, but it is not limited to this. For example, a marker or similar object is attached to the trainee's hand. The determination unit 634A then detects the marker using image recognition and recognizes the position coordinates of the trainee's hand. For example, a light-emitting unit that emits infrared light is attached to the trainee's hand. Then, the determination unit 634A recognizes the position coordinates of the trainee's hand based on the infrared images generated by the two infrared cameras 51, similar to how the position and orientation of the head-mounted display 4 are calculated. [Explanation of symbols]

[0079] 1.1A Simulation System 2 Training seat 3 Simulated operating device 3M Control Device Model 4. Head-mounted display 5 Base Stations 6,6A Control Unit 7. Imaging device 31 Simulation operation section 31a First simulated operating unit 31b Second simulated operating unit 31c Third simulated operating section 31d Fourth simulated operating unit 31e Fifth Simulated Operating Unit 31f Sixth Simulated Control Unit 31g 7th Simulated Operating Unit 31h 8th Simulated Operation Unit 31i 9th Simulated Operating Unit 31j 10th Simulated Operating Unit 31k 11th Simulated Operating Unit 31l 12th Simulated Operating Unit 31m 13th Simulated Control Unit 31n 14th Simulated Operating Unit 31o 15th Simulated Operating Unit Page 31, Section 16: Simulated Control Unit 31q 17th Simulated Operating Unit 31r 18th Simulated Operating Unit 41 Photography Department 42 Display section 43 Infrared output section 51 Infrared Camera 61 Input section 62,62A storage section 63,63A Control Unit 100 Drilling Machine 110 Bogie 111 Trucks 112 base 112M Base Model 120 Boom 120M Boom Model 121 Boom body 121M Boom Unit Model 122 Boom Pedestal 122M Boom Pedestal Model 123 Boom Lift Cylinder 123M Boom Lift Cylinder Model 124 Swivel axis 124M Swivel Axis Model 125 Relief axis 125M elevation axis model 130 rock-cutting units 131 Guide Shell 132 Rock drilling machine 133 Tools 140 Cockpit 141 Driver's seat 621 3D Model Database 622 Related Information Database 623 Learning Model DB 631 First image acquisition unit 632 Second image acquisition unit 633 Calculation Unit 634,634A Judgment section 635, 635A Display Control Unit 636 Third image acquisition unit CI1 First captured image CI2 Second set of captured images F1~F4 Display Images HA hand Pa First position coordinate Pb Second position coordinate Pc Third position coordinate Pd 4th position coordinate Pe's fifth position coordinate Pf 6th position coordinate Pg 7th position coordinate Ph's 8th position coordinate Pi's 9th position coordinate Pj 10th position coordinate Pk 11th position coordinates Pl 12th position coordinate Pm 13th position coordinate Pn 14th position coordinate Po 15th position coordinates Pp 16th position coordinates Pq 17th position coordinate Pr 18th position coordinates SC Screen

Claims

1. A simulated operating device having multiple simulated operating parts that are provided on a drilling machine and each simulated reproduces one of the multiple operating parts used to operate the drilling machine, A display device that displays an image of a virtual model corresponding to the drilling machine, The system includes a device that displays an image of the virtual model on the display device, and a control device that simulates the operation of a part of the virtual model corresponding to the simulated operation unit operated by the user in response to the operation of the simulated operation unit. The control device is A determination unit determines, among the plurality of simulated operation units, that the simulated operation unit that the user intends to operate and whose positional relationship with the position of the user's hand is a specific positional relationship is designated as the target operation unit. The system includes a display control unit that causes the part of the virtual model corresponding to the target operation unit to be displayed on the display device in a way that allows it to be identified from other parts, The aforementioned plurality of simulated operation units are, The control device outputs operation signals corresponding to the user's actions. The display control unit, The first identification display process and the second identification display process are executed, The first identification and display process is: This process involves displaying the part of the virtual model corresponding to the target operation unit on the display device in a way that allows it to be identified from other parts. The second identification display process described above is: A simulation device, which is executed after the first identification display process when an operation is performed on the target operation unit, and which displays on the display device an identification state that identifies the part of the virtual model corresponding to the target operation unit from other parts in a manner different from when the first identification display process is performed.

2. The aforementioned plurality of simulated operation units are, The control device outputs operation signals corresponding to the user's actions. The determination unit, The simulation apparatus according to claim 1, wherein, among the plurality of simulated operation units, the simulated operation unit that outputs the operation signal is determined to be the target operation unit.

3. The display control unit, The simulation apparatus according to claim 2, in which, if a simulation operation unit among the plurality of simulation operation units that accepts multiple operations is determined to be the target operation unit, the part of the virtual model that is operated in response to the operation on the target operation unit is displayed on the display device in a manner that allows it to be identified from other parts.

4. The determination unit, The simulation device according to claim 1, which measures the position coordinates of the user's hand, analyzes the positional relationship between the position coordinates of the user's hand and the pre-set position coordinates of the plurality of simulated operating units, and determines that the simulated operating unit that has the specific positional relationship based on the results of the analysis is the target operating unit.

5. The imaging device further comprises a device that captures an area including the plurality of simulated operation units and outputs the captured image data, The control device is The system further includes an image acquisition unit that acquires data from the captured image, The determination unit, The simulation device according to claim 4, which measures the position coordinates of the user's hand based on the captured image, analyzes the positional relationship between the position coordinates of the user's hand and the position coordinates of the preset plurality of simulated operation units, and determines the simulated operation unit that has the specific positional relationship based on the results of the analysis as the target operation unit.

6. The determination unit, The simulation device according to claim 5, which recognizes the position of the user's hand by detecting the skeletal structure of the user's hand based on the captured image.

7. The determination unit, The simulation device according to claim 5 or 6, wherein, among the plurality of simulated operation units, the simulated operation unit whose position coordinates in the specific positional relationship are the shortest and whose distance from the position coordinates of the user's hand is below a specific threshold is determined to be the target operation unit.

8. A control method executed by the control device of a simulation device, A determination step in which, among multiple simulated operating parts that each simulates an operating part used to operate a drilling machine, the simulated operating part that the user intends to operate and whose positional relationship with the position of the user's hand is a specific positional relationship is determined to be the target operating part, The display processing step includes displaying an image of a virtual model corresponding to the drilling machine on a display device, and displaying on the display device a part of the virtual model corresponding to the target operation unit in a manner that can be identified from other parts, The aforementioned plurality of simulated operation units are, The control device outputs operation signals corresponding to the user's actions. The aforementioned display processing step is: It includes a first identification display process and a second identification display process, The first identification and display process is: The part of the virtual model corresponding to the target operating unit is displayed on the display device in a way that allows it to be identified from other parts. The second identification display process described above is: A control method that, after the first identification display process, when an operation is performed on the target operation unit, displays on the display device an identification state that identifies the part of the virtual model corresponding to the target operation unit and other parts in a manner different from when the first identification display process was performed.

9. A determination step in which, among multiple simulated operating parts that each simulates an operating part used to operate a drilling machine, the simulated operating part that the user intends to operate and whose positional relationship with the position of the user's hand is a specific positional relationship is determined to be the target operating part, The computer is instructed to perform a display processing step which involves displaying an image of a virtual model corresponding to the drilling machine on a display device, and displaying on the display device the part of the virtual model corresponding to the target operating unit in a manner that allows it to be identified from other parts. The aforementioned plurality of simulated operation units are, The control device outputs operation signals corresponding to the user's actions. The aforementioned display processing step is: It includes a first identification display process and a second identification display process, The first identification and display process is: The part of the virtual model corresponding to the target operating unit is displayed on the display device in a way that allows it to be identified from other parts. The second identification display process described above is: A control program that, after the first identification display process, when an operation is performed on the target operation unit, displays on the display device an identification state that identifies the part of the virtual model corresponding to the target operation unit from other parts, in a manner different from when the first identification display process was performed.

Images