Simulation device

Abstract

To accurately match a virtual space coordinate system with a real space coordinate system.SOLUTION: A simulation device 1 comprises: a simulation operating device 3 that has a plurality of simulation operating parts 31; a display device 4 that allows a user to visually recognize a real image according to the user's field of vision and a predetermined image; a control device 7 that causes the display device 4 to display an image of a virtual model corresponding to a drilling machine; and an acquisition device 8. The acquisition device 8 includes a mounting tool 5 that can be mounted on at least any one simulation operating part 31 of the plurality of simulation operating parts 31, and a position detection unit 6 that detects the position of the mounting tool 5. The control device 7 executes calibration processing of matching a virtual space coordination system in the virtual model with a real space coordination system based on the position of the mounting tool 5 detected by the position detection unit 6 and the position of the simulation operating part 31 on which the mounting tool 5 is mounted in the virtual space coordinate system.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This invention relates to a simulation device. [Background technology]

[0002] Conventionally, drilling machines have been used to drill blast holes in rock formations at quarries, limestone mines, etc. (see, for example, Patent Document 1). In this type of drilling machine, a feeding mechanism presses a tool attached to the tip of the rock drill against the object to be crushed, and the impact force generated by the striking mechanism and the rotational force generated by the rotating mechanism are transmitted to the object to be crushed via this tool, thereby drilling a blast hole. During this drilling operation, the boom and guide shell are positioned, the feeding mechanism, striking mechanism, and rotating mechanism are operated, and the centralizer and rod changer are operated, etc. In other words, this drilling machine operates on a wide range of objects, and many operating parts are provided to operate each of these objects.

[0003] Furthermore, as mentioned above, since a drilling machine has many operating parts, it is not easy to acquire the skills to operate such a machine. Therefore, it is conceivable to use a simulation device to acquire skills in operating a drilling machine without using an actual drilling machine. For example, the simulation device allows trainees to view a real image corresponding to their field of view and an image of a virtual model of the drilling machine (hereinafter referred to as the model image) via a display device such as a head-mounted display worn by the trainee. Furthermore, in response to the trainee's operation on one of several simulated operating parts that simulate the operation parts of the drilling machine, the simulation device simulates the operation of a part of the model image, such as the boom, corresponding to the simulated operating part, on the display screen of the display device. Conventionally, simulation devices for work vehicles such as forklifts have been proposed as such (see, for example, Patent Document 2).

[0004] If the virtual space coordinate system of the drilling machine's virtual model is misaligned with the real space coordinate system, the real image and the model image will be misaligned according to the trainee's field of view. In such cases, the trainee, viewing the real image and the model image through a head-mounted display, will experience a sense of unease. To prevent the trainee from experiencing this unease, it is necessary to align the virtual space coordinate system of the virtual model with the real space coordinate system. Conventionally, systems using alignment tools have been proposed to align the virtual space coordinate system of a virtual model with the real space coordinate system (see, for example, Patent Document 3). The alignment tool described in Patent Document 3 comprises a pointer with a sharpened tip and a reference marker used to detect the position of the tip of the pointer. In addition, the system described in Patent Document 3 comprises an augmented reality imaging device worn by the operator, in addition to the alignment tool.

[0005] In the system described in Patent Document 3, the virtual space coordinate system of the virtual model is aligned with the real space coordinate system as shown below. First, the operator holds the alignment tool and brings the tip of the pointer to the real-world position corresponding to a predetermined virtual reference point in the virtual space coordinate system of the virtual model (hereinafter referred to as the corresponding real-world position). After this, the operator presses the selection button provided on the alignment tool. In response to the pressing of the selection button, the augmented reality imaging device attached to the operator reads the reference marker using a sensor provided on the augmented reality imaging device and calculates the real-world position corresponding to the virtual reference point. Then, the augmented reality imaging device aligns the virtual space coordinate system of the virtual model with the real-world coordinate system so that the virtual reference point matches the corresponding real-world position. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2000-303768 [Patent Document 2] Japanese Patent Publication No. 2004-252024 [Patent Document 3] Japanese Patent Publication No. 2019-87241 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] In the technology described in Patent Document 3, when calculating the corresponding real-space position, the operator must hold the alignment tool and maintain contact between the sharpened pointer tip and the corresponding real-space position. If the corresponding real-space position is calculated with the pointer tip away from the corresponding real-space position, a different position will be calculated as the corresponding real-space position. In such cases, the virtual reference point is aligned to a different corresponding real-space position, resulting in the virtual space coordinate system of the virtual model being misaligned with the real-space coordinate system. Therefore, there is a need for a technology that can accurately align the virtual spatial coordinate system of a virtual model with the real spatial coordinate system.

[0008] The present invention has been made in view of the above, and aims to provide a simulation device that can accurately align a virtual spatial coordinate system with a real spatial coordinate system. [Means for solving the problem]

[0009] In order to solve the above problems and achieve the object, a simulation apparatus according to the present invention is provided in a drilling machine and includes a simulation operation device having a plurality of simulated operation parts that respectively and pseudo-replicate a plurality of operation parts used to operate the drilling machine, a display device that is worn by a user and causes the user to visually recognize a real image corresponding to the user's field of view and a predetermined image, a control device that displays an image of a virtual model corresponding to the drilling machine on the display device and pseudo-acts a part of the virtual model corresponding to the simulated operation part operated by the user among the images of the virtual model in accordance with an operation on the simulated operation part, and an acquisition device that acquires information necessary for a calibration process for aligning a virtual space coordinate system in the virtual model with a real space coordinate system. The acquisition device includes a mounting tool that is configured to be attachable to at least one of the plurality of simulated operation parts, and a position detection part that detects the position of the mounting tool. The control device executes the calibration process based on the position of the mounting tool detected by the position detection part and the position of the simulated operation part to which the mounting tool is attached in the virtual space coordinate system.

[0010] Further, in the simulation apparatus according to the present invention, the mounting tool includes an annular fitting part that allows at least one of the plurality of simulated operation parts to be fitted therein.

[0011] Further, in the simulation apparatus according to the present invention, an inclined surface is provided on an inner peripheral surface of the fitting part, the inner diameter dimension of which increases as it approaches an end surface in a direction along a central axis of the fitting part.

[0012] Further, in the simulation apparatus according to the present invention, the simulation operation device further includes an acquisition operation part that receives a user operation, and the control device acquires the position of the mounting tool detected by the position detection part in accordance with a user operation on the acquisition operation part.

[0013] Furthermore, in the simulation apparatus according to the present invention, the calibration process includes a first calibration process that aligns the virtual space coordinate system with the real space coordinate system based on the position of the mounting device detected by the position detection unit while it is mounted on the first simulated operation unit among the plurality of simulated operation units, and the position of the first simulated operation unit in the virtual space coordinate system; and a second calibration process that aligns the virtual space coordinate system with the real space coordinate system based on the position of the mounting device detected by the position detection unit while it is mounted on a second simulated operation unit different from the first simulated operation unit among the plurality of simulated operation units, and the position of the second simulated operation unit in the virtual space coordinate system.

[0014] Furthermore, in the simulation apparatus according to the present invention, in the first calibration process, the virtual space coordinate system is moved so that the position of the first simulated operation unit in the virtual space coordinate system matches the position of the mounting device detected by the position detection unit while it is mounted on the first simulated operation unit. In the second calibration process, the virtual space coordinate system is rotated around a predetermined axis with the position of the first simulated operation unit in the virtual space coordinate system as the center, based on the position of the mounting device detected by the position detection unit while it is mounted on the second simulated operation unit and the position of the second simulated operation unit in the virtual space coordinate system. [Effects of the Invention]

[0015] According to the simulation device of the present invention, a virtual spatial coordinate system can be accurately aligned with a real spatial coordinate system. [Brief explanation of the drawing]

[0016] [Figure 1] Figure 1 shows a drilling machine that is the target of skill acquisition for operation using a simulation device according to the embodiment. [Figure 2] Figure 2 shows the configuration of the simulation device according to the embodiment. [Figure 3] Figure 3 shows the configuration of the device. [Figure 4] Figure 4 shows the configuration of the device to be worn. [Figure 5] Figure 5 is a block diagram showing the configuration of the control device. [Figure 6] Figure 6 is a flowchart showing the calibration method. [Figure 7] Figure 7 shows the device attached to the first simulated operating unit. [Figure 8] Figure 8 shows the device attached to the second simulated operating unit. [Figure 9] Figure 9 illustrates the first calibration process (step S7). [Figure 10] Figure 10 illustrates the second calibration process (step S13). [Modes for carrying out the invention]

[0017] The embodiments for carrying out the present invention (hereinafter referred to as "embodiments") will be described below with reference to the drawings. However, the present invention is not limited to the embodiments described below. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0018] [Outline configuration of a drilling machine] Figure 1 shows a drilling machine 100 that is the target of skill acquisition for operation using the simulation device 1 according to the embodiment. First, before describing the configuration of the simulation device 1 according to this embodiment, we will describe the configuration of the drilling machine 100, which is the target of skill acquisition for operation using the simulation device 1. In describing the drilling machine 100, the term "front side" below refers to the right side as shown in Figure 1.

[0019] The drilling machine 100 drills blast holes in the rock mass according to the operator's instructions. As shown in Figure 1, the drilling machine 100 comprises a traveling carriage 110 having a pair of left and right tracks 111 (only one track 111 is shown in Figure 1), and a boom 120, a rock drilling unit 130, and a control room 140 mounted on the traveling carriage 110, respectively.

[0020] The boom 120 is mounted on the right-front side of the traveling carriage 110 so as to be able to rotate (rotate) around a pivot axis 121 (Figure 1) that is aligned vertically, and so as to be able to rotate (raise and lower) around a luffing axis 122 (Figure 1) that is aligned horizontally (forward and backward).

[0021] As shown in Figure 1, the rock-drilling unit 130 is attached to the tip of the boom 120. This rock-drilling unit 130 comprises a guide shell 131 attached to the tip of the boom 120, a rock-drilling machine 132 that is provided to move forward and backward along the longitudinal direction of the guide shell 131, and a tool 133 attached to the tip of the rock-drilling machine 132. Here, the rock drill 132 moves forward and backward on the guide shell 131 by a feed mechanism (not shown) provided on the guide shell 131, and is equipped with a known striking mechanism (not shown) and a rotation mechanism (not shown). Although not specifically shown, the tool 133 consists of a shank rod, sleeve, rod, and bit connected in this order from the base end. Then, in the drilling machine 100, the tool 133 is pressed against the object to be crushed by a feeding mechanism (not shown), and the impact force generated by the striking mechanism (not shown) and the rotational force generated by the rotation mechanism (not shown) are transmitted to the object to be crushed via the tool 133 to drill a blast hole.

[0022] As shown in Figure 1, the control room 140 is located on the left-front side of the traveling chassis 110. Inside the control room 140, as shown in Figure 1, there is a driver's seat 141 where an operator who operates the drilling machine 100 sits. On both sides of the driver's seat 141, although not shown in detail, there are several operating parts used to operate the drilling machine 100.

[0023] [Outline configuration of the simulation device] Next, we will describe the configuration of the simulation device 1. Figure 2 shows the configuration of the simulation device 1 according to the embodiment. As shown in Figure 2, the simulation device 1 comprises a training seat 2, a simulated operation device 3, a head-mounted display 4, a wearable device 5, a base station 6, and a control device 7.

[0024] Training seat 2 is the area where a trainee, who is a user of the simulation device 1, sits to acquire skills in operating the drilling machine 100. As shown in Figure 2, the simulated operation device 3 is provided on both sides of the training seat 2. This simulated operation device 3 has multiple simulated operation units 31 that simulate the operation units (not shown) provided on the drilling machine 100, and an acquisition operation unit 32 (see Figure 7). These multiple simulated operation units 31 and acquisition operation unit 32 are each connected to the control device 7 wirelessly or via wired communication, and each outputs a signal to the control device 7 in response to user operations by a trainee or the like.

[0025] For example, among the multiple simulated operation units 31, the first simulated operation unit 311 is located to the right of the training seat 2. This first simulated operation unit 311 is a tiltable lever that simulates the operation units for the feed mechanism (not shown), the impact mechanism (not shown), and the rotation mechanism (not shown) of the drilling machine 100 (see Figure 7). Also, among the multiple simulated operation units 31, the second simulated operation unit 312 is located to the left of the training seat 2. The second simulated operation unit 312 may be any other simulated operation unit 31, as long as it is different from the first simulated operation unit 311. Similarly, the first simulated operation unit 311 may be any other simulated operation unit 31, as long as it is different from the second simulated operation unit 312. The acquisition operation unit 32 receives a user operation that triggers the acquisition of the position of the attached device 5. The acquisition operation unit 32 then outputs a signal corresponding to the user operation to the control device 7.

[0026] The head-mounted display 4 corresponds to the display device according to the present invention and is connected to the control device 7 wirelessly or via a wired connection for communication. The head-mounted display 4 has the appearance of eyeglasses and is worn by the trainee. The head-mounted display 4 allows the trainee to view a real image and a predetermined image corresponding to the trainee's field of vision while wearing the head-mounted display 4. The head-mounted display 4 comprises an imaging unit 41, a display unit 42, and a plurality of infrared output units 43 (see Figure 5).

[0027] The imaging unit 41 is a camera that includes an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) that receives incident light and converts it into an electrical signal. The imaging unit 41 also generates an image by capturing an area corresponding to the field of view of the trainee wearing the head-mounted display 4 under the control of the control device 7. The imaging unit 41 then outputs the data of the generated image to the control device 7.

[0028] The display unit 42 is composed of a display using liquid crystal or organic EL (Electro Luminescence), and is positioned opposite at least one of the left or right eyes of the trainee wearing the head-mounted display 4. The display unit 42 then displays various images under the control of the control device 7.

[0029] Multiple infrared output units 43 are used to determine the position (3D position) and orientation (the gaze direction of the trainee wearing the head-mounted display 4) of the head-mounted display 4. These multiple infrared output units 43 are each positioned at different locations and each emits (irradiates) infrared light.

[0030] Figures 3 and 4 show the configuration of the attachment device 5. In the following explanation of the configuration of the mounting device 5, we will use the XYZ coordinate axes, which are mutually orthogonal to each other: the X, Y, and Z axes. The Z axis is the axis that runs along the vertical direction in Figures 3 and 4. The mounting device 5 is used in the calibration process described later, which is performed by the control device 7, and is configured to be attachable to at least one of the multiple simulated operation units 31 (in this embodiment, the first and second simulated operation units 311 and 312 (Figure 2)). As shown in Figures 3 and 4, the mounting device 5 comprises a fitting portion 51, a grip portion 52, and multiple infrared output units 53.

[0031] As shown in Figures 3 and 4, the fitting portion 51 is formed in a generally frustoconical shape. The fitting portion 51 is also formed in an annular shape with a through hole 511 that penetrates linearly along the central axis (Z-axis) of the frustoconical shape. The first and second simulated operating portions 311 and 312 can be fitted into the interior of the fitting portion 51, respectively. In this fitting portion 51, the inner circumferential surface of the through hole 511 is provided with a first inclined surface 512 whose inner diameter increases toward the end face 51A on the +Z axis side, and a second inclined surface 513 whose inner diameter increases toward the end face 51B on the -Z axis side, as shown in Figure 3 or Figure 4. These first and second inclined surfaces 512 and 513 correspond to the inclined surfaces according to the present invention.

[0032] The grip portion 52 is the part that is held when carrying the attachment device 5, and as shown in Figures 3 and 4, it extends linearly toward the -Z axis from the -Z axis end face 51B of the fitting portion 51. As shown in Figure 3, the grip portion 52 is provided with a power button 521 for operating multiple infrared output units 53.

[0033] Multiple infrared output units 53 are used to detect the position (three-dimensional position) of the wearable device 5. These multiple infrared output units 53 are located at different positions on the fitting portion 51 and operate when the power is turned ON by operating the power button 521, and each emits (irradiates) infrared light.

[0034] The base station 6 is used to detect the position (3D position) and orientation (the gaze direction of the trainee wearing the head-mounted display 4) of the head-mounted display 4, as well as the position (3D position) of the wearable device 5. The base station 6 is also connected to the control device 7 wirelessly or via a wired connection for communication. The base station 6 corresponds to the position detection unit according to the present invention. The wearable device 5 and the base station 6 correspond to the acquisition device 8 according to the present invention. The base station 6 is composed of two infrared cameras 61 that detect infrared rays output from multiple infrared output units 43 and multiple infrared output units 53. The number of infrared cameras 61 is not limited to two; any other number may be provided. The two infrared cameras 61 each output infrared image data generated by the capture to the control device 7.

[0035] Figure 5 is a block diagram showing the configuration of the control device 7. For the sake of explanation, only one simulated operation unit 31, infrared output unit 43, and infrared camera 61 are shown in Figure 5. The control device 7 controls the operation of the entire simulation device 1. As shown in Figure 5, the control device 7 comprises an input unit 71, a storage unit 72, and a control unit 73. The input unit 71 consists of buttons, switches, touch panels, etc., that accept user operations by trainers, etc., and outputs signals corresponding to such user operations to the control unit 73. The memory unit 72 stores various programs executed by the control unit 73, as well as data necessary for the control unit 73 to perform processing. Here, the data necessary for the control unit 73 to perform processing includes, for example, data of a 3D model (virtual model) corresponding to the drilling machine 100. This 3D model is, for example, a 3D model generated by CAD (Computer-Aided Design) software, and the position and angle of each part of the 3D model are associated in a virtual space coordinate system that is aligned with the real space coordinate system on which the simulation device 1 is installed.

[0036] The control unit 73 is implemented by a controller such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs stored in the memory unit 72, and controls the operation of the entire simulation device 1. The control unit 73 is not limited to a CPU or MPU, but may also be composed of integrated circuits such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array). As shown in Figure 5, the control unit 73 includes a mode switching unit 731, a first image acquisition unit 732, a second image acquisition unit 733, a first calculation unit 734, a display image generation unit 735, a second calculation unit 736, and a calibration processing unit 737.

[0037] The mode switching unit 731 switches the simulation device 1 between normal mode and calibration mode in response to user operation on the input unit 71. Here, the normal mode is a mode in which the simulation device 1 allows the trainee to acquire skills in operating the drilling machine 100. The calibration mode is a mode in which the calibration processing unit 737 performs the calibration process described later.

[0038] The first image acquisition unit 732 acquires the data of the captured image generated by the imaging unit 41. The second image acquisition unit 733 acquires the data of the infrared image generated by each of the two infrared cameras 61.

[0039] The first calculation unit 734 calculates the position and orientation of the head-mounted display 4 (the direction of the trainee's gaze while wearing the head-mounted display 4) when the simulation device 1 is in normal mode. Specifically, the first calculation unit 734 recognizes the arrival time and angle of infrared light that has reached each infrared camera 61 from the multiple infrared output units 43 based on the data of each infrared image acquired by the second image acquisition unit 733. Then, the first calculation unit 734 calculates the position and orientation of the head-mounted display 4 (the direction of the trainee's gaze while wearing the head-mounted display 4) from the arrival time and angle.

[0040] The display image generation unit 735 generates a display image to be displayed on the display unit 42 when the simulation device 1 is in normal mode. Specifically, the display image generation unit 735 recognizes the position and orientation of the head-mounted display 4 (the direction of the trainee's gaze while wearing the head-mounted display 4) calculated by the first calculation unit 734. Then, based on the 3D model data stored in the memory unit 72, the display image generation unit 735 generates an image of the 3D model (hereinafter referred to as the model image) that is recognized from the position of the head-mounted display 4 to the direction of the trainee's gaze while wearing the head-mounted display 4. The display image generation unit 735 also generates a display image by superimposing the model image with the data (captured image) acquired by the first image acquisition unit 732, and displays the display image on the display unit 42. Furthermore, the display image generation unit 735 simulates the operation of the 3D model corresponding to the simulated operation unit 31 operated by the trainee wearing the head-mounted display 4 in accordance with the operation of the simulated operation unit 31. Furthermore, the displayed image is not limited to an image created by superimposing a model image and a captured image; it may also be an image in which a portion of the field of view of the trainee wearing the head-mounted display 4 is composed of the model image, and the remaining area is composed of the captured image.

[0041] The second calculation unit 736 calculates the position of the mounting device 5 in response to user operations on the acquisition operation unit 32 when the simulation device 1 is in calibration mode. Specifically, the second calculation unit 736 recognizes the arrival time and angle of infrared rays that have reached each infrared camera 61 from the multiple infrared output units 53 based on the data of each infrared image acquired by the second image acquisition unit 733. Then, the second calculation unit 736 calculates the position of the mounting device 5 from the arrival time and angle. For example, the center position of the fitting portion 51 can be cited as the position of the mounting device 5.

[0042] Incidentally, as mentioned above, the virtual spatial coordinate system of the 3D model of the drilling machine 100 stored in the memory unit 72 is set to match the real spatial coordinate system on which the simulation device 1 is installed. However, depending on the installation position of the simulation device 1, the virtual spatial coordinate system may deviate from the real spatial coordinate system. In such cases, the real image (captured image) corresponding to the trainee's field of view and the model image will be misaligned, causing the trainee, who views the real image and the model image through the head-mounted display 4, to feel a sense of unease. Then, when the simulation device 1 is in calibration mode, the calibration processing unit 737 performs a calibration process to align the virtual space coordinate system with the real space coordinate system. The details of the functions of the calibration processing unit 737 will be explained later in the "Calibration Method" section.

[0043] [Calibration method] Next, we will explain a calibration method for aligning a virtual space coordinate system with a real space coordinate system. Figure 6 is a flowchart showing the calibration method. In the following, it is assumed that the device 5 is powered ON by operating the power button 521. First, the mode switching unit 731 switches the simulation device 1 to calibration mode in response to user operation on the input unit 71 by the operator (step S1).

[0044] After step S1, the control unit 73 determines whether the termination flag is set to "ON" (step S2). Then, if the control unit 73 determines that the termination flag is set to "ON" (step S2: Yes), it terminates the calibration method flow and switches the simulation device 1 to normal mode. For example, if the termination flag is set to "ON" by a user operation on the input unit 71 after the simulation device 1 has been switched to calibration mode, or if the termination flag is set to "ON" in another parallel processing program, the calibration method flow will terminate.

[0045] After step S2, the second calculation unit 736 starts calculating the position of the attachment device 5 (step S3).

[0046] Here, the operator switches the simulation device 1 to calibration mode by operating the input unit 71, and then attaches the mounting device 5 to the first simulated operation unit 311. Figure 7 shows the mounting device 5 attached to the first simulated operating unit 311. Specifically, as shown in Figure 7, the operator positions the mounting device 5 with the grip portion 52 below the fitting portion 51, and then fits the tip of the lever on the first simulated operation unit 311 into the fitting portion 51. This attaches the mounting device 5 to the first simulated operation unit 311. After this, the operator operates the acquisition operation unit 32.

[0047] After step S3, the control unit 73 determines whether the position of the mounting device 5 calculated by the second calculation unit 736 is positioned to correspond to the first simulated operation unit 311 (step S4). Specifically, in step S4, the control unit 73 calculates the distance between the position of the first simulated operation unit 311 in a preset real-space coordinate system and the position of the attachment device 5 calculated by the second calculation unit 736. If this distance is within a preset threshold, the control unit 73 determines that the position of the attachment device 5 calculated by the second calculation unit 736 corresponds to the position of the first simulated operation unit 311 (step S4: Yes). In other words, if the operator attaches the attachment device 5 to the first simulated operation unit 311, the determination in step S4 is "Yes". On the other hand, if this distance exceeds a preset threshold, the control unit 73 determines that the position of the attachment device 5 calculated by the second calculation unit 736 does not correspond to the position of the first simulated operation unit 311 (step S4: No).

[0048] If "Yes" is determined in step S4, the control unit 73 determines whether or not the acquisition operation unit 32 has been operated (step S5). When it is determined that the acquisition operation unit 32 is not being operated (step S5: No), the control unit 73 returns to step S2. On the other hand, when it is determined that the acquisition operation unit 32 has been operated (step S5: Yes), the control unit 73 causes the storage unit 72 to store data indicating the position of the wearing device 5 calculated by the second calculation unit 736 at the timing when it is determined that the acquisition operation unit 32 has been operated (step S6). Hereinafter, for convenience of explanation, the position of the wearing device 5 based on the data stored in the storage unit 72 in step S6 (the position of the wearing device 5 calculated by the second calculation unit 736 in the state where the wearing device 5 is attached to the first simulation operation unit 311 (the state shown in FIG. 7)) is described as the first real space position. The first real space position corresponds to the position of the first simulation operation unit 311.

[0049] After step S6, the calibration processing unit 737 executes the following first calibration process (position calibration process) (step S7). FIG. 9 is a diagram for explaining the first calibration process (step S7). In FIG. 9, the portion shown by the solid line shows the training seat 2 and the simulation operation device 3 in the real space coordinate system in a plan view (a plan view seen along the Z-axis (vertical axis)). The portion shown by the alternate long and short dash line shows the virtual space coordinate system of the 3D model of the punching machine 100 stored in the storage unit 72. Further, P1(X P1 ,Y P1 ,Z P1 ) shown in FIG. 9 is the first real space position (three-dimensional position) corresponding to the position of the first simulation operation unit 311 in the real space coordinate system. Also, P1'(X P1 ',Y P1 ',Z P1 ') shown in FIG. 9 is the position of the 3D model corresponding to the first simulation operation unit 311 among the 3D models in the virtual space coordinate system stored in the storage unit 72 (hereinafter described as the first virtual space position (three-dimensional position)).

[0050] Specifically, in step S7, as shown by the arrow in FIG. 9, the calibration processing unit 737 sets the first real space position P1(X P1 ,Y P1 ,Z P1) at the first virtual space position P1'(X P1 ',Y P1 ',Z P1 The virtual spatial coordinate system of the 3D model of the drilling machine 100 stored in the memory unit 72 is moved to match the ').

[0051] After step S7, the calibration processing unit 737 sets the first calibration processing completion flag to "ON" (step S8). After this, the control unit 73 returns to step S2.

[0052] Here, the operator attaches the mounting device 5 to the first simulated operation unit 311 and performs an operation on the acquisition operation unit 32, and then attaches the mounting device 5 to the second simulated operation unit 312. Figure 8 shows the mounting device 5 attached to the second simulated operating unit 312. Specifically, as shown in Figure 8, the operator positions the mounting device 5 with the grip portion 52 above the fitting portion 51, and then fits the second simulated operating portion 312 into the fitting portion 51. This attaches the mounting device 5 to the second simulated operating portion 312. After this, the operator operates the acquisition operating portion 32 again.

[0053] On the other hand, if "No" is determined in step S4, the control unit 73 determines whether the position of the mounting device 5 calculated by the second calculation unit 736 is positioned to correspond to the second simulated operation unit 312 (step S9). Specifically, the control unit 73 calculates the distance between the position of the second simulated operation unit 312 in a preset real-space coordinate system and the position of the attachment device 5 calculated by the second calculation unit 736. If this distance is within a preset threshold, the control unit 73 determines that the position of the attachment device 5 calculated by the second calculation unit 736 corresponds to the position of the second simulated operation unit 312 (Step S9: Yes). In other words, if the operator attaches the attachment device 5 to the second simulated operation unit 312, the determination in Step S9 is "Yes". On the other hand, if this distance exceeds a preset threshold, the control unit 73 determines that the position of the attachment device 5 calculated by the second calculation unit 736 does not correspond to the position of the second simulated operation unit 312 (Step S9: No).

[0054] If the result in step S9 is "No", the control unit 73 returns to step S2. On the other hand, if "Yes" is determined in step S9, the control unit 73 determines whether or not the acquisition operation unit 32 has been operated (step S10). If the control unit 73 determines that the acquisition operation unit 32 has not been operated (step S10: No), the control unit 73 returns to step S2. On the other hand, if the control unit 73 determines that the acquisition operation unit 32 has been operated (step S10: Yes), the control unit 73 stores in the storage unit 72 data indicating the position of the attachment device 5 calculated by the second calculation unit 736 at the time it was determined that the acquisition operation unit 32 had been operated (step S11). For the sake of explanation, below, the position of the attachment device 5 based on the data stored in the storage unit 72 in step S11 (the position of the attachment device 5 calculated by the second calculation unit 736 when the attachment device 5 is attached to the second simulated operation unit 312 (the state shown in Figure 8)) will be referred to as the second real-space position. This second real-space position corresponds to the position of the second simulated operation unit 312.

[0055] After step S11, the calibration processing unit 737 determines whether the first calibration processing completion flag is set to "ON" (step S12).

[0056] If it is determined that the first calibration process completion flag is set to "OFF" (step S12: No), the control unit 73 returns to step S2. On the other hand, if it is determined that the first calibration process completion flag is set to "ON" (step S12: Yes), the calibration processing unit 737 executes the second calibration process (angle calibration process) as shown below (step S13). After this, the control unit 73 returns to step S2. In other words, the second calibration process is executed only if the first calibration process is completed.

[0057] Figure 10 is a diagram illustrating the second calibration process (step S13). Note that Figure 10 corresponds to Figure 9. Also, P2(X) shown in Figure 10... P2 ,Y P2 ,Z P2 ) is the second real-space position (3D position) which corresponds to the position of the second simulated operation unit 312 in the real-space coordinate system. Furthermore, P2'(X) shown in Figure 10 is the second real-space position (3D position). P2 ',Y P2 ',Z P2 ') is the position of the 3D model corresponding to the second simulated operation unit 312 among the 3D models in the virtual space coordinate system stored in the memory unit 72 (hereinafter referred to as the second virtual space position (3D position)).

[0058] Specifically, in step S13, the calibration processing unit 737, as indicated by the arrow in Figure 10, determines the second real space position P2(X P2 ,Y P2 ,Z P2 ) at the second virtual space position P2'(X P2 ',Y P2 ',Z P2 The first virtual space position P1'(X) matches P1 ',Y P1 ',Z P1 The virtual space coordinate system is rotated around the Z-axis (the axis perpendicular to the plane of the paper in Figure 10) with the ') as the center. This causes the 3D position of the 3D model in the virtual space coordinate system to match the 3D position of the corresponding part in the real space coordinate system, thus aligning the virtual space coordinate system with the real space coordinate system.

[0059] According to the embodiment described above, the following effects are achieved. The simulation apparatus 1 according to this embodiment includes an acquisition device 8 having a mounting device 5 configured to be attached to at least one of a plurality of simulated operation units 31, and a base station 6 for detecting the position of the mounting device 5. The control device 7 then performs a calibration process based on the position of the mounting device 5 detected by the base station 6 and the position of the simulated operation unit 31 to which the mounting device 5 is attached in the virtual space coordinate system. In other words, since the mounting device 5 is attached to the simulated operating unit 31, it is possible to avoid the mounting device 5 shifting away from the simulated operating unit 31, and the position of the simulated operating unit 31, which is the position of the mounting device 5, can be calculated with high accuracy. Therefore, according to the simulation device 1 of this embodiment, the position of the simulated operating unit 31 to which the mounting device 5 is attached can be calculated with high accuracy, and the virtual spatial coordinate system of the virtual model can be accurately aligned with the real spatial coordinate system.

[0060] Furthermore, in the simulation device 1 according to this embodiment, the mounting device 5 is provided with an annular fitting portion 51 into which the first and second simulated operating parts 311 and 312 can be fitted inside, respectively. Therefore, the mounting device 5 can be easily attached to the first and second simulated operation units 311 and 312, and the operator's work in the calibration method can be simplified. In particular, the fitting portion 51 is provided with first and second inclined surfaces 512 and 513. As a result, these first and second inclined surfaces 512 and 513 act as guide surfaces, allowing the first and second simulated operating portions 311 and 312 to be in contact with these first and second inclined surfaces 512 and 513 while being well guided into the through hole 511. Therefore, the mounting device 5 can be attached to the first and second simulated operating portions 311 and 312 even more easily.

[0061] Now, let's consider the case where the acquisition operation unit 32 is provided on the mounting device 5. In this case, there is a risk that the position of the mounting device 5 may shift in response to user operation on the acquisition operation unit 32. In contrast, in the simulation device 1 according to this embodiment, the acquisition operation unit 32 is provided in the simulated operation device 3. Therefore, the position of the mounting device 5 does not shift in response to user operation on the acquisition operation unit 32. Consequently, the first and second real-space positions P1 and P2 can be calculated with high accuracy, and the virtual space coordinate system of the virtual model can be aligned with the real-space coordinate system with even greater accuracy.

[0062] Furthermore, in the simulation device 1 according to this embodiment, the calibration process includes the first and second calibration processes described above. Therefore, it is possible to efficiently and accurately align the virtual spatial coordinate system of a virtual model with the real spatial coordinate system.

[0063] (Other embodiments) While embodiments for carrying out the present invention have been described so far, the present invention should not be limited to the embodiments described above. In the embodiment described above, a real image corresponding to the field of view of a trainee wearing the head-mounted display 4, which is a display device according to the present invention, was captured by the shooting unit 41, and the captured image obtained from the shooting was displayed on the head-mounted display 4. However, the embodiment is not limited to this. The real image corresponding to the field of view of the trainee may be made visible to the trainee through the head-mounted display 4, in other words, the head-mounted display 4 may be configured with a see-through structure.

[0064] In the above-described embodiment, a base station 6 was used as the position detection unit according to the present invention, but the invention is not limited to this. For example, infrared light may be emitted from the base station 6 and detected by the wearable device 5, that is, the position detection unit according to the present invention may be mounted on the wearable device 5.

[0065] By the way, if the virtual space coordinate system of the 3D model of the drilling machine 100 stored in the memory unit 72 deviates from the real space coordinate system, the amount of deviation will be greater in the rotational direction around the Z axis than in the rotational direction around the X axis and the rotational direction around the Y axis. For this reason, in the embodiment described above, in the second calibration process, the second real space position P2(X P2 ,Y P2 ,Z P2 ) at the second virtual space position P2'(X P2 ',Y P2 ',Z P2 The first virtual space position P1'(X) matches P1 ',Y P1 ',Z P1 The virtual space coordinate system was rotated around the Z-axis with (') as the center. However, in the second calibration process, the axis on which the virtual space coordinate system is rotated is not limited to the Z-axis; the X-axis or Y-axis may also be used. Furthermore, in the second calibration process, two or more axes from the X-axis, Y-axis, and Z-axis may be used as the axes on which the virtual space coordinate system is rotated.

[0066] In step S4 described above, when determining whether the mounting device 5 is positioned to correspond to the first simulated operating unit 311, the following method may be used. For example, in step S3, the second calculation unit 736 calculates the position of the device 5 as well as its orientation. Specifically, the second calculation unit 736 recognizes the arrival time and angle of infrared rays that reached each infrared camera 61 from the multiple infrared output units 53 based on the data of each infrared image acquired by the second image acquisition unit 733. Then, the second calculation unit 736 calculates the position and orientation of the device 5 from the arrival time and angle. The calibration processing unit 737 determines that the mounting device 5 is positioned to correspond to the first simulated operating unit 311 if the orientation of the mounting device 5 calculated by the second calculation unit 736 is the orientation shown in Figure 7 (the orientation in which the grip portion 52 is positioned below the fitting portion 51) (Step S4: Yes). On the other hand, the calibration processing unit 737 determines that the mounting device 5 is not positioned to correspond to the first simulated operating unit 311 if the orientation of the mounting device 5 calculated by the second calculation unit 736 is not the orientation shown in Figure 7 (Step S4: No).

[0067] Similarly, in step S9 described above, when determining whether the mounting device 5 is positioned to correspond to the second simulated operating unit 312, the following method may be used. For example, in step S3, the second calculation unit 736 calculates the position of the attachment device 5, as well as the orientation of the attachment device 5, as described above. The calibration processing unit 737 determines that the mounting device 5 is positioned to correspond to the second simulated operating unit 312 if the orientation of the mounting device 5 calculated by the second calculation unit 736 is the orientation shown in Figure 8 (the orientation in which the grip portion 52 is positioned above the fitting portion 51) (Step S9: Yes). On the other hand, the calibration processing unit 737 determines that the mounting device 5 is not positioned to correspond to the second simulated operating unit 312 if the orientation of the mounting device 5 calculated by the second calculation unit 736 is not the orientation shown in Figure 8 (Step S9: No). [Explanation of symbols]

[0068] 1. Simulation device 2 Training seat 3 Simulated operating device 4. Head-mounted display 5. Wearing device 6 Base Stations 7 Control device 8 Acquisition device 31 Simulation operation section 32 Acquisition operation section 41 Photography Department 42 Display section 43 Infrared output section 51 Fitting part 51A,51B End face 52 Grip section 53 Infrared output section 61 Infrared Camera 71 Input section 72 Memory section 73 Control Unit 100 Drilling Machine 110 Bogie 111 Trucks 120 Boom 121 Swivel axis 122 Relief axis 130 rock-cutting units 131 Guide Shell 132 Rock drilling machine 133 Tools 140 Cockpit 141 Driver's seat 311 First Simulated Operating Unit 312 Second Simulated Operating Unit 511 Through hole 512 First Inclined Surface 513 Second Inclined Surface 521 Power button 731 Mode switching section 732 First image acquisition unit 733 Second image acquisition unit 734 First calculation unit 735 Display Image Generation Unit 736 Second calculation unit 737 Calibration Processing Unit P1 First real-space position P1' First virtual space position P2 Second real-space position P2' Second virtual space position

Claims

1. A simulated operating device having multiple simulated operating parts that are provided on a drilling machine and each simulated reproduces one of the multiple operating parts used to operate the drilling machine, A display device worn by a user, which allows the user to see a real image corresponding to the user's field of vision and a predetermined image; A control device that displays an image of a virtual model corresponding to the drilling machine on the display device, and simulates the operation of a part of the virtual model corresponding to the simulated operation unit operated by the user in response to the operation of the simulated operation unit, The system includes an acquisition device that acquires information necessary for a calibration process to align the virtual spatial coordinate system in the virtual model with the real spatial coordinate system, The acquisition device is, A mounting device having an annular fitting portion into which at least one of the plurality of simulated operating parts can be fitted, The system includes a position detection unit that detects the position of the aforementioned mounting device, The control device is A simulation device that performs the calibration process based on the position of the mounting device detected by the position detection unit and the position of the simulated operating unit on which the mounting device is attached in the virtual space coordinate system.

2. The inner circumferential surface of the fitting portion is, The simulation device according to claim 1, wherein the fitting portion is provided with an inclined surface that increases in inner diameter towards the end face in the direction along the central axis.

3. The aforementioned simulated operation device is It further includes an acquisition operation unit that accepts user input, The control device is The simulation apparatus according to claim 1 or 2, which acquires the position of the mounting device detected by the position detection unit in response to a user operation on the acquisition operation unit.

4. The aforementioned calibration process is A first calibration process is performed to align the virtual space coordinate system with the real space coordinate system based on the position of the mounting device detected by the position detection unit while it is attached to the first of the plurality of simulated operating units, and the position of the first simulated operating unit in the virtual space coordinate system. The simulation apparatus according to claim 1, further comprising: a second calibration process that aligns the virtual space coordinate system with the real space coordinate system based on the position of the mounting device detected by the position detection unit while it is mounted on a second mounting device, which is different from the first mounting device among the plurality of mounting devices, and the position of the second mounting device in the virtual space coordinate system.

5. In the first calibration process described above, The virtual space coordinate system is moved so that the position of the first simulated operating unit in the virtual space coordinate system matches the position of the mounting device detected by the position detection unit while the mounting device is attached to the first simulated operating unit. In the second calibration process described above, The simulation apparatus according to claim 4, which rotates the virtual space coordinate system around a predetermined axis with respect to the position of the first simulated operation unit in the virtual space coordinate system, based on the position of the mounting device detected by the position detection unit while it is mounted on the second simulated operation unit and the position of the second simulated operation unit in the virtual space coordinate system.

Images