Information processing system, information processing device, information processing method, and program

Abstract

To smoothly carry out construction.SOLUTION: An information processing system comprises: a transmission type display 100; a position information acquisition unit 210 for acquiring position information indicating a position of a heavy machine operated by an operator; an input unit 220 for inputting three-dimensional drawing information, from the outside; a calculation unit 230 for calculating a position where the transmission type display 100 is caused to display a three-dimensional drawing indicated by the three-dimensional drawing information inputted by the input unit 220 on the basis of the position information acquired by the position information acquisition unit 210; and a display control unit 240 for displaying the three-dimensional drawing indicated by the three-dimensional drawing information at the position calculated by the calculation unit 230, in the transmission type display 100. The transmission type display 100 is arranged at a position where the operator can visually recognize an object subjected to construction through the transmission type display 100.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing device, an information processing method, and a program. [Background technology]

[0002] Generally, when a worker uses heavy machinery or the like to carry out a specified construction work on an object, the worker carries out the work while looking at a construction drawing. In addition, a technique has been devised in which guidance according to the construction situation is displayed on a monitor device installed near the driver's seat where the worker operates the machine, and the worker carries out the work according to the guidance displayed on the monitor device (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-136985 A Summary of the Invention [Problem to be solved by the invention]

[0004] When a worker performs construction while looking at a construction drawing, the worker must judge for himself whether the actual construction is proceeding according to the construction drawing. Therefore, even if the technology described in Patent Document 1 is used, the worker must temporarily stop construction in order to look at the guidance displayed on the monitor device. Therefore, there is a problem that it is difficult to perform construction smoothly.

[0005] An object of the present invention is to provide an information processing system, an information processing device, an information processing method, and a program that enable smooth construction work. [Means for solving the problem]

[0006] The information processing system of the present invention comprises: A transmissive display; a position information acquisition unit that acquires position information indicating a position of the heavy equipment operated by the worker; An input unit for inputting three-dimensional drawing information of a construction target from an external device; a calculation unit that calculates a position at which the three-dimensional drawing indicated by the three-dimensional drawing information input by the input unit is to be displayed on the transmissive display based on the position information acquired by the position information acquisition unit; a display control unit that displays the three-dimensional drawing indicated by the three-dimensional drawing information at the position calculated by the calculation unit of the transmissive display, The transmissive display is disposed at a position where the worker can visually recognize an object to be worked on through the transmissive display.

[0007] The information processing device of the present invention comprises: a position information acquisition unit that acquires position information indicating a position of the heavy equipment operated by the worker; An input unit for inputting three-dimensional drawing information of a construction target from an external device; A calculation unit that calculates a position for displaying the three-dimensional drawing indicated by the three-dimensional drawing information input by the input unit on a transparent display mounted on the heavy equipment, the transparent display being arranged at a position where the worker can view the target to be constructed through the transparent display mounted on the heavy equipment, based on the position information acquired by the position information acquisition unit; and a display control unit that displays the three-dimensional drawing indicated by the three-dimensional drawing information at the position calculated by the calculation unit on the transmissive display.

[0008] The information processing method of the present invention further comprises: A process of acquiring location information indicating a location of a heavy machine operated by a worker; A process of inputting three-dimensional drawing information of the construction target from an external source; A process of calculating a position for displaying the three-dimensional drawing indicated by the input three-dimensional drawing information on a transparent display mounted on the heavy equipment, the transparent display being arranged at a position where the worker can visually recognize the target to be constructed through the transparent display mounted on the heavy equipment, based on the acquired position information; and displaying the three-dimensional drawing indicated by the three-dimensional drawing information at the calculated position on the transmissive display.

[0009] In addition, the program of the present invention is A program executed by a computer, On the computer, A step of acquiring location information indicating a location of a heavy machine operated by a worker; A procedure for inputting 3D drawing information of the construction target from outside; A step of calculating a position where the three-dimensional drawing indicated by the input three-dimensional drawing information is displayed on a transparent display mounted on the heavy equipment, the transparent display being arranged at a position where the worker can visually recognize the target to be constructed through the transparent display mounted on the heavy equipment, based on the acquired position information; and a procedure of displaying the three-dimensional drawing indicated by the three-dimensional drawing information at the calculated position on the transmissive display. Effect of the Invention

[0010] In the present invention, construction can be carried out smoothly. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a first embodiment of an information processing system according to the present invention. [Diagram 2] 2 is a diagram illustrating an example of an internal configuration of the information processing device illustrated in FIG. 1. [Diagram 3] 2 is a flowchart illustrating an example of an information processing method in the information processing system shown in FIG. 1. [Figure 4] FIG. 11 is a diagram illustrating a second embodiment of an information processing system according to the present invention. [Diagram 5] 5 is a diagram showing an example of an internal configuration of the information processing device shown in FIG. 4 and a connection with components other than the information processing device. [Figure 6] 6 is a diagram showing an example of how the camera shown in FIG. 5 captures an image of a marker. [Figure 7] 11 is a diagram for explaining an example of a method for correcting a measured position of a heavy machine. FIG. [Figure 8]13 is a diagram showing an example of measuring the direction from a moving heavy equipment to a marker and the distance between the heavy equipment and the marker. FIG. [Figure 9] FIG. 13 is a diagram showing an example of a state in which there is an angle between a horizontal plane and an actual construction surface. [Figure 10] 1A to 1C are diagrams illustrating examples of lines of sight when a construction target area is viewed from viewpoints at mutually different positions. [Figure 11] 5 is a diagram showing an example of an AR display displayed by the transmissive display shown in FIG. 4. [Figure 12] 5 is a flowchart illustrating an example of an information processing method in the information processing system shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. (First embodiment)

[0013] Fig. 1 is a diagram showing a first embodiment of an information processing system of the present invention. As shown in Fig. 1, the information processing system in this embodiment has a transmissive display 100 and an information processing device 200. The transmissive display 100 and the information processing device 200 can exchange information with each other. In other words, the transmissive display 100 and the information processing device 200 are connected to each other by wire or wirelessly.

[0014] The transparent display 100 is a display capable of displaying AR (Augmented Reality) with a structure that allows information (screen) to be displayed on the display while allowing the other side of the display to be seen through. The transparent display 100 is disposed at a position where a worker who uses heavy machinery to perform work can view the target to be constructed through the transparent display 100. For example, when a worker who uses heavy machinery to perform work gets into the driver's seat of the heavy machinery to perform work, the transparent display 100 may be disposed along the windshield of the heavy machinery on the worker's side of the windshield. If the transparent member that performs AR display is a goggle-type member worn by the worker, there is a concern that the safety of the worker wearing the goggle-type member may be reduced. For this reason, in the present invention, the transparent display that performs AR display is disposed in front of the heavy machinery, particularly along the windshield. This makes it easier for the worker to check the surrounding situation without only viewing the AR display, and thus prevents safety from being reduced.

[0015] The information processing device 200 displays a predetermined screen on a transmissive display. Fig. 2 is a diagram showing an example of the internal configuration of the information processing device 200 shown in Fig. 1. As shown in Fig. 2, the information processing device 200 shown in Fig. 1 has a position information acquisition unit 210, an input unit 220, a calculation unit 230, and a display control unit 240. Note that Fig. 2 shows only main components related to this embodiment among the components included in the information processing device 200 shown in Fig. 1.

[0016] The position information acquisition unit 210 acquires position information indicating the position of the heavy machinery operated by the worker. The position information acquisition unit 210 acquires the position information using a Global Navigation Satellite System (GNSS). The position information acquisition unit 210 acquires the position information using an antenna mounted on the heavy machinery. The position information acquired by the position information acquisition unit 210 may be information indicating a position in a three-dimensional space using three-dimensional coordinates.

[0017] The input unit 220 inputs three-dimensional drawing information of the construction target from the outside. The input unit 220 inputs the three-dimensional drawing information based on an operation received from the outside. The input unit 220 may use software for reading a predetermined drawing, or may use a general mechanism for inputting information from the outside, such as a keyboard, a mouse, or a touch panel. The three-dimensional drawing information input by the input unit 220 is information showing a construction drawing, and includes information showing each dimension and position of the construction target.

[0018] The calculation unit 230 calculates a position where the three-dimensional drawing indicated by the three-dimensional drawing information input by the input unit 220 is displayed on the transmissive display 100 based on the position information acquired by the position information acquisition unit 210. That is, the calculation unit 230 calculates a position on the transmissive display 100 where the three-dimensional drawing is displayed so that an object seen by the worker through the transmissive display 100 and the object on the three-dimensional drawing displayed on the transmissive display 100 overlap. The position calculated by the calculation unit 230 in this way is a position where the object on the three-dimensional drawing displayed by the transmissive display 100 and the object seen by the worker through the transmissive display 100 overlap when the transmissive display 100 displays the three-dimensional drawing.

[0019] The display control unit 240 displays the three-dimensional drawing indicated by the three-dimensional drawing information at the position calculated by the calculation unit 230 on the transmissive display 100. By the display control unit 240 controlling the display of the three-dimensional drawing in this manner, the worker can see the object in the three-dimensional drawing and the object seen through the transmissive display 100 superimposed on each other.

[0020] The following describes an information processing method in the information processing system shown in Fig. 1. Fig. 3 is a flowchart for explaining an example of the information processing method in the information processing system shown in Fig. 1.

[0021] First, the position information acquisition unit 210 acquires position information indicating the position of the heavy equipment operated by the worker (step S1). The input unit 220 inputs three-dimensional drawing information from the outside (step S2). Here, either the process of step S1 or the process of step S2 may be performed first. Then, the calculation unit 230 calculates the position at which the three-dimensional drawing indicated by the three-dimensional drawing information input by the input unit 220 is to be displayed on the transmissive display 100 based on the position information acquired by the position information acquisition unit 210 (step S3). Then, the display control unit 240 displays the three-dimensional drawing indicated by the three-dimensional drawing information on the transmissive display 100 at the position calculated by the calculation unit 230 (step S4).

[0022] In this way, the construction drawing is displayed on the transparent display 100, and the worker can check the actual construction object and the object on the construction drawing by overlapping them. This allows the worker to check the construction drawing on the transparent display 100 while performing the actual work, and construction can be carried out smoothly. (Second embodiment)

[0023] Fig. 4 is a diagram showing a second embodiment of the information processing system of the present invention. As shown in Fig. 4, the information processing system in this embodiment includes a transmissive display 100, an information processing device 201, a camera 301, a direction calculation unit 311, a distance measurement unit 321, a marker 401, an acceleration sensor 501, a viewpoint position calculation unit 601, and a detection unit 701. The transmissive display 100 is the same as that in the first embodiment. Each component will be described with reference to Fig. 5.

[0024] Fig. 5 is a diagram showing an example of an internal configuration of the information processing device 201 shown in Fig. 4 and a connection with components other than the information processing device 201. As shown in Fig. 5, the information processing device 201 shown in Fig. 4 has a position information acquisition unit 210, an input unit 220, a calculation unit 231, and a display control unit 241. The position information acquisition unit 210 and the input unit 220 are the same as those in the first embodiment. Moreover, Fig. 5 shows only main components related to this embodiment among the components included in the information processing device 201 shown in Fig. 4.

[0025] Marker 401 is a member placed at a predetermined position. Marker 401 may be anything that can be imaged by camera 301 and whose placement (installation) position can be recognized, and may be, for example, an LED marker that uses an LED (Light Emitting Diode).

[0026] The camera 301 captures an image of the surroundings centered on the heavy equipment. The camera 301 captures still images and videos at a predetermined timing. In particular, the camera 301 captures an image of a marker 401. The camera 301 is mounted on, for example, a heavy equipment used by a worker for work. It is preferable that the camera 301 is capable of capturing an image of 360 degrees.

[0027] Fig. 6 is a diagram showing an example of how camera 301 shown in Fig. 5 captures marker 401. In the example shown in Fig. 6, there are two markers 401, marker 401-1 and marker 401-2. As shown in Fig. 6, camera 301 is installed in a position where it can capture images of markers 401-1 and 401-2. The reason why camera 301 captures images of markers 401-1 and 401-2 will be described below.

[0028] FIG. 7 is a diagram for explaining an example of a method for correcting the measured position of heavy machinery. The GNSS antenna shown in FIG. 7 is one of the components constituting the position information acquisition unit 210, and is mounted on the heavy machinery. As shown in FIG. 7, a slight error occurs between the position of the heavy machinery measured using the GNSS and the actual position of the heavy machinery in the system. This is because a certain range of error is allowed in the GNSS. Therefore, if the position indicated by the position information acquired by the position information acquisition unit 210 is used as the position of the heavy machinery and processing is performed as it is, a deviation occurs from the actual position of the heavy machinery, and more accurate processing (AR display) cannot be performed. Therefore, a camera 301 mounted on the heavy machinery captures an image of a marker 401 installed at a predetermined position, a relative position of the marker 401 with respect to the camera 301 is calculated, and the position information acquired by the position information acquisition unit 210 is corrected using the calculated value. Specifically, a direction calculation unit 311 and a distance measurement unit 321 are used for this correction.

[0029] The direction calculation unit 311 calculates the direction of the marker captured by the camera 301 relative to the camera 301. The direction calculation unit 311 calculates the imaging direction of the camera 301 relative to the direction in which the heavy equipment faces. At this time, the direction calculation unit 311 calculates the imaging direction of the camera 301 relative to the facing direction of the heavy equipment, assuming that the moving direction of the heavy equipment is the facing direction of the heavy equipment.

[0030] Distance measurement unit 321 measures the distance from camera 301 to marker 401 captured by camera 301. In this case, distance measurement unit 321 may measure the distance using, for example, a laser radar. Note that this system may not include distance measurement unit 321.

[0031] FIG. 8 is a diagram showing an example of how the direction from a moving heavy machine to a marker and the distance between the heavy machine and the marker are measured. As shown in FIG. 8, when the heavy machine moves from position A to position B, the calculation unit 231 calculates the direction (facing direction) of the heavy machine based on the movement. For example, if the three-dimensional coordinates of position A are (x0, y0, z0) and the three-dimensional coordinates of position B are (x1, y1, z1), the components of the vector of the facing direction when moving from position A to position B are (x1-x0, y1-y0, z1-z0). The calculation unit 231 regards this direction as the facing direction of the heavy machine at that time. The direction calculation unit 311 calculates the directions of the markers 401-1 and 401-2 captured by the camera 301 with respect to the facing direction of the heavy machine calculated by the calculation unit 231. In addition, the distance measurement unit 321 measures the distance between the moving heavy machine and each of the markers 401-1 and 401-2.

[0032] The acceleration sensor 501 measures the inclination of the heavy equipment relative to a horizontal plane. Specifically, the acceleration sensor 501 is mounted on the heavy equipment to measure the gravitational acceleration, calculates the inclination in the vertical direction based on the measured value, and measures the inclination in the horizontal direction (the inclination of the construction surface), i.e., the inclination of the heavy equipment, based on the calculated inclination in the vertical direction.

[0033] FIG. 9 is a diagram showing an example of a state when there is an angle between the horizontal plane and the actual construction surface. As shown in FIG. 9, the position that can be acquired by the position information acquisition unit 210 is a position on the horizontal plane. Therefore, in a conventional system that detects a position using only the position information acquisition unit 210, when there is an angle between the actual construction surface and the horizontal plane, a difference occurs between the coordinate position of the target on the system and the position where the target is actually seen by the worker. As a result, the accurate position of the target cannot be recognized due to the inclination of the construction surface. Therefore, the inclination of the heavy equipment is measured using the acceleration sensor 501, and the measurement result is used.

[0034] The viewpoint position calculation unit 601 calculates the relative position of the viewpoint of the operator with respect to the transmissive display 100. The viewpoint position calculation unit 601 may be attached to the transmissive display 100.

[0035] Fig. 10 is a diagram showing an example of the line of sight when the construction target area is viewed from viewpoints at different positions. As shown in Fig. 10, the line of sight positions on the transparent display 100 are different when the construction target area is viewed from viewpoint A through the transparent display 100 and when the construction target area is viewed from viewpoint B through the transparent display 100. In other words, if the display position of the AR image displayed on the transparent display 100 is not determined according to the viewpoint, the overlap between the image displayed on the transparent display 100 and the construction target area visible on the other side will not be as desired.

[0036] The detection unit 701 detects the status of the construction. The detection unit 701 detects whether the construction has been completed for the construction target area. For example, the detection unit 701 may detect whether a heavy machine has performed an operation for performing a predetermined construction. More specifically, when the construction involves compaction using a vibrating roller, the detection unit 701 may detect whether a heavy machine has performed an operation for compaction.

[0037] The calculation unit 231 has the following functions in addition to the functions of the calculation unit 230 in the first embodiment. The calculation unit 231 calculates the orientation of the heavy equipment based on the position information acquired by the position information acquisition unit 210 in association with the movement of the heavy equipment. A specific method for calculating the orientation of the heavy equipment is as described with reference to FIG. 8. Then, the calculation unit 231 calculates the position at which the three-dimensional drawing is to be displayed on the transmissive display 100 based on the calculated orientation of the heavy equipment and the position information acquired by the position information acquisition unit 210. Furthermore, the calculation unit 231 calculates the positions at which the markers 401-1 and 401-2 are installed based on the calculated orientation of the heavy equipment, the directions of the markers 401-1 and 401-2 relative to the camera 301 calculated by the direction calculation unit 311, and the distances between the heavy equipment and the markers 401-1 and 401-2 measured by the distance measurement unit 321. Then, the calculation unit 231 corrects the position of the heavy equipment acquired by the position information acquisition unit 210 based on the difference between the calculated positions of the markers 401-1 and 401-2 and the positions of the markers 401-1 and 401-2 that are set (stored) in advance. Furthermore, the calculation unit 231 corrects the position information using the tilt measured by the acceleration sensor 501. Furthermore, the calculation unit 231 corrects the position information using the position of the viewpoint calculated by the viewpoint position calculation unit 601. Then, the calculation unit 231 calculates the position at which the three-dimensional drawing is to be displayed on the transmissive display 100 based on the corrected position information. Note that, if the present system does not include the distance measurement unit 321, the calculation unit 231 calculates the positions at which the markers 401-1 and 401-2 are installed based on the calculated orientation of the heavy equipment and the directions of the markers 401-1 and 401-2 relative to the camera 301 calculated by the direction calculation unit 311.

[0038] The display control unit 241 has the following functions in addition to the functions of the display control unit 240 in the first embodiment. The display control unit 241 changes the display mode of the three-dimensional drawing displayed on the transmissive display 100 based on the construction status detected by the detection unit 701. For example, the display control unit 241 may color in a predetermined color a portion where the detection unit 701 detects that construction has been completed.

[0039] 11 is a diagram showing an example of an AR display displayed by the transmissive display 100 shown in FIG. 4. As shown in FIG. 11, the display control unit 241 causes the transmissive display 100 to display the area to be worked on from the three-dimensional drawing surrounded by a dashed line, and causes the transmissive display 100 to display the area of ​​the work target that the detection unit 701 detects as being worked on in color. Note that the display control unit 241 may use a method other than coloring as long as it is a display mode that allows the area that the detection unit 701 detects as being worked on to be distinguished from other areas. By the transmissive display 100 performing such a display, the worker can recognize the area where work has been completed and the area where work has not been completed while looking at the transmissive display 100 through which the work area is seen.

[0040] In addition, the heavy machinery may be equipped with a function to turn off the AR display of the transmissive display 100 in an emergency, or a function to prevent the mounted mechanism from malfunctioning due to the vibration of the heavy machinery. An abnormality detection mechanism may be provided to detect an abnormality in the operation of the heavy machinery, and when the abnormality detection mechanism detects an abnormality in the heavy machinery, the AR display of the transmissive display 100 may be turned off. The timing to turn off the AR display of the transmissive display 100 may be according to the level of the abnormality detected by the abnormality detection mechanism. For example, if the level of the abnormality is high, the AR display may be turned off immediately when the abnormality is detected, and if the level of the abnormality is low, the AR display may be turned off depending on the number of times the abnormality is detected. In addition, a vibration-proof rubber may be used as a function to prevent the mounted mechanism from malfunctioning due to the vibration of the heavy machinery. For example, the vibration-proof rubber may be installed on the transmissive display 100 or the mounting portion of the transmissive display 100 to reduce the vibration that the heavy machinery applies to the transmissive display 100.

[0041] The following describes an information processing method in the information processing system shown in Fig. 4. Fig. 12 is a flowchart for explaining an example of the information processing method in the information processing system shown in Fig. 4.

[0042] First, the input unit 220 inputs three-dimensional drawing information from the outside (step S11). The position information acquisition unit 210 acquires position information indicating the position of the heavy equipment operated by the worker (step S12). Here, either the process of step S11 or the process of step S12 may be performed first. Next, the acceleration sensor 501 measures the inclination of the heavy equipment with respect to the horizontal plane (step S13). The viewpoint position calculation unit 601 calculates the relative position of the viewpoint of the worker with respect to the transmissive display 100 (step S14). Here, either the process of step S13 or the process of step S14 may be performed first. Then, the calculation unit 231 calculates the position at which the three-dimensional drawing indicated by the three-dimensional drawing information input by the input unit 220 is to be displayed on the transmissive display 100 based on the position information acquired by the position information acquisition unit 210. Furthermore, the calculation unit 231 corrects the calculated position based on the inclination of the heavy machine measured by the acceleration sensor 501 and the position of the viewpoint calculated by the viewpoint position calculation unit 601 to create an AR model (step S15). The display control unit 241 causes the transmissive display 100 to display the AR model calculated by the calculation unit 231 (step S16). This AR model is a three-dimensional drawing indicated by the three-dimensional drawing information displayed on the transmissive display 100 at the position corrected by the calculation unit 230.

[0043] Thereafter, when the heavy equipment moves (step S17), the position information acquisition unit 210 further acquires position information indicating the position of the heavy equipment (step S18). Next, the calculation unit 231 calculates the orientation of the heavy equipment based on the position information acquired by the position information acquisition unit 210 as the heavy equipment moves. Furthermore, the calculation unit 231 corrects the position information using the direction calculated by the direction calculation unit 311 based on the calculated orientation of the heavy equipment and the distance measured by the distance measurement unit 321 (step S19). Then, the process of step S13 is performed. Note that if the present system does not include the distance measurement unit 321, in step S19, the calculation unit 231 corrects the position information using the direction calculated by the direction calculation unit 311 based on the calculated orientation of the heavy equipment.

[0044] In this way, the construction drawing is displayed on the transparent display 100 in AR, so that the worker can check the actual construction object and the object of the construction drawing by overlapping them. Therefore, the worker who operates the heavy equipment can check the construction status while facing forward in the driver's seat, and the work does not have to be temporarily stopped due to checking the construction area on the system management screen in a typical rolling compaction system. This enables continuous construction and improves the construction speed. In addition, by displaying the completed area so that it can be distinguished from the area where construction has not yet been performed, it is possible to prevent construction omissions in the unlikely event of an accident. Furthermore, the display position of the construction drawing is calculated using not only the position acquired by the position information acquisition unit 210, but also the relative position and direction with respect to the marker 401 captured by the camera 301 mounted on the heavy equipment, the inclination of the construction surface, and the position of the worker's viewpoint relative to the transparent display 100, and the AR display is performed. Therefore, more accurate AR display can be performed. This allows construction to be performed smoothly.

[0045] Although the above description has been given by allocating each function (process) to each component, this allocation is not limited to the above. Furthermore, the configuration of the components is not limited to the above-described form, and the above-described form is merely an example.

[0046] The processing performed by the above-mentioned information processing devices 200, 201 and other components (hereinafter referred to as information processing devices) may be performed by logic circuits created according to the purpose. Also, a computer program (hereinafter referred to as a program) in which the processing contents are described as procedures may be recorded on a recording medium readable by the information processing device, and the program recorded on this recording medium may be read and executed by the information processing device. The recording medium readable by the information processing device refers to a removable recording medium such as a floppy (registered trademark) disk, a magneto-optical disk, a DVD (Digital Versatile Disc), a CD (Compact Disc), a Blu-ray (registered trademark) Disc, and a USB (Universal Serial Bus) memory, as well as a memory such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a HDD (Hard Disc Drive) built into the information processing device. The program recorded on this recording medium is read by a CPU provided in the information processing device, and the same processing as that described above is performed under the control of the CPU. Here, the CPU operates as a computer that executes the program read from the recording medium on which the program is recorded. [Explanation of symbols]

[0047] 10 Heavy Machinery 100 Transparent display 200, 201 Information processing device 210 Location information acquisition unit 220 Input section 230,231 Calculation section 240,241 Display control unit 301 Camera 311 Direction Calculation Unit 321 Distance measurement unit 401, 401-1, 401-2 Marker 501 Acceleration Sensor 601 Viewpoint position calculation unit 701 Detection unit

Claims

1. A transmissive display; a position information acquisition unit that acquires position information indicating a position of the heavy equipment operated by the worker; An input unit for inputting three-dimensional drawing information of a construction target from an external device; a calculation unit that calculates an orientation of the heavy equipment based on position information acquired by the position information acquisition unit in association with the movement of the heavy equipment, and calculates a position for displaying the three-dimensional drawing indicated by the three-dimensional drawing information input by the input unit on the transmissive display based on the calculated orientation and the position information acquired by the position information acquisition unit; a display control unit that displays the three-dimensional drawing indicated by the three-dimensional drawing information at the position of the transmissive display calculated by the calculation unit; A camera that captures an image of the surroundings centered on the heavy machinery; A marker installed at a predetermined position; a direction calculation unit that calculates a direction of the marker captured by the camera relative to the camera based on the direction of the heavy equipment calculated by the calculation unit; A detection unit that detects the status of construction, The transmissive display is disposed at a position where the worker can visually recognize an object to be constructed through the transmissive display, The calculation unit corrects the position information acquired by the position information acquisition unit based on the direction calculated by the direction calculation unit and a predetermined position where the marker is installed, and calculates a position for displaying the three-dimensional drawing on the transmissive display based on the corrected position information; The display control unit is an information processing system that displays in color the portion where the detection unit detects that construction has been completed.

2. An acceleration sensor is provided to measure the inclination of the heavy machinery relative to a horizontal plane, 2. The information processing system according to claim 1, wherein the calculation unit corrects the position information acquired by the position information acquisition unit using the tilt measured by the acceleration sensor, and calculates a position at which the three-dimensional drawing is to be displayed on the transmissive display based on the corrected position information.

3. a viewpoint position calculation unit that calculates a position of a viewpoint of the operator with respect to the transmissive display, 3. The information processing system according to claim 1, wherein the calculation unit corrects the position information acquired by the position information acquisition unit using the viewpoint position calculated by the viewpoint position calculation unit, and calculates a position at which the three-dimensional drawing is to be displayed on the transmissive display based on the corrected position information.

4. a position information acquisition unit that acquires position information indicating a position of the heavy equipment operated by the worker; An input unit for inputting three-dimensional drawing information of a construction target from an external device; A calculation unit that calculates the orientation of the heavy equipment based on the position information acquired by the position information acquisition unit as the heavy equipment moves, and calculates a position for displaying the three-dimensional drawing indicated by the three-dimensional drawing information input by the input unit on a transparent display mounted on the heavy equipment, the transparent display being disposed at a position where the worker can view the target to be constructed through the transparent display; a display control unit that displays the three-dimensional drawing indicated by the three-dimensional drawing information at the position calculated by the calculation unit of the transmissive display, The calculation unit corrects the position information acquired by the position information acquisition unit based on a direction of a marker installed at a predetermined position captured by a camera that captures an image of the surroundings centered on the heavy machine, the direction being calculated based on the calculated orientation of the heavy machine, and a predetermined position at which the marker is installed, and calculates a position for displaying the three-dimensional drawing on the transmissive display based on the corrected position information; The display control unit is an information processing device that displays in color a portion where a detection unit that detects the status of the construction has detected that construction has been completed.

5. A process of acquiring location information indicating a location of a heavy machine operated by a worker; A process of inputting three-dimensional drawing information of the construction target from an external source; A process of calculating a direction of the heavy equipment based on position information acquired in association with the movement of the heavy equipment; A process of calculating a position for displaying the three-dimensional drawing indicated by the input three-dimensional drawing information on a transparent display mounted on the heavy equipment, the transparent display being disposed at a position where the worker can visually recognize the target to be constructed through the transparent display mounted on the heavy equipment, based on the calculated orientation and the acquired position information; A process of displaying the three-dimensional drawing indicated by the three-dimensional drawing information at the calculated position on the transmissive display; a process of correcting the acquired position information based on a direction of a marker installed at a predetermined position captured by a camera capturing an image of the surroundings centered on the heavy equipment, the direction being calculated based on the calculated orientation of the heavy equipment, and the predetermined position at which the marker is installed, and calculating a position for displaying the three-dimensional drawing on the transmissive display based on the corrected position information; and coloring the parts of the three-dimensional drawing that a detection unit that detects the status of construction detects as having been completed and displaying them on the transmissive display.

6. On the computer, A step of acquiring location information indicating a location of a heavy machine operated by a worker; A procedure for inputting 3D drawing information of the construction target from outside; A step of calculating a direction of the heavy equipment based on position information acquired in association with the movement of the heavy equipment; A step of calculating a position where the three-dimensional drawing indicated by the input three-dimensional drawing information is displayed on a transparent display mounted on the heavy equipment, the transparent display being arranged at a position where the worker can visually recognize the target to be constructed through the transparent display mounted on the heavy equipment, based on the calculated orientation and the acquired position information; a step of displaying the three-dimensional drawing indicated by the three-dimensional drawing information at the calculated position on the transmissive display; a step of correcting the acquired position information based on a direction of a marker installed at a predetermined position captured by a camera capturing an image of the surroundings centered on the heavy equipment, the direction being calculated based on the calculated orientation of the heavy equipment, and the predetermined position at which the marker is installed, and calculating a position for displaying the three-dimensional drawing on the transmissive display based on the corrected position information; and a procedure for displaying on the transmissive display a portion of the three-dimensional drawing that has been detected as completed by a detection unit that detects the status of construction.

Images