Remote Instruction System

Abstract

[Problem] To provide a system capable of projecting an appropriate instruction image from a remote location.　[Solution] A helmet worn by an on-site person in charge is provided with an imaging unit 12 and a projection unit 14 via a drive unit 16. An imaging direction of the imaging unit 12 and a projection direction of the projection unit 14 are detected by a sensor 28. A direction control means 20 controls the drive unit 16 on the basis of an output of the sensor 28, and maintains the directions of the imaging unit 12 and the projection unit 14 at prescribed directions centered on the on-site person in charge, regardless of motion of the on-site person in charge. A captured image from the imaging unit 12 of an on-site device 10 is transmitted to an instruction device 30 by a transmission unit 22 through control of a captured image transmission means 18. An instructor inputs an instruction image from an instruction image input unit 44 while watching an on-site reference captured image displayed on a captured image display unit 40. An instruction image transmission means 38 transmits the input instruction image to the on-site device 10 by means of a transmission unit 34. The on-site device 10 receives the instruction image by means of a reception unit 24, and projects the instruction image from the projection unit 14. Therefore, an instruction image 62 is projected on a target object 52.

Description

[Technical Field]

[0001] The present invention relates to a system for remotely projecting instruction images onto a site or for projecting pre-recorded instruction images onto a site. [Background technology]

[0002] Systems have been proposed for projecting instructions remotely onto a work site. For example, Patent Document 1 discloses a system in which an on-site worker wears a camera and a laser projector around his neck and projects annotations input by a remote instructor onto an object on the work site.

[0003] In this system, a camera worn by the worker captures an image of a marker placed near the target object, and the annotation is displayed in relation to the marker. Therefore, even if the worker moves and changes the orientation of the laser projector, the annotation is displayed in the correct position.

[0004] In this way, the above system makes it possible to accurately display annotations on-site from a remote location. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2019-5095 Summary of the Invention [Problem to be solved by the invention]

[0006] However, with the conventional technology described above, annotations are displayed in the correct position while the on-site worker is facing the object (i.e., while the marker is being captured by the camera), but if the worker turns their body in a different direction during work (when the marker is no longer being captured by the camera), the annotations will no longer be displayed.

[0007] As a result, if a worker on-site makes a large change in direction, the annotation disappears, and although the annotation reappears when the worker returns to the normal direction, this causes stress for the worker on-site.

[0008] Furthermore, when multiple people are working together, if the worker wearing the laser projector changes direction, the annotations disappear, making it impossible for the other workers to see the annotations.

[0009] The present invention aims to solve the above-mentioned problems and provide a system that allows an instruction image (annotation) to be continuously displayed even if the worker changes his / her direction significantly. [Means for solving the problem]

[0010] The following are some independently applicable features of the present invention.

[0011] (1) to (5) A remote instruction system according to the present invention is a remote instruction system including an instruction device used by an instructor and an on-site device used by an on-site person in charge, the on-site device comprises an imaging unit attached to the on-site person in charge or a mobile body, and imaging the on-site space to generate an image; an imaged image transmitting means for transmitting the imaged image to the instruction device by a transmitting means; a projection unit attached to the on-site person in charge or a mobile body, and projecting an instruction image into the on-site space based on given instruction image data; a drive unit for changing the imaging direction of the imaging unit and the projection direction of the projection unit; a direction control means for receiving outputs from sensors that detect the orientations of the imaging unit and the projection unit and controlling the drive unit so that the imaging unit and the projection unit face in a predetermined direction with the on-site person as the center, regardless of the movement of the on-site person in charge or the mobile body; and a correction means for correcting the projection of the instruction image by the projection unit without using the drive unit, based on a comparison of a characteristic portion image in the reference image when the fixation command was given and a characteristic portion image in the current image, using the image captured when the fixation command was given as a reference image, so that the instruction image is correctly displayed with reference to a predetermined portion of the on-site space; The instruction device is characterized by comprising an image receiving means for receiving the image transmitted by a receiving unit, an image display unit for displaying the received image, a fixed command means for issuing a fixed command to the on-site device by a transmitting unit so that an image of the desired site space is captured, an instruction image input unit for inputting an instruction image at a desired position in the site space by operation of the instructor in the displayed image, and an instruction image transmitting means for transmitting instruction image data that identifies the position of the instruction image to the on-site device by a transmitting unit so that the projection unit of the on-site device correctly projects the instruction image based on a characteristic partial image of the site space contained in the image.

[0012] Therefore, the instruction image data can be projected at the correct position regardless of the orientation of the field worker.

[0013] (6) The remote instruction system according to the present invention is characterized in that the imaging unit and the projection unit are fixed to the drive unit via a member that absorbs high-frequency vibrations.

[0014] Therefore, it is possible to obtain a captured image with less vibration and to project an instruction image with less vibration.

[0015] (7) The remote instruction system according to the present invention is characterized in that the imaging unit and the projection unit are fixed to the helmet of the on-site worker via the drive unit.

[0016] Therefore, the imaging unit and the projection unit can be stably fixed.

[0017] (8) The remote instruction system according to the present invention is characterized in that the direction control means changes the predetermined direction based on a direction instruction from the instruction device.

[0018] Therefore, the instructor can remotely control the imaging direction and check the situation at the site.

[0019] (9) In the remote instruction system according to the present invention, the characteristic part image is a marker provided in the site space or a characteristic point of the captured image.

[0020] Therefore, the instruction image can be displayed correctly based on the markers or feature points.

[0021] (10) The remote instruction system according to the present invention is characterized in that the drive unit has a multi-axis drive mechanism.

[0022] Therefore, the orientation of the imaging unit and the projection unit can be freely controlled.

[0023] (11)(12) The on-site instruction device of this invention comprises an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space to generate an imaged image; a projection unit that is attached to a site worker or a mobile body and projects an instruction image onto the site space based on given instruction image data; a drive unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit; a direction control means that receives output from a sensor that detects the orientation of the imaging unit and the projection unit and controls the drive unit so that the imaging unit and the projection unit face in a predetermined direction centered on the site worker regardless of the movement of the site worker or the mobile body; and a correction means that corrects the projection of the instruction image by the projection unit without using the drive unit, based on a characteristic partial image of the site space contained in the captured image, so that the instruction image is correctly displayed based on a predetermined part of the site space.

[0024] Therefore, the instruction image data can be projected at the correct position regardless of the orientation of the field worker.

[0025] (13) to (17) A remote instruction system according to the present invention is a remote instruction system including an instruction device used by an instructor and an on-site device used by an on-site person, the on-site device comprises an imaging unit attached to the on-site person in charge or the mobile body, and imaging the on-site space to generate an image; an imaged image transmitting means for transmitting the imaged image to the instruction device by a transmitting means; a projection unit attached to the on-site person in charge or the mobile body, and projecting an instruction image into the on-site space based on given instruction image data; a drive unit for changing the imaging direction of the imaging unit and the projection direction of the projection unit; and a tracking control means for controlling the drive unit based on a comparison between a characteristic part image in the reference image when the fixing command was given and a characteristic part image in the current image, using the image when the fixing command was given as a reference image, so that the instruction image is correctly displayed based on a predetermined part of the on-site space, regardless of the movement of the on-site person in charge or the mobile body; The instruction device is characterized by comprising an image receiving means for receiving the image transmitted by a receiving unit, an image display unit for displaying the received image, a fixed command means for issuing a fixed command to the on-site device by a transmitting unit so that an image of the desired site space is captured, an instruction image input unit for inputting an instruction image at a desired position in the site space by operation of the instructor in the displayed image, and an instruction image transmitting means for controlling a drive unit based on a characteristic partial image of the site space contained in the image to transmit instruction image data identifying the position of the instruction image to the on-site device by a transmitting unit so that the projection unit of the on-site device correctly projects the instruction image based on a specified portion of the site space as a reference.

[0026] Therefore, the instruction image data can be projected at the correct position regardless of the orientation of the field worker.

[0027] (18) The remote instruction system according to the present invention is characterized in that the imaging unit and the projection unit are fixed to the drive unit via a member that absorbs high-frequency vibrations.

[0028] Therefore, it is possible to obtain a captured image with less vibration and to project an instruction image with less vibration.

[0029] (19) The remote instruction system according to the present invention is characterized in that the imaging unit and the projection unit are fixed to the helmet of the on-site worker via the drive unit.

[0030] Therefore, the imaging unit and the projection unit can be stably fixed.

[0031] (20) In the remote instruction system according to the present invention, the direction control means changes the predetermined direction based on a direction instruction from the instruction device.

[0032] Therefore, the instructor can remotely control the imaging direction and check the situation at the site.

[0033] (21) In the remote instruction system according to the present invention, the characteristic part image is a marker provided in the site space or a characteristic point of the captured image.

[0034] Therefore, the instruction image can be displayed correctly based on the markers or feature points.

[0035] (22) The remote instruction system according to the present invention is characterized in that the drive unit has a multi-axis drive mechanism.

[0036] Therefore, the orientation of the imaging unit and the projection unit can be freely controlled.

[0037] (23) The remote instruction system of the present invention is characterized in that it further comprises a correction means for correcting the projection of the instruction image by the projection unit without relying on the drive unit, using the image captured when a fixation command was given as a reference image, and comparing the characteristic part image in the reference image captured when the fixation command was given with the characteristic part image in the current image captured, so that the instruction image is correctly displayed based on a specified part of the site space.

[0038] Therefore, the instruction image can be projected more accurately.

[0039] (24)(25) The on-site instruction device of this invention comprises an imaging unit that is attached to a site staff member or a mobile body and captures an image of the site space to generate an imaged image; a projection unit that is attached to a site staff member or a mobile body and projects an instruction image into the site space based on given instruction image data; a drive unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit; and a tracking control means that controls the drive unit based on a characteristic partial image of the site space contained in the captured image so that the instruction image is correctly displayed based on a specified part of the site space, regardless of the movement of the site staff member or the mobile body.

[0040] Therefore, the instruction image data can be projected at the correct position regardless of the orientation of the field worker.

[0041] (26) to (30) A remote instruction system according to the present invention is a remote instruction system including an instruction device used by an instructor and an on-site device used by an on-site person, The on-site device includes an imaging unit attached to the on-site staff or a mobile body, which captures images of the on-site space in a wide-angle direction to generate an image, an imaged image transmission means which transmits the captured image to the instruction device by a transmission unit, a projection unit attached to the on-site staff or a mobile body, which projects an instruction image into the on-site space based on given instruction image data, a drive unit which changes the projection direction of the projection unit, and a direction control means which controls the drive unit so that the projection unit faces in a predetermined direction centered on the on-site staff, regardless of the movement of the on-site staff or the mobile body. a correction unit that corrects the projection of the instruction image by the projection unit without using the drive unit so that the instruction image is correctly displayed based on a predetermined portion of the site space; The instruction device is characterized by comprising: an image receiving means for receiving the transmitted image by a receiving unit; a fixation command means for issuing a fixation command to the on-site device by a transmitting unit; an instruction image input unit for, upon receiving a fixation command, designating the vicinity of a characteristic portion image of the image as a target image, and inputting an instruction image at a desired position in the site space in the target image by operation of the instructor; and an instruction image transmitting means for controlling a drive unit so that the projection unit projects the instruction image based on a predetermined portion of the site space regardless of the movement of the on-site person in charge or the mobile body, and transmitting instruction image data identifying the position of the instruction image to the instruction device by a transmitting unit to control the projection of the projection unit.

[0042] Therefore, the instruction image data can be projected at the correct position regardless of the orientation of the field worker.

[0043] (31) The remote instruction system according to the present invention is characterized in that the projection unit is fixed to the drive unit via a member that absorbs high-frequency vibrations.

[0044] Therefore, it is possible to obtain a captured image with less vibration and to project an instruction image with less vibration.

[0045] (32) The remote instruction system according to the present invention is characterized in that the projection unit is fixed to the helmet of the field worker via the drive unit.

[0046] Therefore, the projection unit can be stably fixed.

[0047] (33) The remote instruction system according to the present invention is characterized in that the characteristic part image is a marker provided in the site space or a characteristic point of the captured image.

[0048] Therefore, the instruction image can be correctly projected based on the markers or feature points.

[0049] (34) The remote instruction system according to the present invention is characterized in that the drive unit has a multi-axis drive mechanism.

[0050] Therefore, the orientation of the imaging unit and the projection unit can be freely controlled.

[0051] (35)36) The on-site instruction device of this invention comprises an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space in a wide-angle direction to generate an image; a projection unit that is attached to a site worker or a mobile body and projects an instruction image into the site space based on given instruction image data; a drive unit that changes the projection direction of the projection unit; a direction control means that controls the drive unit so that the projection unit faces a predetermined direction centered on the site worker regardless of the movement of the site worker or the mobile body; and a correction means that corrects the projection of the instruction image by the projection unit without using the drive unit so that the instruction image is correctly displayed based on a predetermined location in the site space.

[0052] Therefore, the instruction image can be projected more accurately.

[0053] (37) to (41) A remote instruction system according to the present invention is a remote instruction system including an instruction device used by an instructor and an on-site device used by an on-site person in charge, The on-site device comprises an imaging unit that is attached to the on-site person in charge or a mobile body and captures an image of the on-site space in a wide-angle direction to generate an image, an imaged image transmission means that transmits the imaged image to the instruction device by a transmission unit, a projection unit that is attached to the on-site person in charge or a mobile body and projects an instruction image into the on-site space based on given instruction image data, a drive unit that changes the projection direction of the projection unit, and a tracking control means that controls the drive unit so that the instruction image is correctly displayed based on a predetermined part of the on-site space, The instruction device is characterized by comprising: an image receiving means for receiving the transmitted image by a receiving unit; a fixation command means for issuing a fixation command to the on-site device by a transmitting unit; an instruction image input unit for, when a fixation command is received, designating the vicinity of a characteristic partial image of the image as a target image, and inputting an instruction image at a desired position in the site space in the target image by operation of the instructor; and an instruction image transmitting means for transmitting instruction image data specifying the position of the instruction image to the instruction device by a transmitting unit, in order to control the drive unit so that the projection unit projects the instruction image based on a predetermined part of the site space regardless of the movement of the on-site person in charge or the mobile body.

[0054] Therefore, the instruction image data can be projected at the correct position regardless of the orientation of the field worker.

[0055] (42) The remote instruction system according to the present invention is characterized in that the projection unit is fixed to the drive unit via a member that absorbs high-frequency vibrations.

[0056] Therefore, it is possible to obtain a captured image with less vibration and to project an instruction image with less vibration.

[0057] (43) The remote instruction system according to the present invention is characterized in that the projection unit is fixed to the helmet of the field worker via the drive unit.

[0058] Therefore, the projection unit can be stably fixed.

[0059] (44) The remote instruction system according to the present invention is characterized in that the characteristic part image is a marker provided in the site space or a characteristic point of the captured image.

[0060] Therefore, the instruction image can be correctly projected based on the markers or feature points.

[0061] (45) The remote instruction system according to the present invention is characterized in that the drive unit has a multi-axis drive mechanism.

[0062] Therefore, the orientation of the imaging unit and the projection unit can be freely controlled.

[0063] (46) The remote instruction system of the present invention is characterized in that it further comprises a correction means for correcting the projection of the instruction image by the projection unit without relying on the drive unit, using the image captured when a fixation command was given as a reference image, and comparing the characteristic part image in the reference image captured when the fixation command was given with the characteristic part image in the current image captured, so that the instruction image is correctly displayed based on a specified part of the site space.

[0064] Therefore, the instruction image can be projected more accurately.

[0065] (47)(48) The on-site instruction device of this invention comprises an imaging unit that is attached to a site staff member or a mobile body and captures an image of the site space in a wide-angle direction to generate an image; a projection unit that is attached to a site staff member or a mobile body and projects an instruction image into the site space based on given instruction image data; a drive unit that changes the projection direction of the projection unit; and a tracking control means that controls the drive unit so that the instruction image is correctly displayed based on a specified location in the site space.

[0066] Therefore, the instruction image data can be projected at the correct position regardless of the orientation of the field worker.

[0067] (49) to (53) A remote instruction system according to the present invention is a remote instruction system including an instruction device used by an instructor and an on-site device used by an on-site person in charge, The on-site device includes an imaging unit that is attached to the on-site person in charge or a mobile body and captures an image of the site space in a wide-angle direction to generate a captured image; an imaged image transmission means that transmits the captured image to the instruction device by a transmission unit; a projection unit that is attached to the on-site person in charge or a mobile body and is capable of projecting onto the site space in a wide-angle direction and projects an instruction image onto the site space based on given instruction image data; and a tracking control means that, upon receiving a fixing command from the instruction device, controls the projection unit to project the instruction image based on a specific part of the site space as a reference, regardless of the movement of the on-site person in charge or the mobile body, based on a characteristic partial image of the site space included in the captured image. The instruction device is characterized by comprising an image receiving means for receiving the transmitted image by a receiving unit, a fixation command means for issuing a fixation command to the on-site device by a transmitting unit, an instruction image input unit for, upon receiving a fixation command, designating the vicinity of a characteristic partial image of the image as a target image, and inputting an instruction image at a desired position in the on-site space in the target image by operation of the instructor, and an instruction image transmission means for transmitting instruction image data specifying the position of the instruction image to the instruction device by a transmitting unit.

[0068] Therefore, the instruction image data can be projected at the correct position regardless of the orientation of the field worker.

[0069] (54) The remote instruction system according to the present invention is characterized in that the imaging unit and the projection unit are fixed via a member that absorbs high-frequency vibrations.

[0070] Therefore, it is possible to obtain a captured image with less vibration and to project an instruction image with less vibration.

[0071] (55) The remote instruction system according to the present invention is characterized in that the imaging unit and the projection unit are fixed to the helmet of the on-site worker.

[0072] Therefore, the imaging unit and the projection unit can be stably fixed.

[0073] (57) The remote instruction system according to the present invention is characterized in that the characteristic part image is a marker provided in the site space or a characteristic point of the captured image.

[0074] Therefore, the instruction image can be correctly projected based on the markers or feature points.

[0075] (57)(58) The on-site instruction device of this invention comprises an imaging unit that is attached to a site staff member or a mobile body and captures an image of the site space in a wide-angle direction to generate an imaged image; a projection unit that is attached to a site staff member or a mobile body and is capable of projecting onto the site space in a wide-angle direction and projects an instruction image onto the site space based on given instruction image data; and a tracking control means that controls the projection unit to project the instruction image based on a specified part of the site space, regardless of the movement of the site staff member or the mobile body, based on a characteristic partial image of the site space contained in the captured image.

[0076] Therefore, the instruction image data can be projected at the correct position regardless of the orientation of the field worker.

[0077] (59) The combined imaging unit and projection unit of the present invention comprises an imaging unit, a projection unit that projects at an angle of view substantially the same as the imaging angle of view of the imaging unit, a structure that can orient the imaging unit and projection unit as a unit in at least two axial directions, and a drive unit that controls the orientation of the structure.

[0078] Therefore, when projecting an instruction image or the like onto an object, even if the device is attached to a moving person, it is easy to control the device so that the instruction image is displayed accurately.

[0079] In the embodiment, steps S1 and S2 correspond to the "direction control means."

[0080] In the embodiment, the "correction means" corresponds to step S34 and step S75.

[0081] In this embodiment, step S51 corresponds to the "fixed command means."

[0082] In the embodiment, step S53 corresponds to the "instruction image transmitting means."

[0083] In the embodiment, steps S35, S76, and S79 correspond to the "follow-up control means."

[0084] The concept of "device" includes not only what is constituted by one computer, but also what is constituted by multiple computers connected via a network, etc. Therefore, when the means of the present invention (or even a part of the means) is distributed among multiple computers, these multiple computers correspond to the device.

[0085] The term "program" is a concept that includes not only programs that can be executed directly by a CPU, but also programs in source format, compressed programs, encrypted programs, and programs that work in conjunction with an operating system to perform their functions. [Brief explanation of the drawings]

[0086] [Figure 1] 1 is a functional configuration diagram of a remote instruction system according to an embodiment of the present invention. [Figure 2] FIG. 1 shows a field worker 54 wearing a field device. [Figure 3] This is the external appearance of the laser projector-camera combination 58. [Figure 4] 10 is a diagram showing the mounting structure of the laser projector-camera combination 58 to the mount member 97. FIG. [Figure 5] 10 is a diagram showing the attachment position of a silicon gel bushing 120 (high-frequency vibration absorbing member) relative to a unit 80. FIG. [Figure 6] 1 shows the system configuration of a remote instruction system. [Figure 7] 1 shows the hardware configuration of the indicator 30. [Figure 8] 1 shows the hardware configuration of a motor control circuit 400. [Figure 9] 1 shows the hardware configuration of a smartphone 200. [Figure 10] 10 is a flowchart of direction fixing control. [Figure 11] 10 is a flowchart of instruction image display control. [Figure 12] 10 is a captured image displayed on the display 306 of the instruction device 30. [Figure 13] 10 is a diagram showing a captured image on which an instruction image 62 is drawn by the instruction device 30. FIG. [Figure 14] FIG. 10 is a diagram showing the data structure of an instruction image. [Figure 15] 10 is a diagram showing the relationship between the movement of a site worker and the projection direction of a laser projector 84. FIG. [Figure 16] 10 is a diagram showing the relationship between the movement of a site worker and the projection direction of a laser projector 84. FIG. [Figure 17] 10 is a diagram showing the relationship between the movement of a site worker and the projection direction of a laser projector 84. FIG. [Figure 18] This is an example in which feature points 512 are used instead of markers. [Figure 19] 1 shows the functional configuration of the on-site indicator 11. [Figure 20] 10 is a flowchart of an instruction process by the on-site instruction device 11. [Figure 21] FIG. 10 is a functional configuration diagram of a remote instruction system according to a second embodiment. [Figure 22] 10 is a flowchart of instruction image display control. [Figure 23] 10 is a diagram showing the relationship between the movement of a site worker and the projection direction of a laser projector 84. FIG. [Figure 24] 1 shows the functional configuration of the on-site indicator 11. [Figure 25] 10 is a flowchart of an instruction process by the on-site instruction device 11. [Figure 26] FIG. 10 is a functional configuration diagram of a remote instruction system according to a third embodiment. [Figure 27] This is the external appearance of the laser projector-camera combination 58. [Figure 28] 10 is a flowchart of direction fixing control. [Figure 29] 10 is a flowchart of instruction image display control. [Figure 30] 10 is a flowchart of direction fixing control. [Figure 31] 10 is a flowchart of instruction image display control. [Figure 32] FIG. 2 is a functional configuration diagram of the on-site indicator 11. [Figure 33] 10 is a flowchart of an instruction process by the on-site instruction device 11. [Figure 34] FIG. 10 is a functional configuration diagram of a remote instruction system according to a fourth embodiment. [Figure 35] 10 is a flowchart of direction fixing control. [Figure 36] 10 is a flowchart of instruction image display control. [Figure 37] FIG. 2 is a functional configuration diagram of the on-site indicator 11. [Figure 38] 10 is a flowchart of an instruction process by the on-site instruction device 11. DETAILED DESCRIPTION OF THE INVENTION

[0087] 1. First embodiment 1.1 Functional configuration The functional configuration of a remote instruction system according to an embodiment of the present invention is shown in Figure 1. This system includes a field device 10 used by a field worker and an instruction device 30 used by a remote instructor.

[0088] The helmet worn by the on-site worker is provided with an imaging unit 12 and a projection unit 14 via a drive unit 16. The imaging area of ​​the imaging unit 12 and the projection area of ​​the projection unit 14 are arranged so as to be substantially the same.

[0089] The imaging unit 12 and projection unit 14 are configured as a single unit so that their imaging direction and projection direction can be changed by a drive unit 16. The imaging direction and projection direction of the imaging unit 12 and projection unit 14 are detected by a sensor 28. A direction control means 20 controls the drive unit 16 based on the output of the sensor 28, and maintains the orientation of the imaging unit 12 and projection unit 14 in a predetermined direction centered on the on-site staff member, regardless of the movements of the on-site staff member.

[0090] The imaging unit 12 of the on-site device 10 captures an image of the on-site space including the target object 52 and generates a captured image. As described above, the imaging direction of the imaging unit 12 is fixed in a predetermined direction centered on the on-site person in charge of the work, so that a substantially fixed captured image can be obtained even if the on-site person turns his / her face in a different direction on the spot, unless the on-site person himself / herself moves to another location.

[0091] This captured image is transmitted to the instruction device 30 by the transmission unit 22 under the control of the captured image transmission means 18. The captured image receiving means 36 of the instruction device 30 receives the captured image by the reception unit 32. The captured image display unit 40 displays the received captured image, thereby allowing the instructor to view an image of the site space.

[0092] When giving an instruction, the instructor inputs a fixation command. When the fixation command is input, the captured image display 40 uses the captured image at that time as a reference captured image and displays it as a still image. Note that since the captured image is displayed in a substantially fixed state by the direction control means 20, the captured image may be displayed as is.

[0093] The instructor inputs an instruction image from the instruction image input unit 44 while viewing the reference captured image of the site displayed on the captured image display unit 40. The instruction image transmission means 38 transmits the input instruction image to the on-site device 10 via the transmission unit 34.

[0094] The on-site device 10 receives the instruction image via the receiving unit 24 and projects the instruction image from the projection unit 14. As a result, an instruction image 62 is projected onto the target object 52. As described above, the projection direction of the projection unit 14 is fixed, so even if the on-site staff member changes the direction of their face, the instruction image will be projected at the location intended by the instructor. However, if the on-site staff member moves to another location, the projection position of the instruction image will shift.

[0095] Therefore, the correction means 26 of the on-site device 10 corrects and controls the projection position of the instruction image by comparing the characteristic part image (such as a marker) in the reference captured screen with the characteristic part image in the current captured image so that the instruction image is correctly projected at the intended position. As a result, even if the on-site worker moves, the instruction image is displayed at the correct position.

[0096] To achieve the above, when the instruction device 30 receives a fixation command, the transmitter 34 transmits the fixation command to the on-site device 10. The receiver 24 of the on-site device 10 receives the fixation command and records the captured image at that time as a reference captured image. In addition, the on-site staff member places a marker near the target object 52 so that an image can be taken.

[0097] The on-site worker can confirm the position where he or she should work based on the instruction image 62 actually projected in the work site space. This instruction image 62 is displayed in the correct position by the direction control means 20 even if the worker turns his or her head. Therefore, the instruction image 62 does not disappear depending on the direction of the worker's head, making it easier for the worker to work. Furthermore, when multiple workers are working together, even if the on-site worker wearing the on-site device 10 moves his or her head significantly, the instruction image 62 continues to be projected, and the work of the other workers is not interrupted. Furthermore, even if the worker moves, the instruction image 62 is displayed correctly.

[0098] 1.2 Appearance and hardware configuration 2 shows a field worker 54 equipped with the field device 10. The field worker 54 is performing a task of drilling a hole in the side of a chimney 52 on a roof 50, and this task will be described as being remotely instructed by an instructor.

[0099] In this embodiment, the field device 10 is configured by a smartphone (not shown), a laser projector-camera combination unit 58, and a rear housing 59. The field worker 54 wears a helmet 56, with the laser projector-camera combination unit 58 fixed to the front visor and the rear housing 59 housing a battery and a motor control circuit at the rear end. Both are connected by signal lines and power lines (not shown). Furthermore, the field worker 54 holds a smartphone (not shown) in his breast pocket. The smartphone, the laser projector-camera combination unit 58, and the rear housing 59 are connected by signal lines and power lines (not shown).

[0100] 3 shows the external appearance of the laser projector-camera combination 58. The laser projector-camera combination 58 is provided with a clip 96. The clip 96 is biased by a spring member (not shown) in the direction of closing the members 93 and 95 around the axis 91. By pressing the lever 98 in the direction of arrow A, the members 93 and 95 open, and the visor of the helmet 56 is clamped in place. Then, by releasing the pressure, the spring member holds the visor in place.

[0101] The combined laser projector-camera unit 58 is preferably attached so that the camera 82 and laser projector 84 are located near the front of the dominant eye (the right eye for a right-handed person). That is, the combined laser projector-camera unit 58 is held on the visor of the helmet 56 so that the camera 82 and laser projector 84 are positioned downward from the visor (in a state upside down from the state shown in Figure 3). If the combined laser projector-camera unit 58 obstructs the line of sight when attached, it is better to hold the combined laser projector-camera unit 58 on the visor so that the camera 82 and laser projector 84 are positioned upward from the visor (as shown in Figure 3).

[0102] A unit 80 containing a camera 82 and a laser projector 84 is fixed to the clip 96 via a three-axis structure 90 (or other multi-axis structure) that serves as a drive unit. In this embodiment, a member 93 of the clip 96 serves as a base 93 of the three-axis structure 90.

[0103] A motor 92 is fixed to a base 93 of the three-axis structure 90, and one end of an intermediate member 92A that is rotated in the XY plane by the motor 92 is connected to the motor 92. The intermediate member 92A is formed in an L-shape, and a motor 94 is fixed to the other end. One end of an intermediate member 94A that is rotated in the ZX plane by the motor 94 is connected to the motor 94. The intermediate member 94A is formed in an L-shape, and a motor 96 is fixed to the other end. A mount member 97 that is rotated in the ZY plane by the motor 96 is connected to the motor 96. The X, Y, and Z axes shown in FIG. 3 change as the members 92A, 94A, and 97 rotate.

[0104] In this way, the three-axis structure 90 can adjust the orientation of the mount member 97 with three degrees of freedom by driving the motors 92, 94, and 96.

[0105] The base 93 is also provided with a three-axis gyro sensor JS and a three-axis acceleration sensor AS as sensors 28. The motors 92, 94, and 96 are controlled by a motor control circuit (not shown) based on the outputs of the three-axis gyro sensor JS and the three-axis acceleration sensor AS.

[0106] A unit 80 containing a camera 82 and a laser projector 84 is fixed to a mount member 97 of the triaxial structure 90. As shown in Fig. 4, a camera control circuit 102 and a laser projector control circuit 104 that control the camera 82 and the laser projector 84 are provided inside a housing 81 of the unit 80.

[0107] The camera control circuit 102 and the laser projector control circuit 104 may be provided in the rear housing 59, but it is preferable that at least the MEMS circuit of the laser projector 104 be provided in the unit 80.

[0108] The housing 81 is attached to the mount top surface 101, mount side surfaces 97, and mount bottom surface 99 of the triaxial structure 90 via silicone gel bushings 120 (for example, Taica vibration-isolating gel bushing B-1). Note that the mount top surface 101 is omitted in Figure 3 for ease of understanding.

[0109] As shown in FIG. 4, the silicone gel bushing 120 includes a ring-shaped silicone gel 114 that is inserted onto the outside of the upper portion of a ring-shaped silicone gel 116. The upper portion of the silicone gel 116 is inserted into a hole provided in the housing 81. The housing 81 is sandwiched between the silicone gel 114 and the silicone gel 116. The silicone gels 114 and 116 are screwed to the mount bottom surface 99 with a bolt 110 and a washer 112. With this structure, the housing 81 is held in place by the silicone gels 116 and 114. This prevents high-frequency vibrations from the outside from being transmitted to the housing 81.

[0110] In this embodiment, as shown in FIG. 5, silicone gel bushings 120 are provided at two locations on each of the top, side, and bottom surfaces of the housing 81.

[0111] Figure 6 shows the system configuration of the remote instruction system. The on-site device 10 and the instruction device 30 are connected via the Internet. The on-site device 10 and the instruction device 30 may exchange data directly via the Internet, or may exchange data via a server device. The on-site device 10 includes a laser projector / camera combination unit 58, a rear housing (motor control circuit, battery) 59, and a smartphone 200. These are connected to each other via signal lines and power lines.

[0112] 7 shows the hardware configuration of the instruction device 30. Connected to the CPU 302 are a memory 304, a display 306, a microphone 308, a communication circuit 310, an SSD 312, a DVD-ROM drive 314, a mouse / keyboard 316, and a speaker 318. The communication circuit 310 is for connecting to the Internet.

[0113] An operating system 320 and an instruction program 322 are recorded on the SSD 312. The instruction program 322 works in cooperation with the operating system 320 to perform its functions. These programs were recorded on a DVD-ROM 324 and then installed via the DVD-ROM drive 314. The microphone 308 and speaker 318 are for communicating with the on-site personnel.

[0114] 8 shows the hardware configuration of the motor control circuit 400. Connected to the CPU 402 are a memory 404, a gyro sensor JS, an acceleration sensor AS, a camera 82, a laser projector 84, motors 92, 94, and 96, and a non-volatile memory 406. The camera 82 is connected via a camera control circuit 102, and the laser projector 84 is connected via a laser projector control circuit 104, but these are omitted from the drawing.

[0115] The nonvolatile memory 406 stores an operating system 31 and a motor control program 32. The motor control program 32 cooperates with the operating system 31 to perform its functions.

[0116] 9 shows the hardware configuration of the smartphone 200. Connected to the CPU 202 are a memory 204, a touch display 206, a short-range communication circuit 208, a camera 82, a laser projector 84, an SSD 212, a speaker 214, a microphone 216, and a communication circuit 218. Note that the normal communication circuitry is omitted from the diagram.

[0117] The communication circuit 218 is a circuit for connecting to the Internet. The speaker 214 and microphone 216 are for communicating with an instructor via the Internet. The SSD 212 stores an operating system 222 and an image control program 224. The image control program 224 works in cooperation with the operating system 224 to fulfill its functions.

[0118] 1.3 Remote instruction processing 10 and 11 show flowcharts of the motor control program 32 of the motor control circuit 400, the image control program 224 of the smartphone 200, and the instruction program 322 of the instruction device 30. Fig. 10 is a flowchart of the direction fixing control, and Fig. 11 is a flowchart of the instruction image display control.

[0119] When the on-site worker 54 arrives in front of the target object, the chimney 52, he attaches the laser projector-camera combination 58 to the visor of his helmet 58 and turns it on. As a result, the camera 82 captures an image of the vicinity of the chimney 52.

[0120] The CPU 402 of the motor control circuit 400 (hereinafter sometimes abbreviated as the motor control circuit 400) acquires the outputs of the gyro sensor JS and acceleration sensor AS of the laser projector-camera combination 58 (step S1). In this embodiment, gyro sensors and acceleration sensors in three orthogonal axes are used.

[0121] The motor control circuit 400 calculates the position and orientation of the base 93 (see FIG. 3) in three-dimensional space based on the outputs of the gyro sensor JS and the acceleration sensor AS. Then, the motor control circuit 400 controls the rotation angles of the motors 92, 94, and 96 so that the unit 80 faces in a fixed direction regardless of the position or orientation of the base 93 (step S2). Therefore, the unit 80 is maintained in a fixed orientation regardless of the orientation of the head of the field worker 54. This type of control is similar to that of a gimbal used as a stabilizer for a camera or the like.

[0122] As described above, regardless of the direction of the head of the on-site worker 54, the camera 82 is directed in the direction of the vicinity of the chimney 52, and a stable captured image can be obtained as a moving image.

[0123] The CPU 202 of the smartphone 200 (hereinafter sometimes abbreviated as smartphone 200) acquires an image captured by the camera 82 via a signal line (or short-range communication), and transmits the image to the instruction device 30 via the Internet (step S21).

[0124] The CPU 302 of the instruction device 30 (hereinafter sometimes abbreviated as instruction device 30) displays the received captured image on the display 306 (step S41). FIG. 12 shows an example of the captured image displayed on the display 306. As shown in FIG. 12, the chimney 52 and its vicinity are displayed as the captured image. This allows the instructor operating the instruction device 30 to check the situation at the site in real time as a video.

[0125] The instructor can talk to the on-site person in charge of the work via the smartphone 200 of the on-site person in charge of the work via internet communication using the microphone 308 and the speaker 318. This allows the instructor to guide the on-site person in charge of the work to a desired location by conversation while looking at the display 306.

[0126] 12, the instruction device 30 displays a direction change button 500 on the display 306. When the instructor wishes to change the imaging direction of the camera 82, the instructor clicks on the circumference of the direction change button 500 with the mouse 316. When the instruction device 30 detects the click, it generates an imaging direction change command for changing the imaging direction in the corresponding direction (up, down, left, right) and transmits the command to the smartphone 200 (step S42).

[0127] The smartphone 200 transmits the received imaging direction change command to the motor control circuit 400 (step S22). The motor control circuit 400 controls the motors 92, 94, and 96 based on the received imaging direction change command to change the orientation of the unit 80 (i.e., the imaging direction) (step S3). As a result, the orientation of the unit is fixed to the changed orientation thereafter, regardless of the orientation of the head of the on-site person in charge. Therefore, the captured image changed to the orientation instructed by the instructor is displayed on the display 306 of the instruction device 30. In this way, the instructor can obtain an image in the desired direction at the site.

[0128] The direction fixing control of Fig. 10 is repeatedly executed. In parallel with this direction fixing control, the instruction image display control process of Fig. 11 is performed. Based on verbal instructions from the instructor, the on-site person in charge takes out a marker 60 prepared in advance and attaches it to the chimney 52, which is the target object. The size and shape of an image of the marker 60 (characteristic part image) are recorded in advance in the non-volatile memory 212 of the smartphone 200. Therefore, the smartphone 200 can calculate the distance and direction from the camera 82 to the marker 60, based on the captured image of the marker 60.

[0129] 10, the captured image is displayed as a moving image on the display 306 of the instruction device 30. While viewing this captured image, the instructor clicks the instruction input mode button 501 with the marker 60 captured as shown in FIG.

[0130] In response to this, when the instruction input mode button 501 is clicked to give a fixed command, the instruction device 30 uses the captured image at that time as a reference captured image and displays it as a still image on the display 306 (step S52). The instructor inputs instructions to the on-site staff as an instruction image on this still image using the mouse 316 (step S53). For example, as shown in Fig. 13, in the image of the chimney 52 displayed on the display 306, the instructor draws and inputs an instruction image 62 (in this example, a cross image indicating the hole drilling position with a cross) with the mouse 316 to indicate the position where a hole should be drilled.

[0131] Furthermore, the instruction device 30 transmits the fixation command to the smartphone 200 (step S32). Upon receiving the fixation command, the smartphone 200 records the captured image at the time of receiving the fixation command in the nonvolatile memory 212 as a reference captured image (step S32).

[0132] Therefore, the instruction device 30 and the smartphone 200 can recognize the image captured at the same time as the reference image. In order to avoid a time lag due to communication, when transmitting a fixation command from the instruction device 30 to the smartphone 200, information specifying a frame (such as a frame number) may be attached to the command. By determining the reference image in the smartphone 200 based on the information specifying the frame, it is possible to prevent a discrepancy due to a time lag.

[0133] When the instructor has finished inputting the instruction image, he or she clicks the instruction image transmission button 502 (which appears when the instruction input mode is entered) displayed at the bottom right of the reference captured image on the display 306. This causes the instruction device 30 to transmit the instruction image to the smartphone 200 (step S53).

[0134] The instruction device 30 also cancels the instruction input mode, stops displaying the still image as the reference captured image, and displays the transmitted captured image as a moving image (step S54), thereby allowing the instructor to once again become aware of the situation at the site.

[0135] The data structure of the instruction image transmitted to smartphone 200 is shown in FIG. 14A. The instruction image data is entity data of the instruction image input by the instructor, as shown in FIG. 14B. The reference coordinate position is the XY coordinate values ​​of the reference point of the instruction image when the reference point of the marker image (for example, the point at the bottom center of M) is set as the origin, as shown in FIG. 14C. In this embodiment, as shown in FIG. 14B, the reference point is the upper left corner of the rectangle circumscribing the instruction image input by the instructor.

[0136] In the above example, the instruction image is transmitted as image data, but the parameters may be transmitted as numerical values ​​depending on the predetermined shape of the image. For example, the center coordinates and radius for a perfect circle, the upper left coordinates and side lengths for a square, etc. may be transmitted as numerical values.

[0137] 14A, the smartphone 200 stores the instruction image data in the memory 204. Furthermore, the smartphone 200 acquires the currently captured image from the camera 82 (step S33).

[0138] As described above, the imaging range of camera 82 and the projection range of laser projector 84 are configured to be the same. Therefore, if the current captured image is exactly the same as the recorded reference captured image (i.e., if the site staff member has not moved at all since the reference captured image was captured), then by projecting instruction image data by laser projector 84 at a position based on the reference coordinate position (FIG. 14C), instruction image 62 will be projected onto chimney 52 as shown in FIG.

[0139] The position of this instruction image 62 matches the position input by the instructor on the display 306, so the work position can be accurately shown to the site worker. The site worker performs the work of drilling a hole at that position using this instruction image 62 as a landmark.

[0140] As shown in FIG. 15A, even if the helmet 56 is rotated horizontally (vertically) around the vertical axis (horizontal axis) of the head of the field worker 54, the instruction image 62 will be displayed in the correct position as long as the distance and direction to the instruction image of the camera 82 and the laser projector 84 do not change due to the direction fixing control of FIG. 10 (although there are mechanical limitations depending on the amount of rotation).

[0141] However, as shown by the dashed line in Figure 15B, when the site worker 54 (helmet 56) moves closer to or further away from the target object 52, the instruction image 62 projected on the site becomes larger or smaller.

[0142] 15C, when the site staff 54 (helmet 56) moves left or right, the direction fixing control in FIG. 10 simply keeps the image capturing direction indicated by the arrow constant with the site staff 54 as the reference. As a result, the image capturing range of the camera 82 shifts left or right, and the instruction image 62 projected by the laser projector 84 is also displayed at a position shifted left or right. In this situation, the instruction image 62 also shifts up or down when the site staff 54 moves up or down (for example, when standing up from a crouching position).

[0143] In order to resolve these issues and correctly project the instruction image 62, the following processing is carried out in this embodiment.

[0144] The smartphone 200 calculates the distance and direction between the camera 82 and the marker 60 (and the location where the instruction image 62 should be projected) based on the image of the marker 60 in the reference captured image. As described above, a known pattern is printed on the marker 60 in advance, so the smartphone 200 can calculate the distance and direction to the marker 60 affixed to the chimney 52 (and the location where the instruction image 62 should be projected) based on the captured image.

[0145] Furthermore, the smartphone 200 calculates the distance and direction to the marker 60 attached to the chimney 52 (and the location where the instruction image 62 should be projected) based on the current captured image acquired in step S33.

[0146] Therefore, the smartphone 200 deforms the instruction image 62 and controls the position at which the instruction image 62 is projected based on a comparison between the distance and direction to the marker 60 (and the location where the instruction image 62 should be projected) in the reference captured image and the distance and direction to the marker 60 (and the location where the instruction image 62 should be projected) in the current captured image (step S34).

[0147] For example, in the case of FIG. 15B, the projection control is performed so that the instruction image 62 is enlarged or reduced in accordance with the change in the distance between the camera 82 and the marker 60 (or the instruction image 62).

[0148] In the case of FIG. 15C, the position where the instruction image 62 is projected is controlled to move in accordance with the movement of the marker 60.

[0149] In this embodiment, direction fixing control (see Figure 10) is performed separately using the three-axis structure 90, so in many cases, the instruction image 62 can be displayed in the correct position by performing control for Figures 15B and 15C.

[0150] In this embodiment, the above control is performed in addition to the direction fixing control performed by the three-axis structure 90, so the instruction image 62 can be stably displayed in the correct position. Furthermore, even if the site worker 54 turns his head and takes his line of sight away from the target object 52, the direction fixing control keeps the instruction image 62 displayed. This reduces stress on the site worker 54 when he performs his work.

[0151] However, when the limits of directional fixation control by the three-axis structure 90 are exceeded, for example, as shown schematically in Figure 16, the inclination of the surface 510 on which the marker 60 of the object 52 is attached relative to the camera 82 (laser projector 84) may change from the time of the reference captured image in Figure 16A to the time of the current captured image in Figure 16B.

[0152] In this case, the smartphone 200 calculates the tilt of the surface 510 of the object 52 based on the image of the marker 60 in the reference captured image (FIG. 16A). As a result, the smartphone 200 calculates the actual distance LL between the marker 60 and the instruction image 62 based on the reference coordinate position PL1 (X or Y) sent from the instruction device 30.

[0153] Next, the inclination of the surface 510 of the object 52 is calculated based on the image of the marker 60 in the currently captured image (FIG. 16B). As a result, the position where the instruction image 62 should be displayed is determined based on the calculated actual distance LL, and the reference coordinate position PL2 (X or Y) is calculated. The smartphone 200 controls the position where the instruction image 62 is projected based on this reference coordinate position PL2, and can project the instruction image 62 at the correct position. Furthermore, the instruction image 62 is deformed so that the projected instruction image 62 is not distorted.

[0154] The above processing can be performed in the same manner for both the vertical and horizontal directions.

[0155] Furthermore, as shown in Fig. 17, the imaging range 504 in the reference captured image may be tilted at an angle, as shown in imaging range 506. Although Fig. 17 shows the tilt in the direction horizontal to the paper, such tilt can occur in any three-dimensional direction. This will also distort the projected instruction image 62.

[0156] In these cases, too, the correct instruction image can be projected by deforming (deforming in the opposite direction to the distortion) and projecting the instruction image 62 so as to eliminate the above distortion based on the image of the marker 60 in the reference captured image and the image of the marker 60 in the current captured image.

[0157] The above processing can be summarized as follows: The smartphone 200 calculates the distance and direction between the camera 82 and the marker 60 based on the image of the marker 60 in the reference captured image. The smartphone 200 also calculates the distance and direction to the marker 60 affixed to the chimney 52 based on the current captured image acquired in step S33. The smartphone 200 transforms the instruction image 62 and controls the position at which the instruction image 62 is projected based on a comparison between the distance and direction to the marker 60 in the reference captured image and the distance and direction to the marker 60 in the current captured image.

[0158] In this way, the instruction image intended by the instructor is projected and displayed on the object 52 at the site.

[0159] In order to display the instruction image 60 correctly, it is preferable that the marker 60 be attached to the plane on which the instruction image 60 is to be displayed.

[0160] In addition, if the surface on which the marker 60 is attached is different from the surface on which the instruction image 62 should be displayed (for example, if there is a step), it is preferable to combine it with estimating the accurate position using feature points of the image, such as SLAM.

[0161] That is, the smartphone 200 analyzes the captured image to calculate feature points (such as points on the boundary of the object) near the target object (near the marker 60). By comparing the feature points in the reference captured image with the feature points in the current captured image, the positional relationship between the marker 60 and the surface on which the instruction image 62 should be displayed is determined.

[0162] In this way, even if the surface on which the marker 60 is attached and the surface on which the instruction image 62 is to be displayed are different, the instruction image 62 can be projected correctly.

[0163] 1.4 Variations (1) In the above embodiment, the on-site device 10 is attached to the on-site worker 54 .

[0164] However, it may be attached to a mobile object (or a fixed object) near the site staff, such as a bicycle or car ridden by the site staff, or it may be attached to a mobile object (or a fixed object) located at a distance from the site staff.

[0165] Alternatively, the on-site device 10 may be attached to a robot instead of the on-site staff member 54. For example, a supervisor at a remote location can use the on-site instruction device 11 attached to the robot to display text and images, thereby communicating with people around the robot. The same applies to the following embodiments.

[0166] (2) In the above embodiment, the instruction image 62 is always projected onto the target object 52 by the laser projector 84. However, if there is a person in the projection direction, the laser projector 84 may not irradiate the object.

[0167] This can be achieved by having the smartphone 200 determine whether or not a person is present in the captured image (for example, using a trained AI such as YOLO), and if it determines that a person is present, stopping irradiation by the laser projector 84. When no person is detected, irradiation is resumed again.

[0168] Also, when a person is detected, instead of stopping laser irradiation entirely, it is possible to stop irradiation in the area of ​​the recognized person (a rectangular area in the case of YOLO) and continue irradiation in other areas.

[0169] Furthermore, it may be possible to detect a person's eyes and stop illuminating only the eye area (and the surrounding area).

[0170] (3) In the above embodiment, the laser projector 84 is used as the projection unit. However, a normal projector may also be used.

[0171] (4) In the above embodiment, a three-axis structure 90 (gimbal) is used, but a one-axis, two-axis structure (gimbal), or a structure with four or more axes may also be used.

[0172] (5) In the above embodiment, the site staff 54 attaches the markers 60 to the object 52. However, the markers 60 may be placed on the object 52 in advance at the site.

[0173] (6) In the above embodiment, the markers 60 are used to grasp the distance and direction from the camera 82. However, it is also possible to grasp the distance and direction from only the feature points of the captured image using SLAM or the like without using the markers 60, and perform similar processing.

[0174] In this case, feature points 512 (such as vertices that characterize the image) are recognized by the smartphone 200, transmitted to the instruction device 30, and displayed on the display 306 as shown in Fig. 18. The instructor views this image and operates the mouse 316 to select feature points 512 to be used for position identification. It is preferable to select feature points 512 related to the object 52 (feature points on the object) as feature points 512 to be used for position identification.

[0175] When the instruction input mode button 501 is clicked, information on the selected feature points 512 is sent to the smartphone 200. Based on these feature points 512, the smartphone 200 can identify the position and direction.

[0176] (7) In the above embodiment, the instructor checks the screen of Fig. 13 on the instruction device 30 and clicks the instruction image send button 502. However, the instruction device 30 or the smartphone 200 may detect that the marker 60 in the captured image has entered a predetermined area (for example, a predetermined central area) in the captured image and automatically enter instruction input mode. The same applies when processing is performed using the feature point 512 without using the marker 60.

[0177] (8) In the above embodiment, the motor control circuit 400 controls the three-axis structure 90, and the smartphone 200 controls the projection position and the like based on image processing.

[0178] However, the control of the triaxial structure 90 may also be performed by the smartphone 200. Alternatively, a circuit for controlling the triaxial structure 90 and controlling the projection position based on image processing may be provided in the rear housing 59. In this case, the smartphone 200 is used only for making calls. Furthermore, the call function may also be provided in the rear housing 59.

[0179] (9) In the above embodiment, the smartphone 200 deforms the pointing image so that it is not distorted (or changed in size) when projected. However, when the shape of the pointing image is not important and it is important to indicate a specific position (for example, when indicating a position by the center point of a cross mark), there is no problem even if the pointing image is distorted (changed in size) as long as the position can be correctly indicated. In such a case, the process of deforming the pointing image may not be performed.

[0180] (10) In the above embodiment, in addition to the control of the triaxial structure 90 (FIG. 10), the smartphone 200 controls the projection position and the like based on image processing (FIG. 11). However, the smartphone 200 may not control the projection position and the like based on image processing, and only the processing by the triaxial structure 90 may be performed.

[0181] This can be done when the site staff 54 does not move much or when it is acceptable for the projection position of the instruction image at the site to be slightly shifted. For example, when an instruction image (such as an arrow indicating a direction) is displayed on the ground from the integrated laser projector-camera 58 worn by the site staff 54 and route guidance is provided from the pointing device 30, control by the three-axis structure 90 alone is sufficient.

[0182] (11) In the above embodiment, the smartphone 200 not only performs the control corresponding to Figures 15B and 15C but also performs the control corresponding to Figures 16 and 17. However, it is also possible to perform only the control corresponding to Figures 15B and 15C.

[0183] (12) In the above embodiment, when a fixation command is given to the pointing device 30, the reference captured image is displayed as a still image and the mode for inputting the pointed image is set. However, because the imaging direction is fixed in the same direction by the control of the triaxial structure 90, even if the reference image is displayed as a moving image, a substantially fixed image can be obtained.

[0184] In this state, the instructor inputs an instruction image and clicks the instruction image transmission button 502. In response to this, the instruction device 30 and the on-site device 10 may use the image captured at that time as the reference captured image.

[0185] (13) In the above embodiment, the instruction image is projected by the field device 10 and the instruction device 30 that can communicate with the field device 10.

[0186] However, if the instruction image to be projected on-site is predetermined, the instruction image 62 may be recorded in the on-site device 10 and configured as the on-site instruction device 11.

[0187] The functional configuration of the on-site indicator 11 constructed in this manner is shown in FIG. 11. The processing of the direction control means 20 is the same as steps S1 and S2 in FIG. 10. An indicator image 62 is recorded in the recording unit 24. When the marker 60 enters a predetermined area in the captured image, the correction means 26 sets the captured image at that time as the reference captured image. Furthermore, based on the markers or feature points in the reference captured image and the current captured image, the correction means 26 corrects the indicator image 62 and controls the projection position. Therefore, the indicator image 62 is projected onto the object 52.

[0188] The on-site instruction device 11 may be worn by the on-site staff 54 or may be a mobile body (or a fixed body) near the on-site staff.

[0189] In this embodiment, the instruction image 62 is recorded in the recording unit 24, but it may also be acquired from outside the on-site instruction device 11 by communication or the like.

[0190] A flowchart of the instruction process is shown in Fig. 20. In this example, the on-site instruction device 11 is configured by a motor control circuit 400 and a smartphone 200. The process of the motor control circuit 400 is the same as in the above-described embodiment.

[0191] The smartphone 200 acquires a captured image (step S71), and determines whether a predetermined marker 60 is captured and falls within a predetermined range (for example, within a predetermined central area) of the captured image (step S72).

[0192] When the marker 60 is captured within a predetermined range, the smartphone 200 records the captured image at that time as a reference captured image (step S73). Subsequently, the smartphone 200 acquires the current captured image, compares it with the reference captured image, corrects the instruction image, and controls the projection position (step S75).

[0193] (14) In the above embodiment, a case where work on a chimney on site is remotely instructed has been described. However, the present invention can be used to give instructions at various sites, such as factories, roads, squares, buildings, and stadiums.

[0194] Furthermore, when requesting a shop assistant to help you with your purchases from a remote location, this system can be used to project an instruction image onto the item you wish to purchase.

[0195] It can also project arrows and other instructions onto the road ahead of the on-site staff to guide them.

[0196] It also makes it easier to give instructions to on-site personnel remotely in virtual tourism, etc. The remote instructor can look around regardless of the on-site personnel's orientation by moving the camera's imaging direction, and can project necessary instructions as instruction images.

[0197] (15) In the above embodiment, a still image is used as the instruction image. However, a moving image may be used as the instruction image. In this case, the on-site device may be configured to repeatedly play back the moving image.

[0198] (16) In the above embodiment, the on-site device is configured with a smartphone, a laser projector / camera combination 58, and a rear housing 59. However, these may be configured as an integrated unit. Also, if the rear housing 59 (motor control circuit) is given the functions performed by a smartphone, a smartphone may not be necessary. For example, a dedicated device, a PC, a tablet, a stick-type PC, etc. may be used.

[0199] (17) In the above embodiment, the direction of the displayed captured image is changed by operating the direction change button 500 as shown in Fig. 13. However, the direction of the displayed captured image may also be changed by dragging the screen (moving the cursor while holding down the mouse button).

[0200] (18) The above-mentioned modifications may be applied in combination with each other as long as it does not contradict the essence of the modifications. They may also be applied in combination with other embodiments and their modifications.

[0201] 2. Second embodiment 2.1 Functional configuration 21 shows the functional configuration of a remote instruction system according to the second embodiment. This system includes a field device 10 used by a field staff member and an instruction device 30 used by a remote instructor.

[0202] The helmet worn by the on-site worker is provided with an imaging unit 12 and a projection unit 14 via a drive unit 16. The imaging area of ​​the imaging unit 12 and the projection area of ​​the projection unit 14 are arranged so as to be substantially the same.

[0203] The functions of the imaging unit 12, projection unit 14, and drive unit 16 are the same as those in the first embodiment. The tracking control means 21 performs the same function as the direction control means 20 in the first embodiment, and maintains the orientation of the imaging unit 12 and projection unit 14 in a predetermined direction.

[0204] The captured image by the on-site device 10 is transmitted to the instruction device 30 by the transmission unit 22 under the control of the captured image transmission means 18. The captured image receiving means 36 of the instruction device 30 receives the captured image by the reception unit 32. The captured image display unit 40 displays the received captured image, thereby allowing the instructor to view an image of the on-site space.

[0205] When giving an instruction, the instructor inputs a fixation command. When the fixation command is input, the captured image display 40 uses the captured image at that time as a reference captured image and displays it as a still image. Note that since the captured image is displayed in a substantially fixed state by the direction control means 20, the captured image may be displayed as is.

[0206] The instructor inputs an instruction image from the instruction image input unit 44 while viewing the reference captured image of the site displayed on the captured image display unit 40. The instruction image transmission means 38 transmits the input instruction image to the on-site device 10 via the transmission unit 34.

[0207] The on-site device 10 receives the instruction image via the receiving unit 24 and projects the instruction image from the projection unit 14. In addition to the above-mentioned direction control, the tracking control means 21 of the on-site device 10 controls the driving unit 16 so that the imaging unit 12 tracks and captures the characteristic part image (such as a marker).

[0208] This allows the instruction image 62 to be projected onto the object 52.

[0209] 2.2 Appearance and hardware configuration The external appearance and hardware configuration are the same as those of the first embodiment.

[0210] 2.3 Remote instruction processing The direction fixing control for fixing the imaging direction of the camera in a predetermined direction is the same as that in the first embodiment shown in Fig. 10. Fig. 22 shows a flowchart of the instruction image display control after the marker is placed.

[0211] Steps S31 to S33 and steps S51 to S54 are the same as those in embodiment 1. In step S35, the direction fixing control shown in Fig. 10 is released, and thereafter, control for following the marker is performed.

[0212] The smartphone 200 compares the marker 60 in the reference captured image with the marker 60 in the current captured image, and calculates how to control the motors 92, 94, and 96 to change the orientation of the unit 80 so that the position of the marker 60 in the current captured image matches the position of the marker 60 in the reference captured image (step S35). Note that when calculating this control signal, the calculation is performed so that the inclination of the marker 60 in the image (see FIG. 17) is the same as that in the reference captured image.

[0213] The calculated control signal is transmitted to the motor control circuit 400, which controls the motors 92, 94, and 96 (step S5). In this way, the imaging direction of the camera 82 (the irradiation direction of the laser projector 94) is controlled to follow the marker 60.

[0214] Therefore, in the case shown in Fig. 15A, the imaging direction is changed in the direction of the arrow, and the marker 60 is always imaged at a predetermined position, as shown in Fig. 23. Note that since there is a slight change in angle, the marker 60 may not be displayed completely correctly (it may be slightly distorted), but it can be used as long as it is within an acceptable range depending on the intended use.

[0215] Also, even in the case shown in FIG. 17, the tilt is adjusted and the marker 60 is displayed in the correct position.

[0216] In the cases shown in Figure 15 and Figure 16, the size of the marker 60 may change slightly or it may be slightly distorted when projected, but this can also be used as long as it is within an acceptable range depending on the intended use.

[0217] 2.4 Variations (1) In the above embodiment, in the indication image display control of Fig. 22, the triaxial structure 90 is controlled to follow and capture an image of the marker 60. In addition to this, the indication image may be corrected or the projection position may be controlled by image processing shown in step S34 of the first embodiment.

[0218] (2) In the above embodiment, the instruction image is projected by the field device 10 and the instruction device 30 that can communicate with the field device 10.

[0219] However, if the instruction image to be projected on-site is predetermined, the instruction image 62 may be recorded in the on-site device 10 and configured as the on-site instruction device 11.

[0220] The functional configuration of the on-site indicator 11 constructed in this manner is shown in Fig. 24. The tracking control means 21 first performs the processes of steps S1 and S2 in Fig. 10. An indicator image 62 is recorded in the recording unit 24. The tracking control means 21 controls the driving unit 16 based on the marker 60 or feature points in the reference captured image and the current captured image, and captures an image by tracking the marker 60 so that it is at a predetermined position in the captured image. Therefore, the indicator image 62 is projected onto the target object 52.

[0221] The on-site instruction device 11 may be worn by the on-site staff 54, or may be attached to a mobile body (or a fixed body) near the on-site staff.

[0222] In this embodiment, the instruction image 62 is recorded in the recording unit 24, but it may also be acquired from outside the on-site instruction device 11 by communication or the like.

[0223] A flowchart of the instruction process is shown in Fig. 25. In this example, the on-site instruction device 11 is configured by a motor control circuit 400 and a smartphone 200. The process of the motor control circuit 400 is the same as in the above-described embodiment.

[0224] The smartphone 200 acquires a captured image (step S71), and determines whether a predetermined marker 60 is captured and falls within a predetermined range (for example, within a predetermined central area) of the captured image (step S72).

[0225] When the marker 60 is captured within a predetermined range, the smartphone 200 records the captured image at that time as a reference captured image (step S73). Next, the smartphone 200 acquires the current captured image, compares it with the reference captured image, and generates a motor control signal for tracking and capturing the marker 60 (step S76). This motor control signal is provided to the motor control circuit 400.

[0226] In response to this, the motor control circuit 400 controls the motors 92, 94, 96, and controls the triaxial structure 90 so as to follow the marker 60 and capture and project an image of the marker 60.

[0227] (3) The above-described modifications may be applied in combination with one another as long as it does not contradict the essence of the modifications. They may also be applied in combination with other embodiments and their modifications.

[0228] 3. Third embodiment 3.1 Functional configuration In the first and second embodiments, the imaging unit 12 and the projection unit 14 are integrally held by the drive unit 16.

[0229] In this third embodiment, the projection unit 14 is held by a three-axis structure 90, but the imaging unit 12 is capable of wide-angle imaging and is provided fixedly.

[0230] 26 shows the functional configuration of a remote instruction system according to an embodiment of the present invention. This system includes a field device 10 used by a field staff member and an instruction device 30 used by a remote instructor.

[0231] The on-site worker wears the projection unit 14 via the drive unit 16. The imaging unit 12 is fixedly attached. In this embodiment, a 360-degree camera is used as the imaging unit 12, allowing it to capture images of the entire periphery.

[0232] The projection unit 14 is configured so that its projection direction can be changed by a drive unit 16. The projection direction of the projection unit 14 is detected by a sensor 28. A direction control means 20 controls the drive unit 16 based on the output of the sensor 28, and maintains the direction of the projection unit 14 in a predetermined direction centered on the on-site staff member, regardless of the movement of the on-site staff member.

[0233] The imaging unit 12 of the on-site device 10 captures an image of the on-site space including the target object 52 to generate a captured image. As described above, the imaging unit 12 captures images in a wide-angle direction (for example, in a 360-degree circumferential direction), so that a captured image including the on-site space can be obtained even if the on-site person moves or changes direction.

[0234] This captured image is transmitted to the instruction device 30 by the transmission unit 22 under the control of the captured image transmission means 18. The captured image receiving means 36 of the instruction device 30 receives the captured image by the reception unit 32. The captured image display unit 40 displays the received captured image. Note that since the captured image is an image captured in a 360-degree circumferential direction, it is not displayed all at once, but rather partially. The instructor can look around the site worker by changing the direction up, down, left, or right. This makes it possible to display a captured image in the direction in which the object in the site space is reflected.

[0235] When giving instructions, the instructor inputs a fixation command while a captured image including the target object is displayed. When a fixation command is input, the captured image display 40 displays a partial captured image in that direction as a reference captured image and displays it as a still image. The direction when the fixation command is given is transmitted to the field device 10.

[0236] The instructor inputs an instruction image from the instruction image input unit 44 while viewing the reference captured image of the site displayed on the captured image display unit 40. The instruction image transmission means 38 transmits the input instruction image to the on-site device 10 via the transmission unit 34.

[0237] The on-site device 10 receives the instruction image via the receiving unit 24, controls the driving unit 16 to track the characteristic part image (such as a characteristic point of the image) projected in the reference captured image when the fixation command was given, and projects the instruction image from the projection unit 14. As a result, the instruction image 62 is projected onto the target object 52.

[0238] The projection direction of the projection unit 14 is controlled so as to follow the image of the characteristic part, so even if the on-site worker changes the direction of his / her face, the instruction image will be projected at the location intended by the instructor. However, if the on-site worker moves to another location, the projection position of the instruction image will shift.

[0239] Therefore, the correction means 26 of the on-site device 10 corrects and controls the projection position of the instruction image by comparing the characteristic part image (such as the characteristic points of the image) in the reference captured screen with the characteristic part image in the current captured image so that the instruction image is correctly projected at the intended position. As a result, even if the on-site worker moves, the instruction image is displayed at the correct position.

[0240] The on-site worker can confirm the position where he or she should work based on the instruction image 62 actually projected in the work site space. This instruction image 62 is displayed in the correct position by the direction control means 20 even if the worker turns his or her head. Therefore, the instruction image 62 does not disappear depending on the direction of the worker's head, making it easier for the worker to work. Furthermore, when multiple workers are working together, even if the on-site worker wearing the on-site device 10 moves his or her head significantly, the instruction image 62 continues to be projected, and the work of the other workers is not interrupted. Furthermore, even if the worker moves, the instruction image 62 is displayed correctly.

[0241] 3.2 Appearance and hardware configuration 27 shows the appearance of the integrated laser projector and camera 57. No camera is provided inside the unit 80, but a laser projector 84 is provided. Also, a 360-degree camera 81 is provided as a camera and is fixed to a clip member 93.

[0242] In this embodiment, the field device 10 is made up of a smartphone (not shown), a laser projector / camera combination unit 57, and a rear storage unit 59. The field worker 54 wears a helmet 56, and in this embodiment, the laser projector / camera combination unit 57 is held at the top of the helmet 56. A rear storage unit 59 that stores a battery and a motor control circuit is provided on the rear end side of the helmet 56.

[0243] The laser projector-camera combination unit 57 and the rear housing 59 are connected by signal and power lines (not shown). Furthermore, the field worker 54 carries a smartphone (not shown) in his breast pocket. The smartphone, the laser projector-camera combination unit 57, and the rear housing 59 are connected by signal and power lines (not shown).

[0244] The system configuration and hardware configuration are the same as those in the first embodiment.

[0245] 3.3 Remote instruction processing 28 and 29 show flowcharts of the motor control program 32 of the motor control circuit 400, the image control program 224 of the smartphone 200, and the instruction program 322 of the instruction device 30. Fig. 28 is a flowchart of the direction change control, and Fig. 29 is a flowchart of the instruction image display control.

[0246] When the site worker 54 arrives in front of the target object, the chimney 52, he attaches the laser projector case 57 to the top of his helmet 58 and turns it on. It is preferable that the helmet 58 has metal fittings on the top for attachment.

[0247] The smartphone 200 acquires an image captured by the 360-degree camera 81 via a signal line (or short-range communication) and transmits the image to the instruction device 30 via the Internet (step S21). At this time, the smartphone 200 acquires the projection direction of the laser projector 84 (the direction relative to the helmet 58, i.e., the direction on the helmet 58) and transmits it to the instruction device 30.

[0248] The instruction device 30 stores the received captured image in memory and displays it on the display 306 (step S41). The captured image is an omnidirectional image captured by a 360-degree camera. Therefore, the instruction device 30 displays only a partial image of the received captured image that matches the projection direction of the laser projector 84. In this embodiment, the instruction device 30 is controlled so that the area of ​​the partial image displayed on the instruction device 30 matches the projection area of ​​the laser projector 84.

[0249] As shown in FIG. 12, the instruction device 30 displays a direction change button 500 on the display 306. When the instructor wants to display a partial captured image in a different direction, he or she clicks on the circumference of the direction change button 500 with the mouse 316. When the instruction device 30 detects the click, it displays a partial captured image in the corresponding direction (up, down, left, right) (step S44). At this time, the direction is provided from the instruction device 30 to the on-site device 10 via the smartphone 200, and the orientation of the laser projector 84 is controlled to match this. Therefore, the area of ​​the partial image displayed on the instruction device 30 and the projection area of ​​the laser projector 84 are controlled to match. In this way, the instructor can obtain an image of the site in the desired direction.

[0250] 10, the partial captured image in the direction selected by the instructor is displayed as a moving image on the display 306 of the instructor device 30. While viewing this partial captured image, the instructor clicks the instruction input mode button 501 with the chimney 52, which is the target object, displayed.

[0251] In response to this, when the instruction input mode button 501 is clicked to give a fixed command, the instruction device 30 uses the partial captured image at that time as a reference captured image and displays it as a still image on the display 306 (step S52). The instructor inputs instructions to the on-site staff as an instruction image on this still image using the mouse 316 (step S53). For example, as shown in Fig. 13, in the image of the chimney 52 displayed on the display 306, an instruction image 62 (in this example, a cross image) indicating the position where a hole should be drilled is drawn and input using the mouse 316.

[0252] Furthermore, the instruction device 30 transmits the fixation command and the direction at the time when the fixation command was given to the smartphone 200 (step S51). Having received the fixation command, the smartphone 200 determines a reference captured image based on the captured image and the direction at the time when the fixation command was received, and records the reference captured image in the non-volatile memory 212 (step S32). After receiving the fixation command, the smartphone 200 does not transmit all captured images to the instruction device 30, but transmits only a partial captured image in the fixed direction to the instruction device 30.

[0253] As described above, the instruction device 30 and the smartphone 200 can recognize the partial captured image at the same time as the reference captured image. In order to avoid a time lag due to communication, when transmitting a fixation command from the instruction device 30 to the smartphone 200, information specifying a frame (such as a frame number) may be attached to the command. By determining the reference captured image in the smartphone 200 based on the information specifying the frame, it is possible to prevent a discrepancy due to a time lag.

[0254] When the instructor has finished inputting the instruction image, he or she clicks the instruction image transmission button 502 (which appears when the instruction input mode is entered) displayed at the bottom right of the reference captured image on the display 306. This causes the instruction device 30 to transmit the instruction image to the smartphone 200 (step S53).

[0255] The instruction device 30 also cancels the instruction input mode, stops displaying the still image as the reference captured image, and displays the transmitted partial captured image as a moving image (step S54), thereby allowing the instructor to know the situation at the site again.

[0256] 14A, the smartphone 200 stores the instruction image data in the memory 204. Furthermore, the smartphone 200 acquires the current captured image from the camera 82, and extracts a partial captured image based on the received fixed direction (step S33).

[0257] The smartphone 200 compares the image feature points of the reference captured image with the image feature points of the current partial captured image, generates signals to control the motors 92, 94, and 96 of the three-axis structure 90 to track the image feature points, and projects an instruction image from the laser projector 84 (step S35).

[0258] The generated signal is transmitted to the motor control circuit 400, which controls the motors 92, 94, and 96. This allows the laser projector 84 to project the instruction image at the correct position even if the field worker moves.

[0259] 3.4 Variations (1) In the above embodiment, the direction of the unit 80 (laser projector 84) is controlled by the triaxial structure 90. However, in addition to this, by using the same processing as in the first embodiment, the instruction image may be deformed or the projection position may be controlled based on a partial characteristic image (feature points or markers), separately from the control by the triaxial structure 90. This allows the instruction image to be projected in a more accurate position.

[0260] (2) In the above embodiment, the 360-degree camera 81 that captures images in all directions is used. However, a camera that captures images 360 degrees (or a predetermined degree) in the horizontal direction may also be used.

[0261] (3) In the above embodiment, in step S21, all images captured by the 360-degree camera 81 are transmitted to the instruction device 30. However, only a partial captured image in a direction corresponding to the direction of the unit 80 may be transmitted to the indicating device 30. In this way, it is possible to perform substantially the same processing as in the first and second embodiments.

[0262] Flowcharts for carrying out such processing are shown in FIGS.

[0263] (4) In the above embodiment, the feature points of the image are used as the partial feature images, but markers or the like may also be used.

[0264] (5) In the above embodiment, the instruction image is projected by the field device 10 and the instruction device 30 that can communicate with the field device 10 .

[0265] However, if the instruction image to be projected on-site is predetermined, the instruction image 62 may be recorded in the on-site device 10 and configured as the on-site instruction device 11.

[0266] The functional configuration of the on-site indicator 11 constructed in this manner is shown in Fig. 32. An indicator image 62 is recorded in the recording unit 24. The tracking control means 21 controls the driving unit 16 based on the marker 60 or feature points in the reference captured image and the current captured image, so that the indicator image is projected so as to track the marker 60. Therefore, the indicator image 62 is projected onto the object 52.

[0267] The on-site instruction device 11 may be worn by the on-site staff 54, or may be attached to a mobile body (or a fixed body) near the on-site staff.

[0268] In this embodiment, the instruction image 62 is recorded in the recording unit 24, but it may also be acquired from outside the on-site instruction device 11 by communication or the like.

[0269] A flowchart of the instruction process is shown in Fig. 33. In this example, the on-site instruction device 11 is configured by a motor control circuit 400 and a smartphone 200. The process of the motor control circuit 400 is the same as in the above-described embodiment.

[0270] The smartphone 200 acquires a captured image (step S71) and determines whether a predetermined marker 60 has been captured (step S72).

[0271] When the marker 60 is captured, the smartphone 200 records the partial captured image in which the marker 60 is captured as a reference captured image (step S77). At this time, the direction of the partial captured image is selected so that the marker 60 is in a predetermined area (for example, the center). This direction is also recorded.

[0272] Next, a partial captured image in the above direction is acquired from the current captured image, and compared with the reference captured image to generate a motor control signal for capturing an image while tracking the marker 60 (step S78). This motor control signal is provided to the motor control circuit 400.

[0273] In response to this, the motor control circuit 400 controls the motors 92, 94, and 96, and controls the three-axis structure 90 so as to follow and image the marker 60.

[0274] The control for following the marker 60 may be performed in the motor control circuit.

[0275] (6) The above-mentioned modifications may be applied in combination with each other as long as it does not contradict the essence of the modifications. They may also be applied in combination with other embodiments and their modifications.

[0276] 4. Fourth Embodiment 4.1 Functional configuration In the third embodiment, the projection unit 14 is held by a triaxial structure 90, and the wide-angle imaging unit 12 is fixedly provided. In this embodiment, not only the imaging unit 12 but also the projection unit 14 has a wide angle.

[0277] 34 shows the functional configuration of a remote instruction system according to an embodiment of the present invention. This system includes a field device 10 used by a field staff member and an instruction device 30 used by a remote instructor.

[0278] The on-site worker wears a wide-angle projection unit 14 and a wide-angle imaging unit 12. In this embodiment, a 360-degree camera is used as the wide-angle imaging unit 12, allowing for all-around imaging. Also, a laser projector capable of projecting in all directions 360 degrees is used as the wide-angle projection unit 14, allowing for all-around projection.

[0279] The imaging unit 12 of the on-site device 10 captures an image of the on-site space including the target object 52 to generate a captured image. As described above, the imaging unit 12 captures images in all directions of 360 degrees, so that even if the on-site person moves or changes direction, a captured image including the on-site space can be obtained.

[0280] This captured image is transmitted to the instruction device 30 by the transmission unit 22 under the control of the captured image transmission means 18. The captured image receiving means 36 of the instruction device 30 receives the captured image by the reception unit 32. The captured image display unit 40 displays the received captured image. Note that since the captured image is an image captured in a 360-degree circumferential direction, it is not displayed all at once, but rather partially (partial captured image). The instructor can look around the site worker by changing the direction up, down, left, or right. This makes it possible to display a partial captured image in the direction in which the object in the site space is displayed.

[0281] When giving an instruction, the instructor inputs a fixation command while a captured image including the target object is displayed. When a fixation command is input, the captured image display unit 40 displays a partial captured image in that direction as a reference captured image and displays it as a still image. The direction when the fixation command is given is transmitted to the field device 10.

[0282] The instructor inputs an instruction image from the instruction image input unit 44 while viewing the reference captured image of the site displayed on the captured image display unit 40. The instruction image transmission means 38 transmits the input instruction image to the on-site device 10 via the transmission unit 34.

[0283] The on-site device 10 receives the instruction image via the receiving unit 24, and controls the projection unit 14 to project the instruction image in the direction determined when the fixation command was given. As a result, the instruction image 62 is projected onto the target object 52.

[0284] The projection direction of the instruction image by the projection unit 14 is matched with the direction of the reference captured image, so the instruction image is projected at the location intended by the instructor. Although this control alone is feasible, if the on-site personnel moves to another location, the projection position of the instruction image will shift.

[0285] Therefore, the correction means 26 of the on-site device 10 compares the characteristic part image (such as a marker) in the reference captured screen with the characteristic part image in the current captured image, and transforms the instruction image or corrects and controls the projection position of the instruction image so that the instruction image is correctly projected at the intended position. As a result, even if the on-site worker moves, the instruction image is displayed in the correct position.

[0286] The on-site worker can confirm the position where he or she should work based on the instruction image 62 actually projected in the on-site space. This instruction image 62 is displayed in the correct position by the tracking control means 21 and the correction means 26 even if the worker changes the direction of his or her head. Therefore, the instruction image 62 does not disappear depending on the direction of the worker's head, making it easier for the worker to work. Furthermore, when multiple workers are working together, even if the on-site worker wearing the on-site device 10 moves his or her head significantly, the instruction image 62 continues to be projected, and the work of the other workers is not interrupted. Furthermore, even if the worker moves, the instruction image 62 is displayed correctly.

[0287] 4.2 Appearance and hardware configuration In this embodiment, the three-axis structure 90 is not used, and a 360-degree camera 81 and a 360-degree laser projector 83 are fixed to the top of the helmet 56.

[0288] The hardware configuration of the instruction device 30 is the same as that of the first embodiment (see FIG. 7). Furthermore, since the three-axis structure 90 is not used, the motors 92, 94, and 96 that control it are not required, and the motor control circuit 400 is also not required. The hardware configuration of the smartphone 200 is the same as that of the first embodiment (see FIG. 8). However, instead of the camera 82 and the laser projector 94, a 360-degree camera 81 and a 360-degree laser projector 83 are connected.

[0289] 4.3 Remote instruction processing 35 and 36 show flowcharts of the image control program 224 of the smartphone 200 and the instruction program 322 of the instruction device 30. Fig. 35 is a flowchart of the direction change control, and Fig. 36 is a flowchart of the instruction image display control.

[0290] When the site worker 54 arrives at the site and reaches the target object, the chimney 52, he attaches the laser projector housing 57 to the top of his helmet 58 and turns it on.

[0291] The smartphone 200 acquires an image captured by the 360-degree camera 81 via a signal line (or short-range communication), and transmits the image to the instruction device 30 via the Internet (step S21).

[0292] The instruction device 30 stores the received captured image in memory and displays it on the display 306 (step S41). The captured image is an omnidirectional image captured by a 360-degree camera. Therefore, the instruction device 30 displays only a partial image in a predetermined direction from the received captured image.

[0293] As shown in Fig. 12, the instruction device 30 displays a direction change button 500 on the display 306. When the instructor wants to display a partial captured image in a different direction, the instructor clicks on the circumference of the direction change button 500 with the mouse 316. When the instruction device 30 detects the click, it displays a partial captured image in the corresponding direction (up, down, left, right) (step S44). In this way, the instructor can obtain an image of the site in the desired direction.

[0294] Through the above-described processing, the partial captured image in the direction selected by the instructor is displayed as a moving image on the display 306 of the instructor device 30. While viewing this partial captured image, the instructor clicks the instruction input mode button 502 with the chimney 52, which is the target object, displayed.

[0295] In response to this, when the instruction input mode button 501 is clicked to give a fixed command, the instruction device 30 uses the partial captured image at that time as a reference captured image and displays it as a still image on the display 306 (step S52). The instructor inputs instructions to the on-site staff as an instruction image on this still image using the mouse 316 (step S53). For example, as shown in Fig. 13, in the image of the chimney 52 displayed on the display 306, an instruction image 62 (in this example, a cross image) indicating the position where a hole should be drilled is drawn and input using the mouse 316.

[0296] Furthermore, the instruction device 30 transmits the fixation command and the direction at the time when the fixation command was given to the smartphone 200 (step S51). Having received the fixation command, the smartphone 200 determines a reference captured image based on the captured image and the direction at the time when the fixation command was received, and records the reference captured image in the non-volatile memory 212 (step S32). After receiving the fixation command, the smartphone 200 does not transmit all captured images to the instruction device 30, but transmits only a partial captured image in the fixed direction to the instruction device 30.

[0297] As described above, the instruction device 30 and the smartphone 200 can recognize the partial captured image at the same time as the reference captured image. In order to avoid a time lag due to communication, when transmitting a fixation command from the instruction device 30 to the smartphone 200, information specifying a frame (such as a frame number) may be attached to the command. By determining the reference captured image in the smartphone 200 based on the information specifying the frame, it is possible to prevent a discrepancy due to a time lag.

[0298] When the instructor has finished inputting the instruction image, he or she clicks the instruction image transmission button 502 (which appears when the instruction input mode is entered) displayed at the bottom right of the reference captured image on the display 306. This causes the instruction device 30 to transmit the instruction image to the smartphone 200 (step S53).

[0299] The instruction device 30 also cancels the instruction input mode, stops displaying the still image as the reference captured image, and displays the transmitted partial captured image as a moving image (step S54), thereby allowing the instructor to know the situation at the site again.

[0300] 14A, the smartphone 200 stores the instruction image data in the memory 204. Furthermore, the smartphone 200 acquires the current captured image from the camera 82, and extracts a partial captured image based on the received fixed direction (step S33).

[0301] The smartphone 200 compares the markers in the reference captured image with the markers in the current partially captured image, and controls the projection direction of the instruction image by the 360-degree laser projector so that the instruction image is projected correctly by tracking the markers (step S34). Furthermore, based on the comparison of the markers, the smartphone 200 deforms the instruction image so that the instruction image is projected correctly, and controls the projection position.

[0302] 4.4 Variations (1) In the above embodiment, a 360-degree camera 81 that captures images in all directions and a 360-degree laser projector 83 that projects images in all directions are used. However, a camera that captures images in 360 degrees (or a predetermined degree) in the horizontal direction and a laser projector that projects images may also be used.

[0303] (2) In the above embodiment, markers are used as partial feature images, but feature points of an image may also be used.

[0304] (3) In the above embodiment, the instruction image is projected by the field device 10 and the instruction device 30 that can communicate with the field device 10 .

[0305] However, if the instruction image to be projected on-site is predetermined, the instruction image 62 may be recorded in the on-site device 10 and configured as the on-site instruction device 11.

[0306] The functional configuration of the on-site indicator 11 thus constructed is shown in Fig. 37. An indicator image 62 is recorded in the recording unit 24. The tracking control means 21 controls the indicator image so that it is projected at a desired position based on the markers 60 or feature points in the reference captured image and the current captured image. Therefore, the indicator image 62 is projected onto the target object 52.

[0307] The on-site instruction device 11 may be worn by the on-site staff 54, or may be attached to a mobile body (or a fixed body) near the on-site staff.

[0308] In this embodiment, the instruction image 62 is recorded in the recording unit 24, but it may also be acquired from outside the on-site instruction device 11 by communication or the like.

[0309] A flowchart of the instruction process is shown in Figure 38. In this example, the on-site instruction device 11 is made up of a 360-degree camera 81, a 360-degree laser projector 83, and a smartphone 200.

[0310] The smartphone 200 acquires a captured image (step S71) and determines whether a predetermined marker 60 has been captured (step S72).

[0311] When the marker 60 is captured, the smartphone 200 records the partial captured image in which the marker 60 is captured as a reference captured image (step S77). At this time, the direction of the partial captured image is selected so that the marker 60 is in a predetermined area (for example, the center). This direction is also recorded.

[0312] The smartphone 200 compares the markers in the reference captured image with the markers in the current partially captured image, and controls the projection direction of the instruction image by the 360-degree laser projector so that the instruction image is projected correctly by tracking the markers (step S79). Furthermore, based on the comparison of the markers, the smartphone 200 deforms the instruction image so that the instruction image is projected correctly, and controls the projection position.

[0313] (4) In the above embodiment, the camera 81 and the projector 83 are attached directly to the helmet. However, they may be attached via a cushioning material such as silicon gel.

[0314] (5) The above-mentioned modifications may be applied in combination with each other as long as it does not contradict the essence of the modifications. They may also be applied in combination with other embodiments and their modifications.

Claims

1. A remote instruction system including an instruction device used by an instructor and an on-site device used by an on-site person, The field device is an imaging unit that is attached to a site staff member or a mobile body and captures an image of the site space to generate a captured image; a captured image transmitting means for transmitting the captured image to the instruction device by a transmitting unit; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a driving unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit; a direction control means for receiving an output from a sensor that detects the orientation of the image capturing unit and the projection unit, and controlling a drive unit so that the image capturing unit and the projection unit face in a predetermined direction with the site worker at the center, regardless of the movement of the site worker or the moving object; a correction means for correcting the projection of the instruction image by the projection unit without using the drive unit, based on a comparison between a characteristic portion image in the reference image taken when the fixation command was given and a characteristic portion image in the current image taken, using the image taken when the fixation command was given as a reference image, so that the instruction image is correctly displayed with reference to a predetermined portion of the site space; The indicating device a captured image receiving means for receiving the captured image transmitted by the receiving unit; a captured image display unit that displays the received captured image; a fixation command means for issuing a fixation command to the on-site device via a transmission unit so that a desired image of the on-site space is captured; an instruction image input unit for inputting an instruction image at a desired position in the scene space by an instruction user's operation in the displayed captured image; an instruction image transmitting means for transmitting, to the on-site device, instruction image data in which the position of the instruction image is specified by a transmitting unit, so that the projection unit of the on-site device correctly projects the instruction image based on a characteristic partial image of the on-site space included in the captured image, with a predetermined portion of the on-site space as a reference; A remote instruction system comprising:

2. A field device for constructing a remote instruction system together with an instruction device, an imaging unit that is attached to a site staff member or a mobile body and captures an image of the site space to generate a captured image; a captured image transmitting means for transmitting the captured image to the instruction device by a transmitting unit; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a driving unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit; a direction control means for receiving an output from a sensor that detects the orientation of the image capturing unit and the projection unit, and controlling a drive unit so that the image capturing unit and the projection unit face in a predetermined direction with the site worker at the center, regardless of the movement of the site worker or the moving object; a correction means for correcting the projection of the instruction image by the projection unit without using the drive unit, based on a comparison between a characteristic portion image in the reference image when the fixation command is given from the instruction device and a characteristic portion image in the current image, using the image when the fixation command is given as a reference image, so that the instruction image is correctly displayed with reference to a predetermined portion of the site space; Field equipment equipped with

3. A field device correction program for realizing, by a computer, a field device including an imaging unit that is attached to a field staff member or a mobile body and that captures an image of a field space to generate a captured image, an imaged image transmission means that transmits the captured image to an instruction device by a transmission unit, a projection unit that is attached to the field staff member or a mobile body and that projects an instruction image into the field space based on given instruction image data, and a drive unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit, the program comprising: a direction control means for receiving an output from a sensor that detects the orientation of the image capturing unit and the projection unit, and controlling a drive unit so that the image capturing unit and the projection unit face in a predetermined direction with the site worker at the center, regardless of the movement of the site worker or the moving object; A field device correction program for functioning as a correction means for correcting the projection of an instruction image by the projection unit without using the drive unit, so that the instruction image is correctly displayed based on a specified part of the field space, based on a comparison between the characteristic part image in the reference image when a fixation command is given from the instruction device and the characteristic part image in the current image.

4. An instruction device for constructing a remote instruction system together with a field device, a captured image receiving means for receiving the captured image transmitted by the receiving unit; a captured image display unit that displays the received captured image; a fixation command means for issuing a fixation command to the on-site device via a transmission unit so that a desired image of the on-site space is captured; an instruction image input unit for inputting an instruction image at a desired position in the scene space by an instruction user's operation in the displayed captured image; an instruction image transmitting means for transmitting, to the on-site device, instruction image data in which the position of the instruction image is specified by a transmitting unit, so that the projection unit of the on-site device correctly projects the instruction image based on a characteristic partial image of the on-site space included in the captured image, with a predetermined portion of the on-site space as a reference; An indicating device comprising:

5. An instruction device program including a captured image display unit that displays a received captured image, and an instruction image input unit that inputs an instruction image at a desired position in a site space in the displayed captured image by an instruction person's operation, the program comprising: a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixation command means for issuing a fixation command to the on-site device via a transmission unit so that a desired image of the on-site space is captured; An instruction device program that functions as an instruction image transmission means that transmits instruction image data that identifies the position of the instruction image to the on-site device using a transmission unit, so that the projection unit of the on-site device correctly projects the instruction image based on a characteristic partial image of the site space contained in the captured image, using a specified part of the site space as a reference.

6. In any one of the systems, devices, or programs of claims 1 to 3, A system, device, or program, wherein the imaging unit and the projection unit are fixed to the drive unit via a member that absorbs high-frequency vibrations.

7. In any one of the systems, devices, or programs of claims 1 to 3, A system, device, or program characterized in that the imaging unit and the projection unit are fixed to a helmet of a field worker via the drive unit.

8. In any one of the systems, devices, or programs of claims 1 to 3, The direction control means changes the predetermined direction based on a direction instruction from the instruction device.

9. In any one of the systems, devices, or programs of claims 1 to 3, The system, device, or program is characterized in that the characteristic part image is a marker provided in the site space or a characteristic point of the captured image.

10. In the system, device, or program according to any one of claims 1 to 3, The system, device, or program is characterized in that the drive unit has a multi-axis drive mechanism.

11. an imaging unit that is attached to a site staff member or a mobile body and captures an image of the site space to generate a captured image; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a driving unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit; a direction control means for receiving an output from a sensor that detects the orientation of the image capturing unit and the projection unit, and controlling a drive unit so that the image capturing unit and the projection unit face in a predetermined direction with the site worker at the center, regardless of the movement of the site worker or the moving object; a correction means for correcting the projection of the indication image by the projection unit without using the drive unit, based on a characteristic partial image of the site space included in the captured image, so that the indication image is correctly displayed with reference to a predetermined portion of the site space; and A field indicating device equipped with the

12. An on-site instruction program for implementing, by a computer, an on-site instruction device including an imaging unit that is attached to a site staff member or a mobile body and that images a site space to generate an imaged image, a projection unit that is attached to a site staff member or a mobile body and that projects an instruction image into the site space based on given instruction image data, and a drive unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit, the program comprising: a direction control means for receiving an output from a sensor that detects the orientation of the image capturing unit and the projection unit, and controlling a drive unit so that the image capturing unit and the projection unit face in a predetermined direction with the site worker at the center, regardless of the movement of the site worker or the moving object; A site instruction program that functions as a correction means for correcting the projection of an instruction image by the projection unit without using the drive unit, so that the instruction image is correctly displayed based on a specified portion of the site space, based on a characteristic partial image of the site space contained in the captured image.

13. A remote instruction system including an instruction device used by an instructor and an on-site device used by an on-site person, The field device is an imaging unit that is attached to a site staff member or a mobile body and captures an image of the site space to generate a captured image; a captured image transmitting means for transmitting the captured image to the instruction device by a transmitting unit; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a driving unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit; a tracking control means for controlling the drive unit so that the image captured when a fixation command is given is used as a reference image, and based on a comparison between a characteristic portion image in the reference image captured when the fixation command is given and a characteristic portion image in the current captured image, the instruction image is correctly displayed based on a predetermined portion of the work site space, regardless of the movement of the work site staff member or the moving body; The indicating device a captured image receiving means for receiving the captured image transmitted by the receiving unit; a captured image display unit that displays the received captured image; a fixation command means for issuing a fixation command to the on-site device via a transmission unit so that a desired image of the on-site space is captured; an instruction image input unit for inputting an instruction image at a desired position in the scene space by an instruction user's operation in the displayed captured image; an instruction image transmitting means for controlling a driving unit based on an image of a characteristic portion of the site space included in the captured image to transmit, via a transmitting unit, instruction image data specifying the position of the instruction image to the on-site device so that a projection unit of the on-site device correctly projects the instruction image based on a predetermined portion of the site space; A remote instruction system comprising:

14. A field device for constructing a remote instruction system together with an instruction device, an imaging unit that is attached to a site staff member or a mobile body and captures an image of the site space to generate a captured image; a captured image transmitting means for transmitting the captured image to the instruction device by a transmitting unit; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a driving unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit; a tracking control means for controlling the drive unit so that the image captured when a fixation command is given is used as a reference image, and the image of the characteristic part in the reference image captured when the fixation command is given is compared with the image of the characteristic part in the current captured image, so that the indication image is correctly displayed based on a predetermined part of the work site space, regardless of the movement of the work site staff or the moving body; Field equipment equipped with

15. 1. A field device program for implementing, by a computer, a field device including an imaging unit that is attached to a field staff member or a mobile body and that captures an image of a field space to generate a captured image, an imaged image transmission means that transmits the captured image to an instruction device by a transmission unit, a projection unit that is attached to the field staff member or a mobile body and that projects an instruction image into the field space based on given instruction image data, and a drive unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit, the program comprising: A field device program for functioning as a tracking control means that controls the drive unit so that the instruction image is correctly displayed based on a specified part of the field space, regardless of the movement of the field staff or the moving body, by using the image captured when a fixing command is given as a reference image and comparing the characteristic part image in the reference image captured when the fixing command is given with the characteristic part image in the current captured image.

16. An instruction device for constructing a remote instruction system together with a field device, a captured image receiving means for receiving the captured image transmitted by the receiving unit; a captured image display unit that displays the received captured image; a fixation command means for issuing a fixation command to the on-site device via a transmission unit so that a desired image of the on-site space is captured; an instruction image input unit for inputting an instruction image at a desired position in the scene space by an instruction user's operation in the displayed captured image; an instruction image transmitting means for controlling a driving unit based on an image of a characteristic portion of the site space included in the captured image to transmit, via a transmitting unit, instruction image data specifying the position of the instruction image to the on-site device so that a projection unit of the on-site device correctly projects the instruction image based on a predetermined portion of the site space; An indicating device comprising:

17. An instruction device for constructing a remote instruction system together with an on-site device, the instruction device having a captured image display unit that displays a received captured image, and an instruction image input unit that inputs an instruction image at a desired position in the on-site space by operation of an instructor in the displayed captured image, the instruction device program for realizing the instruction device by a computer, the computer comprising: a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixation command means for issuing a fixation command to the on-site device via a transmission unit so that a desired image of the on-site space is captured; An instruction device program that functions as an instruction image transmission means that controls a drive unit based on a characteristic partial image of the site space contained in the captured image, and causes a transmission unit to transmit instruction image data that identifies the position of the instruction image to the site device so that the projection unit of the site device correctly projects the instruction image based on a specified part of the site space.

18. In any one of the systems, devices, or programs of claims 13 to 15, A system, device, or program, wherein the imaging unit and the projection unit are fixed to the drive unit via a member that absorbs high-frequency vibrations.

19. In any one of the systems, devices, or programs of claims 13 to 15, A system, device, or program characterized in that the imaging unit and the projection unit are fixed to a helmet of a field worker via the drive unit.

20. In any one of the systems, devices, or programs of claims 13 to 15, The tracking control means changes the imaging direction and the projection direction based on a direction instruction from the instruction device.

21. In any one of the systems, devices, or programs of claims 13 to 17, The system, device, or program is characterized in that the characteristic part image is a marker provided in the site space or a characteristic point of the captured image.

22. In any one of the systems, devices, or programs of claims 13 to 17, The system, device, or program is characterized in that the drive unit has a multi-axis drive mechanism.

23. In any one of the systems, devices, or programs of claims 13 to 17, The system, device, or program further comprises a correction means for correcting the projection of the indication image by the projection unit without using the drive unit, based on a comparison of the image of the characteristic part in the reference image taken when the fixation command was given with the image of the characteristic part in the current image taken, using the image taken when the fixation command was given as a reference image, so that the indication image is correctly displayed based on a specified part of the site space.

24. an imaging unit that is attached to a site staff member or a mobile body and captures an image of the site space to generate a captured image; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a driving unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit; a tracking control means for controlling a drive unit based on a characteristic partial image of the site space included in the captured image so that an instruction image is correctly displayed based on a predetermined portion of the site space, regardless of the movement of the site worker or the moving object; and A field indicating device equipped with the device.

25. An on-site instruction program for implementing, by a computer, an on-site instruction device including an imaging unit that is attached to a site staff member or a mobile body and that images a site space to generate an imaged image, a projection unit that is attached to a site staff member or a mobile body and that projects an instruction image into the site space based on given instruction image data, and a drive unit that changes the imaging direction of the imaging unit and the projection direction of the projection unit, the program comprising: A site instruction program that functions as a tracking control means that controls the drive unit so that the instruction image is displayed correctly based on a specified part of the site space, regardless of the movement of the site staff or the moving body, based on a characteristic partial image of the site space contained in the captured image.

26. A remote instruction system including an instruction device used by an instructor and an on-site device used by an on-site person, The field device is an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space in a wide-angle direction to generate a captured image; a captured image transmitting means for transmitting the captured image to the instruction device by a transmitting unit; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a drive unit that changes the projection direction of the projection unit; a direction control means for controlling the drive unit so that the projection unit faces a predetermined direction centered on the site staff member, regardless of the movement of the site staff member or the moving body; a correction unit that corrects the projection of the instruction image by the projection unit without using the drive unit so that the instruction image is correctly displayed based on a predetermined portion of the site space; The indicating device a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixed command means for giving a fixed command to the field device by a transmission unit; an instruction image input unit that, when a fixing command is received, sets a neighborhood of the characteristic portion image of the captured image as a target captured image, and inputs an instruction image at a desired position in the scene space in the target captured image by an operation of an instructor; an instruction image transmitting means for transmitting instruction image data specifying the position of the instruction image to the instruction device by a transmitting unit in order to control the driving unit so that the projection unit projects the instruction image based on a predetermined portion of the site space regardless of the movement of the site staff or the mobile body, and to control the projection of the projection unit; A remote instruction system comprising:

27. 1. A field device for a remote indication system, comprising: an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space in a wide-angle direction to generate a captured image; a captured image transmitting means for transmitting the captured image to an instruction device by a transmitting unit; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a drive unit that changes the projection direction of the projection unit; a direction control means for controlling the drive unit so that the projection unit faces a predetermined direction centered on the site staff member, regardless of the movement of the site staff member or the moving body; a correction unit that corrects the projection of the instruction image by the projection unit without using the drive unit so that the instruction image is correctly displayed based on a predetermined portion of the site space; and Field equipment equipped with

28. A field device program for implementing, by a computer, a field device including an imaging unit that is attached to a field staff member or a mobile body and captures a field space in a wide-angle direction to generate a captured image, a captured image transmission means that transmits the captured image to an instruction device by a transmission unit, a projection unit that is attached to the field staff member or a mobile body and projects an instruction image into the field space based on given instruction image data, and a drive unit that changes the projection direction of the projection unit, the program comprising: a direction control means for controlling the drive unit so that the projection unit faces a predetermined direction centered on the site staff member, regardless of the movement of the site staff member or the moving body; A field device program for causing the field device to function as a correction means for correcting the projection of the instruction image by the projection unit without relying on the drive unit so that the instruction image is displayed correctly based on a specified portion of the field space.

29. 1. A pointing device for a remote pointing system, comprising: a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixed command means for giving a fixed command to the field device by a transmission unit; an instruction image input unit that, when a fixing command is received, sets a neighborhood of the characteristic portion image of the captured image as a target captured image, and inputs an instruction image at a desired position in the scene space in the target captured image by an operation of an instructor; an instruction image transmitting means for transmitting instruction image data specifying the position of the instruction image to the instruction device by a transmitting unit in order to control the driving unit and the projection unit so that the projection unit projects the instruction image based on a predetermined part of the site space, regardless of the movement of the site worker or the mobile body to which the imaging unit for capturing the image and the projection unit are attached; An indicating device comprising:

30. An instruction device program for implementing an instruction device by a computer, the instruction device including an instruction image input unit that, when a fixation command is received, sets a neighborhood of a characteristic portion image of a captured image as a target captured image, and inputs an instruction image at a desired position in a site space by an instruction user's operation in the target captured image, the program comprising: a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixed command means for giving a fixed command to the field device by a transmission unit; An instruction device program that functions as an instruction image transmission means that controls a drive unit so that the projection unit projects the instruction image based on a specified location in the site space, regardless of the movement of the site staff member or mobile body to which the imaging unit and projection unit that capture the imaged image are attached, and that transmits instruction image data that identifies the position of the instruction image to the instruction device via a transmission unit to control the projection of the projection unit.

31. In the system, device, or program according to any one of claims 26 to 30, A system, device, or program, wherein the projection unit is fixed to the drive unit via a member that absorbs high-frequency vibrations.

32. In the system, device, or program according to any one of claims 26 to 31, A system, device, or program characterized in that the projection unit is fixed to a helmet of a field worker via the drive unit.

33. In any one of the systems, devices, or programs of claims 26, 29, and 30, The system, device, or program is characterized in that the characteristic part image is a marker provided in the site space or a characteristic point of the captured image.

34. In the system, device, or program according to any one of claims 26 to 33, The system, device, or program is characterized in that the drive unit has a multi-axis drive mechanism.

35. an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space in a wide-angle direction to generate a captured image; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a drive unit that changes the projection direction of the projection unit; a direction control means for controlling the drive unit so that the projection unit faces a predetermined direction centered on the site staff member, regardless of the movement of the site staff member or the moving body; a correction unit that corrects the projection of the instruction image by the projection unit without using the drive unit so that the instruction image is correctly displayed based on a predetermined portion of the site space; and A field indicating device equipped with the device.

36. A field instruction program for implementing, by a computer, a field instruction device including an imaging unit that is attached to a field staff member or a mobile body and that captures a field space in a wide-angle direction to generate an image, a projection unit that is attached to a field staff member or a mobile body and that projects an instruction image into the field space based on given instruction image data, and a drive unit that changes the projection direction of the projection unit, the program comprising: a direction control means for controlling the drive unit so that the projection unit faces a predetermined direction centered on the site staff member, regardless of the movement of the site staff member or the moving body; An on-site instruction program that functions as a correction means for correcting the projection of the instruction image by the projection unit without using the drive unit so that the instruction image is displayed correctly based on a specified portion of the site space.

37. A remote instruction system including an instruction device used by an instructor and an on-site device used by an on-site person, The field device is an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space in a wide-angle direction to generate a captured image; a captured image transmitting means for transmitting the captured image to the instruction device by a transmitting unit; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a drive unit that changes the projection direction of the projection unit; a tracking control means for controlling the drive unit so that the instruction image is correctly displayed based on a predetermined portion of the site space; The indicating device a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixed command means for giving a fixed command to the field device by a transmission unit; an instruction image input unit that, when a fixing command is received, sets a neighborhood of the characteristic portion image of the captured image as a target captured image, and inputs an instruction image at a desired position in the scene space in the target captured image by an operation of an instructor; an instruction image transmitting means for transmitting instruction image data specifying the position of the instruction image to the instruction device by a transmitting unit in order to control a driving unit so that the projection unit projects the instruction image based on a predetermined portion of the site space regardless of the movement of the site staff or the mobile body; A remote instruction system comprising:

38. 1. A field device for a remote indication system, comprising: an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space in a wide-angle direction to generate a captured image; a captured image transmitting means for transmitting the captured image to an instruction device by a transmitting unit; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a drive unit that changes the projection direction of the projection unit; a tracking control means for controlling the driving unit so that the instruction image is correctly displayed based on a predetermined portion of the site space; Field equipment equipped with

39. A field device program for implementing, by a computer, a field device including an imaging unit that is attached to a field staff member or a mobile body and captures a field space in a wide-angle direction to generate a captured image, a captured image transmission means that transmits the captured image to an instruction device by a transmission unit, a projection unit that is attached to the field staff member or a mobile body and projects an instruction image into the field space based on given instruction image data, and a drive unit that changes the projection direction of the projection unit, the program comprising: a captured image transmitting means for transmitting the captured image to the instruction device by a transmitting unit; a tracking control means for controlling the driving unit so that the instruction image is correctly displayed based on a predetermined portion of the site space; Field equipment program with.

40. 1. A pointing device for a remote pointing system, comprising: a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixed command means for giving a fixed command to the field device by a transmission unit; an instruction image input unit that, when a fixing command is received, sets a neighborhood of the characteristic portion image of the captured image as a target captured image, and inputs an instruction image at a desired position in the scene space in the target captured image by an operation of an instructor; an instruction image transmitting means for transmitting instruction image data specifying the position of the instruction image to an instruction device by a transmitting unit, in order to control a driving unit so that the projection unit projects the instruction image based on a predetermined portion of the site space, regardless of the movement of a site worker or a mobile body equipped with an imaging unit for capturing an image and a projection unit; An indicating device comprising:

41. An instruction device program for implementing an instruction device by a computer, the instruction device including an instruction image input unit that, when a fixation command is received, sets a neighborhood of a characteristic portion image of a captured image as a target captured image, and inputs an instruction image at a desired position in a site space by an instruction user's operation in the target captured image, the program comprising: a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixed command means for giving a fixed command to the field device by a transmission unit; An instruction device program that functions as an instruction image transmission means that transmits instruction image data that identifies the position of the instruction image to the instruction device via a transmission unit, in order to control a drive unit so that the projection unit projects the instruction image based on a specified location in the site space, regardless of the movement of a site worker or mobile body to which an imaging unit that captures an image and a projection unit are attached.

42. In the system, device, or program according to any one of claims 37 to 41, The projection unit is fixed to the drive unit via a member that absorbs high-frequency vibrations.

43. In the system, device, or program of any one of claims 37 to 41, A system, device, or program characterized in that the projection unit is fixed to a helmet of a field worker via the drive unit.

44. In any one of the systems, devices, or programs of claims 37, 40, and 41, The system, device, or program is characterized in that the characteristic part image is a marker provided in the site space or a characteristic point of the captured image.

45. In the system, device, or program of any one of claims 37 to 41, The system, device, or program is characterized in that the drive unit has a multi-axis drive mechanism.

46. In any one of the systems, devices, or programs of claims 37, 40, and 41, The system, device, or program further comprises a correction means for correcting the projection of the indication image by the projection unit without using the drive unit, based on a comparison of the image of the characteristic part in the reference image taken when the fixation command was given with the image of the characteristic part in the current image taken, using the image taken when the fixation command was given as a reference image, so that the indication image is correctly displayed based on a specified part of the site space.

47. an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space in a wide-angle direction to generate a captured image; a projection unit that is attached to the site staff or the mobile body and projects an instruction image into the site space based on given instruction image data; a drive unit that changes the projection direction of the projection unit; a tracking control means for controlling the driving unit so that the instruction image is correctly displayed based on a predetermined portion of the site space; A field indicating device equipped with the device.

48. A field instruction program for implementing, by a computer, a field instruction device including an imaging unit that is attached to a field staff member or a mobile body and that captures a field space in a wide-angle direction to generate an image, a projection unit that is attached to a field staff member or a mobile body and that projects an instruction image into the field space based on given instruction image data, and a drive unit that changes the projection direction of the projection unit, the program comprising: A site instruction program for functioning as a tracking control means for controlling the drive unit so that the instruction image is displayed correctly based on a predetermined portion of the site space.

49. A remote instruction system including an instruction device used by an instructor and an on-site device used by an on-site person, The field device is an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space in a wide-angle direction to generate a captured image; a captured image transmitting means for transmitting the captured image to the instruction device by a transmitting unit; a projection unit that is attached to a site staff member or a mobile body, is capable of projecting in a wide-angle direction onto a site space, and projects an instruction image onto the site space based on given instruction image data; and a tracking control means for receiving a fixing command from the instruction device and controlling the projection unit to project the instruction image based on a predetermined portion of the work site space, regardless of the movement of the work site worker or the moving object, based on a characteristic partial image of the work site space included in the captured image. The indicating device a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixed command means for giving a fixed command to the field device by a transmission unit; an instruction image input unit that, when a fixing command is received, sets a neighborhood of the characteristic portion image of the captured image as a target captured image, and inputs an instruction image at a desired position in the scene space in the target captured image by an operation of an instructor; an instruction image transmitting means for transmitting instruction image data in which the position of the instruction image is specified to the instruction device by a transmitting unit; A remote instruction system comprising:

50. 1. A field device for a remote indication system, comprising: an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space in a wide-angle direction to generate a captured image; a captured image transmitting means for transmitting the captured image to an instruction device by a transmitting unit; a projection unit that is attached to a site staff member or a mobile body, is capable of projecting in a wide-angle direction onto a site space, and projects an instruction image onto the site space based on given instruction image data; a tracking control means for receiving a fixation command from the instruction device and controlling the projection unit to project the instruction image based on a predetermined portion of the work site space, regardless of the movement of the work site worker or the moving object, based on a characteristic partial image of the work site space included in the captured image; and Field equipment equipped with

51. A field device program for implementing, by a computer, a field device including an imaging unit that is attached to a field staff member or a mobile body and captures a field space in a wide-angle direction to generate a captured image, and a projection unit that is attached to a field staff member or a mobile body and is capable of projecting onto the field space in a wide-angle direction, and projects an instruction image onto the field space based on given instruction image data, the field device program comprising: An on-site device program that functions as an image capturing means for transmitting the captured image to an instruction device via a transmission unit, and a tracking control means that, upon receiving a fixed command from the instruction device, controls the projection unit to project the instruction image based on a specified part of the site space, regardless of the movement of the on-site person in charge or the moving body, based on a characteristic partial image of the site space contained in the captured image.

52. 1. A pointing device for a remote pointing system, comprising: a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixed command means for giving a fixed command to the field device by a transmission unit; an instruction image input unit that, when a fixing command is received, sets a neighborhood of the characteristic portion image of the captured image as a target captured image, and inputs an instruction image at a desired position in the scene space in the target captured image by an operation of an instructor; an instruction image transmitting means for transmitting instruction image data specifying the position of the instruction image to the instruction device by a transmitting unit so that the projection unit projects the instruction image based on a predetermined portion of the site space, regardless of the movement of the site personnel or the mobile body to which the imaging unit for capturing the image and the projection unit are attached; An indicating device comprising:

53. An instruction device program for implementing an instruction device by a computer, the instruction device including an instruction image input unit that, when a fixation command is received, sets a neighborhood of a characteristic portion image of a captured image as a target captured image, and inputs an instruction image at a desired position in a site space by an instruction user's operation in the target captured image, the program comprising: a captured image receiving means for receiving the captured image transmitted by the receiving unit; a fixed command means for giving a fixed command to the field device by a transmission unit; An instruction device program that functions as an instruction image transmission means that transmits instruction image data that identifies the position of the instruction image to the instruction device via a transmission unit, so that the projection unit projects the instruction image based on a specified location in the site space, regardless of the movement of the site staff member or mobile body to which the imaging unit and projection unit that capture the image are attached.

54. In any one of the systems, devices, or programs of claims 49 to 51, A system, device, or program, wherein the imaging unit and the projection unit are fixed via a member that absorbs high-frequency vibrations.

55. In the system, device, or program of any one of claims 49 to 51, A system, device, or program characterized in that the imaging unit and the projection unit are fixed to a helmet of a field worker.

56. In the system, device, or program of any one of claims 49 to 53, The system, device, or program is characterized in that the characteristic part image is a marker provided in the site space or a characteristic point of the captured image.

57. an imaging unit that is attached to a site worker or a mobile body and captures an image of the site space in a wide-angle direction to generate a captured image; a projection unit that is attached to a site staff member or a mobile body, is capable of projecting in a wide-angle direction onto a site space, and projects an instruction image onto the site space based on given instruction image data; a tracking control means for controlling the projection unit to project the instruction image based on a predetermined portion of the work site space, regardless of the movement of the work site staff member or the moving body, based on a characteristic partial image of the work site space included in the captured image; and A field indicating device equipped with the

58. 1. A field instruction program for implementing, by a computer, a field instruction device including an imaging unit that is attached to a field staff member or a mobile body and that captures a field space in a wide-angle direction to generate an image, and a projection unit that is attached to a field staff member or a mobile body and that is capable of projecting onto the field space in a wide-angle direction and that projects an instruction image onto the field space based on given instruction image data, the program comprising: A site instruction program that functions as a tracking control means that controls the projection unit to project the instruction image based on a specified part of the site space, regardless of the movement of the site staff or the moving body, based on a characteristic partial image of the site space contained in the captured image.

Images