Work Support System

Abstract

To provide a work support system capable of making, as small as possible, a data capacity as of when performing data communication between a first user side and a second user side, as well as, allowing a first user to accurately grasp a position instructed by a second user.SOLUTION: In a second system 3, a data generation unit 324 generates instruction reference position data D21 and instruction position data D22 with an image D11 displayed in a second display part 313. The inside of a real space RS in which a first user resides is captured in the image D11. Meanwhile, in a first system 2, a virtual object generation unit 224 generates a work support virtual object VO for supporting work in the real space RS in which the first user resides on the basis of the instruction reference position data D21 and the instruction position data D22. A first display unit 225 displays a work support virtual object VO superimposed on the real space RS in which the first user resides.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a work assistance system for assisting a work. [Background technology]

[0002] Conventionally, there has been known a work support system for supporting a work performed in a space where a first user (a worker) is present, based on an instruction from a second user (a remote user) who is an instructor. This type of work support system is disclosed in, for example, Patent Document 1.

[0003] In the system disclosed in Patent Document 1, data communication is performed between a worker's computer and an instructor's computer, which are connected via a communication network, to exchange work video of the worker and work instruction video of the instructor. In this system, the instructor can give work instructions to the worker through the work instruction video while watching the work video displayed on the instructor's monitor. In addition, the worker can perform work while checking the work instructions through the work instruction video displayed on the worker's monitor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6179927 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when the work video and the work instruction video are transmitted over a communication network, the video has a relatively large data capacity, and depending on the communication network environment, problems such as a long time required for data transmission may occur. Also, for example, if a worker moves without checking the work instruction video, even if the worker checks the work instruction video at the current location of the moved destination, it is difficult for the worker to accurately grasp the position of the work instructions shown in the work instruction video.

[0006] An object of the present disclosure is to provide a work support system that can minimize the amount of data required when communicating data between a first user and a second user, while enabling the first user to accurately grasp the position of an instruction from the second user. [Means for solving the problem]

[0007] A work support system according to one aspect of the present disclosure includes a first system used by a first user and a second system used by a second user who instructs a work in a space where the first user is present. The first system includes an imaging unit that captures an image in the space, a virtual object generation unit that generates a work support virtual object that supports the work in the space, a first display unit that displays the work support virtual object by superimposing it on the space, and a first communication unit that outputs imaging position data indicating an imaging position of the imaging unit in the space and the image. The second system includes a second display unit that displays the image, a data generation unit that generates designation reference position data indicating a designation reference position corresponding to the imaging position data and designation position data indicating the designation position based on an operation of designating a designation position on the image, and a second communication unit that outputs the designation reference position data and the designation position data. The virtual object generation unit generates the work support virtual object based on the designation reference position data and the designation position data. Effect of the Invention

[0008] According to the present disclosure, it is possible to provide a work support system that can minimize the amount of data required when communicating data between a first user and a second user, and that enables a first user to accurately grasp the position of an instruction given by a second user. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of a work assistance system according to the present disclosure. [Diagram 2]FIG. 2 is a block diagram showing a configuration of a first system provided in the work support system. [Diagram 3] FIG. 2 is a diagram showing a state of real space as seen through a first display unit of a first display device provided in the first system. [Figure 4] FIG. 2 is a block diagram showing a configuration of a second system provided in the work support system. [Diagram 5] 11 is a diagram showing a display state of a second display unit in a second display device provided in the second system. FIG. [Figure 6] 13A and 13B are diagrams illustrating a display state of the second display section when an operation for designating a pointed position is performed on an image displayed on the second display section of the second display device. [Figure 7] FIG. 13 is a diagram showing a state of real space as seen through the first display unit in a state in which a work supporting virtual object is displayed on the first display unit of the first display device. [Figure 8] FIG. 11 is a diagram showing a first example of a display state of the first display device when an operation for specifying the same designated position on a plurality of images is performed on the second display device. [Figure 9] FIG. 13 is a diagram showing a second example of the display state of the first display device when an operation is performed on the second display device to specify the same designated position on a plurality of images. [Figure 10] FIG. 11 is a diagram showing a first example of a display state of the first display device when a work supporting virtual object is generated on the basis of three-dimensional scan data by a scan unit in the first display device. [Figure 11] FIG. 11 is a diagram showing a second example of a display state of the first display device when a work supporting virtual object is generated on the basis of three-dimensional scan data by a scan unit in the first display device. [Figure 12] 13A to 13C are diagrams illustrating a display state of the first display device when an operation for specifying a designated position on an image on the second display device is performed successively. [Figure 13] 1 is a flowchart showing a work assistance method using the work assistance system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of a work assistance system according to the present disclosure will be described with reference to the drawings.

[0011] The work support system 1 shown in Fig. 1 is a system for supporting work in a space where a first user is present, based on instructions from a second user who is in a remote location. Examples of the first user include a worker who actually performs the work, a supervisor who supervises the worker's work, and an assistant who assists the worker's work. Examples of the second user include an instructor who gives instructions for the work. The work support system 1 includes a first system 2 used by the first user and a second system 3 used by the second user.

[0012] The first system 2 includes an imaging device 21, a first display device 22, and a first management device 23. In the first system 2, the imaging device 21 and the first display device 22 are connected to the first management device 23, and the first management device 23 is connected to a communication network 5. On the other hand, the second system 3 includes a second display device 31 and a second management device 32. In the second system 3, the second display device 31 is connected to the second management device 32, and the second management device 32 is connected to the communication network 5.

[0013] A file server device 4 is also connected to the communication network 5. The file server device 4 is connected to the first management device 23 and the second management device 32 via the communication network 9 so as to be able to perform data communications. The file server device 4 has a server storage unit 41 that stores various data used in the first system 2 and the second system 3. Details of the file server device 4 will be described later.

[0014] The imaging device 21, the first display device 22, and the first management device 23 that constitute the first system 2 will be described with reference to FIGS.

[0015] The imaging device 21 is a device that captures an image D11 in a real space RS where the first user is present. The image D11 captured by the imaging device 21 is a two-dimensional image. The imaging device 21 includes a control unit 211, an imaging communication unit 212, an imaging unit 213, and an image processing unit 214. In the imaging device 21, the imaging communication unit 212, the imaging unit 213, and the image processing unit 214 are controlled by the control unit 211. Under the control of the control unit 211, the imaging unit 213 and the image processing unit 214 acquire the image D11, and the imaging communication unit 212 transmits the image D11 to the first management device 23.

[0016] The imaging unit 213 is composed of a plurality of imaging bodies each having an imaging optical system and an imaging element. The imaging optical system includes optical elements such as a lens, a prism, a filter, and an aperture stop. The imaging element is an element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). Images captured by the imaging elements of each of the plurality of imaging bodies are transferred to the image processing unit 214.

[0017] The image processing unit 214 performs image processing to combine images from the imaging elements of the multiple imaging bodies. The image processing unit 214 performs image processing to combine the images, thereby generating an omnidirectional image with a solid angle of 4π steradians as image G11.

[0018] That is, the imaging device 21 acquires an omnidirectional image obtained by capturing images in all directions around the imaging position in the real space RS where the first user is located as the image G11. The image G11 is transmitted to the first management device 23 via the imaging communication unit 212.

[0019] The first display device 22 is a display device that displays a virtual object expressed in CG (Computer Graphics) by superimposing it on the real space RS where the first user is present. The first user using the first display device 22 can experience mixed reality (MR) in which a virtual object is mixed with the real world in the real space RS. Examples of the first display device 22 include a mobile terminal such as a tablet terminal that can be carried by the first user, a head-mounted display device that is worn on the head of the first user, and a glasses-type display device that is worn by the first user like glasses. In this embodiment, the first display device 22 is configured by a glasses-type display device. In such a first display device 22, a virtual object expressed in CG is displayed by superimposing it on the real world seen through the display.

[0020] The first display device 22 includes a control unit 221, a display communication unit 222, a scanning unit 223, a virtual object generating unit 224, and a first display unit 225. In the first display device 22, the display communication unit 222, the scanning unit 223, the virtual object generating unit 224, and the first display unit 225 are controlled by the control unit 221.

[0021] The first display unit 225 is a display capable of displaying a virtual object so as to be superimposed on the real space RS. In the first display device 22 configured by a glasses-type display device, the first display unit 225 is configured to have a property of transmitting light. Then, in a state in which the first display device 22 is worn by the first user, the first display unit 225 is disposed so as to cover both eyes of the first user from the outside. In this state, the first user can see inside the real space RS through the first display unit 225. The virtual object displayed on the first display unit 225 is generated by the virtual object generation unit 224.

[0022] The virtual object generation unit 224 generates a work supporting virtual object as a virtual object for supporting the work of the first user in the real space RS. The details of the operation of generating the work supporting virtual object by the virtual object generation unit 224 will be described later. Note that in the first system 2, the virtual object generation unit 224 is not limited to being provided in the first display device 22. The virtual object generation unit 224 may be provided in the first management device 23.

[0023] The scan unit 223 is composed of a plurality of cameras, sensors, etc. arranged in the vicinity of the first display unit 225. The scan unit 223 generates three-dimensional scan data of a real object RO in the real space RS by scanning the three-dimensional real space RS. The first display device 22 has a SLAM (Simultaneous Localization and Mapping) function, which is a self-position estimation function based on the operation of generating the three-dimensional scan data by the scan unit 223.

[0024] 3, when the first user uses the first display device 22, an operation guide virtual object D13 as a virtual object expressed by CG is displayed on the first display unit 225 so as to be superimposed on the real space RS seen through the first display unit 225. The operation guide virtual object D13 is a virtual object for guiding the operation of the first user, and includes a reference position recognition area D131, an imaging position recognition area D132, and a data reading area D133.

[0025] When the first user performs an operation to select the reference position recognition area D131 while the work reference mark M1 placed at the work reference position P11 in the real space RS is visible through the first display unit 225, the scan unit 223 generates three-dimensional scan data relating to the work reference mark M1. The three-dimensional scan data in this case becomes work reference position data indicating the work reference position P11 at which the work reference mark M1 is placed in the three-dimensional real space RS.

[0026] In the first display device 22, the control unit 221 recognizes the position of the first user wearing the first display device 22 in the real space RS relative to the work reference position P11 indicated by the work reference position data based on the three-dimensional scan data from the scanning unit 223, and also recognizes the direction of the first user's viewpoint.

[0027] When capturing an image D11 in the real space RS using the imaging device 21, an operation of selecting an imaging position recognition area D132 is performed by the first user in a state in which the imaging device 21 is visible through the first display unit 225. In this case, the scanning unit 223 generates three-dimensional scan data regarding the imaging mark M2 attached to the imaging device 21. In this case, the three-dimensional scan data becomes imaging position data D12 indicating an imaging position P12 by the imaging device 21 relative to the task reference position P11 indicated by the task reference position data in the real space RS. This imaging position data D12 is transmitted to the first management device 23 via the display communication unit 222.

[0028] When an operation for selecting the data reading area D133 is performed by the first user, the display communication unit 222 receives, via the first management device 23, the designation reference position data D21 and the designation position data D22, which are described below and are stored in the server storage unit 41 of the file server device 4. The designation reference position data D21 and the designation position data D22 received by the display communication unit 222 are referred to by the virtual object generation unit 224.

[0029] The first management device 23 is configured by, for example, a personal computer. The first management device 23 includes a control unit 231, a first communication unit 232, and a storage unit 233. In the first management device 23, the first communication unit 232 and the storage unit 233 are controlled by the control unit 231.

[0030] The first communication unit 232 performs data communication with the imaging device 21 and the first display device 22, and also performs data communication with the file server device 4 via the communication network 5. The first communication unit 232 receives an image D11 from the imaging device 21 and imaging position data D12 from the first display device 22, and transmits the received image D11 and imaging position data D12 to the file server device 4 via the communication network 5. The first communication unit 232 also receives designation reference position data D21 and designation position data D22 from the file server device 4 via the communication network 5, and transmits the received position data D21, D22 to the first display device 22.

[0031] The storage unit 233 stores the image D11 and the imaging position data D12 received by the communication unit 232 in association with each other. The image D11 and the imaging position data D12 are transmitted to the file server device 4 via the first communication unit 232 in an associated state. The file server device 4 stores the received image D11 and imaging position data D12 in the server storage unit 41.

[0032] In the first system 2, the first management device 23 may be incorporated in the first display device 22. In other words, the first display device 22 may be configured to have the functions of the first communication unit 232 and the storage unit 233 described above.

[0033] Next, the second display device 31 and the second management device 32 constituting the second system 3 will be described with reference to FIGS. 4 to 6 in addition to FIG.

[0034] The second management device 32 is configured by, for example, a personal computer. The second management device 32 has a function of performing data communication with the file server device 4 via the communication network 5 and a function of controlling the second display device 31. The second management device 32 includes a control unit 321, a second communication unit 322, a storage unit 323, and a data generation unit 324. In the second management device 32, the second communication unit 322, the storage unit 323, and the data generation unit 324 are controlled by the control unit 321.

[0035] The second communication unit 322 performs data communication with the second display device 31, and also performs data communication with the file server device 4 via the communication network 5. The second communication unit 322 receives an image D11 and imaging position data D12 from the file server device 4 via the communication network 5, and transmits the received image D11 to the second display device 31. In addition, the second communication unit 322 transmits designation reference position data D21 and designation position data D22 generated by a data generation unit 324 described below to the file server device 4 via the communication network 5.

[0036] The storage unit 323 stores the designation reference position data D21 and the designation position data D22 generated by the data generating unit 324 in association with each other. The designation reference position data D21 and the designation position data D22 are transmitted to the file server device 4 via the second communication unit 322 in an associated state.

[0037] The data generation unit 324 performs a data generation process to generate the designation reference position data D21 and the designation position data D22. The data generation process by the data generation unit 324 is performed in conjunction with the display of an image by the second display device 31. For this reason, the data generation process by the data generation unit 324 will be described in relation to the second display device 31.

[0038] The second display device 31 is a display device that displays a virtual reality (VR) image. Examples of the second display device 31 include a mobile terminal such as a tablet terminal that can be carried by the second user, a head-mounted display device that is worn on the head of the second user, a personal computer, and the like. In this embodiment, the second display device 31 is configured by a head-mounted display device.

[0039] The second display device 31 includes a control unit 311, a display communication unit 312, and a second display unit 313. In the second display device 31, the display communication unit 312 and the second display unit 313 are controlled by the control unit 311. The second display device 31 is configured to receive a command signal from a second operation unit 31A. The second operation unit 31A is configured by, for example, a controller operated by the second user. The second operation unit 31A may be configured by a sensor mounted on the second display device 31 and tracking the movement of the hand or fingers of the second user.

[0040] The second display unit 313 is a display capable of displaying an image. When the second user uses the second display device 31, as shown in Fig. 5, an operation guide virtual object D23 as a virtual object expressed by CG is displayed on the second display unit 313. The operation guide virtual object D23 is a virtual object for guiding the second user in an operation via the second operation unit 31A, and includes an image acquisition area D231, an image reading area D232, an instruction creation mode area D233, an image list display area D234, an instruction deletion area D235, and an instruction transmission area D236.

[0041] When a command signal for selecting the image acquisition area D231 is output from the second operation unit 31A, the display communication unit 312 receives and acquires the image D11 and the imaging position data D12 stored in the server storage unit 41 of the file server device 4 via the second management device 32. In this manner, the display communication unit 312 has a function as an acquisition unit that acquires the image D11 and the imaging position data D12.

[0042] When a command signal for selecting the image reading region D232 is output from the second operation unit 31A, the second display unit 313 reads the image D11 and the imaging position data D12 acquired by the display communication unit 312.

[0043] When a command signal selecting the instruction creation mode area D233 is output from the second operation unit 31A, the second display device 31 is set to a mode for creating data regarding an instruction position for instructing a work position by the first user.

[0044] When a command signal for selecting the image list display area D234 is output from the second operation unit 31A, the second display unit 313 displays a list of the images D11 divided according to the imaging position data D12. When a command signal for selecting one image D11 from the list-displayed images D11 is output from the second operation unit 31A, the second display unit 313 displays the selected image D11.

[0045] As shown in FIG. 6, in a state where an image D11 is displayed on the second display unit 313, the data generating unit 324 of the second management device 32 generates designation reference position data D21 and designation position data D22 based on an operation by the second operating unit 31A to designate a designation position P22 on the image D11. The designation reference position data D21 is data indicating the designation reference position P21 corresponding to the imaging position data D12 associated with the image D11. The designation position data D22 is data indicating the designation position P22 for designating the work position of the first user. When the designation reference position data D21 and the designation position data D22 are generated by the data generating unit 324, a work instruction virtual object P2A indicating a virtual object extending from the designation reference position P21 to the designation position P22 is displayed on the second display unit 313 so as to overlap with the image D11. By checking the work instruction virtual object P2A displayed on the second display unit 313, the second user can grasp the designation position P22.

[0046] In the second management apparatus 32, the designated reference position data D21 and the designated position data D22 generated by the data generating unit 324 are stored in the storage unit 323 in a state in which they are associated with each other.

[0047] Returning to Figure 5, when a command signal to select the instruction deletion area D235 on the operation guide virtual object D23 displayed on the second display unit 313 is output from the second operation unit 31A, in the second management device 32, the data generation unit 324 deletes the created instruction reference position data D21 and instruction position data D22.

[0048] When a command signal selecting an instruction transmission area D236 on the operation guide virtual object D23 displayed on the second display unit 313 is output from the second operation unit 31A, in the second management device 32, the second communication unit 322 transmits the instruction reference position data D21 and the instruction position data D22 stored in the memory unit 323 to the file server device 4 via the communication network 5.

[0049] The file server device 4 stores the received designated reference position data D21 and designated position data D22 in the server storage unit 41.

[0050] As described above, in the first display device 22, when the first user performs an operation to select the data reading area D133 on the operation guide virtual object D13 displayed on the first display unit 225, the display communication unit 222 receives the instruction reference position data D21 and the instruction position data D22 stored in the server memory unit 41 of the file server device 4 via the first management device 23.

[0051] 7, in the first display device 22, the virtual object generation unit 224 generates a work support virtual object VO to be displayed on the first display unit 225 based on the designation reference position data D21 and the designation position data D22. The work support virtual object VO is a virtual object for supporting a work by the first user. The virtual object generation unit 224 generates, as the work support virtual object VO, a virtual object along a straight line P212 passing through the designation reference position P21 indicated by the designation reference position data D21 and the designation position P22 indicated by the designation position data D22. At this time, the designation reference position P21 indicated by the designation reference position data D21 may coincide with the imaging position P12 indicated by the imaging position data D12, or may be a position shifted from the imaging position P12.

[0052] As described above, in the first system 2, the imaging position data D12 of the imaging device 21 and the image D11 captured by the imaging device 21 are output via the first communication unit 232 of the first management device 23. On the other hand, in the second system 3, in a state in which the image D11 is displayed on the second display device 31, the designation reference position data D21 and the designation position data D22 are generated by the data generation unit 324 of the second management device 32. The generated position data D21, D22 are output via the second communication unit 322 of the second management device 32. Then, in the first display device 22 of the first system 2, the virtual object generation unit 224 generates a work supporting virtual object VO based on the designation reference position data D21 and the designation position data D22, and the first display unit 225 displays the work supporting virtual object VO.

[0053] In the work support system 1 configured with such a first system 2 and second system 3, the imaging position data D12 and the image D11 are transmitted from the first system 2 side to the second system 3 side, and the indication reference position data D21 and the indication position data D22 are transmitted from the second system 3 side to the first system 2 side. By using the imaging position data D12, indication reference position data D21, and indication position data D22 and the image D11 as the subject of data communication between the first system 2 and the second system 3, the data capacity during data communication can be made as small as possible compared to the case where video images are the subject of data communication as in the prior art.

[0054] Moreover, in the first display device 22 of the first system 2, the first display unit 225 displays the work support virtual object VO generated by the virtual object generation unit 224 by superimposing it on the real space RS where the first user is located. As described above, the work support virtual object VO is a virtual object generated based on each of the position data of the designation reference position data D21 and the designation position data D22. Therefore, the first display unit 225 displays the work support virtual object VO so that it is located at a fixed position based on the designation reference position P21 indicated by the designation reference position data D21 and the designation position P22 indicated by the designation position data D22 in the real space RS where the first user is located. In other words, even if the first user who uses the first display device 22 moves in the real space RS or changes the direction of the viewpoint, the work support virtual object VO is located at a fixed position in the real space RS. This allows a worker using the first display device 22 to accurately grasp the position of the work instructions from the second user by checking the work support virtual object VO displayed on the first display unit 225 so as to be positioned at a certain position in the real space RS.

[0055] Next, assume that an operation is performed to specify the same designated position P22 for each of a plurality of images D11 having different imaging position data D12 displayed on the second display device 31. In this case, the data generating unit 324 of the second management device 32 generates a plurality of combination data of the designated reference position data D21 and the designated position data D22 in response to each operation for each of the plurality of images D11.

[0056] In this case, as shown in Fig. 8 and Fig. 9, the virtual object generation unit 224 of the first display device 22 calculates a first intersection P231 between straight lines P212 passing through the positions P21, P22 indicated by the indication reference position data D21 and the indication position data D22 of each of the multiple combination data. Then, the virtual object generation unit 224 generates a work support virtual object VO based on the calculated first intersection P231. In the first display device 22, such a work support virtual object VO is displayed on the first display unit 225 so as to be located at a certain position based on the first intersection P231 in the real space RS where the first user is located. In this case, the indication position P22 in the three-dimensional real space RS can be expressed by the position of the work support virtual object VO based on the first intersection P231. As a result, the first user using the first display device 22 can more accurately grasp the position of the work instructions from the second user by checking the work support virtual object VO displayed on the first display unit 225 so as to be positioned at a certain position based on the first intersection point P231 in the real space RS.

[0057] 8, the virtual object generation unit 224 generates, as the work support virtual object VO, each virtual object along each of a plurality of line segments P212 extending from each of a plurality of instruction reference positions P21 indicated by each of the instruction reference position data D21 of a plurality of combination data to the first intersection P231. In this case, the instruction position P22 in the three-dimensional real space RS can be expressed by the position of the end portion corresponding to the first intersection P231 in the work support virtual object VO. This allows the first user using the first display device 22 to more accurately grasp the position of the work instruction by the second user by checking the position of the end portion corresponding to the first intersection P231 in the work support virtual object VO displayed on the first display unit 225 so as to be located at a certain position in the real space RS.

[0058] 9, the virtual object generation unit 224 generates a virtual mark placed at the first intersection P231 as the work supporting virtual object VO. In this case, the position of the virtual mark placed at the first intersection P231 can represent the instruction position P22 in the three-dimensional real space RS. This allows the first user using the first display device 22 to more accurately grasp the position of the work instruction by the second user by checking the work supporting virtual object VO made of the virtual mark displayed on the first display unit 225 so as to be located at a certain position in the real space RS corresponding to the first intersection P231.

[0059] 10 and 11, in the first display device 22, the virtual object generation unit 224 may be configured to generate a work supporting virtual object VO based on three-dimensional scan data generated by the scan unit 223. Specifically, the virtual object generation unit 224 calculates a second intersection P232 between a straight line P212 passing through the designation reference position P21 indicated by the designation reference position data D21 and the designation position P22 indicated by the designation position data D22, and an outer surface of a specific real object ROS in the real space RS based on the three-dimensional scan data generated by the scan unit 223. Then, the virtual object generation unit 224 generates the work supporting virtual object VO based on the calculated second intersection P232.

[0060] In the first display device 22, such a work support virtual object VO is displayed on the first display unit 225 so as to be located at a fixed position based on the second intersection P232 in the real space RS where the first user is located. In this case, the instruction position P22 in the three-dimensional real space RS can be represented by the position of the work support virtual object VO based on the second intersection P232. This allows the first user using the first display device 22 to more accurately grasp the position of the work instruction by the second user by checking the work support virtual object VO displayed on the first display unit 225 so as to be located at a fixed position based on the second intersection P232 in the real space RS.

[0061] 10, the virtual object generation unit 224 generates a virtual object along a line segment P212 extending from the instruction reference position P21 indicated by the instruction reference position data D21 to the second intersection P232 as the work support virtual object VO. In this case, the instruction position P22 in the three-dimensional real space RS can be expressed by the position of the end portion corresponding to the second intersection P232 in the work support virtual object VO. This allows the first user using the first display device 22 to more accurately grasp the position of the work instruction by the second user by checking the position of the end portion corresponding to the second intersection P232 in the work support virtual object VO displayed on the first display unit 225 so as to be located at a certain position in the real space RS.

[0062] 11, the virtual object generation unit 224 generates a virtual mark located at the second intersection P232 as the work support virtual object VO. In this case, the position of the virtual mark located at the second intersection P232 can represent the instruction position P22 in the three-dimensional real space RS. This allows the first user using the first display device 22 to more accurately grasp the position of the work instruction by the second user by checking the work support virtual object VO consisting of the virtual mark displayed on the first display unit 225 so as to be located at a certain position in the real space RS corresponding to the second intersection P232.

[0063] Next, assume that an operation of designating a designated position P22 on an image D11 displayed on the second display device 31 is performed continuously, for example by drawing an arc. In this case, the data generating unit 324 of the second management device 32 generates designated reference position data D21 corresponding to the imaging position data D12 associated with the image D11, and a plurality of continuous designated position data D22 corresponding to the continuous operation.

[0064] In this case, as shown in Fig. 12, the virtual object generating unit 224 of the first display device 22 generates, as the work support virtual object VO, a plurality of virtual objects along a plurality of straight lines P212 extending from the instruction reference position P21 indicated by the instruction reference position data D21 toward the instruction position P22 indicated by each of the plurality of instruction position data D22. In the first display device 22, such a plurality of work support virtual objects VO are displayed on the first display unit 225 so as to be located at a fixed position in the real space RS where the first user is present. In this case, the end portions of the plurality of work support virtual objects VO on the opposite side to the instruction reference position P21 side are arranged along the trajectory VC of the continuous operation when the instruction position P22 is specified on the image D11 displayed on the second display device 31. This allows the first user using the first display device 22 to more accurately grasp the position of the work instruction by the second user by checking the arrangement state of the end portions of the plurality of work support virtual objects VO located at a fixed position in the real space RS.

[0065] As described above, the image D11, the imaging position data D12, the designation reference position data D21, and the designation position data D22 are stored in the server storage unit 41 of the file server device 4. The file server device 4 is connected to the first communication unit 232 of the first management device 23 and the second communication unit 322 of the second management device 32 via the communication network 5 so as to be able to perform data communication. This allows the second user to read the image D11 and the imaging position data D12 from the file server device 4 using the second management device 32 at a desired timing that suits the user's convenience, and to create a work instruction while checking the situation in the real space RS where the first user is located based on the image D11 using the second display device 31. On the other hand, the first user can read the designation reference position data D21 and the designation position data D22 from the file server device 4 using the first management device 23 at a desired timing that suits the user's convenience, and to perform a work while checking the work support virtual object VO using the first display device 22.

[0066] Next, a work assistance method using the work assistance system 1 will be described with reference to the flowchart of FIG.

[0067] When the first user uses the first display device 22, an operation guide virtual object D13 is displayed on the first display unit 225 so as to be superimposed on the real space RS seen through the first display unit 225 (FIG. 3). When the first user performs an operation to select the reference position recognition area D131 in a state in which the work reference mark M1 arranged at the work reference position P11 in the real space RS is seen through the first display unit 225, the scan unit 223 generates three-dimensional scan data regarding the work reference mark M1. As a result, the control unit 221 in the first display device 22 recognizes the work reference position P11 where the work reference mark M1 is arranged in the three-dimensional real space RS (step a1).

[0068] When capturing an image D11 in the real space RS using the imaging device 21, an operation of selecting an imaging position recognition area D132 is performed by the first user in a state in which the imaging device 21 is visible through the first display unit 225. In this case, the scanning unit 223 generates three-dimensional scan data regarding the imaging mark M2 attached to the imaging device 21. As a result, the control unit 221 in the first display device 22 recognizes an imaging position P12 by the imaging device 21 relative to the work reference position P11 in the real space RS (step a2). The imaging position data D12 indicating this imaging position P12 is transmitted to the first management device 23 via the display communication unit 222.

[0069] The imaging device 21 captures an image D11 by capturing an image in the real space RS at the imaging position P12 indicated by the imaging position data D12 (step a3). In the imaging device 21, the imaging communication unit 212 transmits the image D11 to the first management device 23.

[0070] In the first management device 23 that receives the image D11 from the imaging device 21 and the imaging position data D12 from the first display device 22, the storage unit 233 associates the image D11 with the imaging position data D12 and stores them. Then, the first communication unit 232 transmits the image D11 and the imaging position data D12 to the file server device 4 via the communication network 5 (step a4). The file server device 4 stores the received image D11 and imaging position data D12 in the server storage unit 41.

[0071] When the second user uses the second display device 31, an operation guide virtual object D23 is displayed on the second display unit 313 (FIG. 5). In the second display device 31, when a command signal for selecting the image acquisition area D231 is output from the second operation unit 31A, the display communication unit 312 receives and acquires the image D11 and the imaging position data D12 stored in the server storage unit 41 of the file server device 4 via the second management device 32 (step b1).

[0072] In the second display device 31, when a command signal to select the image reading area D232 and the instruction creation mode area D233 and then a command signal to select the image list display area D234 are output from the second operation unit 31A, the second display unit 313 displays a list of the images D11 classified according to the imaging position data D12. When a command signal to select one image D11 from the list-displayed images D11 is output from the second operation unit 31A, the second display unit 313 displays the selected image D11 (step b2).

[0073] With the image D11 displayed on the second display unit 313, an operation of designating a designated position P22 on the image D11 is performed by the second operation unit 31A (step b3). In this case, the data generation unit 324 of the second management device 32 generates designated reference position data D21 and designated position data D22 (FIG. 6, step b4).

[0074] When a command signal for selecting the instruction transmission area D236 on the operation guide virtual object D23 displayed on the second display unit 313 is output from the second operation unit 31A, in the second management device 32, the second communication unit 322 transmits the instruction reference position data D21 and the instruction position data D22 to the file server device 4 via the communication network 5 (step b5). The file server device 4 stores the received instruction reference position data D21 and the instruction position data D22 in the server storage unit 41.

[0075] In the first display device 22, when the first user performs an operation to select the data reading area D133 on the operation guide virtual object D13 displayed on the first display unit 225, the display communication unit 222 receives the instruction reference position data D21 and the instruction position data D22 stored in the server memory unit 41 of the file server device 4 via the first management device 23.

[0076] In the first display device 22, the virtual object generation unit 224 generates a work supporting virtual object VO to be displayed on the first display unit 225 based on the designated reference position data D21 and the designated position data D22 (FIGS. 7 to 12, step a5).

[0077] Then, the first display unit 225 of the first display device 22 displays the work support virtual object VO superimposed on the real space RS where the first user is located (step a6). In this case, the first display unit 225 displays the work support virtual object VO so as to be located at a fixed position in the real space RS based on the instruction reference position P21 indicated by the instruction reference position data D21 and the instruction position P22 indicated by the instruction position data D22. This allows the first user using the first display device 22 to accurately grasp the position of the work instruction by the second user by checking the work support virtual object VO displayed on the first display unit 225 so as to be located at a fixed position in the real space RS.

[0078] In the work support system 1, the file server device 4 is connected to the first communication unit 232 of the first system 2 and the second communication unit 322 of the second system 3 via the communication network 5 so as to be able to communicate data with each other. However, the work support system 1 is not limited to such a configuration. The work support system 1 may include another server device such as a mail server device instead of the file server device 4. The work support system 1 may also be configured not to include a server device such as the file server device 4. In this case, the first communication unit 232 of the first system 2 and the second communication unit 322 of the second system 3 are connected to each other via the communication network 5 so as to be able to communicate data with each other.

[0079] [Summary of this disclosure] The specific embodiments described above include disclosures having the following configurations.

[0080] The work support system according to the present disclosure includes a first system used by a first user and a second system used by a second user who instructs a work in a space where the first user is present. The first system includes an imaging unit that captures an image in the space, a virtual object generation unit that generates a work support virtual object that supports the work in the space, a first display unit that displays the work support virtual object by superimposing it on the space, and a first communication unit that outputs imaging position data indicating an imaging position of the imaging unit in the space and the image. The second system includes a second display unit that displays the image, a data generation unit that generates designation reference position data indicating a designation reference position corresponding to the imaging position data and designation position data indicating the designation position based on an operation of designating a designation position on the image, and a second communication unit that outputs the designation reference position data and the designation position data. The virtual object generation unit generates the work support virtual object based on the designation reference position data and the designation position data.

[0081] According to this work support system, in the first system, the imaging position data of the imaging unit and the image captured by the imaging unit are output via the first communication unit. Meanwhile, in the second system, the designation reference position data and the designation position data are generated by the data generation unit while the image is displayed on the second display unit. Each of the generated position data is output via the second communication unit. Then, in the first system, the virtual object generation unit generates a work support virtual object based on the designation reference position data and the designation position data, and the first display unit displays the work support virtual object. In such a work support system, the imaging position data and the image are transmitted from the first system side to the second system side, and the designation reference position data and the designation position data are transmitted from the second system side to the first system side. By using each of the position data of the imaging position data, the designation reference position data, and the designation position data and the image as the target of data communication between the first system and the second system, the data capacity during data communication can be reduced as much as possible compared to the case where video is used as the target of data communication as in the conventional technology.

[0082] Also, in the first system, the first display unit displays the work support virtual object generated by the virtual object generation unit by superimposing it on the space where the first user is present. As described above, the work support virtual object is a virtual object generated based on each position data of the instruction reference position data and the instruction position data. Therefore, the first display unit displays the work support virtual object so as to be located at a fixed position based on each position indicated by each position data in the space where the first user is present. In other words, even if the first user using the first display unit moves in the space or changes the direction of the viewpoint, the work support virtual object is located at a fixed position in the space. As a result, the first user using the first display unit can accurately grasp the position of the work instruction by the second user by checking the work support virtual object displayed on the first display unit so as to be located at a fixed position in the space.

[0083] In the above work support system, when operations for specifying a designated position for each of the plurality of images having different imaging position data are operations for specifying the same designated position, the data generation unit generates a plurality of combination data of the designated reference position data and the designated position data corresponding to each of the operations. In this case, the virtual object generation unit calculates a first intersection point between straight lines passing through each position indicated by the designated reference position data and the designated position data of each of the plurality of combination data, and generates the work support virtual object based on the first intersection point.

[0084] In this aspect, when an operation is performed to specify the same designated position for each of a plurality of images having different imaging position data displayed on the second display unit of the second system, the data generating unit generates a plurality of combination data of the designated reference position data and the designated position data in response to each operation. In this case, the virtual object generating unit of the first system calculates a first intersection between straight lines passing through each position indicated by the designated reference position data and the designated position data of each of the plurality of combination data, and generates a work support virtual object based on the first intersection. In the first system, such a work support virtual object is displayed on the first display unit so as to be located at a fixed position based on the first intersection in the space where the first user is present. In this case, the designated position in the three-dimensional space can be represented by the position of the work support virtual object based on the first intersection. This allows the first user using the first display unit to more accurately grasp the position of the work instruction by the second user by checking the work support virtual object displayed on the first display unit so as to be located at a fixed position based on the first intersection in the space.

[0085] In the above work support system, the virtual object generation unit generates, as the work support virtual object, a virtual object along a line segment extending from the instruction reference position indicated by the instruction reference position data of each of the multiple combination data to the first intersection.

[0086] In this aspect, the virtual object generating unit of the first system generates a virtual object along a line segment extending from the instruction reference position indicated by the instruction reference position data of each of the multiple combination data to the first intersection as a work support virtual object. In this case, the instruction position in the three-dimensional space can be represented by the position of the end portion corresponding to the first intersection in the work support virtual object. This allows the first user using the first display unit to more accurately grasp the position of the work instruction by the second user by checking the position of the end portion corresponding to the first intersection in the work support virtual object displayed on the first display unit so as to be located at a certain position in the space.

[0087] In the above task support system, the virtual object generation unit generates a virtual mark to be placed at the first intersection as the task support virtual object.

[0088] In this aspect, the virtual object generating unit of the first system generates a virtual mark placed at the first intersection as a work support virtual object. In this case, the position of the virtual mark placed at the first intersection can represent the indicated position in a three-dimensional space. This allows the first user using the first display unit to more accurately grasp the position of the work instruction by the second user by checking the work support virtual object made of the virtual mark displayed on the first display unit so as to be located at a certain position in the space corresponding to the first intersection.

[0089] In the above work support system, the first system further includes a scanning unit that generates 3D scan data of a real object in the space by scanning the space. In this case, the virtual object generating unit calculates a second intersection point between an outer surface of the real object based on the 3D scan data and a line passing through each position indicated by the designation reference position data and the designation position data, and generates the work support virtual object based on the second intersection point.

[0090] In this aspect, the virtual object generating unit of the first system calculates a second intersection between an outer surface of a real object in a space based on the three-dimensional scan data generated by the scanning unit and a straight line passing through each position indicated by the designated reference position data and the designated position data, and generates a work support virtual object based on the second intersection. In the first system, such a work support virtual object is displayed on the first display unit so as to be located at a fixed position based on the second intersection in the space where the first user is present. In this case, the designated position in the three-dimensional space can be represented by the position of the work support virtual object based on the second intersection. This allows the first user using the first display unit to more accurately grasp the position of the work instruction by the second user by checking the work support virtual object displayed on the first display unit so as to be located at a fixed position based on the second intersection in the space.

[0091] In the above work support system, the virtual object generation unit generates, as the work support virtual object, a virtual object along a line segment extending from the designated reference position indicated by the designated reference position data to the second intersection point.

[0092] In this aspect, the virtual object generating unit of the first system generates a virtual object along a line segment extending from the instruction reference position indicated by the instruction reference position data to the second intersection as a work support virtual object. In this case, the instruction position in the three-dimensional space can be represented by the position of the end portion corresponding to the second intersection in the work support virtual object. This allows the first user using the first display unit to more accurately grasp the position of the work instruction by the second user by checking the position of the end portion corresponding to the second intersection in the work support virtual object displayed on the first display unit so as to be located at a certain position in the space.

[0093] In the above task support system, the virtual object generation unit generates a virtual mark to be placed at the second intersection as the task support virtual object.

[0094] In this aspect, the virtual object generating unit of the first system generates a virtual mark located at the second intersection as a work support virtual object. In this case, the position of the virtual mark located at the second intersection can represent the indicated position in a three-dimensional space. This allows the first user using the first display unit to more accurately grasp the position of the work instruction by the second user by checking the work support virtual object consisting of the virtual mark displayed on the first display unit so as to be located at a certain position in the space corresponding to the second intersection.

[0095] In the above-mentioned work support system, when an operation of designating a designated position on the image is performed continuously, the data generation unit generates a continuous plurality of the designated position data in response to the operation. In this case, the virtual object generation unit generates, as the work support virtual object, a plurality of virtual objects along a plurality of straight lines extending from the designation reference position indicated by the designation reference position data to the designated positions indicated by each of the plurality of the designation position data.

[0096] In this aspect, when the operation of designating the designated position on the image displayed on the second display unit is performed continuously, for example, by drawing an arc, the data generating unit generates a plurality of designated position data corresponding to the continuous operations. In this case, the virtual object generating unit of the first system generates a plurality of virtual objects along a plurality of straight lines extending from the designated reference position indicated by the designated reference position data toward the designated positions indicated by each of the plurality of designated position data as work support virtual objects. In the first system, such a plurality of work support virtual objects are displayed on the first display unit so as to be located at a certain position in the space where the first user is present. In this case, the end portions of the plurality of work support virtual objects on the opposite side to the designated reference position side are arranged along the trajectory of the continuous operations when designating the designated position on the image displayed on the second display unit. This allows the first user using the first display unit to more accurately grasp the position of the work instruction by the second user by checking the arrangement of the end portions of the plurality of work support virtual objects located at a certain position in the space.

[0097] The work support system further includes a server device connected to the first communication unit and the second communication unit via a communication network so as to be capable of data communication, the server device having a server storage unit that stores the imaging position data, the image, the indication reference position data, and the indication position data.

[0098] In this aspect, the imaging position data and the image, the indication reference position data and the indication position data are stored in a server storage unit of the server device. This server device is connected to the first communication unit and the second communication unit via a communication network so that data can be communicated. This allows the second user to read the imaging position data and the image from the server device using the second system at a desired timing that suits him / her, and to create a work instruction while checking the situation in the space where the first user is located based on the image using the second display unit. On the other hand, the first user can read the indication reference position data and the indication position data from the server device using the first system at a desired timing that suits him / her, and to perform a work while checking the work support virtual object using the first display unit. [Explanation of symbols]

[0099] 1 Work support system 2. System 1 21 Imaging device 213 Imaging unit 22 1st display device 223 Scanning section 224 Virtual Object Generation Unit 225 1st display section 23 1st management device 232 First Communications Department 3. Second System 31 2nd display device 313 2nd display section 32 Second management device 322 2nd Communications Department 324 Data Generation Department 4 File server device 5. Communication Network D11 Images D12 Imaging position data D21 Reference position data D22 Indicated position data P12 Imaging position P21 Indication reference position P22 Indicated position RO Real Object RS Real Space VO Work Support Virtual Object

Claims

1. a first system used by a first user; A second system used by a second user to instruct an operation in the space where the first user is present; The first system comprises: An imaging unit that captures an image in the space; a virtual object generation unit that generates a work support virtual object that supports a work in the space; a first display unit that displays the work support virtual object by superimposing it on the space; a first communication unit that outputs imaging position data indicating an imaging position of the imaging unit in the space and the image, The second system is A second display unit that displays the image; a data generating unit that generates, based on an operation of designating a designated position on the image, designated reference position data indicating a designated reference position corresponding to the imaging position data and designated position data indicating the designated position; a second communication unit that outputs the indication reference position data and the indication position data, The virtual object generation unit generates the work support virtual object based on the indication reference position data and the indication position data.

2. when each operation of designating a designated position for each of the plurality of images having different imaging position data is an operation of designating the same designated position, the data generation unit generates a plurality of combination data of the designation reference position data and the designation position data corresponding to each operation; 2. The work support system according to claim 1, wherein the virtual object generation unit calculates a first intersection point between straight lines passing through each position indicated by the indication reference position data and the indication position data of each of the plurality of combination data, and generates the work support virtual object based on the first intersection point.

3. 3. The work support system according to claim 2, wherein the virtual object generation unit generates, as the work support virtual object, a virtual object along a line segment extending from the instruction reference position indicated by the instruction reference position data of each of the plurality of combination data to the first intersection.

4. The work support system according to claim 2 , wherein the virtual object generation unit generates a virtual mark to be placed at the first intersection as the work support virtual object.

5. the first system further includes a scanning unit configured to generate three-dimensional scan data regarding a real object in the space by scanning the space, 2. The work support system according to claim 1, wherein the virtual object generation unit calculates a second intersection between an outer surface of the physical object based on the 3D scan data and a straight line passing through each position indicated by the indication reference position data and the indication position data, and generates the work support virtual object based on the second intersection.

6. The work support system according to claim 5 , wherein the virtual object generation unit generates, as the work support virtual object, a virtual object along a line segment extending from the instruction reference position indicated by the instruction reference position data to the second intersection point.

7. The work support system according to claim 5 , wherein the virtual object generation unit generates a virtual mark to be placed at the second intersection as the work support virtual object.

8. the data generating unit generates a plurality of pieces of the designated position data in response to successive operations of designating designated positions on the image; 2. The work support system according to claim 1, wherein the virtual object generation unit generates, as the work support virtual objects, a plurality of virtual objects along a plurality of straight lines extending from the instruction reference position indicated by the instruction reference position data toward the instruction positions indicated by each of the plurality of instruction position data.

9. a server device connected to the first communication unit and the second communication unit via a communication network so as to be capable of data communication; 9. The work support system according to claim 1, wherein the server device includes a server storage unit that stores the imaging position data, the image, the indication reference position data, and the indication position data.

Images