Remote work support device, remote work support system, remote work support method, and remote work support program

Abstract

A remote work support device included in a remote work support system according to one embodiment comprises: a generation unit that generates a spatial model of a work site on the basis of first image data obtained by capturing a work site from a first position in a first direction and second image data obtained by capturing the work site in a second direction from a second position; an arrangement unit that arranges an indicator in the space model; and a first superimposition unit that superimposes first indicator data, representing the indicator when viewed in the first direction from the first position, on the first image data, and superimposes second indicator data, representing the indicator when viewed in the second direction from the second position, on the second image data.

Description

Remote work support device, remote work support system, remote work support method, and remote work support program

[0001] The embodiments relate to a remote work support device, a remote work support system, a remote work support method, and a remote work support program.

[0002] In recent years, with the spread of remote work, there has been an increase in work styles in which a supervisor with work knowledge and a worker who actually performs the work collaborate remotely to perform the work. This type of collaborative work has also been introduced in the repair of office automation (OA) terminals, servers, etc. For example, there is a work style in which a skilled technician in a remote location different from the work site acts as a supervisor and gives instructions to a worker who does not have the know-how to perform the work on site.

[0003] When giving instructions from a remote location, there are cases where the instructor communicates with the worker over the phone or uses on-site video. To give more complex instructions, a method has been devised in which a hand model showing the shape or appearance of the instructor's fingers is superimposed on the image captured by a wearable camera attached to the worker (hereinafter referred to as the "worker camera") using mixed reality (MR) glasses or the like, thereby efficiently communicating the work content using hand gestures.

[0004] Shohei Yamada, P. Chandrasiri Naiwala, "Verification of Work Efficiency in Remote Work Support Using Smart Glasses and Hand Gestures," Journal of the Institute of Image Electronics Engineers of Japan, Vol. 47, No. 4, pp. 401-404, 2018, doi: 10.11371 / iieej.47.401

[0005] However, in conventional remote work support systems, the instructor gives instructions based on the image captured by the worker camera, so the instructor's viewpoint depends on the viewpoint of the worker camera. Therefore, the instructor cannot voluntarily move to a viewpoint that makes it easier to give instructions. Furthermore, the angle at which a hand model is attached when communicating work content using hand gestures also depends on the viewpoint of the worker camera. Therefore, it is not possible to attach a hand model from a different angle. In addition, when the viewpoint of the worker camera moves, the position indicated by the attached hand model changes, so the position of the hand model must be manually corrected each time. For these reasons, it is difficult for an instructor to easily give instructions to a worker during remote work.

[0006] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a means that enables an instructor to easily give instructions in remote work.

[0007] A remote work support device provided in a remote work support system according to one embodiment includes a generation unit that generates a spatial model of a work site based on first video data obtained by photographing the work site from a first position in a first direction and second video data obtained by photographing the work site from a second position in a second direction, a placement unit that places an indicator within the spatial model, and a first superimposition unit that superimposes first indicator data showing the indicator when viewed in the first direction from the first position onto the first video data and superimposes second indicator data showing the indicator when viewed in the second direction from the second position onto the second video data.

[0008] According to an embodiment of the present invention, it is possible to provide a means that enables an instructor to easily give instructions in remote work.

[0009] FIG. 1 is a diagram showing an example of the system configuration of a remote work support system according to the first embodiment. FIG. 2 is a diagram showing an example of data transmitted and received via a network by each component of the remote work support system according to the first embodiment. FIG. 3 is a block diagram showing an example of software functions provided in a worker terminal according to the first embodiment, together with an example of a hardware configuration. FIG. 4 is a block diagram showing an example of software functions provided in a shooting point terminal according to the first embodiment, together with an example of a hardware configuration. FIG. 5 is a block diagram showing an example of software functions provided in an instructor terminal according to the first embodiment, together with an example of a hardware configuration. FIG. 6 is a sequence diagram showing an example of the operation of a worker terminal, a shooting point terminal, and an instructor terminal in the remote work support system according to the first embodiment. FIG. 7 is a diagram showing an example of an instructor superimposing a hand model on video data in the remote work support system according to the first embodiment. FIG. 8 is a diagram showing an example of video data on which a hand model is superimposed in the remote work support system according to the first embodiment. FIG. 9 is a diagram showing an example of video data on which a hand model is superimposed in the remote work support system according to the first embodiment. 10 is a block diagram showing an example of a software function provided in a worker terminal according to the second embodiment together with an example of a hardware configuration. FIG. 11 is a block diagram showing an example of a software function provided in a shooting point terminal according to the second embodiment together with an example of a hardware configuration.

[0010] Hereinafter, embodiments will be described with reference to the drawings. In the following description, components having the same functions and configurations will be denoted by the same reference numerals.

[0011] 1. First Embodiment 1.1 Configuration 1.1.1 Remote Work Support System Fig. 1 is a diagram showing an example of the system configuration of a remote work support system according to the first embodiment. Fig. 2 is a diagram showing an example of data transmitted and received via a network by each component of the remote work support system according to the first embodiment. The configuration of the remote work support system according to the first embodiment will be described using Figs. 1 and 2.

[0012] 1, the remote work support system according to the first embodiment communicates via a network NW between, for example, an instructor terminal IT used by an instructor IU at an instruction point IS, a worker terminal WT used by a worker WU at a remote work site WS, and photographing point terminals ST1 to ST4 installed at the work site WS. With this configuration, the instructor IU can give work instructions to the worker WU while checking the video of the work site WS in real time.

[0013] The network NW includes, for example, a wide area network with the Internet at its core and an access network for accessing this wide area network. Examples of the access network include a public communication network using wired or wireless connections, a local area network (LAN) using wired or wireless connections, or a cable television (CATV) network. Note that the access network is not limited to the above. Furthermore, if the service provision area is limited to a company's office or offices, the network NW may be configured solely with a wired LAN or wireless LAN.

[0014] The worker terminal WT is, for example, a head-mounted display (HMD) or smart glasses-type terminal worn on the head of the worker WU. The following description will be given assuming that the worker terminal WT is an HMD-type terminal. The worker terminal WT is connected to a worker camera WCM. The worker camera WCM is, for example, a wearable camera attached to the worker WU, and captures video from the worker WU's viewpoint. The worker camera WCM may be built into the worker terminal WT or may be attached externally to the worker terminal WT. The following description will be given assuming that the worker camera WCM is built into the worker terminal WT. As shown in FIG. 2 , the worker terminal WT transmits worker video data WVD captured by the worker camera WCM, time information when the worker video data WVD was captured, and three-dimensional position information and direction information of the worker camera WCM to the instructor terminal IT via the network NW.

[0015] The worker terminal WT also receives instructions from the instructor IU transmitted from the instructor terminal IT via the network NW and transmits them to the worker WU by displaying them on a display. Instructions from the instructor IU are given, for example, by superimposing a hand model on an image captured by the worker camera WCM. The worker WU performs work while checking the instructions displayed on the worker terminal WT.

[0016] The shooting point terminals ST1 to ST4 are installed at different locations in the work site WS from the worker terminal WT. In the example shown in FIG. 1, four shooting point terminals are installed, but the number of shooting point terminals may be one or more. Each shooting point terminal ST1 to ST4 is connected to one shooting point camera SCM1 to SCM4. Each shooting point camera SCM1 to SCM4 captures video of the work site WS from a different viewpoint. Each shooting point camera SCM1 to SCM4 may be built into each shooting point terminal ST1 to ST4, or may be attached externally separately from the shooting point terminals ST1 to ST4. The following explanation will be given assuming that each shooting point camera SCM1 to SCM4 is built into each shooting point terminal ST1 to ST4. As shown in Figure 2, each shooting point terminal ST1 to ST4 transmits shooting point video data SVD1 to SVD4 captured by the shooting point cameras SCM1 to SCM4 connected to it, time information when each shooting point video data SVD1 to SVD4 was captured, and three-dimensional position information and direction information of each shooting point camera SCM1 to SCM4 to the instructor terminal IT via the network NW.

[0017] The instructor terminal IT includes, for example, a computer used by the instructor. The instructor terminal IT may be a personal computer, or a server computer partly or entirely located within the instruction point IS. The instructor terminal IT may also be in the form of a VR device or the like. The instructor terminal IT is connected to an instructor camera ICM. The instructor camera ICM is, for example, a camera externally attached to the instructor terminal IT, and is positioned so as to be able to capture the movements of the instructor IU's fingers.

[0018] The instructor terminal IT receives video data sent from the worker terminal WT and the shooting point terminals ST1 to ST4 and shows it to the instructor IU via a display device. The instructor IU determines an instruction based on the video data shown on the instructor terminal IT and shows it, for example, using a hand gesture. The instructor camera ICM reads the hand gesture made by the instructor IU in the capture area and transmits the video data to the instructor terminal IT. The instructor terminal IT captures the hand gesture and generates a hand model representing the shape or appearance of the fingers. As shown in FIG. 2, the instructor terminal IT transmits time information of the video data to which the hand model is attached, as well as the shape, three-dimensional position information, and direction information of the hand model, to the worker terminal WT via the network NW. As described above, the instructor IU can issue instructions to the worker WU.

[0019] 1.1.2 Worker Terminal Fig. 3 is a block diagram showing an example of software functions provided in the worker terminal according to the first embodiment, together with an example of the hardware configuration. As shown in Fig. 3, the worker terminal WT includes a worker camera WCM, a control unit 10, a program storage unit 11, a communication I / F unit 12, a tracking sensor 13, and a display device 14, which are connected via a bus (not shown).

[0020] The control unit 10 includes, for example, a central processing unit (CPU) and a random access memory (RAM). The CPU of the control unit 10 controls the entire worker terminal WT. The RAM of the control unit 10 is used as a working area for the CPU of the control unit 10.

[0021] The program storage unit 11 includes, for example, an SSD (Solid State Drive) and a ROM (Read Only Memory), etc. The SSD and ROM of the program storage unit 11 are used in combination to store middleware such as an OS (Operating System), and programs used in various processes performed by the control unit 10.

[0022] The communication I / F unit 12 transmits and receives data to and from the instructor terminal IT using a communication protocol defined in the network NW.

[0023] The tracking sensor 13 includes, for example, an acceleration sensor and a gyro sensor. The tracking sensor 13 acquires three-dimensional position information of the worker camera WCM and information on the direction in which the worker camera WCM is facing, and transmits this information to the control unit 10. The information acquired by the tracking sensor 13 can be considered to be equivalent to three-dimensional position information and viewpoint direction information of the viewpoint of the worker WU. The tracking sensor 13 may be built into the worker camera WCM.

[0024] The display device 14 includes, for example, a liquid crystal display or an HMD. The display device 14 outputs video data captured by the worker camera WCM or composite video data generated by the control unit 10 and on which a hand model image is superimposed.

[0025] As processing functions for implementing the first embodiment, control unit 10 includes a video data acquisition unit 101, a position and direction information acquisition unit 102, a video data transmission unit 103, a hand model information reception unit 104, a hand model image generation unit 105, and a composite video data generation unit 106. These processing functions are implemented by expanding programs stored in program storage unit 11 into the RAM of control unit 10 and having the CPU of control unit 10 interpret and execute these programs.

[0026] The video data acquisition unit 101 acquires the worker video data WVD obtained by the worker camera WCM capturing the work site WS, and associates each image frame with information indicating the time of capture. The acquired worker video data WVD may be acquired at any timing of the worker WU using a capture button or the like, or the video data may be acquired automatically and continuously.

[0027] The position and direction information acquisition unit 102 acquires the three-dimensional position information of the worker camera WCM and the direction information of the worker camera WCM obtained from the tracking sensor 13, and associates this information with information indicating the acquisition time.

[0028] The video data transmission unit 103 references the shooting time of the worker video data WVD acquired by the video data acquisition unit 101, and associates it with the three-dimensional position information and direction information of the worker camera WCM acquired by the position and direction information acquisition unit 102. Then, the worker video data WVD, its shooting time information, and the three-dimensional position information and direction information of the worker camera WCM are transmitted to the instructor terminal IT via the communication I / F unit 12.

[0029] The hand model information receiving unit 104 receives hand model information transmitted from an instructor terminal IT (described later) via the communication I / F unit 12 .

[0030] The hand model image generating unit 105 generates a 2D or 3D hand model representing the shape or appearance of the fingers based on the hand model information received by the hand model information receiving unit 104 .

[0031] The composite video data generation unit 106 superimposes the hand model generated by the hand model image generation unit 105 at the coordinate position specified by the hand model information of the worker video data WVD acquired by the video data acquisition unit 101, to generate worker viewpoint composite video data, and displays it on the display device 14. Note that if no hand model information is received, the worker video data WVD acquired by the video data acquisition unit 101 is displayed on the display device 14.

[0032] 1.1.3 Shooting Point Terminal Figure 4 is a block diagram showing an example of the software functions of the shooting point terminal according to the first embodiment, along with an example of the hardware configuration. In the following description, the configuration of the shooting point terminal ST1 is shown as a representative of the shooting point terminals ST, but the configurations of the shooting point terminals ST2 to ST4 are similar. As shown in Figure 4, the shooting point terminal ST1 includes a shooting point camera SCM1, a control unit 20, a program storage unit 21, a communication I / F unit 22, and a tracking sensor 23, which are connected via a bus (not shown).

[0033] The control unit 20 includes, for example, a CPU, a RAM, etc. The CPU of the control unit 20 controls the entire shooting point terminal ST1. The RAM of the control unit 20 is used as a work area for the CPU of the control unit 20.

[0034] The program storage unit 21 includes, for example, an SSD, a ROM, etc. The SSD and the ROM of the program storage unit 21 are used in combination to store programs and the like used in various processes performed by the control unit 20.

[0035] The communication I / F unit 22 transmits data to the instructor terminal IT using a communication protocol defined in the network NW.

[0036] The tracking sensor 23 includes, for example, an acceleration sensor and a gyro sensor. The tracking sensor 23 acquires three-dimensional position information of the shooting point camera SCM1 connected to the shooting point terminal ST1 and information on the direction in which the shooting point camera SCM1 is facing, and transmits this information to the control unit 20. Note that the tracking sensor 23 may be built into the shooting point camera SCM1.

[0037] The control unit 20 includes, as processing functions for implementing the first embodiment, a video data acquisition unit 201, a position and direction information acquisition unit 202, and a video data transmission unit 203. These processing functions are implemented by expanding programs stored in the program storage unit 21 into the RAM of the control unit 20 and having the CPU of the control unit 20 interpret and execute these programs.

[0038] The video data acquisition unit 201 acquires the shooting point video data SVD1 obtained by capturing an image of the work site WS from the shooting point camera SCM1, and associates each image frame with information indicating the shooting time. The acquired shooting point video data SVD1 may be acquired at a timing selected by the worker WU using a shooting button or the like, or may automatically acquire video data continuously.

[0039] The position and direction information acquisition unit 202 acquires three-dimensional position information of the shooting point camera SCM1 and information on the direction in which the shooting point camera SCM1 is facing, obtained from the tracking sensor 23, and associates this information with information indicating the acquisition time.

[0040] The video data transmission unit 203 refers to the shooting time of the shooting point video data SVD1 acquired by the video data acquisition unit 201, and associates it with the three-dimensional position information and direction information of the shooting point camera SCM1 acquired by the position and direction information acquisition unit 202. Thereafter, the shooting point video data SVD1, its shooting time information, and the three-dimensional position information and direction information of the shooting point camera SCM1 are transmitted to the instructor terminal IT via the communication I / F unit 12.

[0041] 5 is a block diagram showing an example of the software functions of the instructor terminal according to the first embodiment, together with an example of the hardware configuration. As shown in Fig. 5, the instructor terminal IT includes a control unit 30, a program storage unit 31, a communication I / F unit 32, a selection device 33, and a display device 34, which are connected via a bus (not shown).

[0042] The control unit 30 includes, for example, a CPU and a RAM. The CPU of the control unit 30 controls the entire instructor terminal IT. The RAM of the control unit 30 is used as a work area for the CPU of the control unit 30.

[0043] The program storage unit 31 includes, for example, an SSD, a ROM, etc. The SSD and ROM of the program storage unit 31 are used in combination to store middleware such as an OS, and programs used in various processes performed by the control unit 30, etc.

[0044] The communication I / F unit 32 uses a communication protocol defined in the network NW to transmit and receive data to and from the worker terminal WT and to receive data from a plurality of shooting point terminals ST.

[0045] The selection device 33 includes, for example, a button, a keyboard, a touch panel, a mouse, etc. The selection device 33 selects and switches between the video data to be displayed on the display device 34 and the reference video data B_VD (described later in FIG. 7 ) that serves as a reference for providing hand model information, which will be described later, from the multiple video data received from the worker terminal WT and the multiple shooting point terminals ST.

[0046] The display device 34 includes, for example, a liquid crystal display, an HMD, or a tabletop display. The display device 34 outputs the video data selected by the selection device 33. The display device 34 may simultaneously display a plurality of video data side by side. In this case, the plurality of video data are displayed so that the shooting times of the respective video data are the same. Furthermore, when hand model information is added to the video data, the display device 34 outputs composite video data on which a hand model is superimposed. The display device 34 may simultaneously display a plurality of composite video data side by side, each with hand model information superimposed thereon.

[0047] As processing functions for implementing the first embodiment, the control unit 30 includes a video data receiving unit 301, a display video data selecting unit 302, a display video data generation processing unit 303, a motion capture processing unit 304, a reference video data selecting unit 305, a space model generating unit 306, a hand model information generating unit 307, and a hand model image generating unit 308. These processing functions are implemented by loading programs stored in the program storage unit 31 into the RAM of the control unit 30 and having the CPU of the control unit 30 interpret and execute these programs.

[0048] The video data receiving unit 301 receives the worker video data WVD transmitted from the worker terminal WT, the shooting time information of the worker video data WVD, and the three-dimensional position information and direction information of the worker camera WCM via the communication I / F unit 32. It also receives the shooting point video data SVD1 to SVD4 transmitted from the shooting point terminals ST1 to ST4, the shooting time information of each of the shooting point video data SVD1 to SVD4, and the three-dimensional position information and direction information of each of the shooting point cameras SCM1 to SCM4 via the communication I / F unit 32.

[0049] The display video data selection unit 302 receives an input from the selection device 33 and selects display video data to be displayed on the display device 34 from the plurality of video data received by the video data reception unit 301. At this time, a plurality of pieces of display video data to be displayed on the display device 34 may be selected.

[0050] Display video data generation processing unit 303 receives the video data sent from video data receiving unit 301 and displays the display video data selected by display video data selection unit 302 on display device 34. At this time, if multiple display video data are selected, display video data is generated so that the multiple video data for the same time are displayed simultaneously, and the display video data is arranged side by side on display device 34. Furthermore, when hand model information described below is generated, display video data generation processing unit 303 receives hand model information corresponding to each piece of display video data selected by display video data selection unit 302. Next, composite video data is generated by superimposing corresponding images from multiple hand models representing the shape or appearance of fingers generated by hand model image generation unit 308 described below on the coordinate positions of each piece of display video data specified by the hand model information. Thereafter, the composite video data is displayed on display device 34. At this time, if multiple composite video data are selected by display video data selection unit 302, composite video data is generated so that the multiple composite video data for the same time are displayed simultaneously, and the composite video data is arranged side by side on display device 34.

[0051] The motion capture processing unit 304 captures the gesture movements of the instructor IU's fingers based on the video data captured by the instructor camera ICM, and uses hand tracking technology to obtain the shape or form of the instructor IU's fingers and spatial position information in the capture area.

[0052] The reference video data selection unit 305 receives an input from the selection device 33 and selects one piece of reference video data B_VD, which will be the reference for adding a hand model, from the display video data selected by the display video data selection unit 302 .

[0053] The spatial model generation unit 306 acquires the three-dimensional position information and direction information of the worker camera WCM received by the video data receiving unit 301 and the three-dimensional position information and direction information of each of the shooting point cameras SCM1 to SCM4, and generates a virtual three-dimensional spatial model of the work site WS.

[0054] The hand model information generation unit 307 generates hand model information from the shape or form of the instructor IU's fingers and position information acquired by the motion capture processing unit 304. The hand model information is, for example, information indicating the shape of the hand model, the position where the hand model is superimposed, and the direction of the hand model from the viewpoints of the worker camera WCM and each of the shooting point cameras SCM1 to SCM4. Specifically, first, information indicating the shape of the hand model is generated from the shape or form of the instructor IU's fingers. Next, the position where the hand model is superimposed on the reference video data B_VD is determined based on the spatial position information of the instructor IU's fingers in the capture area. Next, the hand model is projected onto the three-dimensional spatial model of the work site WS generated by the spatial model generation unit 306 based on the three-dimensional position information and direction information of the camera that captured the reference video data B_VD. In other words, coordinate transformation is performed on the hand model on the reference video data B_VD to calculate the three-dimensional position and direction of the hand model, and the hand model is placed on the three-dimensional spatial model showing the work site WS. Then, hand model information indicating the shape of the hand model, the position where the hand model is superimposed, and the direction of the hand model from the viewpoint of each of worker camera WCM and each of shooting point cameras SCM1 to SCM4 is calculated and output. The generated hand model information is then sent together with time information to display video data generation processing unit 303 and hand model image generation unit 308, respectively. In addition, hand model information indicating the shape of the hand model, the position where the hand model is superimposed, and the direction of the hand model from the viewpoint of worker video data WVD is sent to worker terminal WT via communication I / F unit 32.

[0055] Based on the hand model information sent from hand model information generation unit 307, hand model image generation unit 308 generates a plurality of hand models representing the shape or appearance of fingers as seen from the viewpoint of worker camera WCM and each of the viewpoints of each of shooting point cameras SCM1 to SCM4 on a three-dimensional space model of work site WS. Hand model image generation unit 308 then transmits the generated plurality of hand models to display video data generation processing unit 303. Note that the plurality of hand models representing the shape or appearance of fingers may be generated only from the viewpoint of the camera capturing the display video data to be displayed on display device 34, selected by display video data selection unit 302, and transmitted to display video data generation processing unit 303.

[0056] 1.2 System Operation FIG. 6 is a sequence diagram showing an example of the operation of the worker terminal, the imaging point terminal, and the instructor terminal in the remote work support system according to the first embodiment. Note that in FIG. 6, one of the multiple imaging point terminals ST1 to ST4 is shown as the imaging point terminal ST. FIG. 7 is a diagram showing an example of an instructor superimposing a hand model on video data in the remote work support system according to the first embodiment. FIG. 8 is a diagram showing an example of video data with a hand model superimposed thereon in the remote work support system according to the first embodiment. An example of the operation of the remote work support system according to the first embodiment will be described below using FIGS. 6, 7, and 8.

[0057] Remote work support using the remote work support system according to the first embodiment is started by a command from the worker WU or instructor IU. When remote work support is started, in step S11, the control unit 10 of the worker terminal WT acquires video data of the work site WS that is output in real time from the worker camera WCM under the control of the video data acquisition unit 101.

[0058] Next, in step S12, the control unit 10 of the worker terminal WT, under the control of the position and direction information acquisition unit 102, acquires the three-dimensional position information of the worker camera WCM and the direction information in which the worker camera WCM is facing, which are output in real time from the tracking sensor 13.

[0059] Then, in step S13, the control unit 10 of the worker terminal WT, under the control of the video data transmission unit 103, adds three-dimensional position information and direction information of the worker camera WCM to the video data, and transmits the worker video data WVD with this information from the communication I / F unit 12 to the instructor terminal IT.

[0060] In step S21, the control unit 20 of the shooting point terminal ST, under the control of the video data acquisition unit 201, acquires video data of the work site WS that is output in real time from the shooting point camera SCM.

[0061] Next, in step S22, the control unit 20, under the control of the position and direction information acquisition unit 202, acquires the three-dimensional position information of the shooting point camera SCM and the direction information in which the shooting point camera SCM is facing, which are output in real time from the tracking sensor 23.

[0062] Then, in step S23, the control unit 20, under the control of the video data transmission unit 203, adds three-dimensional position information and direction information of the shooting point camera SCM to the video data, and transmits the shooting point video data SVD with this information from the communication I / F unit 22 to the instructor terminal IT.

[0063] Under the control of the video data receiving unit 301, the control unit 30 of the instructor terminal IT receives the worker video data WVD and the shooting point video data SVD1 to SVD4 with information from the worker terminal WT and the multiple shooting point terminals ST. Then, in step S31, under the control of the display video data selecting unit 302, the selection device 33 selects display video data to be displayed on the display device 34 from the worker video data WVD and the shooting point video data SVD1 to SVD4. Multiple display video data may be selected. Under the control of the reference video data selecting unit 305, the selection device 33 selects reference video data B_VD from the display video data, which serves as a reference for adding a hand model. If only one display video data is selected, the selected display video data is automatically selected as the reference video data B_VD.

[0064] In the example shown below, a case will be described in which the worker video data WVD and the shooting point video data SVD1 are selected as the display video data, and the shooting point video data SVD1 is selected as the reference video data B_VD.

[0065] Thereafter, in step S32, the control unit 30 displays the selected display video data on the display device 34 under the control of the display video data generation processing unit 303. Note that if multiple pieces of video data are selected as display video data in step S31, the selected multiple pieces of display video data are displayed side by side simultaneously. It is preferable that the display video data displayed on the display device 34 be synchronized with the video captured by the worker camera WCM and the video captured by the shooting point cameras SCM1 to SCM4. This allows the instructor IU to check the state of the work site WS in real time.

[0066] In step S33, the control unit 30, under the control of the space model generation unit 306, generates a three-dimensional space model of the work site WS from the three-dimensional position information and direction information of each camera attached to each video data.

[0067] In this state, suppose that the instructor IU makes a hand gesture within the capture area CA to give a work instruction while looking at the display image data displayed on the display device 34. For example, suppose that the instructor IU makes a hand gesture HG as shown in part (A) of Figure 7 within the capture area CA to point at the object OBJ1, which is the work target, with respect to the shooting point image data SVD1, which is the reference image data B_VD.

[0068] When the control unit 30 detects the hand gesture HG from the video data captured by the instructor camera ICM (S34: Yes), in step S35, under the control of the motion capture processing unit 304, the control unit 30 captures the hand gesture HG by the instructor IU.

[0069] Next, under the control of the hand model information generator 307, the control unit 30 generates hand model information in step S36. Specifically, first, information indicating the shape of the hand model acquired by motion capture is generated. Then, the hand model is projected onto a three-dimensional spatial model of the work site WS, and information indicating the shape of the hand model, the position where the hand model is superimposed, and the orientation of the hand model from each viewpoint of the worker video data WVD and each of the shooting point video data SVD1 to SVD4 is calculated and output. That is, in video data different from the reference video data B_VD, the position of the hand model is moved to a position corresponding to the viewpoint from the camera that captured that video data. Furthermore, the shape and orientation of the hand model are rotated so that they correspond to the shape and orientation from the viewpoint from the camera that captured that video data.

[0070] Then, in step S37, under the control of the hand model information generating unit 307, the control unit 30 transmits, from the hand model information, the hand model information from the viewpoint of the worker video data WVD to the worker terminal WT from the communication I / F unit 32. The hand model information specifies the shape, position, and orientation of the hand model when the hand model is superimposed on the worker video data WVD.

[0071] Thereafter, in step S38, the control unit 30, under the control of the display video data generation processing unit 303, generates a 2D or 3D hand model image, for example, representing the shape or appearance of fingers, corresponding to each display video data based on the hand model information. The generated hand model image is then superimposed on each display video data. Specifically, the hand model image is superimposed on the display video data, to which the three-dimensional position information and direction information was added, displayed in S32, by referring to the coordinate information and direction information included in the hand model information. In this manner, composite video data is generated. For example, as shown in part (B) of FIG. 7, composite video data SCVD1 is generated by superimposing a hand model HM on the shooting point video data SVD1.

[0072] Thereafter, in step S39, the control unit 30, under the control of the display video data generation processing unit 303, displays the composite video data on which the hand model image is superimposed on the display device 34. For example, for a hand gesture HG as shown in part (A) of FIG. 7, the composite video data SCVD1 corresponding to the shooting point video data SVD1 is displayed as shown in part (B) of FIG. 7. The hand model HM in the composite video data SCVD1 is synchronized with the movement of the hand gesture HG captured by the instructor camera ICM. In other words, it is possible to give instructions using the hand gesture HG while checking the composite video data SCVD1 in real time.

[0073] If the control unit 30 does not detect the hand gesture HG from the video data captured by the instructor camera ICM (S34: No), the processes of steps S35 to S39 are skipped.

[0074] When the control unit 10 of the worker terminal WT receives the hand model information transmitted from the instructor terminal IT under the control of the hand model information receiving unit 104 (S14: Yes), the control unit 10 performs the following processing under the control of the composite video data generating unit 106.

[0075] First, in step S15, the hand model information is read, and a 2D or 3D hand model image representing, for example, the shape or appearance of the fingers is generated. The generated hand model image is then superimposed on the worker video data WVD. Specifically, the hand model image is superimposed on the worker video data WVD, which was generated in S13 and to which the three-dimensional position information and direction information of the worker camera WCM have been added, by referring to the coordinate information and direction information included in the hand model information.

[0076] Thereafter, in step S16, the video data on which the hand model image is superimposed is displayed on the display device 14.

[0077] FIG. 8 shows an example of composite video data in which a hand model image is superimposed when an operation of pointing at an object OBJ1 is performed on the shooting point video data SVD1 transmitted from the shooting point terminal ST1, which is the reference video data B_VD. Part (A) of FIG. 8 is an example of composite video data SCVD1 in which a hand model HM is superimposed on the shooting point video data SVD1, and corresponds to part (B) of FIG. 7. Part (B) of FIG. 8 is an example of composite video data WCVD in which a hand model HM is superimposed on worker video data WVD. As shown in FIG. 8, all hand model images point to the same coordinate position (the position of the object OBJ1) from the same direction. The hand model shown in part (B) of FIG. 7 is generated by moving and rotating the hand model shown in part (A) of FIG. 7.

[0078] If the control unit 10 does not receive the hand model information (S14: No), the processes of steps S15 and S16 are skipped.

[0079] In this way, remote work support using the hand model is performed. In reality, each terminal repeats the above series of processes to continuously perform remote work support. When the work is completed, or when the instructor IU or worker WU determines that remote work support is no longer necessary, the remote work support ends.

[0080] 1.3 Effects The use of the remote operation support system according to the first embodiment makes it possible to easily give instructions for remote work. This effect will be described in detail with reference to Figures 8 and 9. Figure 9 is a diagram showing an example of video data on which a hand model is superimposed in the remote operation support system according to the first embodiment.

[0081] When providing remote work support, it may be difficult to give instructions using only the image from the worker camera. For example, in the example shown in FIG. 8 , as shown in part (B), from the viewpoint of the worker camera WCM, part of object OBJ1 is hidden behind object OBJ2. Therefore, it is difficult for the instructor IU to confirm the state of object OBJ1. Furthermore, when pointing at object OBJ1 using a hand model from the viewpoint of the worker camera WCM, it may be recognized as pointing at another target (e.g., object OBJ2), which may result in an erroneous instruction.

[0082] In the remote work support system according to the first embodiment, the instructor IU can check the state of the work site WS from the viewpoints of multiple imaging point cameras SCM1 to SCM4 in addition to the worker camera WCM. This allows the instructor IU to check the state of the work site WS that could not be checked from the viewpoint of the worker camera WCM. Furthermore, the instructor IU can give instructions using a hand model from the viewpoints of multiple imaging point cameras SCM1 to SCM4 in addition to the worker camera WCM. For example, in the example shown in FIG. 8 , an object OBJ1 is pointed to using a hand model HM from the viewpoint of imaging point camera SCM1. This allows the instructor IU to more clearly point to the object he or she wants to point out, reducing the occurrence of erroneous instructions to the object being pointed out.

[0083] Furthermore, in the remote operation support system according to the first embodiment, the worker WU can view the hand model instructed from the viewpoint of the instructor IU from the viewpoint of the worker camera WCM. Therefore, the worker WU can view the instructions from approximately the same viewpoint as the worker WU, and can intuitively recognize the instructions given by the instructor IU's hand model HM.

[0084] Additionally, in the remote work system according to the first embodiment, the hand model is superimposed in accordance with the coordinate information and direction information of the worker camera WCM. Therefore, when the position and direction of the worker camera WCM change, the position and direction of the hand model are re-rendered accordingly. FIG. 9 shows an example in which the viewpoint of the worker camera WCM changes due to the movement of the worker WU. Part (A) of FIG. 9 shows the composite video data WCVD in which a hand model HM is superimposed on the worker video data WVD before the movement. Part (B) of FIG. 9 shows the composite video data WCVD in which a hand model HM is superimposed on the worker video data WVD after the movement. Comparing parts (A) and (B) of FIG. 9 reveals that when the worker WU moves and the viewpoint of the worker camera WCM changes, the hand model HM changes its shape, position, and angle to continue pointing at the object OBJ1 being pointed to from the same direction. In this way, even if the worker WU moves or changes his / her orientation, the hand model can continue to point to the same target while changing its shape, position, and angle in the video data. This eliminates the need to modify the hand model in response to the worker WU's movement or change of viewpoint, making it easier to give instructions. In addition, it is possible to prevent erroneous instructions from occurring due to a lack of hand model modification or an incorrect hand model modification.

[0085] 1.4 Modifications The remote operation support system according to the first embodiment can be modified in various ways. Below, the first, second, and third modifications of the first embodiment will be described, focusing on the differences from the first embodiment.

[0086] As a first modification, the display device 14 of the worker terminal WT may be a translucent MR display. In this case, the composite video data generator 106 in the control unit 10 of the worker terminal WT displays only the hand model generated by the hand model image generator 105 on the display device 14 and superimposes the hand model on the actual work site WS. In this way, the instructor IU can give instructions to the worker WU using the hand model.

[0087] As a second modified example, the instructor terminal IT may transmit to the worker terminal WT composite video data WCVD in which a hand model is superimposed on the worker video data WVD. In this case, the hand model information receiving unit 104 of the worker terminal WT functions as a unit that receives the composite video data WCVD. Furthermore, the control unit 10 does not need to include the hand model image generating unit 105 and the composite video data generating unit 106. The worker terminal WT displays the received composite video data WCVD on the display device 14.

[0088] As a third modified example, the indicator used by the instructor IU to give instructions to the worker WU does not have to be a hand model. For example, an arrow or an icon may be used as the indicator. In this case, the hand model image generation unit 105 in the control unit 10 of the worker terminal WT functions as a unit that generates an image of the indicator. Furthermore, the hand model information generation unit 307 in the control unit 30 of the instructor terminal IT functions as a unit that generates information on the shape, position, and direction of the indicator. The hand model image generation unit 308 functions as a unit that generates an image of the indicator.

[0089] 2. Second Embodiment Next, a remote operation support system according to a second embodiment will be described. The following mainly describes the configuration that differs from the first embodiment.

[0090] 2.1 Configuration 2.1.1 Worker Terminal Fig. 10 is a block diagram showing an example of software functions provided in the worker terminal WT according to the second embodiment, together with an example of the hardware configuration. As shown in Fig. 10, the worker terminal WT according to the second embodiment further includes a depth sensor 15.

[0091] The depth sensor 15 includes, for example, a light emitter and a Time of Flight (ToF) sensor for performing a ToF distance measurement function. The depth sensor 15 is, for example, built into the worker camera WCM. The depth sensor 15 calculates the distance from the viewpoint of the worker camera WCM to a subject shown in the image at the work site WS and outputs the calculated distance as depth information.

[0092] The control unit 10 also includes a depth information acquisition unit 107 as a processing function for implementing the second embodiment. This processing function is implemented by expanding programs stored in the program storage unit 11 into the RAM of the control unit 10 and having the CPU of the control unit 10 interpret and execute these programs.

[0093] The depth information acquisition unit 107 acquires the depth information obtained from the depth sensor 15 .

[0094] The video data transmission unit 103 references the shooting time of the worker video data WVD acquired by the video data acquisition unit 101, and associates it with the three-dimensional position information and direction information of the worker camera WCM acquired by the position and direction information acquisition unit 102, and the depth information acquired by the depth information acquisition unit 107. Then, the worker video data WVD, its shooting time information, the three-dimensional position information and direction information of the worker camera WCM, and the depth information are transmitted to the instructor terminal IT via the communication I / F unit 12.

[0095] 2.1.2 Shooting Point Terminal Figure 11 is a block diagram showing an example of the software functions of the shooting point terminal according to the second embodiment, along with an example of the hardware configuration. In Figure 11, the configuration of the shooting point terminal ST1 is shown as a representative of the shooting point terminals ST, but the configurations of the shooting point terminals ST2 to ST4 are similar. As shown in Figure 10, the shooting point terminal ST1 according to the second embodiment further includes a depth sensor 24.

[0096] The depth sensor 24 includes, for example, a light emitter and a ToF sensor for performing a ToF distance measurement function. The depth sensor 24 is, for example, built into the shooting point camera SCM1. The depth sensor 24 calculates the distance from the shooting point to the subject shown in the image at the work site WS and outputs the calculated distance as depth information.

[0097] The control unit 20 also includes a depth information acquisition unit 204 as a processing function for implementing the second embodiment. This processing function is implemented by expanding programs stored in the program storage unit 21 into the RAM of the control unit 20 and having the CPU of the control unit 20 interpret and execute these programs.

[0098] The depth information acquisition unit 204 acquires the depth information obtained from the depth sensor 24 .

[0099] The video data transmission unit 203 refers to the shooting time of the shooting point video data SVD1 acquired by the video data acquisition unit 201, and associates it with the three-dimensional position information and direction information of the shooting point camera SCM1 acquired by the position and direction information acquisition unit 202, and the depth information acquired by the depth information acquisition unit 204. Thereafter, the shooting point video data SVD1, its shooting time information, the three-dimensional position information and direction information of the shooting point camera SCM1, and the depth information are transmitted to the instructor terminal IT via the communication I / F unit 22.

[0100] 2.1.3 Instructor Terminal The instructor terminal IT according to the second embodiment performs processing in the control unit 30 that is partially different from that in the first embodiment.

[0101] The video data receiving unit 301 receives the worker video data WVD and the shooting point video data SVD1 to SVD4 each including depth information from the worker terminal WT and the plurality of shooting point terminals ST.

[0102] When displaying the video data sent from video data receiving unit 301 on display device 34, display video data generation processing unit 303 may display video data including depth information on display device 34. Furthermore, when hand model information is generated, display video data generation processing unit 303 may generate composite video data by superimposing a hand model on the coordinate position of each piece of display video data including depth information specified by the hand model information, and display the composite video data on display device 34.

[0103] The spatial model generation unit 306 performs three-dimensional spatial mapping of the work site WS from the three-dimensional position information, direction information, and depth information of each camera attached to the multiple video data acquired by the video data receiving unit 301, and generates a three-dimensional spatial model.

[0104] 2.2 Effects The remote work support system according to the second embodiment makes it possible to easily give instructions for work remotely. The effects of this will be described in detail below.

[0105] The remote work support system according to the second embodiment generates a three-dimensional spatial model by spatial mapping and adds a hand model to the three-dimensional spatial model. This allows the instructor IU to grasp the situation at the work site WS in a more three-dimensional manner. Furthermore, when giving instructions to the worker WU, it becomes possible to add a three-dimensional hand model to each piece of video data. Therefore, the instructor IU can give more explicit instructions to the worker WU.

[0106] 2.3 Modifications The remote operation support system according to the second embodiment can be modified in various ways. Below, a first modification of the second embodiment will be described, focusing on the differences from the second embodiment.

[0107] As a first modification, the instructor terminal IT may be an HMD-type or smart glasses-type terminal worn on the head of the instructor IU. The instructor terminal IT may also include a head tracking sensor. The head tracking sensor includes, for example, an acceleration sensor and a gyro sensor. The head tracking sensor detects the direction in which the head of the instructor IU is facing, and changes the angle of the image displayed on the display device 34 according to that direction.

[0108] The remote work support system according to the first modification of the second embodiment can change the viewpoint angle of the instructor IU on the three-dimensional space model. This allows the instructor IU to spontaneously check the situation at the work site WS. Furthermore, the instructor IU can give instructions to the worker WU by attaching a hand model or the like from a viewpoint that makes it easier for the instructor IU to give instructions.

[0109] 3. Other Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0110] WT...worker terminal WS...work site WU...worker WCM...worker camera WVD...worker video data ST, ST1, ST2, ST3, ST4...shooting point terminal SCM, SCM1, SCM2, SCM3, SCM4...shooting point camera SVD, SVD1, SVD2, SVD3, SVD4...shooting point video data IT...instructor terminal IS...instruction point IU...instructor NW...network 10, 20, 30...control unit 11, 21, 31...program storage unit 12, 22, 32...communication I / F unit 13, 23...tracking sensor 14, 34...display device 15, 24...depth sensor 33...selection device 101, 201...video data acquisition unit 102, 202...position and direction information acquisition unit 103, 203...video data transmission unit 104...Hand model information receiving unit 105, 308...Hand model image generating unit 106...Synthesized video data generating unit 107, 204...Depth information acquiring unit 301...Video data receiving unit 302...Display video data selecting unit 303...Display video data generating processing unit 304...Motion capture processing unit 305...Reference video data selecting unit 306...Spatial model generating unit 307...Hand model information generating unit B_VD...Reference video data CA...Capture area HG...Hand gesture HM...Hand model OBJ1, OBJ2...Object WCVD, SCVD1...Synthesized video data

Claims

1. A remote work support device comprising: a generation unit that generates a spatial model of a work site based on first video data obtained by photographing the work site from a first position in a first direction and second video data obtained by photographing the work site from a second position in a second direction; a placement unit that places an indicator within the spatial model; and a first superimposition unit that superimposes first indicator data showing the indicator when viewed in the first direction from the first position onto the first video data, and superimposes second indicator data showing the indicator when viewed in the second direction from the second position onto the second video data.

2. A remote work support device as described in claim 1, further comprising: a capture unit that captures the movement of the instructor's fingers based on third video data of the fingers and acquires the shape or form and position information of the fingers; and an indicator generation unit that generates the indicator based on the shape or form of the fingers acquired by the capture unit, wherein the placement unit places the indicator within the spatial model based on the position information of the fingers acquired by the capture unit.

3. A remote work support device as described in claim 2, further comprising a selection unit that selects reference image data from the first image data and the second image data that serves as a basis for placing the indicator, and the placement unit places the indicator within the spatial model based on the coordinates of the fingers when the fingers are projected onto the reference image data based on the position information of the fingers.

4. A remote work support device as described in claim 1, wherein when fourth image data photographing the work site from a third position in a third direction is acquired instead of the first image data, the first superimposition unit superimposes third indicator data showing the indicator when viewed in the third direction from the third position onto the fourth image data.

5. A remote work support system comprising an instructor terminal, a worker terminal, and a photographing point terminal connected via a network, for transmitting instruction information relating to work from the instructor terminal held by an instructor to the worker terminal held by a worker, wherein the worker terminal comprises: a first acquisition unit that acquires first image data of a work site photographed from a first position in a first direction; the photographing point terminal comprises: a second acquisition unit that acquires second image data of the work site photographed from a second position in a second direction; the instructor terminal comprises: a generation unit that generates a spatial model of the work site based on the first image data transmitted from the worker terminal and the second image data transmitted from the photographing point terminal; a placement unit that places an indicator within the spatial model; and a first superimposition unit that superimposes first indicator data showing the indicator when viewed in the first direction from the first position onto the first image data and superimposes second indicator data showing the indicator when viewed in the second direction from the second position onto the second image data; and the worker terminal comprises: a second superimposing unit that superimposes first indicator data transmitted from the instructor terminal on the first video data; and a display unit that displays the first video data on which the first indicator data is superimposed to the worker.

6. The remote work support system according to claim 5, wherein the worker terminal further comprises a tracking unit that acquires the position and viewpoint direction of the worker, wherein the first position is approximately equal to the viewpoint position of the worker, and the first direction is approximately equal to the viewpoint direction of the worker, and the tracking unit acquires information about the first position and the first direction, and when the position and viewpoint direction of the worker change and the first acquisition unit acquires fourth video data of the work site captured from a third position in a third direction instead of the first video data, the tracking unit acquires information about the third position and the third direction, and the first superimposition unit superimposes third indicator data showing the indicator when viewed in the third direction from the third position onto the fourth video data transmitted from the worker terminal, based on the information about the third position and the third direction transmitted from the worker terminal, and the second superimposition unit superimposes the third indicator data transmitted from the instructor terminal onto the fourth video data, and the display unit displays the fourth video data with the third indicator data superimposed.

7. A remote work support method comprising: generating a spatial model of a work site based on first video data obtained by photographing the work site from a first position in a first direction and second video data obtained by photographing the work site from a second position in a second direction; arranging an indicator within the spatial model; and superimposing first indicator data showing the indicator when viewed in the first direction from the first position onto the first video data, and superimposing second indicator data showing the indicator when viewed in the second direction from the second position onto the second video data.

8. A remote work support program for causing a computer to function as each unit of the remote work support device according to claim 1.

Images