Remote work support system and remote work support method

Abstract

In one aspect of the present invention, when a manager terminal and a worker terminal are connected to each other via a network and instruction information relating to work is transmitted from the manager terminal to the worker terminal, the worker terminal stores video data obtained by capturing a work area in a storage unit in a state where timestamp information indicating a capturing time is attached for each image frame and transmits the video data to the manager terminal. When an instruction operation is performed on the video data in a state where the manager terminal has received and is displaying the video data having the timestamp information attached thereto, the manager terminal transmits instruction information corresponding to the instruction operation to the worker terminal, while attaching the timestamp information attached to the image frame subject to the instruction operation to the instruction information. Upon receiving the instruction information having the timestamp information attached thereto, the worker terminal selects, from the storage unit, the image frame having attached thereto the same timestamp information as the timestamp information attached to the received instruction information and displays the instruction information in a manner combined with the selected image frame.

Description

Remote work support system and remote work support method

[0001] One aspect of the present invention relates to a remote work support system and a remote work support method, for example, by having a controller send instruction information to a worker in a remote location via a network to support the worker's work.

[0002] When a controller at a management center and a worker in a remote location work together to perform a task, instructions from the controller to the worker are often given via telephone, video conference, etc. In this case, the controller's instructions are often given mainly by voice, but because the worker cannot perceive them visually, it is expected that the more complex the task, the more likely it is that instructions will be miscommunicated.

[0003] Therefore, in order for the controller to communicate instructions to the worker more clearly, a method has been proposed in which information on hand movements and pointing fingers is acquired using hand tracking technology, etc., and transmitted to the worker, and this hand and finger instruction information is then superimposed on the video captured during work on the worker's terminal (see, for example, non-patent document 1).

[0004] NTT XR Real Support, "MR technology enables remote support from the scene, as if you were next to someone, even if you are far away," retrieved October 31, 2023, Internet <URL: https: / / group.ntt / jp / nttxr / service / service00028 / >

[0005] However, if a transmission delay occurs in the network when transmitting the video captured by the worker and the controller's finger instruction information between the worker's terminal and the controller's terminal, when the finger instruction information is superimposed on the current work video and displayed on the worker's terminal, the display of the current work video and the finger instruction information will become asynchronous. In this case, the position of the controller's finger instruction on the work video will be shifted from the original position, and as a result, there is a risk that the worker will mistakenly recognize a different object in the work video as the instruction target.

[0006] This invention was made with the above-mentioned circumstances in mind, and aims to provide a technology that synchronizes the display of the supervisor's instruction information with the work video, even if a transmission delay occurs on the network, thereby enabling the supervisor to always accurately instruct the target of the instruction.

[0007] In order to solve the above problems, a first aspect of the present invention is a system in which a controller terminal and a worker terminal are connected via a network, and when instruction information related to work is transmitted from the controller terminal to the worker terminal, the worker terminal stores video data of a target work area in a memory unit with timestamp information indicating the time of shooting for each image frame attached, and performs a process of transmitting the video data with the timestamp information attached to it to the controller terminal; the controller terminal receives and displays the video data with the timestamp information attached, and when an instruction operation related to the work is performed on the displayed video data, generates instruction information corresponding to the instruction operation, attaches to the generated instruction information the timestamp information attached to the image frame of the video data on which the instruction operation was performed, and transmits the instruction information with the timestamp information attached to it to the worker terminal; and when the worker terminal receives the instruction information with the timestamp information attached, it selects from the memory unit an image frame to which the same timestamp information as the timestamp information attached to the received instruction information is attached, and performs a process of combining the instruction information with the selected image frame for display.

[0008] In a second aspect of the present invention, a controller terminal and a worker terminal are connected via a network, and when instruction information related to work is transmitted from the controller terminal to the worker terminal, the controller terminal receives and displays video data of the area to be worked on from the worker terminal, and when an instruction operation related to the work is performed on the displayed video data, generates instruction information representing the position specified by the instruction operation for each image frame of the video data and transmits this instruction information to the worker terminal, and when the worker terminal receives the instruction information transmitted from the controller terminal, predicts the specified position of the instruction information in a third image frame that is later in time than the second image frame based on the movement history of the instruction information between a first image frame and the subsequent second image frame of the video data, and performs processing to synthesize and display the instruction information at the predicted position in the third image frame.

[0009] In a third aspect of the present invention, a controller terminal and a worker terminal are connected via a network, and when instruction information related to work is transmitted from the controller terminal to the worker terminal, the controller terminal receives and displays video data of the area to be worked on from the worker terminal, and when an instruction operation related to the work is performed on the displayed video data, extracts a partial image area related to a position specified by the instruction operation from a first image frame of the video data and transmits image data corresponding to the extracted image area to the worker terminal, and when the worker terminal receives the image data transmitted from the controller terminal, detects an image area similar to the received image data from a second frame that is later in time than the first image frame of the video data, sets the position of the detected image area as a predicted position of the instruction operation in the second image frame, and performs processing to synthesize and display the instruction information representing the instruction operation at the predicted position in the second image frame.

[0010] According to the first to third aspects of the present invention, even if a transmission delay occurs over the network during the process of transmitting video data and instruction information between the operator terminal and the controller terminal, the video data and the instruction information corresponding to the instruction operation performed on the video data are displayed in a synchronized state on the operator terminal. Therefore, the object designated by the instruction operation can always be accurately designated in the video data displayed on the operator terminal, thereby preventing erroneous instructions from being given to the object.

[0011] In other words, according to each aspect of the present invention, even if a transmission delay occurs in the network, it is possible to provide a technology that synchronizes the display of the supervisor's instruction information with the work video, thereby enabling the supervisor to always accurately instruct the target of instruction.

[0012] FIG. 1 is a diagram showing an example of the system configuration of a remote work support system according to a first embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of a worker terminal used in the remote work support system shown in FIG. 1. FIG. 3 is a block diagram showing an example of the configuration of a controller terminal used in the remote work support system shown in FIG. 1. FIG. 4 is a flowchart showing an example of the processing procedure and processing content executed by the control unit of the worker terminal shown in FIG. 2. FIG. 5 is a flowchart showing an example of the processing procedure and processing content executed by the control unit of the controller terminal shown in FIG. 3. FIG. 6 is a diagram for explaining an example of operations realized by each process of the worker terminal and the controller terminal shown in FIGS. 4 and 5, respectively. FIG. 7 is a block diagram showing an example of the configuration of a worker terminal used in a remote work support system according to a second embodiment of the present invention. FIG. 8 is a block diagram showing an example of the configuration of a controller terminal used in the remote work support system according to the second embodiment of the present invention. FIG. 9 is a flowchart showing an example of the processing procedure and processing content executed by the control unit of the worker terminal shown in FIG. 7. FIG. 10 is a flowchart showing an example of the processing procedure and processing content executed by the control unit of the controller terminal shown in FIG. 8. FIG. 11 is a diagram illustrating an example of an operation realized by each process of the worker terminal and the controller terminal shown in FIGS. 9 and 10 , respectively. FIG. 12 is a diagram illustrating an example of a finger indication position prediction process among the series of processes shown in FIG. 9 . FIG. 13 is a block diagram illustrating an example of a configuration of a worker terminal used in a remote work support system according to a third embodiment of the present invention. FIG. 14 is a block diagram illustrating an example of a configuration of a controller terminal used in a remote work support system according to the third embodiment of the present invention. FIG. 15 is a flowchart illustrating an example of a processing procedure and processing content executed by the control unit of the worker terminal shown in FIG. 13 . FIG. 16 is a flowchart illustrating an example of a processing procedure and processing content executed by the control unit of the controller terminal shown in FIG. 14 . FIG. 17 is a diagram illustrating an example of a similarity prediction process among the series of processes shown in FIG. 15 . FIG. 18 is a diagram illustrating another example of a similarity prediction process.

[0013] Hereinafter, several embodiments of the present invention will be described with reference to the drawings.

[0014] First Embodiment (Configuration Example) (1) System FIG. 1 is a diagram showing an example of the system configuration of a remote operation support system according to a first embodiment of the present invention.

[0015] The remote work support system according to the first embodiment of the present invention enables communication between a controller terminal MT used by a controller MU at a management center MS and a worker terminal WT used by a worker WU at a remote work site WS via a network NW, thereby enabling the controller MU to give work instructions to the worker WU while viewing video of the work site WS.

[0016] The network NW comprises, for example, a wide area network centered on the Internet and an access network for accessing this wide area network. Examples of the access network include, but are not limited to, a public communication network using wired or wireless connections, a local area network (LAN) using wired or wireless connections, and a cable television (CATV) network. Note that, if the service provision area is limited to a company's business establishment or office, the network NW may be configured solely with a LAN or wireless LAN.

[0017] (2) Worker Terminal WT The worker terminal WT is, for example, a head-mounted display (HMD) type terminal that is worn on the head of the worker WU.

[0018] FIG. 2 is a block diagram showing the software functions of the worker terminal WT together with the hardware configuration.

[0019] The worker terminal WT includes a control unit 1A that uses a hardware processor such as a central processing unit (CPU), and is connected to this control unit 1A via a bus (not shown) to a program storage unit 2, a communication interface (hereinafter, interface will be referred to as I / F) unit 3, a video data storage unit 41, and a display device 5, and can also be connected to an external camera WCM. The camera WCM may be of a type that is built into the HMD.

[0020] The program storage unit 2 is configured by combining, for example, a nonvolatile memory such as a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a read only memory (ROM), and stores middleware such as an operating system (OS) as well as application programs required to execute various controls according to the first embodiment. Hereinafter, the OS and each application program will be collectively referred to as a program.

[0021] The communication I / F unit 3 transmits and receives data to and from the controller terminal MT using a communication protocol defined in the network NW.

[0022] The video data storage unit 41 is provided in a storage area within the data storage unit using a non-volatile memory such as an SSD that can be written to and read at any time, and stores video data of the work area captured by the camera WCM.

[0023] The control unit 1A has the processing functions necessary to implement the first embodiment of this invention, including a video data acquisition processing unit 11, a timestamp generation processing unit 12, a video data transmission processing unit 13, a finger instruction data reception processing unit 14, and a composite video data generation processing unit 15A.

[0024] The processing units 11 to 15A are all realized by causing a hardware processor in the control unit 1A to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 11 to 15A may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0025] The video data acquisition processing unit 11 acquires video data obtained by photographing the work site WS from the camera WCM, and stores the acquired video data in the video data storage unit 41 by correlating each image frame with information indicating the time of shooting.

[0026] The timestamp generation processing unit 12 generates timestamp information for notifying the controller terminal MT of the shooting time associated with each image frame of the video data. Note that the information representing the shooting time may be used as the timestamp information as is.

[0027] The video data transmission processing unit 13 reads the video data from the video data storage unit 41 in the order of shooting time, adds the timestamp information to each image frame of the read video data, and then transmits it from the communication I / F unit 3 to the controller terminal MT.

[0028] The finger instruction data receiving processor 14 receives finger instruction data of the controller MU transmitted from the controller terminal MT, which will be described later, via the communication I / F unit 3 .

[0029] The composite video data generation processing unit 15A generates image data representing the shape or appearance of the fingers based on the received finger instruction data. At the same time, the composite video data generation processing unit 15A reads, based on the timestamp information added to the finger instruction data, image frames assigned with shooting times corresponding to the timestamp information from the video data storage unit 41. The composite video data generation processing unit 15A then generates composite video data by superimposing the image data representing the shape or appearance of the fingers at the coordinate positions specified by the finger instruction data in the read image frames, and displays the generated composite video data on the display device 5.

[0030] (3) Controller Terminal MT The controller terminal MT is a personal computer used by the controller. Note that the controller terminal MT may be configured as a server computer located in the management center MS, in part or in whole.

[0031] FIG. 3 is a block diagram showing the software functions of the controller terminal MT together with the hardware configuration.

[0032] The controller terminal MT is equipped with a control unit 6A that uses a hardware processor such as a central processing unit (CPU), and this control unit 6A is connected to a program storage unit 7, a communication I / F unit 8, and a display device 9 via a bus (not shown), and is also connected to an external camera MCM. The camera MCM is placed in a position where it can capture images of the movements of the controller MU's fingers.

[0033] The program storage unit 7 is configured by combining, for example, an SSD and a ROM as storage media, and stores middleware such as an OS as well as application programs required to execute various controls according to the first embodiment. Hereinafter, the OS and each application program will be collectively referred to as a program.

[0034] The communication I / F unit 8 transmits and receives data to and from the worker terminal WT using a communication protocol defined in the network NW.

[0035] The control unit 6A has the processing functions necessary to implement the first embodiment of this invention, including a video data reception processing unit 61, a timestamp extraction processing unit 62, a finger instruction data generation processing unit 63A, and a motion capture processing unit 64.

[0036] The processing units 61 to 64 are all realized by causing a hardware processor in the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 61 to 64 may be realized using hardware such as an LSI or an ASIC.

[0037] The video data reception processing unit 61 receives video data of the work area transmitted from the worker terminal WT during remote work support via the communication I / F unit 8 , and displays the received video data on the display device 9 .

[0038] The timestamp extraction processing section 62 extracts, from the video data received by the video data reception processing section 61, timestamp information added to the video data.

[0039] The motion capture processing unit 64 captures gesture movements using the fingers of the controller MU based on the video data captured by the camera MCM, and uses hand tracking technology to generate finger instruction data indicating the coordinates of the indicated position in the image frame specified by the operation of the fingers.

[0040] The finger instruction data generation processing unit 63A generates time-stamped finger instruction data by adding the timestamp information extracted by the timestamp extraction processing unit 62 to the finger instruction data generated by the motion capture processing unit 64. Then, the generated time-stamped finger instruction data is transmitted from the communication I / F unit 8 to the worker terminal WT.

[0041] (Operation Example) Next, an operation example of the remote work support system configured as described above will be described. Fig. 4 is a flowchart showing the processing procedure and processing content of the remote work support processing executed by the control unit 1A of the worker terminal WT, and Fig. 5 is a flowchart showing an example of the processing procedure and processing content of the remote work support processing executed by the control unit 6A of the controller terminal MT. Fig. 6 also shows the operation sequence between the worker terminal WT and the controller terminal MT.

[0042] When the control unit 1A of the worker terminal WT detects the start of the remote work support process in step S10, first in step S11, under the control of the video data acquisition processing unit 11, it acquires video data of the work site WS that is output in real time from the camera WCM. Then, the acquired video data is stored in the video data storage unit 41 with information indicating the shooting time added to each image frame, and is displayed on the display device 5 in step S12.

[0043] Furthermore, when the video data is acquired, in step S13, the control unit 1A of the worker terminal WT generates time stamp information ("1001" in the example of FIG. 6) based on the shooting time of each image frame of the video data under the control of the time stamp generation processing unit 12. Note that the information indicating the shooting time may be used as the time stamp information as is.

[0044] Then, in step S14, the control unit 1A of the worker terminal WT, under the control of the video data transmission processing unit 13, assigns the generated timestamp information to each image frame of the video data, and transmits this time-stamped video data VD11 from the communication I / F unit 3 to the controller terminal MT.

[0045] In response to this, when the control unit 6A of the controller terminal MT detects the start of the remote work support process in step S30, under the control of the video data reception processing unit 61, in step S31 it receives the time-stamped video data transmitted from the worker terminal WT via the communication I / F unit 8 and displays the received video data on the display device 9. At the same time, under the control of the timestamp extraction processing unit 62, the control unit 6A of the controller terminal MT extracts timestamp information for each image frame from the received video data in step S32.

[0046] In this state, suppose that the controller MU makes a gesture using his or her finger to indicate a work object while watching the image of the work site displayed on the display device 9. For example, as shown in Fig. 6, suppose that an operation is made to point at a work object, for example, a vehicle, on an image frame VD11 of the work site WS displayed on the display screen of the display device 9.

[0047] In step S33, the control unit 6A of the controller terminal MT detects the above gesture from the video data captured by the camera MCM, and in step S34, under the control of the motion capture processing unit 64, captures the movement of the above gesture in the video data.

[0048] Then, under the control of the finger instruction data generation processing unit 63A, the control unit 6A of the supervisor terminal MT first generates finger instruction data indicating the coordinates of the instruction position in the image frame specified by the finger movement, for example, using hand tracking technology, in step S35. Next, in step S36, the finger instruction data generation processing unit 63A assigns the timestamp information "1001" extracted from the display image VD11 to the finger instruction data generated by the motion capture processing unit 64. Then, the finger instruction data with the timestamp information assigned, i.e., the time-stamped finger instruction data, is transmitted from the communication I / F unit 8 to the worker terminal WT.

[0049] In response to this, when the control unit 1A of the worker terminal WT receives the time-stamped finger instruction data transmitted from the controller terminal MT in step S15 under the control of the finger instruction data receiving processing unit 14, it performs the following processing under the control of the composite video data generation processing unit 15A.

[0050] That is, the composite video data generation processing unit 15A first generates a 2D or 3D image representing, for example, the shape or appearance of fingers in step S16. Next, in step S17, the composite video data generation processing unit 15A reads out from the video data storage unit 41, based on the timestamp information added to the received finger instruction data, an image frame assigned with a shooting time corresponding to the timestamp information. For example, as shown in FIG. 6, the composite video data generation processing unit 15A selectively reads out from the video data storage unit 41 an image frame VD11 whose shooting time corresponds to the timestamp "1001."

[0051] Then, in step S18, the composite video data generation processing unit 15A generates composite video data by superimposing the image data HD representing the shape or appearance of the fingers at the coordinate position specified by the finger instruction data in the read image frame VD11. Then, in step S19, the generated composite video data is output to the display device 5 for display.

[0052] At this time, the display device 5 displays in real time an image of the work area photographed by the camera WCM, and further displays, for example, the above-mentioned composite image superimposed on a predetermined display area on the display screen. That is, the display device 5 simultaneously displays the real-time image and a composite image showing the position indicated by the controller MU. Note that the display device 5 may be configured to display only the above-mentioned composite image data without displaying the real-time image data.

[0053] Thereafter, the control unit 1A of the worker terminal WT and the control unit 6A of the controller terminal MT similarly repeat the above-described series of processes until they detect the end of the remote work support in step S20 and step S37, respectively.

[0054] (Effects) As described above, in the first embodiment, video data to which timestamp information corresponding to the capture time for each image frame is assigned is transmitted from the worker terminal WT to the controller terminal MT, and when the controller MU indicates a work object using a finger gesture in response to this video data, finger instruction data indicating the coordinates of the indicated position is transmitted from the controller terminal MT to the worker terminal WT together with timestamp information extracted from the image frame of the video data. Then, when the worker terminal WT receives the finger instruction data, it reads from the video data storage unit 41 an image frame whose capture time corresponds to the timestamp information sent together with the finger instruction data, and superimposes a finger image representing the shape or appearance of the fingers on the indicated position coordinates specified by the finger instruction data in the read image frame, and displays it on the display device 5.

[0055] Therefore, even if a transmission delay occurs over the network NW in the process of transmitting the video data and finger instruction data between the worker terminal WT and the controller terminal MT, the video data and the finger instruction image data are displayed in a synchronized state on the display device 5 of the worker terminal WT. Therefore, in the image of the work area presented to the worker WU, the controller MU can always accurately indicate the object indicated by his / her finger, thereby preventing erroneous indication of the indicated object.

[0056] In the first embodiment, the real-time video data and the composite video data are displayed in separate display areas or superimposed on each other on the display device 5 of the worker terminal WT. This allows the worker WU to view the state of the work area in real time on the display device 5 of the worker terminal WT (HMD), and to accurately recognize the position indicated by the controller MU's finger using the composite video data.

[0057] In a second embodiment of the present invention, the operator terminal WT detects a movement history of the designated position coordinates for each image frame of the video data based on the finger instruction data sent from the controller terminal MT, and predicts the designated position coordinates for the latest image frame of the video data based on the detected movement history of the designated position coordinates, and superimposes the finger image data on the predicted designated position coordinates in the latest image frame.

[0058] (Configuration Example) (1) Worker Terminal WT Fig. 7 is a block diagram showing the software functions of the worker terminal WT used in the remote operation support system according to the second embodiment of the present invention, together with the hardware configuration. In Fig. 7, the same parts as in Fig. 2 are designated by the same reference numerals, and detailed explanations thereof will be omitted.

[0059] The worker terminal WT in the second embodiment is provided with a new finger instruction data storage unit 42 in the storage area of ​​the data storage unit in addition to the video data storage unit 41. When finger instruction data is sent from the controller terminal MT across multiple consecutive image frames of video data, the finger instruction data storage unit 42 stores the received finger instruction data in the order of reception.

[0060] The control unit 1B of the worker terminal WT includes a video data acquisition processing unit 11, a video data transmission processing unit 13, a finger instruction data reception processing unit 14, a composite video data generation processing unit 15B, and a finger instruction position prediction processing unit 16.

[0061] Of these, the finger instruction position prediction processing unit 16 predicts the instruction position coordinates in the latest image frame based on the movement history of the instruction position coordinates for multiple consecutive past image frames represented by the finger instruction data stored in the finger instruction data storage unit 42, and provides the predicted coordinates to the composite video data generation processing unit 15B.

[0062] The composite video data generation processing unit 15B generates composite video data by superimposing hand and finger image data on the predicted indication position coordinates in the latest image frame of the video data read from the video data storage unit 41, and displays the generated composite video data on the display device 5.

[0063] (2) Controller Terminal MT Figure 8 is a block diagram showing the software functions and hardware configuration of the controller terminal MT used in the remote operation support system according to the second embodiment of the present invention. In Figure 8, the same parts as in Figure 3 are designated by the same reference numerals and detailed explanations will be omitted.

[0064] The control unit 6B of the controller terminal MT includes a video data reception processing unit 61, a finger instruction data generation processing unit 63B, and a motion capture processing unit 64.

[0065] Of these, the finger instruction data generation processing unit 63B transmits the finger instruction data, which is generated by the motion capture processing unit 64 and indicates the coordinates of the instruction position in the image frame, from the communication I / F unit 8 to the worker terminal WT.

[0066] (Operation Example) Next, an operation example of the remote work support system configured as described above will be described. Fig. 9 is a flowchart showing the processing procedure and processing content of the remote work support processing executed by the control unit 1B of the worker terminal WT, and Fig. 10 is a flowchart showing an example of the processing procedure and processing content of the remote work support processing executed by the control unit 6B of the controller terminal MT. Fig. 11 shows the operation sequence between the worker terminal WT and the controller terminal MT.

[0067] 9 and 10, steps having the same processing content as those in FIGS. 4 and 5 are designated by the same reference numerals, and detailed explanations thereof will be omitted.

[0068] The control unit 1B of the worker terminal WT, under the control of the video data acquisition processing unit 11, acquires video data of the work site WS photographed by the camera WCM in step S11, stores the video data in the video data storage unit 41 for each image frame, and displays it on the display device 5 in step S12.

[0069] At the same time, the control unit 1B of the worker terminal WT, under the control of the video data transmission processing unit 13, transmits the video data stored in the video data storage unit 41 from the communication I / F unit 3 to the controller terminal MT in step S21.

[0070] Meanwhile, in step S38, the control unit 6B of the controller terminal MT, under the control of the video data receiving processing unit 61, receives the video data transmitted from the worker terminal WT via the communication I / F unit 8 and displays the received video data on the display device 9.

[0071] In this state, suppose that the controller MU makes a gesture with his or her finger to indicate the work target while watching the image of the work target area displayed on the display device 9. For example, as shown in Fig. 11, suppose that the controller MU points at the work target, for example, a vehicle, in a plurality of consecutive image frames VD(1-Δt) and VD(t) of the video data displayed on the screen of the display device 9.

[0072] When the control unit 6B of the controller terminal MT detects the gesture in step S33 based on the video data output from the camera MCM, it captures the movement of the gesture in the video data under the control of the motion capture processing unit 64 in step S34. Then, under the control of the finger instruction data generation processing unit 63B, the control unit 6B of the controller terminal MT generates finger instruction data indicating the coordinates of the instruction position in the image frames VD(1-Δt) and VD(t) specified by the finger operation using hand tracking technology in step S35. Then, in step S39, it transmits the generated finger instruction data from the communication I / F unit 8 to the worker terminal WT.

[0073] In response to this, when the finger instruction data is sent from the controller terminal MT, the control unit 1B of the worker terminal WT receives this finger instruction data in step S15 under the control of the finger instruction data reception processing unit 14. Then, the received finger instruction data is stored in the finger instruction data storage unit 42 in step S22.

[0074] Next, in step S23, the control unit 1B of the worker terminal WT reads out finger instruction data from the finger instruction data storage unit 42 under the control of the finger instruction position prediction processing unit 16. Then, the finger instruction position prediction processing unit 16 predicts the instruction position coordinates in subsequent image frames based on the movement history of the instruction position coordinates in the multiple image frames represented by the read-out finger instruction data.

[0075] For example, as shown in FIG. 11, based on the coordinates of the designated positions in the image frames VD(1-Δt) and VD(t), first, the movement vector (x ｔ－Δｔ →x ｔ ) is calculated. Then, the calculated movement vector (x ｔ－Δｔ →x ｔ ), the motion vector (x) from the current image frame VD(t) to the next image frame VD(t+Δt) is calculated. ｔ →x ｔ＋Δｔ ) and calculate the calculated movement vector (x ｔ →x ｔ＋Δｔ ) to predict the pointed position coordinate HD(t+Δt) for the next image frame.

[0076] Next, under the control of the composite video data generation processing unit 15B, the control unit 1B of the worker terminal WT first generates 2D or 3D image data representing, for example, the shape or appearance of fingers in step S16. Subsequently, in step S24, the composite video data generation processing unit 15B reads the latest image frame VD(t+Δt) from the video data storage unit 41 and generates composite video data by superimposing the image data HD representing the shape or appearance of the fingers on the predicted pointing position coordinates HD(t+Δt) in the read image frame VD(t+Δt). Then, in step S19, the composite video data generation processing unit 15B outputs the generated composite video data to the display device 5 for display.

[0077] (Effects) As described above, in the second embodiment, the operator terminal WT detects a movement history of the pointing position coordinates based on the finger pointing data sent from the controller terminal MT across multiple consecutive image frames of the video data, and predicts the pointing position coordinates for the latest image frame of the video data based on the detected movement history of the pointing position coordinates, and superimposes the finger image data on the predicted pointing position coordinates in the latest image frame.

[0078] Therefore, even if a transmission delay occurs over the network NW in the process of transmitting the video data and finger instruction data between the worker terminal WT and the controller terminal MT, the video data and the finger instruction image data are displayed in a synchronized state on the display device 5 of the worker terminal WT. Therefore, in the image of the work area presented to the worker WU, the controller MU can always accurately indicate the object indicated by his / her finger, thereby preventing erroneous indication of the indicated object.

[0079] Furthermore, since the image of the fingers indicating the position is always displayed in the latest image frame, the worker WU can perform the work accurately and smoothly by following the instructions given by the fingers while watching the real-time video.

[0080] [Third Embodiment] The third embodiment of the present invention operates as follows. First, when the supervisor MU performs a pointing operation using his / her fingers on an image frame of video data transmitted from the worker terminal WT, the supervisor terminal MT extracts image data of a peripheral area including the position coordinates specified by the pointing operation from the image frame in which the pointing operation was performed, and transmits the extracted image data of the peripheral area (peripheral image data) to the worker terminal WT. In response, the worker terminal WT extracts an image area similar to the peripheral image data from the latest image frame, for example, using image matching technology. The extracted image area is then predicted as the position where the supervisor MU will point using his / her fingers, and an image of the pointing finger is superimposed on the predicted pointing position in the latest image frame.

[0081] (Configuration Example) (1) Worker Terminal WT Fig. 13 is a block diagram showing the software functions of the worker terminal WT used in the remote operation support system according to the third embodiment of the present invention, together with the hardware configuration. In Fig. 13, the same parts as those in Figs. 2 and 9 are designated by the same reference numerals, and detailed explanations thereof will be omitted.

[0082] The control unit 1C of the worker terminal WT includes a video data acquisition processing unit 11, a video data transmission processing unit 13, a surrounding image data reception processing unit 17, a similarity prediction processing unit 18, and a composite video data generation processing unit 15C.

[0083] Of these, the surrounding image data receiving processing unit 17 receives the surrounding image data, i.e., the surrounding image data, from the controller terminal MT via the communication I / F unit 3 when the image data of the surrounding image area including the position specified by the controller MU by operating his / her fingers is transmitted.

[0084] The similarity prediction processing unit 18 performs image matching between the received peripheral image data and the latest image frame stored in the video data storage unit 41, and extracts an image area from the latest image frame that is similar to the peripheral image data. The similarity prediction processing unit 18 then predicts the extracted image area as the position of the target in the latest image frame.

[0085] The composite video data generation processing unit 15C generates composite video data by superimposing image data representing the shape or appearance of the fingers at the target position in the latest image frame read from the video data storage unit 41, and displays the generated composite video data on the display device 5.

[0086] (2) Controller Terminal MT Figure 14 is a block diagram showing the software configuration of the controller terminal MT used in the remote operation support system according to the third embodiment of the present invention, together with the hardware configuration. In Figure 14, the same parts as those in Figures 3 and 10 are designated by the same reference numerals, and detailed explanations thereof will be omitted.

[0087] The control unit 6C of the controller terminal MT includes a video data reception processing unit 61, a finger instruction data generation processing unit 63C, a motion capture processing unit 64, and a surrounding image generation processing unit 65.

[0088] Of these, the peripheral image generation processing unit 6 extracts image data of a peripheral area of ​​a predetermined range including the pointing position coordinates centered on the pointing position coordinates from the image frame being displayed, in accordance with the pointing position coordinates generated by the finger instruction data generation processing unit 63, and transmits the extracted image data from the communication I / F unit 8 to the worker terminal WT.

[0089] (Operation) Next, an example of the operation of the remote operation support system configured as described above will be described. Fig. 15 is a flowchart showing the procedure and processing content of the control processing executed by the control unit 1C of the worker terminal WT, and Fig. 16 is a flowchart showing an example of the procedure and processing content of the control processing executed by the control unit 6C of the administrator terminal MT.

[0090] 15, steps having the same processing content as those in Figures 4 and 9 are designated by the same reference numerals, and detailed descriptions thereof will be omitted. Similarly, in Figure 16, steps having the same processing content as those in Figures 5 and 10 are designated by the same reference numerals, and detailed descriptions thereof will be omitted.

[0091] When the control unit 1C of the worker terminal WT acquires video data of the work site WS captured by the camera WCM, it stores the video data for each image frame in the video data storage unit 41 in step S11, and displays it on the display device 5 in step S12.

[0092] At the same time, the control unit 1C of the worker terminal WT, under the control of the video data transmission processing unit 13, transmits the video data stored in the video data storage unit 41 from the communication I / F unit 3 to the controller terminal MT in step S21.

[0093] Meanwhile, in step S38, the control unit 6C of the controller terminal MT, under the control of the video data receiving processing unit 61, receives the video data transmitted from the worker terminal WT via the communication I / F unit 8 and displays the received video data on the display device 9.

[0094] In this state, suppose that the controller MU makes a gesture using his or her fingers to indicate a work target on the image while viewing the image of the work site WS displayed on the display device 9. Then, the control unit 6C of the controller terminal MT detects the gesture in step S33 based on the image data output from the camera MCM, and captures the movement of the gesture in the image data under the control of the motion capture processing unit 64 in step S34. Then, under the control of the finger instruction data generation processing unit 63C in step S35, the control unit 6C of the controller terminal MT generates finger instruction data indicating the coordinates of the indicated position in the image frame specified by the finger operation using hand tracking technology.

[0095] Next, under the control of the peripheral image generation processing unit 65, the control unit 6C of the controller terminal MT first extracts, in step S39, image data of a peripheral region of a predetermined size that includes the pointing position coordinates as its center from the image frame being displayed, in accordance with the pointing position coordinates of the fingers generated by the finger instruction data generation processing unit 63. For example, as shown in Fig. 17, the peripheral image generation processing unit 65 extracts, from the image frame VD(t), an image region VG1 that is a rectangular shape of a predetermined size and has the pointing position coordinates HD(t) as its center.

[0096] Then, in step S40, the surrounding image generation processing unit 65 transmits the image data of the extracted surrounding area from the communication I / F unit 8 to the worker terminal WT.

[0097] In response to this, when the image data of the surrounding area is sent from the controller terminal MT, the control unit 1C of the worker terminal WT receives the image data of the surrounding area under the control of the surrounding image data receiving processing unit 17 in step S25.

[0098] Subsequently, in step S26, the control unit 1C of the worker terminal WT reads the latest image frame VD(t+Δt) from the video data storage unit 41 under the control of the similarity prediction processing unit 18, as shown in FIG. 17 . The similarity prediction processing unit 18 then extracts an image region VG2 similar to the peripheral image data VG1 from the image frame VD(t+Δt). As an image recognition technique for extracting this similar image region VG2, for example, a method using comparison of feature points, such as general template matching or SURF (Speed-Up Robust Features), can be used.

[0099] Then, in step S28, the similarity prediction processing unit 18 predicts that the extracted image region VG2 is the pointing target position in the latest image frame.

[0100] The control unit 1C of the worker terminal WT then, under the control of the composite video data generation processing unit 15C, first generates a 2D or 3D image representing, for example, the shape or appearance of fingers in step S16. The composite video data generation processing unit 15C then reads the latest image frame VD(t+Δt) from the video data storage unit 41 in step S24 and sets the center coordinates of the extracted image region VG2 in the read image frame VD(t+Δt) to the pointing target position HD(t+Δt). The composite video data generation processing unit 15C then superimposes image data HD representing the shape or appearance of the fingers on the pointing target position HD(t+Δt) to generate composite video data. The generated composite video data is then output to the display device 5 for display in step S19.

[0101] (Effects) As described above, in the third embodiment, when the supervisor MU performs a pointing operation using his / her fingers on an image frame of video data transmitted from the worker terminal WT, the supervisor terminal MT first extracts an image of a peripheral area including the position coordinates specified by the pointing operation from the image frame in which the pointing operation was performed, and transmits peripheral image data indicating the extracted peripheral area to the worker terminal WT. In response, the worker terminal WT uses image recognition technology to extract an image area similar to the peripheral image data from the latest image frame. The extracted image area is then estimated to be the pointing position specified by the supervisor MU using his / her fingers, and an image of the pointing position of the supervisor MU is superimposed on the estimated pointing position in the latest image frame.

[0102] Therefore, as in the second embodiment described above, even if a transmission delay occurs over the network NW in the process of transmitting the video data and finger instruction data between the worker terminal WT and the controller terminal MT, the video data and the finger instruction image data are displayed in a synchronized state on the display device 5 of the worker terminal WT. Therefore, in the image of the work area presented to the worker WU, the controller MU can always accurately indicate the object indicated by his / her finger, thereby preventing erroneous indication of the indicated object.

[0103] Furthermore, since the image of the fingers indicating the position is always displayed in the latest image frame, the worker WU can perform the work accurately and smoothly by following the instructions given by the fingers while watching the real-time video.

[0104] [Other Embodiments] (1) Both the timestamp-based synchronization method of the first embodiment and the synchronization method using the prediction method of the pointing position of the second or third embodiment may be used simultaneously, and the composite image data obtained by each method may be displayed on the worker terminal WT. In this case, a conceivable image display format may be a method in which the composite image generated by the first embodiment is superimposed on a partial area of ​​the composite image generated by the second or third embodiment.

[0105] In this way, it is possible to simultaneously display a composite image that has real-time properties but may be inaccurately synchronized due to the influence of prediction accuracy, and a composite image that does not have real-time properties but is accurately synchronized.

[0106] (2) In the third embodiment, a case where one similar image region VG2 is extracted from the latest image frame VD(t+Δt) has been described. However, depending on the content of the image of the latest image frame, it is also possible that multiple similar image regions VGA, VGB are extracted from the latest image frame VD(t+Δt), as shown in FIG. 18 . In this case, the similarity prediction processing unit 18 predicts the coordinates of the designated position by, for example, performing the following process.

[0107] That is, the similarity prediction processing unit 18 calculates the similarity (Si) for each of the extracted similar images, and further calculates the distance (Di) between the coordinates predicted from the movement speed of the fingers of the controller MU and the center coordinates of each of the multiple similar images.Then, the calculated similarity (Si) and distance (Di) are substituted into the following evaluation formula to calculate the evaluation value (Pi):

[0108]

[0109] The similarity prediction processing unit 18 then specifies the similar image with the highest calculated evaluation value (Pi) as the target, and sets the coordinates of the center position of the specified similar image as the predicted point.

[0110] In the evaluation formula, a is a parameter that determines the priority of the predicted position based on the finger movement speed and the image similarity, and can be set arbitrarily by a system administrator or the like.

[0111] (3) In addition, the types and configurations of the worker terminals and controller terminals, and the processing procedures and processing contents related to the remote work support processing executed by these terminals can be modified in various ways without departing from the spirit of this invention.

[0112] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0113] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0114] WS...Work site MS...Management center WU...Worker MU...Controller WT...Worker terminal MT...Controller terminal NW...Network WCM, MCM...Camera 1A, 1B, 1C...Controller 2, 7...Program storage 3, 8...Communication I / F 41...Video data storage 42...Hand instruction data storage 5, 9...Display device 11...Video data acquisition processor 12...Timestamp generation processor 13...Video data transmission processor 14...Hand instruction data reception processor 15A, 15B, 15C...Synthetic video data generation processor 16...Hand instruction position prediction processor 17...Peripheral image data reception processor 18...Similarity prediction processor 61...Video data reception processor 62...Timestamp extraction processor 63A, 63B, 63C...Hand instruction data generation processor 64...Motion capture processor 65...Peripheral image generation processor

Claims

1. A remote work support system that connects between a controller terminal and a worker terminal via a network and transmits instruction information related to work from the controller terminal to the worker terminal, wherein the worker terminal includes: A first processing unit that stores video data obtained by photographing a work target area in a storage unit in a state where time stamp information indicating the photographing time is attached to each image frame; A second processing unit that transmits the video data with the time stamp information attached thereto to the controller terminal. The controller terminal includes: A third processing unit that receives and displays the video data with the time stamp information attached thereto; A fourth processing unit that generates the instruction information corresponding to the instruction operation when the instruction operation related to the work is performed on the displayed video data; A fifth processing unit that attaches the time stamp information attached to the image frame of the video data on which the instruction operation is performed to the generated instruction information, and transmits the instruction information with the time stamp information attached thereto to the worker terminal. The worker terminal includes: A sixth processing unit that receives the instruction information with the time stamp information attached thereto; A seventh processing unit that selects, from the storage unit, the image frame with the same time stamp information as the time stamp information attached to the received instruction information, and synthesizes and displays the instruction information on the selected image frame. A remote work support system.

2. A remote work support system that connects a controller terminal and an operator terminal via a network and transmits instruction information related to work from the controller terminal to the operator terminal, wherein: The controller terminal includes: A first processing unit that receives and displays video data obtained by photographing a work target area from the operator terminal; A second processing unit that generates the instruction information representing the position specified by the instruction operation for each image frame of the video data when an instruction operation related to the work is performed on the displayed video data; A third processing unit that transmits the generated instruction information to the operator terminal. The operator terminal includes: A fourth processing unit that receives the instruction information transmitted from the controller terminal; A fifth processing unit that predicts the specified position of the instruction information in a third image frame whose time position is behind the second image frame based on the movement history of the instruction information between the first image frame and the subsequent second image frame of the video data; A sixth processing unit that synthesizes and displays the instruction information at the predicted position of the third image frame. A remote work support system comprising the above components.

3. A remote work support system that connects between a controller terminal and a worker terminal via a network and transmits instruction information related to work from the controller terminal to the worker terminal, wherein: The controller terminal includes: A first processing unit that receives and displays video data obtained by photographing a work target area from the worker terminal; A second processing unit that extracts a partial image area related to a position specified by the instruction operation from a first image frame of the video data when an instruction operation related to the work is performed on the displayed video data; A third processing unit that transmits image data corresponding to the extracted image area to the worker terminal. The worker terminal includes: A fourth processing unit that receives the image data transmitted from the controller terminal; A fifth processing unit that detects a similar image area similar to the received image data from a second image frame whose time position is behind the first image frame of the video data, and sets the position of the detected similar image area as a predicted position of the instruction operation in the second image frame; A sixth processing unit that synthesizes and displays the instruction information representing the instruction operation at the predicted position in the second image frame. A remote work support system.

4. A remote work support method executed by a system that connects a controller terminal and a worker terminal via a network and transmits instruction information related to work from the controller terminal to the worker terminal, wherein the worker terminal performs: a process of storing video data obtained by photographing a work target area in a storage unit with time stamp information indicating the photographing time added to each image frame; and a process of transmitting the video data with the time stamp information added thereto to the controller terminal. The controller terminal performs: a process of receiving and displaying the video data with the time stamp information added thereto; a process of generating instruction information corresponding to an instruction operation when the instruction operation related to the work is performed on the displayed video data; a process of adding the time stamp information added to the image frame of the video data on which the instruction operation is performed to the generated instruction information, and transmitting the instruction information with the time stamp information added thereto to the worker terminal. The worker terminal performs: a process of receiving the instruction information with the time stamp information added thereto; and a process of selecting, from the storage unit, the image frame with the same time stamp information as the time stamp information added to the received instruction information, and synthesizing and displaying the instruction information on the selected image frame.

Images