Work assistance system, work assistance device, work assistance method, and program

Abstract

A work assistance system according to an aspect of the present invention is provided with a worker terminal and a work assistance device that can communicate with each other. The worker terminal comprises: a camera that acquires video frames in a time series by imaging an area that includes the direction of a worker's line of sight; and a transmission unit that transmits video data containing the video frames to the work assistance device. The work assistance device comprises: a reception unit that receives the video data and reproduces the video frames; a marker detection unit that detects a marker with a known pattern from the reproduced video frames; a video correction unit that generates corrected video by performing conversion processing on the video frames to revert the pattern of the detected marker back to the original pattern; and a display unit that displays the corrected video.

Description

Work support system, work support device, work support method, and program

[0001] The embodiments relate to a work assistance system, a work assistance device, a work assistance method, and a program.

[0002] A work support system is a system that remotely supports on-site workers to ensure that work is carried out smoothly. In other words, a supervisor who assists the workers communicates with the workers via a network, and by working together, it is possible to expect benefits such as speed, safety, and cost reduction. In recent years, video conferencing, which involves video, has become more common than just phone calls.

[0003] Video footage of the work site is captured by cameras, converted into data, and sent to a control center as a video stream. Because fixed or stationary cameras are inconvenient, small cameras that can be attached to workers' helmets, etc., and smart glasses with cameras are becoming more common.

[0004] "MR technology enables remote support from the scene, as if you were next to someone, even if you are far away," [online], [searched October 31, 2023], Internet <https: / / group.ntt / jp / nttxr / service / service00028 / >

[0005] In actual work, many of the locations that need to be checked are physically fixed. However, the images captured by cameras worn by workers move in accordance with the worker's movements. If the image from the camera is unstable, the location (orientation, size, etc.) of the location that needs to be checked at the control center becomes unstable and not fixed. This can make the controller, who must constantly watch the image, feel ill or unable to maintain focus on the target. In many cases, even turning on the camera's image stabilization function is not enough to compensate. There is a demand for technology that can reduce the burden on controllers without being affected by the worker's movements or blurred line of sight.

[0006] The present invention has been made in light of the above circumstances, and aims to provide a technique that can reduce the burden on controllers.

[0007] A work assistance system according to one aspect of the present invention includes a worker terminal and a work assistance device that can communicate with each other. The worker terminal includes a camera that captures an area including the worker's line of sight to acquire time-series video frames, and a transmitter that transmits video data including the video frames to the work assistance device. The work assistance device includes a receiver that receives the video data and plays the video frames, a marker detector that detects markers of known patterns from the played video frames, an image corrector that performs a conversion process on the video frames to return the detected marker pattern to the original pattern and generate a corrected image, and a display that displays the corrected image.

[0008] According to the embodiment, it is possible to reduce the burden on the controller who supports the worker.

[0009] FIG. 1 is a system diagram showing an example of a work support system according to an embodiment. FIG. 2 is a diagram showing an example of image distortion at a control center. FIG. 3 is a functional block diagram showing an example of a work support system according to an embodiment. FIG. 4 is a flowchart showing an example of a processing procedure related to the initial setting of the worker terminal 20 and the controller terminal 10. FIG. 5 is a diagram for explaining projective transformation parameters. FIG. 6 is a flowchart showing an example of a processing procedure of the controller terminal 10 in an embodiment. FIG. 7 is a diagram for explaining an example of image processing according to an embodiment. FIG. 8 is a diagram for explaining another example of image processing according to an embodiment. FIG. 9 is a diagram for explaining image complementation. FIG. 10 is a block diagram showing an example of the hardware configuration of the controller terminal 10.

[0010] FIG. 1 is a system diagram showing an example of a work support system according to an embodiment. The work support system is a system that forms a communication environment between workers and controllers and supports workers at work sites with video, audio, and the like from a remote control center. Video information acquired by cameras at the work site is sent to the control center via a network communication line. At the control center, images within the camera's field of view are displayed on a computer display or on a virtual reality (VR) device worn by the controller. The controller stationed at the control center gives instructions to the workers by audio, etc., while referring to the displayed images.

[0011] Figure 2 shows an example of video distortion at the control center. Because the camera worn by the worker moves with the worker, the video at the control center may be blurred or it may be difficult to focus on a specific area. In severe cases, the controller may experience motion sickness, so some countermeasure is required. Below, we will disclose technology that can solve this type of problem.

[0012] 3 is a functional block diagram showing an example of a work support system according to an embodiment. A worker terminal 20 worn by a worker and a controller terminal 10 provided in a control center are connected to each other so as to be able to communicate with each other via a network 100, such as an IP (Internet Protocol) network.

[0013] The worker terminal 20 is a computer, and includes a camera 22 with a lens 21, a communication unit 23, and a fixed display range setting unit 24. The camera 22 is worn on the worker's head, for example, and captures an image of an area including the worker's line of sight to acquire a time series of video frames.

[0014] In the embodiment, the marker 200 is placed or affixed around the work location. Preferably, the marker 200 is placed near the item that is the target of the worker's work, in a position that is within the field of view of the camera 22 together with the item. For example, an infrared marker, an AR (Augmented Reality) marker, or the like can be used as the marker 200, but this is not limitative. The marker 200 has a pre-set known design or pattern, such as a two-dimensional barcode. The same pattern may be set for different work objects, or a different pattern may be set for each work object, allowing the work objects to be identified.

[0015] The communication unit 23 communicates with the controller terminal 10 via the network 100. The communication unit 23 also functions as a transmission unit, and performs data processing such as compression, encoding, and packetization on the time-series video frames acquired by the camera 22 to generate video data, and transmits this video data to the controller terminal 10 via the network 100.

[0016] The fixed display range setting unit 24 executes processing related to the initial setting of the worker terminal 20 and the controller terminal 10. That is, the fixed display range setting unit 24 transmits image data including a reference still image captured by the camera 22 to the controller terminal 10. The procedure related to the initial setting will be described in detail later with reference to FIGS. 4 and 5.

[0017] The controller terminal 10 serving as a work support device is a computer and includes a communication unit 70, a processor 30, a display unit 51, and a storage unit 80. Of these, the communication unit 70 communicates with the worker terminal 20 via the network 100. The communication unit 70 also functions as a receiving unit, receiving video data transmitted from the worker terminal 20 and reproducing video frames.

[0018] The storage unit 80 includes a fixed display range recording unit 80a, which is an area for recording the fixed display range created during initial setup. The storage unit 80 also stores a program 80b for causing the computer to function as the controller terminal 10.

[0019] The processor 30 includes, as processing functions according to the embodiment, a marker detection unit 31, an image correction unit 32, and an image completion unit 33. These processing functions are realized by the processor 30 executing a sequence of instructions included in the program 80b.

[0020] The marker detection unit 31 detects the markers 200 from the video frames played back by the communication unit 70 using a known image processing method or an image determination technique using AI (Artificial Intelligence). The markers 200 detected here are often deformed compared to the original.

[0021] The image correction unit 32 performs a transformation process on the image frame to generate a corrected image in order to restore the pattern of the detected marker 200 to its original pattern. A typical transformation process is a projective transformation (homography transformation), but this is not limiting, and an affine transformation or the like may also be used.

[0022] The image complementing unit 33 complements areas not captured by the camera 22, i.e., areas outside the field of view of the camera 22, in the corrected image generated by the image correcting unit 32. For example, past image data is referenced during the complementing. Image complementing will be described in detail later with reference to FIG. 9.

[0023] The display unit 51 is a computer display that provides the controller with a GUI (Graphical User Interface) environment. The display unit 51 also visually displays the corrected image generated by the image corrector 32 and the interpolated corrected image. Next, the operation of the above configuration will be described.

[0024] 4 is a flowchart showing an example of a processing procedure related to the initial setting of the operator terminal 20 and the controller terminal 10. Setting of a fixed display range is started in response to an instruction from the operator terminal 10 or a notification from the operator terminal 20 (step S11). The controller terminal 10 then waits for the arrival of a still image frame to be used for the initial setting from the operator terminal 20 (step S12). When an image frame arrives, the controller terminal 10 detects the position of the marker 200 included in this image frame (step S13).

[0025] If the marker 200 cannot be detected (NO in step S14), an error message is displayed on the display unit 51 of the controller terminal 10, and the initial setting must be redone. On the other hand, if the marker 200 can be detected (YES in step S14), the controller terminal 10 calculates the projection transformation parameters of the marker and stores them in the storage unit 80 together with the positional relationship of the fixed display range (step S15).

[0026] 5 is a diagram illustrating the projective transformation parameters. An image frame captured by the camera 22 includes a work object (e.g., a distribution board) and a marker attached to the back cover of the distribution board. Depending on the orientation of the camera 22, a distorted image may be captured, and therefore the marker may also be distorted.

[0027] The controller at the control center who receives this operates the controller terminal 10 to set the coordinates (X １ , Y １ ), (X ２ , Y ２ ), (X ３ , Y ３ ), (X ４ , Y ４ ) is specified by clicking the mouse or the like. Here, the fixed display range is the area that is desired to be displayed in a fixed state on the display unit 51. This area also includes the marker 200, and the coordinates of its four corners can be calculated by the processor 30 (Fig. 3). These coordinates are (x １ , y １ ), (x ２ , y ２ ), (x ３ , y ３ ) (x ４ , y ４ ) Furthermore, the coordinates of the four corners of the marker 200 in the original image frame can also be calculated by the processor 30 (FIG. 3), and are respectively (x Ａ１ , y Ａ１ ), (x Ａ２ , y Ａ２ ), (x Ａ３ , y Ａ３ ) (x Ａ４ , y Ａ４ )

[0028] Now, obtaining the projection transformation parameters is done by calculating the two-dimensional coordinates (x Ａ１ , y Ａ１ ), (x Ａ２ , y Ａ２ ), (x Ａ３ , y Ａ３ ), (x Ａ４ , y Ａ４ ) into the two-dimensional coordinates (x １ , y １ ), (x ２ , y ２ ), (x ３ , y ３ ), (x ４ , y ４ ) This calculation can be performed by the processor 30. That is, if the coordinate vector of the original image frame is x' and the coordinate vector of the initial setting image is x, the matrix H is calculated so that x = Hx'. In other words, the controller terminal 10 sets parameters for the conversion process that returns the pattern of the detected marker to the original pattern.

[0029] After that, by performing transformation using the matrix H on each image frame of the video frame received from the operator terminal 20, the image can be displayed in a fixed state on the display unit 51 of the operator terminal 10. １ , Y １ ), (X ２ , Y ２ ), (X ３ , Y ３ ), (X ４ , Y ４ ) is enough to display only the image inside, so these four points (X １ , Y １ ), (X ２ , Y ２ ), (X ３ , Y ３ ), (X ４ , Y ４ ) transformed points (pixels) outside the

[0030] 6 is a flowchart showing an example of a processing procedure of the controller terminal 10 in an embodiment. In FIG. 6, the controller terminal 10 acquires a video frame from the worker terminal 20 (step S1). If a fixed display range has been set (YES in step S2), the processing procedure in FIG. 5 is completed, and the controller terminal 10 detects the position of the marker in the acquired video frame (step S3).

[0031] If the marker is present in the video frame (YES in step S4), the controller terminal 10 calculates projective transformation parameters to match the marker in the video frame with the initial marker (step S5), and performs projective transformation on the video frame using the parameters (step S6).Furthermore, the controller terminal 10 updates the image data of the fixed display range using the transformed video frame and generates a corrected image (step S7).

[0032] 7 is a diagram illustrating an example of image processing according to the embodiment. By generating a corrected image through image conversion, an image with a fixed viewpoint is always displayed on the display unit 51 of the controller terminal 10, regardless of how the worker's line of sight changes.

[0033] In the above process, the image delivered from the worker terminal 20 is converted in real time so that it matches the position of the markers in the reference image, and even if the worker's viewpoint changes, the viewpoint of the relevant location is maintained as if it were taken with a fixed camera. This makes the work much easier for the supervisor.

[0034] Returning to FIG. 6 , the explanation continues. If no marker is present in the video frame in step S4 (NO), the controller terminal 10 cannot update the image data in the fixed display range and continues to display the most recent video frame at that time. In other words, the display on the display unit 51 is frozen (step S8). From this point on, a period in which no marker can be detected in the video frame (non-detection period) begins. As the non-detection period elapses, the controller terminal 10 changes the display mode of the corrected image. For example, the processor 30 (video correction unit 32) changes the saturation of the corrected image as the non-detection period elapses (step S8).

[0035] If the video frame contains both updated and non-updated parts, the controller terminal 10 displays the boundary between the updated and non-updated parts of the image data in the fixed display range in the corrected image (step S9).The corrected image is then displayed on the display unit 51 (step S10).

[0036] FIG. 8 is a diagram illustrating another example of image processing according to the embodiment. Suppose the worker's attention is directed to, for example, a tool box, and his / her line of sight moves away from the marker. This causes the entire fixed display range to be captured, and the marker also moves out of the capture range. In such a case, the corrected image is complemented using the most recent image information and displayed on the display unit 51. This allows the supervisor to confirm the viewpoint to which the worker should focus even if the worker has a different viewpoint.

[0037] Furthermore, as time passes, the saturation of areas that are not currently being filmed is reduced, allowing the controller to recognize at a glance that the latest work status is not being reflected in the video.

[0038] FIG. 9 is a diagram illustrating image complementation. While FIG. 8 shows an example of changing the saturation of all parts of the image, FIG. 9 shows an example of changing the saturation of only a portion of the image. For example, assume that a marker and only a portion of the work object are captured in the field of view of the camera 22. The presence of the marker allows the controller terminal 10 to determine the position of the work object, so the controller unfolds the image frame to generate a corrected image. If only part of the fixed display range is captured, the boundary line is displayed. Then, the saturation of only the unupdated portion is gradually reduced. This allows the controller to clearly distinguish between the portion currently being captured and the complemented portion, making it easy to understand the portion currently being captured.

[0039] As described above, in this embodiment, a marker 200 with a predetermined pattern is placed near a work location. During initial setup, an image frame of a fixed display area including the marker 200 is transmitted to the controller terminal 10. The controller terminal 10 calculates in advance projective transformation parameters for returning the deformed marker 200 in the image frame to its original pattern, and stores the calculated parameters along with the positional relationship of the fixed display range. The controller terminal 10 then detects the marker 200 from the video frame from the worker terminal 20, and performs projective transformation on the video frame using the projective transformation parameters and the positional relationship of the marker 200 to match the marker at the time of initial setup.

[0040] By converting the received video frames into images, a fixed area can always be displayed on the controller terminal 10, regardless of blurring or movement of the field of view of the camera 22. This makes it easier for the controller to focus on a fixed area, eliminating concerns about visually-induced motion sickness and reducing the burden on the controller. As a result, according to the embodiment, it is possible to reduce the burden on the controller who remotely supports the worker.

[0041] The functions of the controller terminal 10 can be realized by installing a program on a computer. For example, a program provided as package software or online software can be executed on a computer to implement the functions of the controller terminal 10. The computer is not limited to a so-called server-type computer, but may also be a desktop or laptop personal computer. Portable terminals such as smartphones and tablets are also included in the category of computers. Furthermore, computer resources virtualized on the cloud can also function as the controller terminal 10.

[0042] Fig. 10 is a block diagram showing an example of the hardware configuration of the controller terminal 10. As shown in Fig. 10, the controller terminal 10 includes a CPU (Central Processing Unit) 30A, a bridge circuit 102, a memory 60, a GPU (Graphics Processing Unit) 30B connected to a display unit 51, a communication unit 70, a storage 85, a USB connector 90, an input unit 40, and an output unit 50.

[0043] The storage 85 is a non-volatile storage medium (block device), such as a hard disk drive (HDD) or a solid state drive (SSD), and stores basic programs such as an operating system (OS) 62 and device drivers, as well as a program 61 for implementing the functions of the controller terminal 10 as a work support device.

[0044] The memory 60 includes a ROM (Read Only Memory) and a RAM (Random Access Memory). The CPU 30A and GPU 30B are arithmetic elements related to the processor 30. The CPU 30A mainly controls the controller terminal 10. The GPU 30B mainly performs calculations related to image processing (such as product-sum calculations) at high speed. The CPU 30A loads the program 61 from the storage 85 into the memory 60 and executes it. The same is true for the GPU 30B.

[0045] The bridge circuit 102 relays data transmission between the CPU 30A and the GPU 30B and each unit. The bridge circuit 102 is located between the CPU 30A and the GPU 30B and an input unit 40 and an output unit 50 connected to a PCI (Peripheral Component Interconnect) bus or a hardware device connected to a PCIe bus (not shown), and relays communication between the CPU 30A and the GPU 30B.

[0046] The communication unit 70 communicates with the operator terminal 20 via the network 100. The USB connector 90 connects a USB device or the like. For example, the program 61 may be installed in the controller terminal 10 via a USB device. The input unit 40 connects the controller terminal 10 to a communication network such as a LAN (Local Area Network) or a WAN (Wide Area Network).

[0047] The program 61 and various data may be stored in a removable storage medium other than the storage 85, and may be read by the CPU 30A from a disk drive or the like. Alternatively, the program 61 and various data may be stored in another computer connected via a communication network, and may be read by the CPU 30A via the input unit 40.

[0048] Furthermore, the present invention is not limited to the above-described embodiment. For example, the presence or absence of the marker 200 in the video frame captured by the camera 22 may be determined on the operator terminal 20 (edge) side, and video frames that do not include the marker 200 may not be sent. This can reduce traffic on the network 100 and lighten the load on the controller terminal 10.

[0049] In short, this invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0050] 10...Controller terminal 20...Worker terminal 21...Lens 22...Camera 23...Communication unit 24...Fixed display range setting unit 30...Processor 30A...CPU 31...Marker detection unit 32...Image correction unit 33...Image completion unit 40...Input unit 50...Output unit 51...Display unit 60...Memory 61...Program 70...Communication unit 80...Storage unit 80a...Fixed display range recording unit 80b...Program 85...Storage 90...USB connector 100...Network 102...Bridge circuit 200...Marker.

Claims

1. A work support system comprising an operator terminal and a work support device capable of communicating with the operator terminal, wherein the operator terminal includes a camera that captures an area including the operator's line of sight direction to obtain time-series video frames, and a transmitter that transmits video data including the video frames to the work support device, and the work support device includes a receiver that receives the video data and plays back the video frames, a marker detector that detects a marker of a known pattern from the played-back video frames, a video corrector that performs conversion processing on the video frames to return the detected marker pattern to its original pattern and generates a corrected video, and a display unit that displays the corrected video.

2. The work support system according to claim 1, wherein the operator terminal includes an initial setting unit that transmits image data including a reference still image captured by the camera to the work support device, the receiver receives the image data and plays back the reference still image, and the video corrector sets parameters for the conversion processing based on the marker pattern included in the played-back reference still image.

3. The work support system according to claim 1, wherein the work support device further includes a video complementer that complements an area outside the field of view of the camera in the corrected video.

4. The work support system according to claim 1, wherein when a non-detection period in which the marker cannot be detected by the marker detector starts, the video corrector freezes the corrected video and changes the display mode of the corrected video as the non-detection period elapses.

5. The work support system according to claim 4, wherein the video corrector changes the saturation of the corrected video as the non-detection period elapses.

6. A work support device including a receiver that receives video data including time-series video frames obtained by capturing an area including the operator's line of sight direction and plays back the video frames, a marker detector that detects a marker of a known pattern from the played-back video frames, a video corrector that performs conversion processing on the video frames to return the detected marker pattern to its original pattern and generates a corrected video, and a display unit that displays the corrected video.

7. A work support method for a work support system including an operator terminal and a work support device capable of communicating with the operator terminal, the method including: a process in which the operator terminal transmits image data including a reference still image captured by a camera that captures an area including the line-of-sight direction of the operator to the work support device; a process in which the work support device receives the image data and reproduces the reference still image; a process in which the work support device detects a marker of a known pattern from the reproduced reference still image; and a process in which the work support device sets parameters for a conversion process that returns the pattern of the detected marker to its original pattern.

8. A program including instructions for causing a computer to function as the work support device according to claim 6.

Images