Remote support system

Abstract

To improve work quality by preventing errors in work and leakage, and to execute a further active training by visualizing unsafe elements in a work field and avoiding danger.SOLUTION: A remote support system includes: a wearable terminal 102 having an image pickup part for photographing space video in a direction that a worker's face looks, a sound collection part for collecting at least a worker's voice, a projection part for projecting viewable video to a worker, and a voice output part for outputting voice and a radio communication part for performing radio communication; a 360-degree camera 104 for photographing space video around a worker; and a managing device 108 for outputting video information or voice information to the wearable terminal while reproducing space video or voice that the wearable terminal acquires and space video that the 360-degree camera acquires. The managing device outputs notification information corresponding to space video that the 360-degree camera acquires to the wearable terminal, and the wearable terminal reproduces notification information by the projection part or the voice output part.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a remote support system for assisting workers working at a remote location.

Background Art

[0002] Conventionally, a remote support system using a wearable terminal has been known. For example, in the remote work support system disclosed in Patent Document 1, a control means receives detection signals from a state detection device and a suspended load position detection device to detect a work start state. Further, the control means acquires imaging information (for example, image information, moving image information, position information, etc.) from an imaging means, for example, a wearable terminal of a worker at the site and / or a mounting camera provided at the site. Next, the control means determines whether or not the working posture of the crane has been changed based on the detection signals from the state detection device and the suspended load position detection device. When the working posture of the crane has been changed, imaging information to be displayed on the display means in the operator's cab is extracted based on the working posture and the information of the imaging means (wearable terminal, mounting camera, etc.). At this time, when the operator in the operator's cab is wearing a wearable terminal, the imaging information to be displayed on the wearable terminal may also be extracted. Then, the extracted imaging information is transmitted to the display means (wearable terminal) and the imaging information is displayed to end the remote work support.

[0003] Further, in the remote support system disclosed in Patent Document 2, it includes a facility device, a portable terminal that transmits and / or receives information regarding the facility device, and a response means for implementing a service for the facility device. The response means remotely supports the implementation of the service via the portable terminal. More specifically, an appropriate response work instruction is given to a worker who responds to an abnormality or the like from a terminal such as a center by a wearable terminal. That is, when there is a notification of an abnormality or the like at the center, the instruction terminal that gives an instruction to the site performs an access detection process of the instruction terminal. When there is an access, image information of a target system, a target line, etc. of an abnormality or the like at the site is received from the wearable terminal of the on-site worker. Also, a situation report of the site may be received by conversation or the like together with the image information.

[0004] In addition, in the remote operation system for material handling and transporting machinery disclosed in Patent Document 3, first, an operator puts on the wearable glasses and starts up the wearable glasses and the large screen monitor. Next, after starting up the wearable glasses and the large screen monitor, the operation power of the continuous unloader as the material handling and transporting machinery is turned on. After turning on the operation power, a determination is made as to whether the material handling and transporting machinery is ready to operate. If it is determined that the material handling and transporting machinery is not ready to operate, the wearable glasses enter an operation start mode, and the operation preparation procedure for starting operation is displayed on the wearable glasses, and audio guidance is played. The operator prepares the material handling and transporting machinery to operate according to the display on the wearable glasses and the audio guidance.

[0005] After completing the preparations for operation, the operator begins to operate the loading and unloading machine by following the voice guidance of the wearable glasses. Since the operator's method of operation differs depending on the state of the loading and unloading machine (normal or abnormal), the state of the loading and unloading machine is judged following the start of operation. If it is determined that the state of the loading and unloading machine is a fault or abnormality such as a motor overload, the wearable glasses enter an abnormality mode, and the fault state (major or minor fault), fault details (error number, error item), and fault recovery procedures are displayed, along with voice guidance. The operator operates the machine by following the abnormality display and voice guidance to clear the abnormality.

[0006] On the other hand, if the condition of the loading and unloading machine is judged to be normal, the wearable glasses will enter normal operation mode, and the operator will select the camera image display function of the wearable glasses as necessary. If the operator does not select the camera image display function of the wearable glasses, the operation status of the loading and unloading machine while stopped / operating, as well as the distance and status quantity will be displayed on the wearable glasses. The operator will remotely operate the loading and unloading machine while viewing the display on the wearable glasses and the images captured by multiple surveillance cameras installed on the loading and unloading machine on a large-screen monitor. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Specification of Utility Model Registration No. 3226637 [Patent Document 2] Japanese Unexamined Patent Application Publication No. 2018-185570 [Patent Document 3] Japanese Unexamined Patent Application Publication No. 2016-199388 [Summary of the Invention] [Problems to be Solved by the Invention]

[0008] In Japan's production systems, "cell production" where workers assemble products individually to create finished products, and "line production" where flow operations are carried out accurately within a specified time, are the mainstream. The factory equipment is subject to preventive maintenance by maintenance workers who require high skills, and is always maintained to ensure high-precision production. In other words, it can be said that the production site is based on the presence of highly skilled workers. However, due to factors such as factory relocations, the departure of skilled workers, and the decline in the working population, it may become difficult to expect high levels of achievement in manual work.

[0009] Moreover, as an active learning method for workers, even if we plan exercises using actual equipment, we cannot use the equipment during production operations for training purposes, and it is necessary to purchase new training equipment. It is not realistic to obtain all the necessary equipment, and such training is likely to be limited. As a result, the penetration rate and effectiveness of training are decreasing. There is a need for an effective training method. In addition, active learning during work has been proven to be very effective, and it is desirable for new workers to receive training while contributing to the productivity of the entire plant. However, in order to learn the factory's working methods, passive learning centered on classroom learning is often observed.

[0010] In addition, in the inspection of manufactured products, except for quantitative inspections using test tools and the like, inspections based on the subjective judgment of inspectors are central. Also, in some cases, omissions in inspection procedures and inspection errors occur, and there are also risks to ensuring manufacturing quality. Further, due to the need to confirm the manufacturing history of products returned as non-conforming products and the fact that the management of members and equipment is individual, a lot of time is spent on confirming scattered data for each, falling into a difficult situation.

[0011] In such manufacturing sites, in order to eliminate variations in work quality, various trainings and handovers such as OJT, work procedure manuals, and manualization are carried out. However, if the departure of skilled workers and the decline in the working population progress, it is expected that it will become impossible to pass on from person to person. Therefore, a mechanism that can be used by anyone, such as new workers and inexperienced workers, for the actions and ways of thinking of skilled workers is required. Also, it is required to be usable not only in training but also in actual work.

[0012] Also, not only the work content of skilled workers but also data representing the state of equipment, etc., are collected by IoT, and by "visualizing" a huge amount of data, issues in work and manufacturing are extracted, and it is required to be used for improving quality and productivity. Further, it is required to avoid the risk of work mistakes and inefficiencies with support from skilled workers or experts from a remote location.

[0013] Furthermore, by analyzing a huge amount of data, simulate what kind of effects will occur when errors that were not known until now or risks of inefficiencies are foreseen or when the risks become apparent, and further, feedback it to workers, supervisors, and equipment maintenance, etc., the realization of an optimal manufacturing site is required.

[0014] In a conventionally known remote support system, such problems have not been sufficiently solved. In particular, when displaying an image acquired from a wearable terminal on a management device or when transmitting an image from the management device to the wearable terminal, there were problems such as low quality of the displayed image, a problem with the transmission capacity, and unstable communication. In such a situation, it was not sufficient to avoid "misdiagnosis" in telemedicine or "accidents" at construction sites.

[0015] The present invention has been made in view of such circumstances, and by sharing the work of an operator with an expert, it prevents work errors and omissions, improves work quality, visualizes unsafe elements at the work site to avoid danger, and further provides a remote support system capable of implementing effective training.

Means for Solving the Problems

[0016] (1) To achieve the above object, the present invention takes the following means. That is, the remote support system of the present invention is a remote support system that supports an operator working at a remote location, and includes an imaging unit that captures a spatial video in the direction in which the operator's face is facing, a sound collection unit that collects at least the operator's voice, a projection unit that projects a video visible to the operator, a voice output unit that outputs voice, and a wearable terminal having a wireless communication unit that performs wireless communication, a 360-degree camera that captures a spatial video around the operator, and a management device that reproduces the spatial video or voice acquired by the wearable terminal and the spatial video acquired by the 360-degree camera, while outputting video information or voice information to the wearable terminal. The management device outputs notification information corresponding to the spatial video acquired by the 360-degree camera to the wearable terminal, and the wearable terminal reproduces the notification information by at least one of the projection unit or the voice output unit.

[0017] (2) Further, the remote support system of the present invention compares the spatial video acquired by the 360-degree camera with a plurality of videos accumulated by deep learning, and when there is a video corresponding to the spatial video, it outputs notification information corresponding to the video to at least one of the wearable terminal or the management device. It is characterized by further comprising an AI unit.

[0018] (3) Further, in the remote support system of the present invention, the wearable terminal acquires video content from the management device and reproduces the video content with the projection unit and the audio output unit.

[0019] (4) Further, the remote support system of the present invention is characterized by further comprising a relay device that relays communication between the wearable terminal, the 360-degree camera, and the management device.

[0020] (5) Further, in the remote support system of the present invention, the wearable terminal acquires work evaluation information from the monitoring device and reproduces the work evaluation information with the projection unit.

Advantages of the Invention

[0021] According to the present invention, it is possible to prevent work errors and omissions, improve work quality, visualize unsafe elements at the work site to avoid danger, and further implement effective training.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 7A

Figure 7B

Embodiments for Carrying Out the Invention

[0023] The GDP (Gross Domestic Product) related to the manufacturing industry in Japan has been on a shrinking trend, decreasing by about 20% from 114 trillion yen in 1997, which was the peak, to around 90 trillion yen in 2017. On the other hand, at the manufacturing site, since the Industrial Revolution, efforts have been made in technological innovation for the advancement, high efficiency, high quality, and low cost in production. In recent years, with the advent of digital transformation (DX), there has been a dramatic increase in the amount of data to be handled, an explosive improvement in hardware performance, and furthermore, the emergence of innovative technologies such as AI (artificial intelligence) and deep learning has made things that were previously considered impossible possible, and the momentum for the transformation of the industrial structure has been growing. The movement to acquire all data from the production site and highly utilize it to transform manufacturing not only addresses the issues of the manufacturing industry so far but is also expected to create new added value such as bringing long-term benefits. The present invention incorporates innovative technologies and is a tool that connects people not only to products but also to machines, systems, and even society.

[0024] [Features of the Present Invention] (1) Features in Remote Monitoring The video taken by the hands-free wearable device (AR glasses) worn by the operator is shared with the supervisor at a remote location. As a result, it becomes possible to prevent mistakes and omissions in work. By converting the work procedures of skilled workers into content and projecting it onto the AR glasses, the work procedures are provided to the operator. As a result, it becomes possible to prevent work errors in advance, and even new or inexperienced workers can perform high-quality work.

[0025] (2) Features in ensuring safety and work quality The video of the space around the operator is shared with the supervisor by the "360-degree camera" installed at the site. This makes it possible to visualize and confirm the unsafe factors of the operator and the work site. In addition, it becomes possible to send instructions from the supervisor to the on-site worker or to communicate by voice, video, and text when there is no contact.

[0026] (3) Features in providing added value to on-site work The video information from the AR glasses, the work status of the operator, and the surrounding environmental information are analyzed by AI. As a result, it becomes possible to utilize the analysis results for accident prevention training and risk prediction before work. Furthermore, it becomes possible to simulate the impact when risks such as accidents occur and to cooperate with related systems.

[0027] [Configuration of the present invention] FIG. 1 is a diagram showing a schematic configuration of a remote support system according to an embodiment of the present invention. This remote support system 100 includes an AR glass 102a as a wearable terminal worn by a field worker A1, a 360-degree camera 104a that captures a spatial video around the field worker A1, a relay device 106a that relays communication between the AR glass 102a and the 360-degree camera 104a, and a management device 108 described later. The management device 108 plays the spatial video or audio acquired by the AR glass 102a and the spatial video acquired by the 360-degree camera 104a, and outputs video information or audio information to the AR glass 102a upon an operation by the administrator B. Further, the remote support system 100 includes an AI (Artificial Intelligence) unit 110 that compares the spatial video acquired by the 360-degree camera 104a with a plurality of videos accumulated by deep learning, and if there is a video corresponding to the spatial video, outputs notification information corresponding to the video to at least one of the AR glass 102a or the management device 108. This remote support system 100 can remotely support the work of a plurality of field workers, and can switch and display the images of each field worker on the management device 108. That is, the remote support system 100 includes an AR glass 102b as a wearable terminal worn by a field worker A2, a 360-degree camera 104b that captures a spatial video around the field worker A2, a relay device 106b that relays communication between the AR glass 102b and the 360-degree camera 104b, and a management device 108 described later. The management device 108 plays the spatial video or audio acquired by the AR glass 102b and the spatial video acquired by the 360-degree camera 104b, and outputs video information or audio information to the AR glass 102b upon an operation by the administrator B. Further, the AI unit 110 compares the spatial video acquired by the 360-degree camera 104b with a plurality of videos accumulated by deep learning, and if there is a video corresponding to the spatial video, outputs notification information corresponding to the video to at least one of the AR glass 102b or the management device 108. With this configuration, when managing and checking a plurality of bases from the management device 108, it is possible to switch the screen with a button or the like from the management device 108 (one PC at the headquarters) and communicate with each base.

[0028] FIG. 2 is a diagram showing the hardware configuration of the management device 108. A memory 204, an OS / application 206, a communication interface 208, a display 210, a disk drive 212, a keyboard / mouse 214, and a microphone / speaker 216 are connected to the CPU 202. The communication interface 208 communicates with the AR glasses 102 and the 360-degree camera 104 via the Internet and the relay device 106. Also, the relay device 106 has a similar configuration, but the relay device 106 may not be provided with the disk drive 212. The relay device 106 can be configured by a smartphone, a tablet, a mobile computer, or the like.

[0029] FIG. 3 is a diagram showing the schematic configuration of the AR glasses according to the present embodiment. In this AR glasses 102, a main body portion 102a is coupled to a frame 102b for wearing like glasses. A camera 412 for photographing a spatial image in the direction in which the operator's face is facing is provided in the main body portion 102a. This camera 412 can be configured to be capable of photographing, for example, a still image of 12.8 megapixels and a 4k30 video. Further, it is desirable to have an autofocus function (PDAF). Furthermore, an LED flash 413 is provided. Also, an optical image stabilization function or a barcode scan function may be provided. Also, an operation button 414 is provided on the side surface of the main body portion 102a, and the operator can operate various functions. Also, a display 410 is provided in the main body portion 102a, and the image projected by the operator's right eye can be visually recognized. Note that this display 410 can also be used for the left eye. In the present embodiment, an example using the frame 102b is shown, but the present invention is not limited to this, and the wearing function can be extended or changed according to the working environment, such as a helmet and a headband mount.

[0030] FIG. 4 is a diagram showing the hardware configuration of the AR glasses 102. The CPU 402 is provided with a memory 404, an OS / application 406, a communication interface 408, a display 410, a camera 412, operation buttons 414, and a microphone / speaker 416. The AR glasses 102 support a plurality of languages by voice control. In addition to the operation buttons 414, a two-axis touch pad that supports multi-fingers may be provided. Further, it can be configured to be equipped with a three-axis gyro, a three-axis accelerometer, and a three-axis magnet / integrated compass so that three-degree-of-freedom head tracking is possible. Regarding audio, in addition to the built-in speaker, it is also possible to wear earphones, and it may be provided with a triple noise canceling microphone.

[0031] [Operation of the present invention] Next, the operation of the remote support system according to this embodiment configured as described above will be explained. FIG. 5 is a flowchart showing an overview of the basic operation. In FIG. 5, when the system is started (step S500), it connects to the backend server (step S502). It accesses the user profile from the backend server via the notification system (step S504) (step S506). This remote support system can support simultaneous connections with multiple users and can switch the display. Here, it will be described assuming that a specific one of the multiple workers is selected. A task is selected between the AR glasses worn by the worker and the management device, and data is acquired by the task manager (step S508). The acquired data is provided to the backend server. Next, when a meeting is selected from the user profile (step S510), a session is started. As a result, two-way communication by video and audio becomes possible between the AR glasses and the management device. Also, when transmitting and receiving live video, video streaming is started (step S512). Note that the management device of the supervisor can also access the user profile (step S514). Also, the backend server can access the AI server (step S516), access the storage (step S518), access the database (step S520), and access the streaming server (step S522).

[0032] FIGS. 6A to 6C are flowcharts showing detailed operations. When the worker wears the AR glasses (step S600) and starts the dedicated application in the relay device (step S602), it connects to the backend server (step S604). When connected to the backend server, it becomes possible to access the database (step S606), and it also becomes possible to access the storage (step S608), and it becomes possible to connect to the AI server (step S610). Next, it is determined whether there is a task (step S612), and if there is no task, this determination is repeated.

[0033] In step S612, if there is a task, obtain the task (step S614), and display the task information (step S616). Next, select one or more tasks (step S618), and initialize the 360-degree camera (step S620). Then, initialize the video recording mode (step S622), and return to the backend server. Also, if a QR code (registered trademark) is scanned due to the task selection in step S618, scan the QR code (registered trademark), check the information in the database, and return to the backend server. Further, if a task is selected in step S618 and the task is completed (step S626), update the status of the task and return to the backend server. Note that it is also possible to adopt a configuration to read the QR code during the task selection in step S618 (step S624).

[0034] In step S604, the AR glasses determine whether to hold a meeting shown in FIG. 6B with the management device while connected to the backend server (step S628). When making a call from the AR glasses, check the contact list (step S630), select a contact (step S632), and make a call request (step S634). On the other hand, in step S628, if there is an incoming call from the management device, determine whether to agree to the push notification (step S636). If not, close the session and end (step S644). On the other hand, in step S636, if agreeing to the push notification, initialize the stream on the host (step S638), and stream and play the live video from the AR glasses (step S640). Here, share the file to be streamed between the AR glasses and the management device (step S642), and return to the backend server. Also, in step S640, if the meeting ends, close the session and end (step S644).

[0035] FIG. 7A is a diagram showing a state where an image captured by a camera of AR glasses is displayed on a display of a management device during a meeting. On a screen 700 of the display, a spatial image in the direction in which the face of the operator is facing is displayed, and at the same time, an image of the administrator is displayed on a sub-screen 702. The administrator can also switch the operator displayed on the display. When the administrator gives an instruction to the operator, the instruction can be indicated by inputting on the screen 700. In addition, the administrator can also give an instruction to the operator by voice. FIG. 7B is a diagram showing an example of image display on a display of AR glasses during a meeting. A frame 704 schematically represents an image that enters the vision of the operator. An image 708 provided from the management device is displayed on a display 706 of the AR glasses. This image 708 includes an instruction from the administrator, and the operator can receive the instruction from the administrator in a hands-free state while working.

[0036] In step S610 of FIG. 6A, the AI performs a surrounding analysis of the operator (step S646). In step S648, the AI determines whether the surroundings of the operator are in a dangerous situation (step S648). If it is not a dangerous situation, this determination is repeated. If it is a dangerous situation, a notification is generated (step S650). Then, the notification is sent to the supervisor (step S656), the notification is sent to the operator of the AR glasses (step S652), and the task is aborted (step S654). That is, the AI determines whether the spatial video acquired by the 360-degree camera 104 matches a plurality of videos accumulated by deep learning, such as videos of past accidents or videos representing dangerous situations. If there is a matching video, the notification information corresponding to the video is output to at least one of the AR glasses 102 or the management device 108. The notification information can be, for example, text indicating a high degree of danger or an alarm sound. Further, in step S650, when a notification is generated, the process transitions to step S616 of FIG. 6A, marks the task information, and displays the task information (step S616). This makes it possible to notify the operator before starting the task that the task has been aborted due to the occurrence of danger.

[0037] [Advantages of the Present Invention] By applying the present invention to the manufacturing site, the actions and experiences of skilled workers are converted into content. Furthermore, in on-site work, by wearing AR glasses, it becomes possible to prompt instructions to workers according to the contentified procedures. Also, by realizing communication with supervisors or experts located remotely, high-quality operations become possible. That is, even for new or inexperienced workers, high-quality work becomes possible with remote support. Also, it becomes possible to manufacture high-quality products. Also, through active training, it becomes possible to achieve a high level of proficiency in a short period of time. Also, skilled workers or experts located remotely can use a dashboard to monitor not only the images displayed on the on-site AR glasses but also the images from a 360-degree camera, and by grasping the workers and the working environment, accurate cause analysis and prompt response become possible. Also, by sharing the history between workers and past experts, prompt response becomes possible. Also, it becomes unnecessary to send skilled workers or experts to the site, and it becomes possible to save transportation costs and travel time. Also, on-site workers can perform highly productive work with hands-free AR glasses.

[0038] By applying the present invention to a logistics warehouse, it becomes possible to centrally manage the data of articles, and efficient article search becomes possible by AR technology. Also, by linking with the transaction of articles, it becomes possible to provide the latest article information in the warehouse. That is, by AR technology, it becomes possible to centrally manage the data of articles in the warehouse. Also, based on the centrally managed data, by instructing the worker about the target article and its location, it becomes possible to shorten the average pick-up time. Also, it becomes possible to eliminate errors in the parts assigned to an order.

[0039] By applying the present invention to the medical field, medical knowledge, operation methods of medical instruments, etc. can be managed as up-to-date information, enabling doctors to perform appropriate medical practices. That is, by formulating medical knowledge into a data platform, it becomes possible to share it with medical staff as always up-to-date data. Also, through real-time communication with patients, attending doctors, and experts, it becomes possible to realize appropriate medical practices. Further, by managing the history of communication and treatment, appropriate medical practices become possible.

Explanation of Signs

[0040] 100…Remote support system 102…AR glasses 102a…Main body part 102b…Frame 104…360-degree camera 106…Relay device 108…Management device 110…AI part 202…CPU 204…Memory 206…OS·Application 208…Communication interface 210…Display 212…Disk drive 214…Keyboard·Mouse 216…Microphone·Speaker 402…CPU 404…Memory 406…OS·Application 408…Communication interface 410…Display 412…Camera 413…LED flash 414…Operation button 416…Microphone·Speaker

Claims

1. A remote support system for assisting an operator working remotely, comprising: a wearable terminal having an imaging unit for capturing a spatial video in the direction in which the operator's face is facing, a sound collection unit for collecting at least the operator's voice, a projection unit for projecting a visible video to the operator, a voice output unit for outputting voice, and a wireless communication unit for performing wireless communication; a 360-degree camera for capturing a spatial video around the operator; a management device that reproduces the spatial video or voice acquired by the wearable terminal and the spatial video acquired by the 360-degree camera, and outputs video information or voice information to the wearable terminal; an AI unit that compares the spatial video acquired by the 360-degree camera with videos of a plurality of past accidents accumulated by deep learning and videos representing dangerous situations, and when there is a video corresponding to the spatial video, outputs notification information indicating the degree of danger around the operator corresponding to the video to at least one of the wearable terminal and the management device; and comprising: the management device outputs the notification information to the wearable terminal; the wearable terminal is characterized in that the notification information is reproduced by at least one of the projection unit and the voice output unit. A remote support system.

2. The remote support system according to claim 1, wherein the wearable terminal acquires video content from the management device and reproduces the video content by the projection unit and the voice output unit.

3. The remote support system according to claim 1 or claim 2, further comprising a relay device for relaying communication between the wearable terminal, the 360-degree camera, and the management device.

Images