Remote work assistance system

Abstract

An operator who performs work can confirm image information from an assistant who is at a remote place with eyes as needed while ensuring a sufficient field of view and safety. A remote work assistance system (1) includes a wearable device (10), a mobile terminal (20), and an assistant terminal (40). The wearable device (10) has a photographing section (11), a sound input section (12), and a sound output section (13). The mobile terminal (20) has a display section (21). The assistant terminal (40) transmits and receives sound with the wearable device (10), receives an image from the wearable device (10), and transmits the image to the mobile terminal (20).

Description

Technical Field

[0001] This application relates to a remote work assistance system for use by remote assistants to assist workers at the work site. Background Technology

[0002] A remote work assistance system is known, in which a worker is equipped with a camera terminal, and an assistant operates the camera terminal from a distance to assist the worker. Patent Document 1 (Japanese Patent Application Publication No. 2018-207420) discloses a work assistance system that allows the worker to identify the work object through voice communication even without a display device. Summary of the Invention

[0003] The technical problem that the invention aims to solve

[0004] Wearable devices have been designed in various types. In glasses-type devices, information can be displayed on or near the lenses, which are equivalent to eyeglasses. When glasses-type devices are used to assist workers, displaying images sent from the assistant on the lenses narrows the worker's field of vision, reduces work efficiency, and could potentially lead to dangerous situations. Conversely, in worker terminals (wearable devices) without displays, workers cannot view images sent from the assistant, creating a dilemma.

[0005] Technical solutions adopted to solve technical problems

[0006] The first-view remote work assistance system includes wearable devices, mobile terminals, and assistant terminals. The wearable devices are installed on the field workers. The wearable devices have a camera, a sound input unit, and a sound output unit. The wearable devices transmit and receive images and sound. The mobile terminals are separate from the wearable devices and are carried by the field workers. The mobile terminals have a display unit that shows the received images. The assistant terminals are used by assistants who provide remote assistance to the field workers. The assistant terminals have a sound input unit, a sound output unit, and a display unit. The assistant terminals transmit and receive sound with the wearable devices, and receive images from the wearable devices and transmit images to the mobile terminals.

[0007] In the first-view remote work assistance system, the wearable device does not include a display, ensuring that on-site workers have sufficient visibility to perform their work safely. Furthermore, since images can be displayed on the mobile terminal when needed, instructions from remote assistance personnel can also be viewed on the display screen.

[0008] Based on the system of the first perspective, in the remote work assistance system of the second perspective, the wearable device is a neck-mounted device.

[0009] In the second perspective of remote work assistance systems, since the wearable device is a neckband-style device, on-site workers can work hands-free and efficiently with unobstructed vision.

[0010] Based on the first or second viewpoint system, in the third viewpoint remote operation assistance system, the display unit of the assistant's terminal displays video and still images captured by the camera unit of the wearable device.

[0011] Based on the system of any one of the first to third viewpoints, in the remote work assistance system of the fourth viewpoint, the image captured by the camera unit of the wearable device is selected, processed or supplemented with information at the assistant's terminal, and sent to the mobile terminal for display on the display unit of the mobile terminal.

[0012] By utilizing a remote work assistance system based on the fourth perspective, on-site workers can process images of the site without needing to pay special attention to taking pictures, utilize images with additional information, and receive support, thereby improving work efficiency.

[0013] Based on the system of any one of the first to fourth viewpoints, in the remote operation assistance system of the fifth viewpoint, the voices of the on-site operators and assistants are recognized by sound and displayed as text information on the display of the assistant's terminal.

[0014] In the fifth perspective's remote operation assistance system, assistants can confirm conversations with on-site workers via text messages while providing assistance, thus eliminating the need for recording and enabling efficient operations.

[0015] Building upon the system of the fifth perspective, in the remote operation assistance system of the sixth perspective, text information is also displayed on the display of the mobile terminal.

[0016] In the sixth point of view's remote work assistance system, on-site workers can confirm their conversations with assistants via text.

[0017] Building upon the systems of any of the first through sixth viewpoints, the remote work assistance system of the seventh viewpoint includes a first line and a second line. The first line connects the assistive worker's terminal to a wearable device. The second line connects the assistive worker's terminal to a mobile terminal. The first line is used continuously during the work performed by the on-site worker. The second line is connected as needed during the work performed by the on-site worker.

[0018] The seventh point of view's remote operation assistance system separates the first and second lines, using the second line only when needed. Therefore, on-site operators only need to use the second line when required, thus eliminating the need for unnecessary operations on the mobile terminal.

[0019] Based on the systems of any one of the first to seventh viewpoints, the remote operation assistance system of the eighth viewpoint automatically extracts the work progress of the on-site workers from images captured by the camera unit and / or the voices of the on-site workers and support personnel. The work progress or completion time is estimated based on the automatically extracted images and voices. The estimated progress or completion time is displayed on the display unit of the support personnel's terminal.

[0020] In the remote work assistance system of the eighth perspective, assistants are able to monitor the progress of the work.

[0021] Based on the system of any one of the first to eighth viewpoints, in the remote operation assistance system of the ninth viewpoint, the assistant terminal accesses the database that stores on-site operation assistance information, extracts and processes the on-site operation assistance information, and sends it to the mobile terminal.

[0022] By utilizing the Ninth Viewpoint's remote operation assistance system, on-site operators receive information after extracting and processing on-site operation assistance information, thus enabling efficient operation.

[0023] Based on the system of any one of the first to ninth viewpoints, in the remote operation assistance system of the tenth viewpoint, the assistance provided from the assistant's terminal to the field operator includes any one of the following: voice-based assistance, image-based assistance, text-based assistance input on the assistant's terminal, or text-based assistance based on the assistant's voice transcribed into text, or a combination thereof.

[0024] Building upon the systems of any of the first through tenth perspectives, the remote work assistance system of the eleventh perspective includes multiple devices capable of connecting to the same server. These multiple devices include wearable devices, mobile terminals, and assistive worker terminals. The server designates at least one of the multiple devices based on defined conditions and, based on the designation, calls out the devices constituting the connection objects.

[0025] Building upon the system described in the eleventh viewpoint, the remote work assistance system in the twelfth viewpoint involves the server acquiring environmental information about the work site from the on-site workers. Based on this environmental information, the server then calls up at least one of multiple devices.

[0026] Building upon the system described in the eleventh viewpoint, the remote work assistance system in the thirteenth viewpoint manages the progress of tasks performed by on-site personnel. The server calls upon at least one of multiple devices based on the progress of the task.

[0027] Building upon the system described in the eleventh viewpoint, in the remote operation assistance system described in the fourteenth viewpoint, the server performs voice recognition on the voices of on-site operators and support personnel. Based on keywords obtained through voice recognition, the server calls up at least one of multiple devices.

[0028] Based on the system of any one of the eleventh to fourteenth viewpoints, in the remote operation assistance system of the fifteenth viewpoint, the server also receives designations from the assistance personnel terminal for at least one of a plurality of devices used by the field operator.

[0029] Based on the systems of any one of the first to fifteenth viewpoints, in the remote operation assistance system of the sixteenth viewpoint, multiple devices are able to communicate with each other.

[0030] The seventeenth point describes an auxiliary personnel terminal used by assistants to remotely assist on-site workers. The auxiliary personnel terminal has a sound input unit, a sound output unit, and a display unit. It transmits and receives sound from and receives images from a wearable device. The wearable device, which is installed on the on-site worker, has a camera unit, a sound input unit, and a sound output unit. The wearable device transmits images and transmits and receives sound. The auxiliary personnel terminal transmits images to a mobile terminal. The mobile terminal is separate from the wearable device and carried by the on-site worker. The mobile terminal has a display unit that displays the images received by the image receiving unit.

[0031] The server in the eighteenth viewpoint is a server connected to multiple devices. These devices include wearable devices, mobile terminals, and auxiliary personnel terminals. Wearable devices are installed on field workers and have a camera, sound input, and sound output unit. The wearable device transmits images and transmits and receives sound. Mobile terminals are separate from the wearable devices and are carried by the field workers, having a display unit that displays the images received by the image receiving unit. Auxiliary personnel terminals are used by assistants who provide remote assistance to field workers and have a sound input, sound output, and display unit. The server designates at least one of the multiple devices according to specified conditions and calls the device constituting the communication target based on the designation. Attached Figure Description

[0032] Figure 1 This is a perspective view of the wearable device 10 according to the first embodiment.

[0033] Figure 2 This is a schematic diagram of the overall structure of the remote work assistance system 1 according to the first embodiment.

[0034] Figure 3 This is a diagram showing an example of the display screen of the display unit 43 of the assistant terminal 40 according to the first embodiment.

[0035] Figure 4 This is a diagram schematically illustrating the remote work assistance system of the second embodiment. Detailed Implementation

[0036] <First Embodiment>

[0037] (1) Overall structure

[0038] like Figure 2 As shown, the remote work assistance system 1 of the first embodiment includes a wearable device 10, a mobile terminal 20, a server 30, an assistant terminal 40, and a database 50. The wearable device 10 and the mobile terminal 20 are respectively held by field workers "AAA_aaa", "BBB_bbb", and "CCC_ccc". The field workers are multiple workers who are dispatched to multiple sites. Figure 2 In this case, three workers were dispatched to three sites; however, the number of sites and workers is not limited to this. The auxiliary personnel terminal 40 is configured for use by auxiliary personnel located at a distance from the on-site workers.

[0039] Wearable device 10, mobile terminal 20, server 30, assistant terminal 40, and database 50 are connected to a shared network and can communicate with each other.

[0040] Additionally, in this manual, on-site workers are sometimes referred to as "operators," which means the same thing.

[0041] In addition, in this specification, "operation" refers to any one of the following: repair, maintenance, or repair of the equipment; on-site investigation, setup, or installation of the equipment.

[0042] (2) Detailed Structure

[0043] (2-1) Wearable devices 10

[0044] like Figure 1 As shown, the wearable device 10 is a neckband-style device. The wearable device 10 is ring-shaped, with a portion of the ring having an opening. The wearable device is installed by inserting it into the neck using the opening portion, with the opening portion facing forward.

[0045] The wearable device 10 is a neckband style, so it can be used hands-free even when worn. Furthermore, it does not obstruct vision like glasses.

[0046] The wearable device 10 has a main body 19, a camera unit 11, a sound input unit 12, a sound output unit 13, a gesture sensor 14, a control unit 15, a communication unit 16, and a power switch (not shown).

[0047] A camera unit 11 and a gesture sensor 14 are provided at both ends of the main body 19.

[0048] The imaging unit 11 is a camera. The camera continuously and constantly captures images of the area in front of the operator during their work. The captured images are either video or still images. The captured images are transmitted to the server 30 via the communication unit 16 and the network. The images transmitted to the server 30 are stored in the server 30's storage unit 32. Simultaneously, the images transmitted to the server 30 are displayed on the display unit 43 of the assistant operator terminal 40.

[0049] The gesture sensor 14 functions as a switch. The wearable device can be operated by the installer (operator) by displaying their signature in front of the gesture sensor. For example, this operation could be turning the camera unit 11 on or off. Furthermore, it can also be used by the operator to call upon an assistant using the assistive device terminal 40. The wearable device 10 may also have a mechanical or electronic switch instead of the gesture sensor 14, or it may have both a gesture sensor 14 and a mechanical or electronic switch.

[0050] The voice input unit 12 is a microphone. The microphone is directional and prioritizes picking up the voices emitted by the installer. The voice input unit 12 can also be used by the operator to call out an assistant using the assistive device terminal 40 instead of the gesture sensor 14. Alternatively, the voice input unit 12 can be used by the operator together with the gesture sensor 14 to call out an assistant using the assistive device terminal 40. For example, the operator emits a voice requesting "help," which reaches the server 30. The server 30 performs voice recognition and transcribes the voice data. Based on the transcribed voice data, the server 30 identifies that the operator is seeking assistance and causes the operator's name "AAA aaa" in the operator identification information 112 of the assistive device terminal 40 to flash (on and off). Thus, the assistant knows that the operator "AAA aaa" is seeking assistance.

[0051] The sound output unit 13 is a speaker. The sound output unit 13 is located on the rear side of the body of the installer (operator) of the wearable device 10.

[0052] The wearable device 10 of this embodiment includes a voice input unit 12, a voice output unit 13, and a communication unit 16 for transmitting and receiving voice. Therefore, the operator who installs the wearable device 10 can use the device as a telephone to communicate with assistants.

[0053] The communication unit 16 is connected to the network via the first line T1. It can transmit sound input via the sound input unit 12 and images captured by the imaging unit 11 as electronic data to the network. In addition, it can receive sound as electronic data via the network.

[0054] The control unit 15 is a computer. The control unit 15 has a processor and a storage unit. The control unit 15 controls the camera unit 11, the audio input unit 12, the audio output unit 13, the gesture sensor 14, and the communication unit 16. The control unit 15 receives and executes control commands input to the gesture sensor 14. The control unit 15 executes commands from the server 30 and commands input by the assistant operator when operating the assistant terminal 40.

[0055] (2-2) Mobile terminal 20

[0056] The mobile terminal 20 is carried or held by the operator. The mobile terminal 20 can also be a smartphone, tablet, PC, etc. The mobile terminal 20 has a display unit 21, a communication unit 22, and a control unit 23. The mobile terminal 20 is connected to the network via a second line T2.

[0057] The communications unit 22 receives images via a network. The images can be either video or still images.

[0058] Display unit 21 is a monitor. Display unit 21 displays images received via the network. Images processed by the assistant at the assistant terminal 40 are sent to the mobile terminal 20 via the network and displayed on display unit 21. The images can be videos or still images.

[0059] The control unit 23 is a computer. The control unit 23 has a processor and a storage unit. The control unit 23 controls the display unit 21 and the communication unit 22.

[0060] Unlike wearable device 10, mobile terminal 20 does not need to be constantly connected to assistant terminal 40 (or server 30) during the operator's work. It only needs to be connected when images (including text information) are sent to mobile terminal 20 through assistant terminal 40.

[0061] (2-3) Server 30

[0062] Server 30 is a computer. Server 30 has a processor 31 and a storage unit 32. The storage unit 32 stores application programs and data. In this embodiment, the application program of the remote work assistance system 1 stored in the storage unit 32 is read and executed by the processor 31. The application program of the remote work assistance system 1 is a web application. A browser is launched on the assistant terminal 40 and the mobile terminal 20, and the application program of the remote work assistance system 1 is accessed through the browser.

[0063] If the wearable device 10 is activated by the operator, it connects to the server 30 and sends the images captured by the camera unit 11 of the wearable device 10 to the server 30. The captured images are accumulated in the storage unit 32 of the server 30. At the same time, by launching the browser of the assistive device 40, the captured images can be viewed on the display unit 43 of the assistive device 40.

[0064] Furthermore, during the operation of the operator, the application of the remote operation assistance system 1 enables two-way communication between the operator and the assistant based on the connection request from the wearable device 10 or the connection request from the assistant terminal 40.

[0065] Server 30 converts the voice received from wearable device 10 and assistive device terminal 40 into text information using voice recognition. The text information is stored in storage unit 32 for each task. Furthermore, the text information is sent to assistive device terminal 40 and displayed on display unit 43 of assistive device terminal 40. Additionally, the text information is also sent to mobile terminal 20 as needed and displayed on display unit 21 of mobile terminal 20.

[0066] Server 30 collects the voice data of all communicating wearable devices 10 and saves it as text information in storage unit 32. If the voice of a worker "BBB bbb" who is currently connected to the assistive worker terminal 40 but not receiving assistance contains a request for assistance, server 30 causes the worker's name "BBB bbb" in the worker identification information 112 of the assistive worker terminal 40 to flash (on and off).

[0067] Server 30 may also have an automatic translation function. It can translate the voice data after voice recognition into other languages. For example, if the on-site worker speaks English while the remote support staff speaks Japanese, the English spoken by the on-site worker can be transmitted to the support staff in Japanese. Alternatively, the voice recording 116 on the display unit 43 of the support staff terminal 40 can be displayed in Japanese. English and Japanese can also be displayed together.

[0068] Images generated during operation assistance, transcribed audio information, and images processed by the assistant are stored in storage unit 32 of server 30. This information can also be stored in other locations. For example, it can be stored in database 50. It can also be stored in storage unit of assistant terminal 40.

[0069] (2-4) Database 50

[0070] Database 50 includes a storage unit for storing terminal data. Database 50 is connected to a network. In this embodiment, database 50 stores on-site operation assistance information for equipment repair, maintenance, and upkeep. The storage of on-site operation assistance information is not limited to database 50. It may also be stored in the storage unit 32 of server 30. It may also be stored in the storage unit of assistant personnel terminal 40.

[0071] (2-5) Auxiliary personnel terminal 40

[0072] The auxiliary personnel terminal 40 is used by an auxiliary personnel to remotely assist on-site workers. The auxiliary personnel terminal 40 can also be a PC. The auxiliary personnel terminal 40 includes a voice input unit 41, a voice output unit 42, a display unit 43, a communication unit 44, and a control unit 45.

[0073] The sound input unit 41 is a microphone. The sound output unit 42 is a speaker. The wearable device 40 of this embodiment includes a sound input unit 41, a sound output unit 42, and a communication unit 44 for transmitting and receiving sound. Therefore, assistants can use the assistant terminal 40 as a telephone to communicate with field workers.

[0074] The communication unit 44 is connected to a network. It receives sound as electronic data via the network. The communication unit 44 also receives text information transcribed from sound recognition data on the server 30. The communication unit 44 receives images captured by the camera unit 11 of the wearable device 10. Furthermore, the communication unit 44 can transmit sound input via the voice input unit 41 and images processed by the assistant at the assistant terminal 40 as electronic data to the network.

[0075] Display unit 43 is a monitor.

[0076] The control unit 45 is a computer. The control unit 45 has a processor and a storage unit. The control unit 45 controls the sound input unit 41, the sound output unit 42, the display unit 43, and the communication unit 44. The storage unit of the control unit 45 stores a browser (program). The control unit 45 reads and executes the browser program. If the application program of the remote work assistance system 1 is executed by the server 30, then by using the browser on the control unit 45 of the assistant terminal 40, the assistant terminal 40 can assist with work.

[0077] In addition, regarding the communication between the wearable device 10, mobile terminal 20, server 30, and assistant terminal 40 in this embodiment, multiple communication protocols can be used separately.

[0078] In this embodiment, for video and audio communication protocols, WebRTC is used, for example. WebRTC is a standard protocol, which, while secure, has limited design freedom (only parameters such as resolution can be changed). For still image communication protocols, HTTPS, MQTT, or WebSocket can be used. These communication protocols are freely designable and can handle video, text, still images, etc., exhibiting high design flexibility on the server side.

[0079] (2-5-1) Display screen of display unit 43 of auxiliary personnel terminal 40

[0080] Figure 3 The image shows an example of the browser display screen of the display unit 43 of the assistant terminal 40 when the application of the remote work assistance system 1 is being executed.

[0081] The display unit 43 displays the following information: work progress 111, worker identification information 112, on-site real-time images 113, images of auxiliary personnel processing 114, a list of references 115, and audio recordings 116.

[0082] The "Work Progress 111" indicates the number of work sites (or the number of workers) that the assistants using the assistant terminal participated in "today". Here, it shows the case where two jobs have been completed "today".

[0083] The progress of the work (which can be shown as a percentage) and the estimated end time of the work can also be displayed at the work progress 111. For this purpose, the server 30 can automatically extract the information from the images captured by the camera unit 11 of the wearable device 10 and / or the voices of the workers and assistants to automatically estimate the progress of the work or the end time of the work.

[0084] The operator identification information 112 describes the operator as "Currently Online Novice". This refers to the operator currently using the wearable device 10 connected to the assistant operator terminal 40. Here, the operator's icon (photo) and the operator's name "AAA aaa" are displayed. In addition, "LIVE" is displayed before the operator "AAA aaa"'s icon. Here, the display of "LIVE" indicates that the connection between the assistant operator terminal 40 and the wearable device 10 is active. Active means that the following state is achieved: the live image 113 of the operator "AAA aaa" is displayed on the screen and the audio is connected to the operator "AAA aaa" in a way that allows for communication.

[0085] In addition, Figure 3 The worker identification information 112 shows five workers; however, some inactive workers can be hidden by scrolling. Alternatively, active workers can be displayed more prominently, while the names of other workers can be partially or completely hidden.

[0086] The worker identification information 112 is not limited to displaying icons and worker names; it can also display worker ID numbers, on-site residences, the model and ID number of the equipment being worked on (in this case, an air conditioner), and the work ID. The worker identification information 112 is information that assists personnel in identifying workers or work.

[0087] Furthermore, the method of displaying active workers is not limited to displaying "LIVE". The assistant using the assistant terminal 40 only needs to be able to identify active workers. Alternatively, the worker identification information in the connection can be enlarged, flashed, its color changed, or it can be made to pop up.

[0088] Live image 113 shows a live video of an active worker being captured by the camera unit 11 of the worker's wearable device 10 "AAA aaa". Figure 3 The image shown is of the outdoor unit of the air conditioner. In the active state, the sound is connected to the operator "AAA aaa", so the assistant gives instructions through sound to move the operator. As a result, the position of the camera unit 11 of the wearable device 10 can be moved, thereby moving the image.

[0089] Image 114, processed by assistant personnel, is derived from and processed from live image 113. Figure 3 In the auxiliary personnel processing image 114, a circular mark is provided in the center of the fan image, indicating "Turn on the fan from here" in a handwritten manner. This is text or markings written by the auxiliary personnel on the auxiliary personnel terminal 40 using a stylus. By sending the auxiliary personnel processing image 114 to the operator's mobile terminal 20 and displaying it on the display unit 21, the operator can be assisted efficiently.

[0090] Reference list 115 is a list of references (on-site operation support information) consulted by support personnel when providing assistance. On-site operation support information may also include on-site investigation and confirmation items, construction operation sequence manuals, or maintenance information. Support personnel can extract the documents required for construction from the on-site operation support information stored in database 50, further extract the required parts from the documents, and, after setting tags as needed, send them to the operator's mobile terminal 20.

[0091] Voice recording 116 displays the voice text of the active operator "AAA aaa" and the voice text of the assistant separately.

[0092] (3) Explanation of on-site worker assistance using remote work assistance system 1

[0093] Next, the method for assisting workers using the remote work assistance system 1 of this embodiment will be described.

[0094] To repair and maintain the equipment, the workers arrive at the site and begin work. Here, the equipment is assumed to be an air conditioner. When work begins, the workers on site attach wearable device 10 to their bodies and turn on the power switch.

[0095] First, the operator sends a signature in front of the gesture sensor 14 to activate the camera unit 11. The camera unit 11 then begins capturing images. Furthermore, the wearable device 10 connects to the application of the remote work assistance system 1 on the server 30. The captured images are transmitted to the server 30 via the communication unit 16 and the network, and are accumulated in the storage unit 32 of the server 30.

[0096] When the assistant begins to assist, a browser is launched on the assistant's terminal 40 to connect to the application of the remote work assistance system 1 on the server 30.

[0097] The power-on of the wearable device 10 and the connection between the assistive personnel terminal 40 and the application of the remote work assistance system 1 on the server 30 can be asynchronous. Either one can be prioritized. The two are connected when the camera unit 11 of the wearable device 10 is activated and the assistive personnel terminal 40 is connected to the application of the remote work assistance system 1 on the server 30 via a browser.

[0098] Using the auxiliary personnel terminal 40, the auxiliary personnel appropriately select the operators and begin providing assistance.

[0099] When a worker requests assistance from a support staff member, a gesture indicating a desire to receive assistance is displayed in front of the gesture sensor 14. Assistance requests can also be made via voice or other methods. The worker's assistance request causes the worker's name "AAA aaa" in the worker identification information 112 to flash (on and off) on the display unit 43 of the support staff terminal 40. When the support staff agrees to the worker's assistance request, they can activate the connection with the worker "AAA aaa" by clicking on their name. In other words, the worker "AAA aaa" in the worker identification information 112 is marked "Live," and the image captured by the wearable device 10 is displayed live on the live image 113, making communication between the worker "AAA aaa" and the support staff possible.

[0100] While communicating with the operator "AAA aaa", the assistant confirmed the video sent from the operator's wearable device 10 and studied the content of the operation assistance.

[0101] When accessing the operator from the assistant's side, if the operator's name "AAA aaa" in the operator identification information 112 is clicked, the connection with the operator "AAA aaa" becomes active, the image captured by the wearable device 10 is displayed live on the live image 113, and the operator "AAA aaa" can communicate with the assistant.

[0102] As a supplementary method, the simplest approach is for the assistant to offer suggestions to the operator while speaking with the operator "AAA aaa". The voices of both the operator and the assistant are transcribed into text by voice recognition on server 30 and stored in storage unit 32 of server 30. Simultaneously, this is displayed on the voice recording 116 of the assistant's terminal 40's display unit 43. The assistant reviews the voice recording 116, considers the suggestions, and continuously offers assistance. Alternatively, the transcribed voice or conversation content of the assistant can be sent to the operator "AAA aaa's" mobile terminal 20 and displayed on display unit 21, thus providing assistance.

[0103] Furthermore, the assistant appropriately selects (crops) an image from the live image 113 as a still image (capture), and displays this still image on the assistant's processed image 114. The assistant further processes the displayed image. Specifically, the required parts are cropped, marked, and text is written on them. The image written by the assistant is sent to the operator's mobile terminal 20. The operator "AAA aaa" confirms the image sent by the assistant on the display 21 of the mobile terminal 20. In this way, the operator receives assistance from the assistant and continuously advances the work.

[0104] The above explains the case where the images created by the assistant are still images; however, they can also be videos.

[0105] The auxiliary information provided by the assistant is not limited to the images captured and processed by the wearable device 10, but can also include images or text (captures) extracted or processed from references (on-site operation assistance information). On-site operation assistance information can also be displayed in the reference list 115 on the screen, or obtained by searching through the window of the reference list 115. The on-site operation assistance information obtained by searching in the reference list 115 is information stored in the database 50. For example, if the assistant determines that they need to confirm the operating instructions for assistance, they click on the operating instructions in the reference list 115. A new window pops up, displaying the pages of the operating instructions. The assistant selects a specific page of the operating instructions, cuts out the required portion, and displays the cut-out image (capture) in the assistant-processed image 114. Here, the assistant marks, adds text, and processes the image (capture) of the specific page of the operating instructions. Similar to the images captured, selected, and processed by the wearable device 10, the processed image is sent to the operator's mobile terminal 20 as the assistant-processed image. The operator can use the assistant-processed image as a reference to advance their work.

[0106] Furthermore, the above explanation addresses the scenario where on-site operational assistance information is stored in database 50. On-site operational assistance information may also be stored in storage unit 32 of server 30. Alternatively, it may be stored in storage unit of assistant personnel terminal 40. On-site operational assistance information may also be information obtained from other sources. For example, it could be information from a website.

[0107] As explained above, the assistant continuously advances the assistance work of worker "AAA aaa". When the assistance work of worker "AAA aaa" ends or when an assistance request from worker "BBB bbb" occurs during the assistance process, the worker's name "BBBbbb" in the worker identification information 112 on the browser display screen of the assistant's terminal 40 flashes (on and off). If the assistant who sees the assistance request can begin assisting worker "BBB bbb", they click on the worker's name "BBB bbb". This activates the connection with worker "BBB bbb". In other words, worker "BBB bbb" in the worker identification information 112 is marked "Live", the image captured by the wearable device 10 is displayed live on the live image 113, and communication between worker "BBB bbb" and the assistant becomes possible.

[0108] As described above, work assistance is provided to multiple workers. When work is performed using the remote work assistance system 1 of this embodiment, images captured by the wearable device 10, audio including conversations between workers and assistants, and processing images created by the assistants are continuously accumulated in the storage unit 32 of the server 30. This information is stored for each job. The server 30 can also automatically generate reports based on the stored information.

[0109] (4) Features

[0110] (4-1)

[0111] The remote work assistance system 1 of this embodiment includes a wearable device 10, a mobile terminal 20, and an assistant personnel terminal 40. The wearable device 10 is installed and used by on-site workers. The wearable device 10 includes a camera unit 11, which can transmit captured images of the area around the worker to the assistant personnel terminal 40. Furthermore, the wearable device 10 and the assistant personnel terminal 40 can be used to make calls, similar to using a telephone.

[0112] Wearable device 10 does not include a display. When wearable devices, such as eyeglasses, include a display, the operator's field of vision is limited, and the operator may encounter danger. Conversely, omitting a display reduces these concerns.

[0113] Furthermore, since the operator carries the mobile terminal 20, auxiliary information from the assistant can be displayed on the display section 21 of the mobile terminal 20 as needed.

[0114] (4-2)

[0115] The wearable device 10 in this embodiment is a neckband-style device. Therefore, the operator can perform the work hands-free.

[0116] (4-3)

[0117] The images captured by the camera unit 11 of the wearable device 10 are either videos or still images.

[0118] By capturing video, the live image 113 can be continuously displayed on the display unit 43 of the auxiliary personnel terminal 40.

[0119] When assistants create images for assistant processing, they can do so more easily by using still images.

[0120] (4-4)

[0121] Images captured by the camera unit 11 of the wearable device 10 are selected, processed, or have additional information added at the assistive terminal 40, and then sent to the mobile terminal 20 for display on the display unit 21 of the mobile terminal 20.

[0122] By utilizing both the wearable device 10 and the mobile terminal 20, operators do not need to view the auxiliary information of assistants, and operators can view it on the mobile terminal 20 when necessary.

[0123] (4-5)

[0124] The voices of the operators and assistants are recognized by sound and displayed as text information on the voice recording 116 of the display unit 43 of the assistant terminal 40.

[0125] Since the voice recording is displayed as text information on the support staff terminal 40, the support staff can easily confirm the content of the call and can also use it to assist the operators. Furthermore, the recording of the voice content as text information can also be used for report preparation.

[0126] (4-6)

[0127] The aforementioned text information can also be displayed on the display unit 21 of the mobile terminal 20. Even operators can confirm the voice communication with support personnel in text form through the display unit 21, which is very useful. In addition, support personnel can also extract important voice content text and display it on the operator's mobile terminal 20.

[0128] (4-7)

[0129] The remote work assistance system 1 includes: a first line T1 connecting the assistant terminal 40 to the wearable device 10; and a second line T2 connecting the assistant terminal 40 to the mobile terminal 20. The first line T1 is used continuously during the operation, and the second line T2 is connected as needed during the operation.

[0130] In this way, operators can work without constantly looking at the screen.

[0131] (4-8)

[0132] The server 30 of the remote work assistance system 1 stores images captured by the camera unit 11 of the wearable device 10 in the storage unit 32 of the server 30. Furthermore, it performs voice recognition on the voices of the operator and assistant, transcribes the transcribed text, and stores this transcribed text in the storage unit 32 of the server 30. The server 30 can also estimate the progress and end time of the work based on these images and transcribed text, and display this information on the display unit 43 of the assistant's terminal 40. The progress of the work can be estimated and displayed in a manner such as "○○% completed." In this way, the assistant can be aware of the progress of the work.

[0133] (4-9)

[0134] The support personnel terminal 40 displays the reference list 115 on the display unit 43. The references refer to on-site operational support information. This on-site operational support information is stored in a database 50 on the network. Support personnel can access the content of the on-site operational support information through the reference list 115. Support personnel extract and process the content of the on-site operational support information and send it to the mobile terminal 20.

[0135] As described above, the support staff processes the on-site operational support information and delivers it to the operators. As a result, the operators do not need to look at a large amount of data, thus improving their efficiency.

[0136] (4-10)

[0137] The remote work assistance system 1 of this embodiment provides work assistance based on various means. In other words, the assistance includes any one of the following: voice-based assistance, image-based assistance, text-based assistance input on the assistant's terminal, and text-based assistance based on the assistant's voice, or a combination thereof.

[0138] Voice-based assistance is the same as telephone-based assistance.

[0139] Image-based assistance includes assistance using images captured by wearable device 10, and assistance using processed on-site operational assistance information.

[0140] Assistance based on text input on the assistive device terminal 40 is implemented in the following manner: the assistive device inputs text on the assistive device terminal 40, and the text is sent to the mobile terminal 20 and displayed on the display unit 21.

[0141] Assistance based on the textualization of the assistant's speech involves transing the assistant's speech into text and sending the transcribed information to the operator's mobile terminal 20. Operators may miss or forget what the assistant says. Therefore, for important speech, sometimes it is transcribed and sent to create effective assistance.

[0142] (5) Variations

[0143] (5-1) Variation 1A

[0144] In the first embodiment, such as Figure 3 As shown, only the live image 113 of the active worker "AAA aaa" is displayed on the display unit 43. In Modification 1A, the live images of all connected workers "AAA aaa", "BBB bbb", "CCC ccc", "DDD ddd", and "EEE eee" are displayed on the live image 113. In other words, the live images of inactive workers are also displayed on the live image 113. However, the active worker "AAA aaa" is displayed larger at the top, while the images of the inactive workers "BBB bbb", "CCC ccc", "DDD ddd", and "EEE eee" are displayed smaller at the bottom. Thus, through the display unit 43 of Modification 1A, the assistant can know the approximate movements of inactive workers, or in other words, workers who are not receiving assistance. For example, when a worker wants to perform an undesirable action, the assistant can use this to draw the worker's attention.

[0145] (5-2) Variation 1B

[0146] In the first embodiment, the sound recognition and textification are performed in the server 30. In Variation 1B, sound recognition and textification are performed in the wearable device 10 or the assistive terminal 40. Therefore, the wearable device 10 or the assistive terminal 40 sends both the audio data and the text data of the sound to the network. Furthermore, the wearable device 10 or the assistive terminal 40 may have not only sound recognition capabilities but also translation capabilities.

[0147] (5-3) Variation 1C

[0148] In the first embodiment, the wearable device 10 is a neckband type. The wearable device 10 can also be in other forms. Preferably, it is mounted on the body and is hands-free. In variation 1C, the wearable device 10 is mounted on the head. Specifically, it is mounted on a helmet, hat, or headband.

[0149] (5-4) Variation 1D

[0150] In the remote work assistance system 1 of the first embodiment, the field worker holds and uses a wearable device 10 and a mobile terminal 20. The remote work assistance system 1 may also utilize other devices. These other devices can be connected to the assistance worker terminal 40 via a network. Other devices may also be connected to a server 30.

[0151] In Modification 1D, the other device is a webcam, which is different from the camera unit 11 of the wearable device 10 used by the field workers. The webcam transmits images taken from locations different from the camera unit 11 of the wearable device 10, as well as images taken from the same location but from a different angle, to the assistant's terminal via the network.

[0152] (5-5) Variation 1E

[0153] In the first embodiment, the wearable device 10 and the mobile terminal 20 are each connected to the network individually via a first line T1 or a second line T2. In a variation 1E, the wearable device 10 and the mobile terminal 20 are connected to each other in a peer-to-peer (P2P) manner.

[0154] In Modification 1E, the wearable device 10 can be connected to the network (or the assistive device terminal 40) via a first line, or the wearable device 10 can temporarily communicate with the mobile terminal 20 in a P2P manner, and then the mobile terminal 20 can be connected to the network via a second line.

[0155] (5-6) Variation 1F

[0156] Another communication device can connect to the wearable device 10 via near-field communication (NFC) technology. NFC technology can be Bluetooth (registered trademark), infrared communication, or Wi-Fi.

[0157] (5-7) Variation 1G

[0158] In the first embodiment, the voice emitted by the assistant is sent to the wearable device 10, and the image sent from the assistant terminal 40 is sent to the mobile terminal 20.

[0159] In variant 1G, field workers can use wearable devices 10, smartphones or mobile terminals 20, or PCs as other devices.

[0160] When sending an image from the assistant's terminal 40, the assistant can choose to send it to the mobile terminal 20 or a PC.

[0161] Alternatively, the configuration may be as follows: server 30 controls the transmission of images from auxiliary personnel terminal 40 to the operator, auxiliary personnel terminal 40 issues instructions to the operator's equipment for transmitting images, and server 30 accepts the instructions.

[0162] (5-8) Variation 1H

[0163] The worker assistance system of this embodiment can be used for other purposes, such as training.

[0164] The training content includes on-site operations such as equipment repair, maintenance, and upkeep. The trainees participate as operators. Ideally, there should be one or two trainees (operators). This is preferred to have the fewest possible participants for operations requiring proficiency.

[0165] The trainers are support staff. Ideally, there should be multiple trainers, each with a different perspective, guiding the trainees. Therefore, the trainers need to observe different places and angles. For example, a trainer familiar with the operation itself might want to confirm the trainee's hand movements. Someone knowledgeable about the equipment might want a bird's-eye view of the overall condition of the equipment, including the trainee, and want to confirm the measuring instruments and other equipment attached to it.

[0166] The instructor (assistant) urges the listeners (operators) to switch to wearable devices 10 and mobile terminals 20, such as smartphones, capable of capturing images of the desired locations and fields of view, or to add such devices. In this way, the listeners (operators) can receive assistance and education related to their own work from the instructor and from various perspectives and angles.

[0167] <Second Implementation>

[0168] (6) Remote operation assistance system of the second embodiment

[0169] Figure 4 The diagram illustrates a remote work assistance system according to the second embodiment.

[0170] The remote work assistance system of the second embodiment basically includes the same structure as the remote work assistance system 1 of the first embodiment. In other words, the remote work assistance system of the second embodiment includes a wearable device 10, a mobile terminal 20, a server 30, a support personnel terminal 40, and a database 50. Similar to the first embodiment, the remote work assistance system provides work assistance, resulting in the continuous accumulation of images captured by the wearable device 10, text digitized from the voices of the operator and the support personnel, support images created by the support personnel, work reports, etc. For newly occurring fault repairs and other tasks, the server 30 of the second embodiment compares the data such as images captured by the wearable device 10 with the previously accumulated data, and automatically identifies similar equipment abnormalities or similar equipment malfunctions that are occurring. Thus, the remote work assistance system 1 of this embodiment can diagnose equipment abnormalities or malfunctions.

[0171] If the server 30 successfully diagnoses a device malfunction or fault, it may suggest the same appropriate assistance method as in the past to the support personnel. Alternatively, it may directly suggest the same appropriate assistance method as in the past to the operators.

[0172] Furthermore, the remote operation assistance system of this embodiment predicts future equipment failures.

[0173] (7) Variation

[0174] (7-1) Variation 2A

[0175] The structure of Modification 2A is basically the same as that of the second embodiment. In Modification 2A, the server 30 continuously accumulates operational assistance data, and sometimes can automatically identify inappropriate parts in the equipment's operation manual and repair and maintenance manual. In such cases, the server 30 proposes to revise and optimize the operation manual and repair and maintenance manual.

[0176] <Third Implementation Method>

[0177] (8) Remote operation assistance system of the third embodiment

[0178] The structure of the remote work assistance system 1 in the third embodiment is as follows: Figure 2The structure is the same as that of the remote work assistance system 1 in the first embodiment. The remote work assistance system 1 includes multiple devices. These devices include a wearable device 10, a mobile terminal 20, and a human assistant terminal 40. Other devices may also be included. These other devices are those other than the wearable device 10 and the mobile terminal 20, which are used by the worker and can be devices that assist in the input or output of sound or images. Examples of specific other devices include cameras, smartphones, smartwatches, smart glasses, headsets, wearable devices, etc. They may also be the same network camera as in Modification 1D, or the same communication device as in Modification 1F.

[0179] In this embodiment, the server 30 is also connected to multiple devices, and the information collected by the multiple devices is collected on the server and accumulated in the storage unit 32. The information includes, for example, images captured by the wearable device 10 and sounds collected by the wearable device 10 or the assistive device terminal 40.

[0180] In this embodiment, server 30 designates at least one device from a plurality of devices based on predetermined conditions, and calls up the devices constituting the connection object based on the designation. AI technology can be used to determine the predetermined conditions of server 30.

[0181] In this embodiment, the server 30 acquires environmental information about the work site of the on-site workers and accumulates it in the storage unit 32. Here, the environmental information includes information such as the illuminance, noise level, temperature, humidity, and whether the work involves working at height. The server 30 can also urge the workers to use and select equipment based on the environmental information, thereby calling up the selected equipment.

[0182] For example, server 30 obtains information such as the worker being in a noisy environment like a factory based on sound information sent from wearable device 10. In this case, server 30 urges the worker to use a device other than wearable device 10 that includes a directional microphone. The worker turns on the device including the directional microphone, thereby triggering server 30 to activate the device.

[0183] As another example, server 30 obtains information such as the worker being in a dark place based on image information sent from wearable device 10. In this case, server 30 urges the worker to use a camera with a light (e.g., mobile terminal 20 or other digital camera) other than wearable device 10. The worker prepares the camera with a light, and server 30 activates the camera with a light.

[0184] (9) Variation

[0185] (9-1) Variation 3A

[0186] In variation 3A, server 30 manages the progress of tasks performed by field personnel and stores the progress in storage unit 32. Server 30 calls up at least one of the plurality of devices based on the progress of the tasks.

[0187] For example, when workers are using smart glasses to perform tasks, if the server 30 determines that the next task is a task at height based on the progress of the task, it will urge the workers to remove the smart glasses that have a narrow field of vision.

[0188] As another example, if the task is determined to require confirmation from more than two viewpoints based on the progress of the task, server 30 urges the operator to use both the overall camera and the camera capturing the area near their hands. When the operator is able to use both cameras, server 30 activates both cameras.

[0189] (9-2) Variation 3B

[0190] In Variation 3B, server 30 performs voice recognition on the voices of on-site workers and support personnel. Server 30 calls up at least one of multiple devices based on keywords obtained through voice recognition.

[0191] For example, when words like "climb" or "on a ladder" appear in a conversation between the operator and the support staff, server 30 prompts the operator to use the headset. When the operator puts on the headset, server 30 activates the headset.

[0192] When phrases like "can't hear clearly" or "speaker is too quiet" appear in conversations between operators and support staff, the operator is urged to use a device equipped with a directional microphone. Once the operator is able to use a device equipped with a directional microphone, server 30 calls the device equipped with the directional microphone.

[0193] When phrases like "can't see clearly" or "zoom in" appear in conversations between operators and support staff, the operator is urged to use a smartphone or smart glasses. Once the operator is able to use a smartphone or smart glasses, server 30 invokes the smartphone or smart glasses.

[0194] (9-3) Variation 3C

[0195] Similar to Modification 1H, the remote work assistance system 1 in Modification 3C is used for training purposes. In this embodiment, based on Modification 1H, the server 30 also receives designations from the assistant terminal 40 for at least one of a plurality of devices used by the field operator.

[0196] The embodiments of this disclosure have been described above, but it should be understood that various changes in form and detail can be made without departing from the spirit and scope of this disclosure as set forth in the claims.

[0197] Symbol Explanation

[0198] 1. Remote Operation Assistance System

[0199] 10 wearable devices

[0200] 11th Filming Department

[0201] 12 sound input sections

[0202] 13 Sound Output Section

[0203] 20 mobile terminals

[0204] 21 Display Section

[0205] 22Ministry of Communications

[0206] 40 Auxiliary Personnel Terminals

[0207] 41. Sound Input Section

[0208] 42 Sound Output Section

[0209] 43 Display Section

[0210] T1 First Line

[0211] T2 Second Line

[0212] Existing technical documents

[0213] Patent documents

[0214] Patent document 1: Japanese Patent Application Publication No. 2018-207420.

Claims

1. A remote operation assistance system, characterized in that, include: Wearable device (10), which will be installed on the field worker, has a shooting unit (11), a sound input unit (12), a sound output unit (13) and a communication unit (16). The communication unit is connected to the network through a first line (T1) to send the image captured by the shooting unit and to send and receive the sound. The wearable device does not have an image display unit. A mobile terminal (20), which is separate from the wearable device and carried by the field worker, has an image receiving unit (22) and a display unit (21). The image receiving unit is connected to the network via a second line (T2) different from the first line to receive images, and the display unit displays the received images. as well as The auxiliary personnel terminal (40) is used by an auxiliary personnel to remotely assist the on-site workers. It has a sound input unit (41), a sound output unit (42), and a display unit (43). It transmits and receives sound with the wearable device through communication via the first line (T1). The auxiliary personnel terminal receives images from the wearable device and transmits images to the mobile terminal through communication via the second line (T2).

2. The remote operation assistance system as described in claim 1, characterized in that, The wearable device also has a control unit (15) that controls the transmission of images, including the video captured by the camera unit, to the communication unit of the assistive personnel terminal.

3. The remote operation assistance system as described in claim 1, characterized in that, The wearable device is a neckband-style device.

4. The remote operation assistance system as described in any one of claims 1 to 3, characterized in that, The display unit of the auxiliary personnel terminal displays videos and still images captured by the camera unit of the wearable device.

5. The remote operation assistance system as described in any one of claims 1 to 3, characterized in that, Images captured by the camera unit of the wearable device are selected, processed, or have additional information added at the assistive device terminal, and then sent to the mobile terminal for display on the mobile terminal's display unit.

6. The remote operation assistance system as described in any one of claims 1 to 3, characterized in that, The voices of the on-site workers and the assistants are recognized by sound and displayed as text information on the display of the assistants' terminal.

7. The remote operation assistance system as described in claim 6, characterized in that, The text information is also displayed on the display of the mobile terminal.

8. The remote operation assistance system as described in any one of claims 1 to 3, characterized in that, The first line is used continuously during the period when the on-site workers are performing their work. The second line is connected as needed during the operation of the on-site personnel.

9. The remote operation assistance system as described in any one of claims 1 to 3, characterized in that, The system automatically extracts the work progress of the on-site workers from the images captured by the camera and / or the voices of the on-site workers and the assistants, estimates the work progress or the end time of the work, and displays the progress or the end time on the display unit of the assistant's terminal.

10. The remote operation assistance system as described in any one of claims 1 to 3, characterized in that, The auxiliary personnel terminal accesses the database (50) that stores on-site operation assistance information, extracts and processes the on-site operation assistance information, and sends it to the mobile terminal.

11. The remote operation assistance system as described in any one of claims 1 to 3, characterized in that, The assistance provided to the field workers through the assistant terminal includes any one of the following: voice-based assistance, image-based assistance, text-based assistance input on the assistant terminal, or text-based assistance based on the assistant's voice, or a combination thereof.

12. The remote operation assistance system as described in any one of claims 1 to 3, characterized in that, The remote operation assistance system includes multiple devices capable of connecting to the same server. The plurality of devices includes the wearable device, the mobile terminal, and the assistive personnel terminal. The server designates at least one of the plurality of devices according to specified conditions, and calls up the devices constituting the connection object according to the designation.

13. The remote operation assistance system as described in claim 12, characterized in that, The server obtains environmental information about the work site of the on-site workers. The server calls up at least one of the multiple devices based on the environmental information.

14. The remote operation assistance system as described in claim 12, characterized in that, The server manages the progress of the work performed by the on-site personnel. The server calls up at least one of the multiple devices based on the progress of the job.

15. The remote operation assistance system as described in claim 12, characterized in that, The server performs voice recognition on the on-site workers and the support staff. The server calls up at least one of the multiple devices based on keywords obtained through the voice recognition.

16. The remote operation assistance system as described in claim 12, characterized in that, The server also receives designations from the assistant personnel terminal for at least one of the multiple devices used by the field operators.

17. The remote operation assistance system (1) as described in claim 12, characterized in that, The multiple devices are able to communicate with each other.

18. A personnel assistance terminal (40), which is used by an assistant to remotely assist on-site workers, and has a voice input unit (41), a voice output unit (42), and a display unit (43), characterized in that, The auxiliary personnel terminal communicates with the wearable device (10) via a first line (T1) to transmit and receive sound, and to receive images from the wearable device. The wearable device is installed on the field worker and has a camera (11), a sound input (12), a sound output (13), and a communication unit (16). The communication unit is connected to the network via the first line (T1) to transmit images captured by the camera and to transmit and receive sound. The wearable device does not have an image display unit. The auxiliary personnel terminal sends images to the mobile terminal (20) via communication via the second line (T2). The mobile terminal is separate from the wearable device and is carried by the field operator. It has a display unit (21) that displays the images received by the image receiving unit (22). The image receiving unit is connected to the network via the second line (T2), which is different from the first line, and receives the images.

19. A server (30) connected to a plurality of devices including a wearable device (10), a mobile terminal (20), and an assistive device terminal (40), The wearable device will be installed on the field worker and has a camera (11), a sound input (12), a sound output (13), and a communication unit (16). The communication unit is connected to the network through a first line (T1) to send images captured by the camera and to send and receive sound. The wearable device does not have an image display unit. The mobile terminal is separate from the wearable device and carried by the field operator. It has an image receiving unit (22) and a display unit (21). The image receiving unit is connected to the network via a second line (T2) different from the first line to receive images. The display unit displays the received images. The auxiliary personnel terminal is used by an assistant to remotely assist on-site workers. It has a sound input unit (41), a sound output unit (42), and a display unit (43). It transmits and receives sound from the wearable device via the first line (T1), and receives images from the wearable device and transmits these images to the mobile terminal via the second line (T2). Its features are, The server designates at least one of the plurality of devices according to specified conditions, and calls out the devices constituting the communication object according to the designation.

20. The server as described in claim 19, characterized in that, The wearable device also has a control unit (15) that controls the transmission of images, including the video captured by the camera unit, to the communication unit of the assistive personnel terminal.

Images