Instructor-side device, method, and program

Abstract

To desirably provide a technique capable of enabling an instructor to more effectively give instructions to a worker.SOLUTION: The instructor-side apparatus is provided with a recognition unit that recognizes a specific input pattern from first sensor data, and a switching signal output unit that outputs a switching signal between a still image and a moving image to a worker-side apparatus based on the recognition of the input pattern; and an image display control unit that controls display of the gesture on a display based on gesture information of an instructor and display of the still image or moving image transmitted from the worker-side apparatus based on the switching signal on the display.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an instructor-side device, method, and program. [Background technology]

[0002] Known conventional remote work support technologies include technologies that display instructions on a small display or a see-through display, but regardless of the technology, conventional remote work support technologies generally display instructions in a two-dimensional format.

[0003] Meanwhile, in recent years, with the development of technologies such as AR (Augmented Reality), work manuals that can be displayed in three dimensions have appeared. Such work manuals can be created in advance and displayed in three dimensions using AR technology. This makes it possible for workers to understand work in three dimensions. However, even when a work manual is displayed in three dimensions using AR technology, it is difficult to provide work support that is appropriate for the work situation at that time, because the work manual itself is created in advance.

[0004] On the other hand, in remote work support, it has already been shown that using both voice output input by an instructor and hand gesture display is effective (see, for example, Non-Patent Document 1). However, when gesture display is applied to AR technology as is, there are some points that need to be improved.

[0005] For example, even if gestures are displayed three-dimensionally using AR technology, if the gestures do not change as the worker moves and the worker can only view the gestures from the same viewpoint, it may be difficult for the worker to understand instructions given by the gestures.On the other hand, if the gestures constantly change as the worker moves, situations may frequently arise where the viewpoint from which the instructor views the gestures does not match the viewpoint from which the instructor views the gestures, making it difficult for the instructor to give instructions by gestures. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Shunsuke Ichihara, Yusuke Suzuki, "Development and Issues of a Remote Work Support System with Hand Gesture Transmission Function," Information Processing Society of Japan Interaction 2019, 2B-36 Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, it is desirable to provide a technique that enables an instructor to give instructions to an operator more effectively. [Means for solving the problem]

[0008] In order to solve the above problem, according to one aspect of the present invention, there is provided an instructor-side device including: a recognition unit that recognizes a specific input pattern from first sensor data; a switching signal output unit that outputs a switching signal between a still image and a moving image to the operator-side device based on the recognition of the input pattern; and an image display control unit that controls the display of a gesture on a display based on gesture information of the instructor, and controls the display of a still image or a moving image transmitted from the operator-side device based on the switching signal.

[0009] The first sensor data may include a gesture, and the specific input pattern may include a first gesture pattern.

[0010] The first sensor data may include audio, and the specific input pattern may include a specific audio pattern.

[0011] The recognition unit recognizes a second gesture pattern from the gesture of the instructor, and when the second gesture pattern is recognized, the switching signal output unit does not need to output the switching signal to the worker-side device even if the specific voice pattern is recognized.

[0012] The recognition unit recognizes a third gesture pattern from the gesture of the instructor, and the switching signal output unit may not output the switching signal to the worker-side device when the third gesture pattern is not recognized, even if the specific voice pattern is recognized.

[0013] The switching signal output unit may output a switching signal from a still image to a moving image to the worker-side device based on recognition of an input pattern indicating switching from a still image to a moving image.

[0014] The switching signal output unit may output a switching signal from a moving image to a still image to the worker-side device based on recognition of an input pattern indicating switching from a moving image to a still image.

[0015] When the worker's behavior recognized from the second sensor data indicates that the worker is moving, the switching signal output unit does not need to output a switching signal from a moving image to a still image to the worker-side device even if an input pattern indicating a switch from a moving image to a still image is recognized.

[0016] According to another aspect of the present invention, there is provided a method comprising: recognizing a specific input pattern from first sensor data; outputting a switching signal between a still image and a moving image to an operator-side device based on the recognition of the input pattern; controlling display of a gesture on a display based on gesture information of an instructor; and controlling display of a still image or a moving image transmitted from the operator-side device on the display based on the switching signal.

[0017] According to another aspect of the present invention, there is provided a program for causing a computer to function as an instructor-side device including: a recognition unit that recognizes a specific input pattern from first sensor data; a switching signal output unit that outputs a switching signal between a still image and a moving image to an operator-side device based on the recognition of the input pattern; and an image display control unit that controls the display of a gesture on a display based on gesture information of an instructor, and controls the display of a still image or a moving image transmitted from the operator-side device based on the switching signal. [Effects of the Invention]

[0018] As described above, the present invention provides a technique that enables an instructor to give instructions to an operator more effectively. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of a remote operation support system according to an embodiment of the present invention. [Figure 2] 10 is a flowchart illustrating an example of a switching operation from a still image state to a video state in the remote operation support system. [Figure 3] 10 is a flowchart illustrating an example of a switching operation from a video state to a still image state of the remote operation support system. [Figure 4] 10 is a diagram for explaining a coordinate system to which gesture information is converted according to Comparative Example 1. FIG. [Figure 5] FIG. 10 is a diagram for explaining a coordinate system into which gesture information is converted according to Comparative Example 2. [Figure 6] 10A and 10B are diagrams illustrating a coordinate system to which gesture information is converted according to an embodiment of the present invention. [Figure 7] 1 is a diagram illustrating a hardware configuration of an information processing device as an example of an instructor-side system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant explanations will be omitted.

[0021] Furthermore, in this specification and drawings, multiple components having substantially the same functional configuration may be distinguished by adding different numbers after the same reference numeral. However, if there is no particular need to distinguish between multiple components having substantially the same functional configuration, only the same reference numeral will be used. Furthermore, similar components in different embodiments may be distinguished by adding different letters after the same reference numeral. However, if there is no particular need to distinguish between similar components in different embodiments, only the same reference numeral will be used.

[0022] (1. Details of the embodiment) The details of the embodiment of the present invention will now be described.

[0023] (1-1. Configuration of the remote work support system) First, a configuration example of a remote work support system according to an embodiment of the present invention will be described. Fig. 1 is a diagram showing a functional configuration example of a remote work support system according to an embodiment of the present invention. As shown in Fig. 1, the remote work support system 1 has a worker-side system and an instructor-side system.

[0024] The worker-side system is a system used by a worker in the remote work support system 1. The worker performs work at a location (i.e., a remote location) away from the instructor who issues instructions to the worker. On the other hand, the instructor-side system is a system used by the instructor in the remote work support system 1. The worker-side system and the instructor-side system are connected to a network 30 and are configured to be able to communicate with each other via the network 30.

[0025] In remote work support, voice calls and the like are generally conducted between the instructor and the worker. However, the configuration necessary for voice calls and the like is not involved in the description of the configuration of the remote work support system 1 according to the embodiment of the present invention, and therefore a description of the configuration necessary for voice calls and the like will be omitted.

[0026] 1, the worker-side system includes an AR display 11, a camera 12, a position and orientation measurement unit 13, a processing unit 14, a speaker 15, and a voice processing unit 16. On the other hand, as shown in FIG. 1, the instructor-side system includes a display 21, a gesture input device 22, a processing unit 24, a microphone 25, a gesture recognition device 26, and a voice recognition device 27.

[0027] (AR Display 11) The AR display 11 displays the gesture. More specifically, the AR display 11 displays the gesture superimposed on the worker's field of view. The AR display 11 is worn by the worker. For example, the AR display 11 may be a head-mounted display worn on the worker's head. However, the type of the AR display 11 is not limited to a head-mounted display. For example, the AR display 11 may be a display other than a head-mounted display.

[0028] (Camera 12) The camera 12 captures moving images (hereinafter simply referred to as "video") of the worker's environment. The camera 12 is preferably provided so as to face the same direction as the worker's line of sight. Therefore, the camera 12 is preferably integrated with the AR display 11. However, the camera 12 may be configured as hardware separate from the AR display 11.

[0029] (Position and Orientation Measurement Unit 13) The position and orientation measurement unit 13 measures the position and orientation of the camera 12. For example, if a sensor is built into the camera 12, the position and orientation measurement unit 13 may measure the position and orientation of the camera 12 based on sensor data detected by the sensor built into the camera 12. The sensor may be an acceleration sensor or a gyro sensor, but the type of sensor is not particularly limited.

[0030] For example, the position and orientation measurement unit 13 may include a sensor that measures a two-dimensional marker installed in the worker's environment. In this case, the position and orientation measurement unit 13 may measure the position and orientation of the camera 12 based on the measured shape of the two-dimensional marker. Alternatively, the position and orientation measurement unit 13 may combine a method of measuring the position and orientation of the camera 12 based on sensor data detected by a sensor built into the camera 12 with a method of measuring the position and orientation of the camera 12 based on the measured shape of the two-dimensional marker.

[0031] (Calculation processing unit 14) The arithmetic processing unit 14 is realized by a computer and functions as an operator-side device that performs various arithmetic processing operations. For example, the arithmetic processing unit 14 can function as a gesture acquisition unit that acquires gesture information from the arithmetic processing unit 24, a signal acquisition unit that acquires a switching signal from the arithmetic processing unit 24, and a presentation control unit that controls the AR display 11 so that gestures are displayed in the field of view of the operator.

[0032] Furthermore, the arithmetic processing unit 14 performs processing such as controlling communication via a communication interface with the arithmetic processing unit 24 in the instructor system. For example, the arithmetic processing unit 14 can function as a transmission control unit that controls transmission of moving images or still images (hereinafter simply referred to as "still images") obtained by the camera 12 to the arithmetic processing unit 24.

[0033] For example, the arithmetic processing unit 14 includes an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and its functions can be realized by the arithmetic device expanding a program stored in a ROM (Read Only Memory) into a RAM and executing it. At this time, a computer-readable recording medium on which the program is recorded can also be provided.

[0034] (Speaker 15) The speaker 15 outputs the instruction voice input by the instructor without passing through the voice processing unit 16. Alternatively, the speaker 15 outputs the instruction voice input by the instructor through the voice processing unit 16. Note that the speaker 15 is configured as a stereo speaker or an array speaker so that the virtual output position of the instruction voice output from the speaker 15 can be controlled.

[0035] (Audio processing unit 16) The audio processing unit 16 acquires position data and audio data and performs processing to make the audio sound to the worker as if it were being output from that position. Typically, HRTF (non-patent document: Fundamentals of Head-Related Transfer Functions and Their Applications to 3D Sound Systems, edited by the Acoustical Society of Japan, written by Kazuhiro Iida, published by Corona Publishing) may be used for such processing. More specifically, the audio processing unit 16 acquires an instruction audio from the arithmetic processing unit 14 and controls the output of the acquired instruction audio from a virtual output position by the speaker 15. It is possible for the instruction audio to be output directly from the speaker 15 without being processed by the audio processing unit 16.

[0036] (Display 21) The display 21 can display moving images obtained by the camera 12. The display 21 can also display snapshots based on the moving images obtained by the camera 12 as still images. This allows the instructor to see the environment of the worker. Furthermore, the display 21 can display the instructor's gestures. This allows the instructor to confirm what gestures are being conveyed to the worker.

[0037] (Gesture input device 22) The gesture input device 22 is an input device that accepts gestures input by an instructor. The gesture input device 22 outputs the accepted gestures to the calculation processing unit 24. Furthermore, the gesture input device 22 outputs the accepted gestures to the gesture recognition device 26. The gesture input device 22 may be an example of a sensor. That is, the gesture input device 22 detects gestures as an example of sensor data (first sensor data).

[0038] For example, the gesture input device 22 may be realized by an optical device. An example of such an optical device is Leap Motion (registered trademark) developed by Ultraleap. Leap Motion (registered trademark) is a technology that tracks hand movements based on the detection results of infrared light irradiated onto the hand by a plurality of light emitting diodes (LEDs) using an infrared stereo camera. Alternatively, the gesture input device 22 may be realized by an input device (e.g., sensor gloves) worn by the instructor.

[0039] The gesture may be expressed in any format as long as it can be represented by a structure including multiple three-dimensional coordinates and can be reproduced in three-dimensional space. For example, the gesture may be represented by skeleton endpoint data or skin mesh data.

[0040] (Microphone 25) The microphone 25 receives voice input from an instructor. The microphone 25 outputs the received voice to the calculation processing unit 24 as an instruction voice. Furthermore, the microphone 25 outputs the received voice to the voice recognition device 27. The microphone 25 may be an example of a sensor. That is, the microphone 25 detects voice as an example of sensor data (first sensor data).

[0041] (Gesture Recognition Device 26) The gesture recognizer 26 performs gesture recognition on the gesture received by the gesture input device 22, and attempts to recognize a specific input pattern from the gesture. For example, the specific input pattern may be a predetermined gesture pattern (e.g., one or more predetermined gesture actions). For example, the predetermined gesture action may be one or more gesture actions indicating switching.

[0042] The gesture recognition device 26 is realized by a computer and functions as an instructor-side device. For example, the gesture recognition device 26 includes an arithmetic unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and its functions can be realized by the arithmetic unit expanding a program stored in a ROM (Read Only Memory) into a RAM and executing the program. In this case, a computer-readable recording medium on which the program is recorded can also be provided.

[0043] (Speech Recognition Device 27) The voice recognition device 27 performs voice recognition on the voice received by the microphone 25, and attempts to recognize a specific input pattern from the voice. For example, the specific input pattern may be a predetermined voice pattern (e.g., one or more predetermined phrases). For example, the predetermined phrase may be one or more phrases indicating a switch.

[0044] The speech recognition device 27 is realized by a computer and functions as an instructor-side device. For example, the speech recognition device 27 includes an arithmetic unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and its functions can be realized by the arithmetic unit expanding a program stored in a ROM (Read Only Memory) into a RAM and executing the program. In this case, a computer-readable recording medium on which the program is recorded can also be provided.

[0045] (arithmetic processing unit 24) The arithmetic processing unit 24 is realized by a computer and functions as an instructor-side device that performs various arithmetic processing. For example, the arithmetic processing unit 24 can function as a switching signal output unit that outputs a switching signal to the arithmetic processing unit 14 based on the recognition of a specific input pattern by a recognition unit (e.g., gesture recognition device 26 or voice recognition device 27).

[0046] The switching signal is a signal that indicates switching from one of two states to the other. In the embodiment of the present invention, it is mainly assumed that the switching signal indicates either a state in which a moving image obtained by camera 12 is displayed on display 21 (hereinafter also referred to as a "moving image state"), or a state in which a snapshot based on the moving image obtained by camera 12 is displayed as a still image on display 21 (hereinafter also referred to as a "still image state").

[0047] The arithmetic processing unit 24 can also function as an image display control unit that controls the display of gestures on the display 21 and also controls the display of still images or moving images on the display 21. The arithmetic processing unit 24 also performs processing such as controlling communication via a communication interface with the arithmetic processing unit 14 in the worker-side system.

[0048] For example, the arithmetic processing unit 24 includes an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and its functions can be realized by the arithmetic device expanding a program stored in a ROM (Read Only Memory) into a RAM and executing it. In this case, a computer-readable recording medium on which the program is recorded can also be provided.

[0049] (Network 30) The network 30 connects the worker-side system and the instructor-side system, and can function as a communication path between the worker-side system and the instructor-side system.

[0050] The above has described an example of the configuration of the remote operation support system 1 according to an embodiment of the present invention.

[0051] (1-2. Operation of the remote work support system) Next, an example of the operation of the remote operation support system 1 according to the embodiment of the present invention will be described with reference to FIGS.

[0052] Note that operations such as voice calls are not relevant to the description of the operation of the remote work support system 1 according to the embodiment of the present invention, and therefore a description of operations such as voice calls will be omitted. Furthermore, it is assumed that communication delays are negligibly small.

[0053] As described above, the remote work support system 1 can be in either a video state or a still image state. The state of the remote work support system 1 is switched based on a switching signal. In the video state, video captured by the camera 12 is displayed on the display 21. On the other hand, in the still image state, a snapshot of the video at the time of switching is displayed as a still image on the display 21.

[0054] Fig. 2 is a flowchart showing an example of a switching operation from a still image state to a video state in the remote work support system 1. Fig. 3 is a flowchart showing an example of a switching operation from a video state to a still image state in the remote work support system 1. First, an example of a switching operation from a still image state to a video state will be described mainly with reference to Fig. 2, and then an example of a switching operation from the video state to a still image state will be described mainly with reference to Fig. 3.

[0055] Here, the instructor-side system transmits to the worker-side system gesture information that has been input from the instructor by the gesture input device 22. The gesture information may include information indicating the displacement of the gesture from a reference position (displacement of each feature point of the hand).

[0056] In the worker-side system, the arithmetic processing unit 14 controls the AR display 11 so that a gesture is placed based on the reference position, the reference direction, and the gesture information. Meanwhile, in the instructor-side system, the arithmetic processing unit 24 controls the display 21 so that a gesture is displayed based on the gesture information.

[0057] Furthermore, in the instructor-side system, the instruction voice is received by the microphone 25 and output to the arithmetic processing unit 24. Then, the instruction voice is constantly transmitted from the arithmetic processing unit 24 to the arithmetic processing unit 14 in the worker-side system via the network 30. The arithmetic processing unit 14 constantly acquires the instruction voice.

[0058] (Switching from still image to video) In the still image state, the reference position and reference direction in which a gesture is placed do not follow the position and direction of the camera 12. That is, in the still image state, the reference position and reference direction in which a gesture is placed in the worker-side system are fixed to the position and direction of the camera 12 recorded at the time of switching from the moving image state to the still image state.

[0059] In the instructor system, the gesture recognition device 26 attempts to recognize a first gesture pattern (a gesture pattern indicating a switch from a still image state to a video state) from the instructor's gesture received by the gesture input device 22. The specific gesture pattern may be any specific gesture pattern. Note that the gesture pattern indicating a switch from a still image state to a video state and the gesture pattern indicating a switch from a video state to a still image state may be the same, but it is preferable that they are different in order to separately recognize the two switch instructions.

[0060] Alternatively, the voice recognition device 27 attempts to recognize a specific voice pattern (a voice pattern indicating a switch from a still image state to a video state) from the voice of the instructor received by the microphone 25. The specific voice pattern may be any specific voice pattern. Note that the voice pattern indicating a switch from a still image state to a video state and the voice pattern indicating a switch from a video state to a still image state may be the same, but it is preferable that they are different in order to separately recognize the two switch instructions.

[0061] The arithmetic processing unit 24 determines whether an instruction to switch from a still image state to a video state has been input from the instructor, depending on whether a first gesture pattern has been recognized by the gesture recognition device 26 (S11). Alternatively, the arithmetic processing unit 24 determines whether an instruction to switch from a still image state to a video state has been input from the instructor, depending on whether a specific voice pattern has been recognized by the voice recognition device 27.

[0062] 2, if an instruction to switch from the still image state to the video state has not been input ("NO" in S11), the arithmetic processing unit 24 proceeds to operation S11. On the other hand, if an instruction to switch from the still image state to the video state has been input ("YES" in S11), the arithmetic processing unit 24 notifies the arithmetic processing unit 14 in the worker-side system of a signal to switch from the still image state to the video state via the network 30 (S12).

[0063] In the operator-side system, when the calculation processing unit 14 receives a signal to switch from a still image state to a video state, it causes the reference position where the gesture is placed by the AR display 11 to follow the position of the camera 12 measured by the position and orientation measurement unit 13, and causes the reference direction where the gesture is placed by the AR display 11 to follow the direction of the camera 12 measured by the position and orientation measurement unit 13.

[0064] Furthermore, the calculation processing unit 14 disables the voice processing unit 16 (that is, stops the operation of the voice processing unit 16), and causes the acquired instruction voice to be output as is from the speaker 15 (S31).

[0065] Furthermore, if there is a still image being displayed on display 21, processing unit 24 erases the still image (S13). Then, when a moving image obtained by camera 12 is transmitted from processing unit 14, processing unit 24 controls display 21 so that the display of the transmitted moving image on display 21 resumes (S14).

[0066] (Switching from video to still image) In the animated state, the reference position and orientation in which the gestures are placed follows the position and orientation of the camera 12 .

[0067] In the instructor system, the gesture recognition device 26 attempts to recognize a first gesture pattern (a gesture pattern indicating a switch from a moving image state to a still image state) from the instructor's gesture received by the gesture input device 22. As described above, the gesture pattern indicating a switch from a still image state to a moving image state and the gesture pattern indicating a switch from a moving image state to a still image state may be the same, but it is preferable that they are different in order to separately recognize the two switch instructions.

[0068] Alternatively, the voice recognition device 27 attempts to recognize a specific voice pattern (a voice pattern indicating a switch from a moving image state to a still image state) from the voice of the instructor received by the microphone 25. As described above, the voice pattern indicating a switch from a moving image state to a still image state and the voice pattern indicating a switch from a still image state to a moving image state may be the same, but it is preferable that they are different in order to separately recognize the two switching instructions.

[0069] The arithmetic processing unit 24 determines whether an instruction to switch from a moving image state to a still image state has been input from the instructor, depending on whether a first gesture pattern has been recognized by the gesture recognition device 26 (S21). Alternatively, the arithmetic processing unit 24 determines whether an instruction to switch from a moving image state to a still image state has been input from the instructor, depending on whether a specific voice pattern has been recognized by the voice recognition device 27.

[0070] If an instruction to switch from the moving image state to the still image state has not been input ("NO" in S21), the arithmetic processing unit 24 proceeds to operation S21. On the other hand, if an instruction to switch from the moving image state to the still image state has been input ("YES" in S21), the arithmetic processing unit 24 notifies the arithmetic processing unit 14 in the worker-side system of a signal to switch from the still image state to the moving image state via the network 30 (S22).

[0071] In the operator-side system, when the calculation processing unit 14 receives a signal to switch from the still image state to the video state, it records the position and direction of the camera 12 at the time of switching from the still image state to the video state (S26), and fixes the reference position and reference direction used by the AR display 11 to place gestures to the position and direction of the camera 12 at the time of switching from the video state to the still image state.

[0072] Furthermore, the calculation processing unit 14 enables the audio processing unit 16 (i.e., starts the operation of the audio processing unit 16) (S41). As a result, the audio processing unit 16 processes the instruction audio (for example, processing using HRTF, etc.) to output the instruction audio from the speaker 15 as if the instruction audio is being output from the reference position where the gesture is placed.

[0073] Furthermore, the processing unit 24 creates a snapshot of the moving image captured by the camera 12 as a still image (S23).

[0074] If there is a moving image being displayed on the display 21, the calculation processing unit 24 erases the moving image (S24). Then, the calculation processing unit 24 controls the display 21 so that the display 21 resumes displaying the generated still image (S25).

[0075] The above describes an example of switching from a still image state to a video state and from a video state to a still image state. Next, the reference position and reference direction in which a gesture is placed will be described in more detail.

[0076] (1-3. Reference position and reference direction where gestures are placed) The gesture information input from the instructor received by the gesture input device 22 is information expressed by a position, a direction, etc. in a coordinate system specific to the instructor's environment. In order to display a gesture on the AR display 11 in the worker's environment based on such gesture information, it is necessary to perform coordinate conversion of the gesture information from the instructor's coordinate system (source coordinate system) to the worker's coordinate system (destination coordinate system).

[0077] Here, as examples of the coordinate system to be converted, Comparative Example 1 (world coordinate system), Comparative Example 2 (camera coordinate system), and the coordinate system according to this embodiment will be described in order. For simplicity of explanation, it is assumed that the position of the gesture is fixed.

[0078] (Comparative Example 1 (World Coordinate System)) Fig. 4 is a diagram for explaining a coordinate system to which gesture information is converted according to Comparative Example 1. Referring to Fig. 4, an operator environment E11 in an initial state is shown. A work object R1 exists in the operator environment E11 in the initial state. The work object R1 is an object that is the target of work by the operator.

[0079] Here, for ease of viewing, the work object R1 is shown as a square plane having a "front" and a "back". However, the shape of the work object R1 is not limited. Typically, the work object R1 is assumed to be a machine such as an ATM (Automated Teller Machine) or a printer, but the work object R1 does not have to be a machine and can be any object used for work.

[0080] Furthermore, a camera 12 is present in the worker environment E11 in the initial state, and a video captured by the camera 12 is displayed on the instructor's screen 121a. The video displayed on the instructor's screen 121a shows a work object R1. The instructor inputs gesture information while viewing the video of the worker environment E11 in the initial state. A gesture J1 is displayed on the instructor's screen 121a based on the gesture information.

[0081] Referring to the worker environment E11 in the initial state, a world coordinate system C1 and a camera coordinate system C2 are shown. Based on gesture information input by the instructor, the AR display on the worker's side is controlled so that a gesture J1 is placed in the world coordinate system C1. The worker can perform work on a work object R1 while viewing the gesture J1 placed in the world coordinate system C1.

[0082] Now, assume that the worker moves behind the work object R1 as shown in the worker environment E12, and the position and direction of the gesture J1 in the world coordinate system C1 are fixed.

[0083] At this time, the "back" of the work object R1 and the gesture J1 are displayed on the instructor's screen 122a. However, the position and direction of the gesture J1 also change according to the amount and direction of movement of the worker. Therefore, the instructor needs to input the gesture before the position and direction change (before the 180-degree rotation) compared to the displayed gesture J1, which makes it difficult for the instructor to input the gesture.

[0084] On the other hand, suppose that the worker moves to the rear side of the work object R1 as shown in the worker environment E13, and the position and direction of the gesture J1 in the world coordinate system C1 are fixed.

[0085] At this time, since the position and direction of the gesture J1 in the world coordinate system C1 are fixed, the gesture J1 is located behind the worker, and the worker cannot see the gesture J1. For example, even if the instructor inputs gesture information for moving to the next work position, the worker cannot see the gesture and therefore cannot know the next work position.

[0086] For the above reasons, Comparative Example 1 is suitable for displaying a pre-created work manual, etc. However, in Comparative Example 1, gestures by the instructor may become invisible as the worker moves, so it is not suitable for situations where work instructions need to be given in real time.

[0087] The coordinate system into which gesture information is converted according to the first comparative example has been described above.

[0088] (Comparative Example 2 (camera coordinate system)) Fig. 5 is a diagram for explaining a coordinate system to which gesture information is converted according to Comparative Example 2. Referring to Fig. 5, a worker environment E21 in an initial state is shown, similar to the example shown in Fig. 4. A work object R1 exists in the worker environment E21 in the initial state.

[0089] Furthermore, a video captured by the camera 12 is displayed on the instructor's screen 123a. The video displayed on the instructor's screen 123a shows the work object R1. The instructor inputs gesture information while viewing the video of the worker environment E11 in the initial state. A gesture J1 is displayed on the instructor's screen 123a based on the gesture information.

[0090] In Comparative Example 2, the AR display on the worker's side is controlled based on gesture information input by the instructor so that gesture J1 is placed in camera coordinate system C2. The worker can perform work on work object R1 while viewing gesture J1 placed in camera coordinate system C2.

[0091] Now, assume that the worker moves behind the work target R1 as shown in the worker environment E22. Unlike Comparative Example 1, the position and direction of the gesture J1 in the world coordinate system C1 change, but the position and direction of the gesture J1 in the camera coordinate system C2 are fixed.

[0092] At this time, the "back" of the work object R1 and the gesture J1 are displayed on the instructor's screen 124a. At this time, even though the worker is moving, the position and direction of the gesture J1 displayed on the screen 124a are fixed. Therefore, the instructor only needs to input the gesture at the same position and direction as the displayed gesture J1, making it easy for the instructor to input the gesture.

[0093] However, since the position and direction of the gesture J1 in the camera coordinate system C2 are fixed, the position and direction of the gesture J1 as seen by the worker are constant, which means that the worker cannot change the position or angle at which the gesture J1 is seen (for example, the worker cannot see the gesture J1 from above).

[0094] On the other hand, as shown in the worker environment E23, suppose the worker moves to the rear side of the work object R1. At this time, the position and direction of the gesture J1 in the world coordinate system C1 change, but the position and direction of the gesture J1 in the camera coordinate system C2 are fixed. At this time, the position and direction of the gesture J1 displayed on the instructor's screen 125a are fixed.

[0095] Furthermore, since the position of the gesture J1 in the camera coordinate system C2 is fixed, the gesture J1 will not remain in front of the worker and will not become invisible to the worker. For example, when an instructor inputs gesture information for moving to the next work position, the worker can see the gesture and know the next work position.

[0096] As described above, Comparative Example 2 is suitable for cases where work instructions need to be given in real time, since gestures by an instructor do not become invisible as the worker moves. However, in Comparative Example 2, the position and angle of the gesture visible to the worker do not change, which can be said to impair the advantage of displaying gestures using AR technology.

[0097] The coordinate system into which gesture information is converted according to Comparative Example 2 has been described above.

[0098] (Coordinate system according to this embodiment) Fig. 6 is a diagram for explaining a coordinate system to which gesture information is converted according to an embodiment of the present invention. Referring to Fig. 6, similar to the example shown in Fig. 4, a worker environment E31 in an initial state is shown. A work object R1 exists in the worker environment E31 in the initial state. In the initial state, it is assumed that the remote work support system 1 is in a video state. As an example, in the video state, an instructor instructs the worker on a work position by gesture.

[0099] The instruction voice received by the microphone 25 is constantly transmitted from the arithmetic processing unit 24 to the arithmetic processing unit 14 via the network 30 .

[0100] When the remote operation support system 1 is in a video state, the calculation processing unit 24 controls the video captured by the camera 12 to be displayed on the instructor's screen 126a. The video displayed on the instructor's screen 126a shows the work object R1. The instructor inputs gesture information while viewing the video of the worker environment E31 in the initial state. A gesture J1 is displayed on the instructor's screen 126a based on the gesture information.

[0101] The arithmetic processing unit 14 acquires gesture information from the arithmetic processing unit 24. When the remote work support system 1 is in a video state, the arithmetic processing unit 14 employs a method (second method) of controlling the AR display 11 so that the gesture J1 is placed in a camera coordinate system C2 (second coordinate system) in which the position and direction of the camera 12 are used as a reference position and reference direction (second reference position and second reference direction). The worker can move towards the work position while looking at the gesture J1 placed in the camera coordinate system C2.

[0102] Furthermore, when the remote work support system 1 is in a video state, the viewpoint of the worker and the viewpoint of the instructor are the same. Therefore, when the remote work support system 1 is in a video state, as shown in the worker environment E31, the arithmetic processing unit 14 outputs the instruction voice received from the arithmetic processing unit 24 in the instructor system directly from the speaker 15 without causing the voice processing unit 16 to process it.

[0103] As an example, assume that the instructor finishes instructing the work position and starts instructing the work content for work object R1 by gesture. At this time, the instructor inputs a switching instruction indicating a switch from the moving image state to the still image state (i.e., a switch from the second technique to the first technique) by a gesture pattern or a voice pattern. When this switching instruction is input, a switching signal indicating a switch from the moving image state to the still image state is notified from arithmetic processing unit 24 to arithmetic processing unit 14.

[0104] When the calculation processing unit 14 acquires a switching signal indicating a switch from the moving image state to the still image state, the calculation processing unit 14 records the position and direction of the camera 12 at the time when the switching signal indicating a switch from the moving image state to the still image state was acquired as the reference position C3 and the reference direction (first reference position and first reference direction) used for arranging the gesture based on the acquisition of the switching signal. Note that the time when the switching signal is acquired is an example of the predetermined time.

[0105] As shown in the worker environment E32, it is assumed that the worker has moved to the rear of the work object R1. The position and orientation measurement unit 13 measures the amount and direction of movement of the worker based on the reference position and reference direction recorded by the calculation processing unit 14.

[0106] The calculation processing unit 14 employs a method (first method) of controlling the AR display 11 so that the gesture J1 is placed in a coordinate system (first coordinate system) based on the recorded reference position and reference direction, based on the movement amount and movement direction of the worker measured by the position and orientation measurement unit 13. In other words, the calculation processing unit 14 places the gesture J1 at a reference position and reference direction in the AR space.

[0107] At this time, as shown on the screen 127a, the position and direction of the gesture J1 in the world coordinate system C1 are fixed, while the gesture J1 is displayed on the AR display 11 at a position and direction that is seen from a position and direction that is moved and rotated by the worker's movement amount and movement direction from the recorded reference position and reference direction. Note that the "back" of the work object R1 that appears on the screen 127a may be a through image or the actual object.

[0108] The calculation processing unit 14 determines a virtual output position and a virtual output direction of the instruction voice in a coordinate system based on the recorded reference position and reference direction, based on the movement amount and movement direction of the worker measured by the position and orientation measurement unit 13. In other words, the calculation processing unit 14 determines the reference position and reference direction in the AR space as the virtual output position and output direction of the instruction voice.

[0109] The audio processor 16 controls the output of the instruction voice from the speaker 15 as if the instruction voice were being output in a virtual output direction from a virtual output position determined by the calculation processor 14 (as if the instruction voice were being output from a virtual speaker K1). This virtual output position may correspond to the viewpoint of the instructor. Therefore, the worker can recognize the viewpoint of the instructor from the instruction voice, and can understand the instructions while experiencing a sense of realism.

[0110] At this time, as shown in the worker environment E32, the reference position and reference direction are fixed to the world coordinate system C1, and the instruction sound is output as if it were being output from a reference position and reference direction that can be seen from a position and direction that is moved and rotated by the worker's movement amount and movement direction from the reference position and reference direction.

[0111] Meanwhile, in the instructor system, the arithmetic processing unit 24 generates a still image, which is a snapshot of the moving image at the timing when a switching signal indicating switching from a moving image state to a still image state is input. Then, as shown on screen 127b, the arithmetic processing unit 24 controls the display 21 to display the generated still image. On screen 127b on the instructor side, gesture J1 is displayed based on the gesture information.

[0112] At this time, even though the worker is moving, the position and direction of the gesture J1 displayed on the screen 127b are fixed. Therefore, the instructor can easily input the gesture because the instructor only needs to input the gesture in the same position and direction as the displayed gesture J1.

[0113] Furthermore, the position and direction of the gesture J1 that can be seen by the worker may change as the worker moves, so that the worker can change the position or angle at which the gesture J1 can be seen (for example, the worker can see the gesture J1 from above).

[0114] On the other hand, as shown in the worker environment E33, suppose the worker moves to the rear side of the work object R1. At this time, the position and direction of the gesture J1 in the world coordinate system C1 change, but the position and direction of the gesture J1 in the camera coordinate system C2 are fixed. At this time, the position and direction of the gesture J1 displayed on the screen 128a on the instructor side are fixed.

[0115] Therefore, the gesture J1 does not remain in front of the worker and become invisible to the worker, as in Comparative Example 2. For example, when an instructor inputs gesture information for moving to the next work position, the worker can see the gesture and know the next work position.

[0116] Although not shown in Fig. 6, it is assumed that the instructor finishes instructing the work content of work object R1 by gesture and starts instructing the next work position. At this time, the instructor inputs a switching instruction indicating a switch from the still image state to the video state (i.e., a switch from the first technique to the second technique) by a gesture pattern or a voice pattern. When such a switching instruction is input, a switching signal indicating a switch from the still image state to the video state is notified from the arithmetic processing unit 24 to the arithmetic processing unit 14.

[0117] When the calculation processing unit 14 receives a switching signal indicating a switch from a still image state to a video state, the calculation processing unit 14 controls the AR display 11 based on the reception of the switching signal so that the gesture J1 is placed in a camera coordinate system C2 whose reference position and reference direction are the position and direction of the camera 12. The worker can move towards the work position while viewing the gesture J1 placed in the camera coordinate system C2.

[0118] Furthermore, when the remote work support system 1 is in a video state, the viewpoint of the worker and the viewpoint of the instructor are the same. Therefore, when the remote work support system 1 is in a video state, the processing unit 14 outputs the instruction voice received from the processing unit 24 in the instructor system directly from the speaker 15 without processing it in the voice processing unit 16, so as to be in the worker environment E31.

[0119] The reference position and reference direction in which a gesture is placed have been described in detail above.

[0120] (1-4. Effects) As described above, according to the embodiment of the present invention, the position and angle of the gesture as seen by the worker can be changed, so the advantage of displaying the gesture using AR technology is not lost. Furthermore, according to the embodiment of the present invention, the gesture by the instructor does not become invisible as the worker moves. Therefore, the technology according to the embodiment of the present invention is suitable for cases where work instructions need to be given in real time.

[0121] Furthermore, according to an embodiment of the present invention, when the remote work support system 1 is in a still image state, the virtual output position of the instruction voice is controlled to be the viewpoint of the instructor, so that the worker can grasp the viewpoint of the instructor through the instruction voice.

[0122] Furthermore, according to an embodiment of the present invention, a command to switch between a moving image state and a still image state can be given by voice or gesture. Therefore, when an instructor wants to give a command to switch between a moving image state and a still image state, there is no need to reach for a switch switch. This eliminates the need for the instructor's gesture command to be interrupted each time a switch command is given (or it is possible to shorten the time during which the instructor's gesture command is interrupted).

[0123] The effects achieved by the remote operation support system 1 according to the embodiment of the present invention have been described above.

[0124] (2. Hardware configuration example) Next, an example of the hardware configuration of the worker-side system according to the embodiment of the present invention will be described.

[0125] Below, an example of the hardware configuration of an information processing device 900 will be described as an example of the hardware configuration of an instructor-side system according to an embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of an instructor-side system. Therefore, the hardware configuration of the instructor-side system may be such that unnecessary components are deleted from the hardware configuration of the information processing device 900 described below, or new components are added. Note that the hardware configuration of the worker-side system may also be realized in the same way as the hardware configuration of the instructor-side system.

[0126] 7 is a diagram showing the hardware configuration of an information processing device 900 as an example of an instructor-side system according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0127] The CPU 901 functions as an arithmetic processing unit and control unit, and controls the overall operation of the information processing device 900 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, etc. These are interconnected by a host bus 904 that is composed of a CPU bus, etc.

[0128] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not necessary to configure the host bus 904, bridge 905, and external bus 906 separately, and these functions may be implemented on a single bus.

[0129] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers that allow the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. By operating this input device 908, the user operating the information processing device 900 can input various data to the information processing device 900 and instruct the information processing device 900 to perform processing operations.

[0130] The output device 909 includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a lamp, and an audio output device such as a speaker.

[0131] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 910 is configured, for example, with an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

[0132] The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network, etc. The communication device 911 may be compatible with either wireless communication or wired communication.

[0133] An example of the hardware configuration of the instructor-side system according to the embodiment of the present invention has been described above.

[0134] (3. Summary) Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications and alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0135] For example, in the above description, there is one instructor and one worker. However, there may be multiple instructors. In such a case, gestures by each of the multiple instructors may be handled. Alternatively, there may be multiple workers. In such a case, a camera corresponding to a camera coordinate system in which the gesture is arranged may be designated, and the gesture may be presented to the worker of the designated camera.

[0136] In the above, the case where the reference position and the reference direction where a gesture is placed are switched based on a switching signal has been mainly described. However, the reference position and the reference direction where an object other than a gesture (for example, an instruction note) is placed may also be switched based on a switching signal in the same manner as the reference position and the reference direction where a gesture is placed.

[0137] The above description mainly deals with a case where, when a switching signal indicates switching from a moving image to a still image, the processing unit 24 in the instructor system creates a snapshot from the moving image and controls the display 21 to display the created snapshot as a still image. However, the processing unit 24 may also display the moving image together with the still image on the display 21. In this case, the still image and the moving image may be displayed side by side.

[0138] Alternatively, in the still image state, the still image and the video do not need to be displayed side by side. In this case, the process of creating a snapshot from the video may be performed by the processor 14 in the worker system instead of the processor 24 in the instructor system. In such a case, the camera 12 in the worker system may stop capturing video, and the transmission of video from the worker system to the instructor system may be stopped. This may reduce the power consumption of the camera 12 and the communication bandwidth.

[0139] In the above description, the reference position where the virtual instruction sound is placed is mainly the position of the camera 12 at the time when a switching signal indicating switching from a moving image state to a still image state is acquired. However, the reference position where the virtual instruction sound is placed may be a position shifted (offset position) from the position of the camera 12 at the time when the switching signal is acquired.

[0140] In the above description, it is mainly assumed that the gestures of the instructor and the voice of the instructor are used separately for switching between the still image state and the video state, but a combination of the gestures of the instructor and the voice of the instructor may be used for switching between the still image state and the video state.

[0141] For example, the gesture recognition device 26 may attempt to recognize a gesture pattern (second gesture pattern) indicating prohibition of switching from the gesture of the instructor received by the gesture input device 22. The gesture pattern indicating prohibition of switching may be any specific gesture pattern.

[0142] When a gesture pattern indicating prohibition of switching is recognized, the arithmetic processing unit 24 may not output a switching signal to the arithmetic processing unit 14 even if a specific voice pattern is recognized by the voice recognition device 27. In other words, when a specific voice pattern is recognized by the voice recognition device 27, the arithmetic processing unit 24 may output a switching signal to the arithmetic processing unit 14 only when a gesture pattern indicating prohibition of switching is not recognized.

[0143] Alternatively, the gesture recognition device 26 may attempt to recognize a gesture pattern (third gesture pattern) indicating the execution of switching from the gesture of the instructor accepted by the gesture input device 22. The gesture pattern indicating the execution of switching may be any specific gesture pattern.

[0144] When a gesture pattern indicating the execution of switching is not recognized, the arithmetic processing unit 24 may not output a switching signal to the arithmetic processing unit 14 even if a specific voice pattern is recognized by the voice recognition device 27. In other words, when a specific voice pattern is recognized by the voice recognition device 27, the arithmetic processing unit 24 may output a switching signal to the arithmetic processing unit 14 only when a gesture pattern indicating the execution of switching is recognized.

[0145] In the above, it is mainly assumed that when the gesture recognition device 26 recognizes a first gesture pattern or when the voice recognition device 27 recognizes a specific voice pattern, the arithmetic processing unit 24 outputs a switching signal between a still image and a moving image to the arithmetic processing unit 14. However, whether or not to output a switching signal to the arithmetic processing unit 14 may be determined taking additional conditions into consideration.

[0146] For example, a sensor (e.g., an acceleration sensor, a vibration sensor, a gyro sensor, etc.) that detects the state of the worker and a behavior recognition device that recognizes the worker's behavior from the sensor data (second sensor data) may be provided. For example, the sensor may be attached to the worker's body (e.g., attached to the AR display 11). The behavior recognition device may be provided in the worker-side system, the instructor-side system, or external to the worker-side system and the instructor-side system.

[0147] For example, if the behavior of the worker recognized from the sensor data indicates that the worker is moving (e.g., walking), it is assumed that the worker is not performing work. Therefore, in such a case, even if the input pattern indicating switching from a moving image to a still image is recognized by the gesture recognition device 26 or the voice recognition device 27, the arithmetic processing unit 24 does not need to output a signal for switching from a moving image to a still image to the arithmetic processing unit 14.

[0148] In the above description, it is mainly assumed that the gesture recognition device 26 and the voice recognition device 27 are provided exclusively for switching between the video state and the still image state. However, a gesture recognition device and a voice recognition device provided for realizing a function other than switching between the video state and the still image state (for example, disconnecting communication, changing the volume, etc.) may be used as the gesture recognition device 26 and the voice recognition device 27 for switching between the video state and the still image state. [Explanation of symbols]

[0149] 1. Remote work support system 11 AR Display 12 Camera 13 Position and Orientation Measurement Unit 14 Processing unit 15 Speaker 16 Audio processing unit 21 Display 22 Gesture input device 24 Processing unit 25 microphones 26 Gesture Recognition Device 27 Voice recognition device 30 Network

Claims

1. a recognition unit that recognizes a specific input pattern from the first sensor data; a switching signal output unit that outputs a switching signal between a still image and a moving image to the operator-side device based on the recognition of the input pattern; an image display control unit that controls display of a gesture on a display based on gesture information of an instructor, and controls display of a still image or a moving image transmitted from the operator-side device on the display based on the switching signal; Equipped with the first sensor data includes audio; the specific input pattern includes a specific voice pattern; the recognition unit recognizes a second gesture pattern from a gesture of an instructor; the switching signal output unit does not output the switching signal to the worker-side device when the second gesture pattern is recognized, even if the specific voice pattern is recognized. Instructor side device.

2. a recognition unit that recognizes a specific input pattern from the first sensor data; a switching signal output unit that outputs a switching signal between a still image and a moving image to the operator-side device based on the recognition of the input pattern; an image display control unit that controls display of a gesture on a display based on gesture information of an instructor, and controls display of a still image or a moving image transmitted from the operator-side device on the display based on the switching signal; Equipped with the first sensor data includes audio; the specific input pattern includes a specific voice pattern; the recognition unit recognizes a third gesture pattern from a gesture of an instructor; the switching signal output unit does not output the switching signal to the worker-side device when the third gesture pattern is not recognized, even if the specific voice pattern is recognized. Instructor side device.

3. the switching signal output unit outputs a switching signal from the moving image to the still image to the worker-side device based on recognition of an input pattern indicating switching from the moving image to the still image.

3. The instructor side device according to claim 1 or 2.

4. a recognition unit that recognizes a specific input pattern from the first sensor data; a switching signal output unit that outputs a switching signal between a still image and a moving image to the operator-side device based on the recognition of the input pattern; an image display control unit that controls display of a gesture on a display based on gesture information of an instructor, and controls display of a still image or a moving image transmitted from the operator-side device on the display based on the switching signal; Equipped with The switching signal output unit outputting a signal for switching from a moving image to a still image to the operator-side device based on recognition of an input pattern indicating switching from a moving image to a still image; when the worker's behavior recognized from the second sensor data indicates that the worker is moving, even if an input pattern indicating switching from a moving image to a still image is recognized, a switching signal from a moving image to a still image is not output to the worker-side device. Instructor side device.

5. the first sensor data includes audio; the specific input pattern includes a specific voice pattern; 5. The instructor side device according to claim 4.

6. the first sensor data includes a gesture; the specific input pattern includes a first gesture pattern; The instructor side device according to any one of claims 1 to 5.

7. the switching signal output unit outputs a switching signal from a still image to a moving image to the worker-side device based on recognition of an input pattern indicating switching from a still image to a moving image. The instructor side device according to any one of claims 1 to 6.

8. Recognizing a particular input pattern from the first sensor data; outputting a signal for switching between a still image and a moving image to an operator-side device based on the recognition of the input pattern; controlling display of the gesture on the display based on gesture information of the instructor, and controlling display of the still image or the moving image transmitted from the operator-side device on the display based on the switching signal; Equipped with the first sensor data includes audio; the specific input pattern includes a specific voice pattern; Recognizing the specific input pattern includes recognizing a second gesture pattern from a gesture of an instructor; outputting the switching signal to the worker-side device includes not outputting the switching signal to the worker-side device when the second gesture pattern is recognized, even if the specific voice pattern is recognized. method.

9. Recognizing a particular input pattern from the first sensor data; outputting a signal for switching between a still image and a moving image to an operator-side device based on the recognition of the input pattern; controlling display of the gesture on the display based on gesture information of the instructor, and controlling display of the still image or the moving image transmitted from the operator-side device on the display based on the switching signal; Equipped with the first sensor data includes audio; the specific input pattern includes a specific voice pattern; recognizing the specific input pattern includes recognizing a third gesture pattern from a gesture of an instructor; outputting the switching signal to the worker-side device includes not outputting the switching signal to the worker-side device when the third gesture pattern is not recognized, even if the specific voice pattern is recognized. method.

10. Recognizing a particular input pattern from the first sensor data; outputting a signal for switching between a still image and a moving image to an operator-side device based on the recognition of the input pattern; controlling display of the gesture on the display based on gesture information of the instructor, and controlling display of the still image or the moving image transmitted from the operator-side device on the display based on the switching signal; Equipped with Outputting the switching signal to the operator-side device outputting a signal for switching from a moving image to a still image to the operator-side device based on recognition of an input pattern indicating switching from a moving image to a still image; when the behavior of the worker recognized from the second sensor data indicates that the worker is moving, even if an input pattern indicating switching from a moving image to a still image is recognized, not output a switching signal from a moving image to a still image to the worker-side device; A method comprising:

11. Computer, a recognition unit that recognizes a specific input pattern from the first sensor data; a switching signal output unit that outputs a switching signal between a still image and a moving image to the operator-side device based on the recognition of the input pattern; an image display control unit that controls display of a gesture on a display based on gesture information of an instructor, and controls display of a still image or a moving image transmitted from the operator-side device on the display based on the switching signal; Equipped with the first sensor data includes audio; the specific input pattern includes a specific voice pattern; the recognition unit recognizes a second gesture pattern from a gesture of an instructor; the switching signal output unit does not output the switching signal to the worker-side device when the second gesture pattern is recognized, even if the specific voice pattern is recognized. A program that functions as the instructor's device.

12. Computer, a recognition unit that recognizes a specific input pattern from the first sensor data; a switching signal output unit that outputs a switching signal between a still image and a moving image to the operator-side device based on the recognition of the input pattern; an image display control unit that controls display of a gesture on a display based on gesture information of an instructor, and controls display of a still image or a moving image transmitted from the operator-side device on the display based on the switching signal; Equipped with the first sensor data includes audio; the specific input pattern includes a specific voice pattern; the recognition unit recognizes a third gesture pattern from a gesture of an instructor; the switching signal output unit does not output the switching signal to the worker-side device when the third gesture pattern is not recognized, even if the specific voice pattern is recognized. A program that functions as the instructor's device.

13. Computer, a recognition unit that recognizes a specific input pattern from the first sensor data; a switching signal output unit that outputs a switching signal between a still image and a moving image to the operator-side device based on the recognition of the input pattern; an image display control unit that controls display of a gesture on a display based on gesture information of an instructor, and controls display of a still image or a moving image transmitted from the operator-side device on the display based on the switching signal; Equipped with The switching signal output unit outputting a signal for switching from a moving image to a still image to the operator-side device based on recognition of an input pattern indicating switching from a moving image to a still image; when the worker's behavior recognized from the second sensor data indicates that the worker is moving, even if an input pattern indicating switching from a moving image to a still image is recognized, a switching signal from a moving image to a still image is not output to the worker-side device. A program that functions as the instructor's device.

Images