Remote control system and remote operation device, image processing device and program thereof

Abstract

VR motion sickness is reduced without complicated image processing. A remote working device having a movable part, the movable part can perform an action and change its posture according to a control signal sent from a user device connected to a display part via a network, the remote working device comprising: an image data acquisition unit that acquires image data obtained by a camera provided on the movable part; a motion detection data acquisition unit that acquires motion detection data indicating changes in the posture of the movable part from a motion sensor; and a data transmission unit that transmits the acquired image data and motion detection data to the user device via the network.

Description

Technical Field

[0001] Embodiments of the present invention relate to, for example, a remote control system in which a user device controls a remote working device located at a remote location via a network, and a remote working device, an image processing device, and a program used in the system. Background Art

[0002] A system has been developed in which a user remotely controls a remote working device via a network while viewing an image displayed on a display device. This system, for example, includes a head-mounted display (HMD) (hereinafter referred to as a head-mounted kit) mounted on the user's head. The system is configured such that the user remotely controls the actions of the robot by performing actions with his or her body or operating a controller while viewing an image captured by a remote robot using the head-mounted kit.

[0003] However, regarding this system, for example, using augmented reality (AR) technology or virtual reality (VR) technology, AR images or VR images are generated based on images transmitted from the robot and displayed on a head-mounted kit. In addition, at this time, the user performs so-called tracking control to change the displayed image by following the movements of his own body or the operation of the controller. As a result, the user can obtain a high degree of immersion during the operation.

[0004] However, there is a scanning delay (Latency) from the generation of AR images or VR images until they are displayed. The scanning delay (Latency) includes the transmission delay when the image generated by a remote robot is transmitted to the device on the head-mounted kit side via the network. Moreover, the transmission delay is not constant. For example, the delay amount varies due to the complexity of the network and the load changes of the processors of each device. Therefore, after the user makes the body perform an action or operates the controller, a time deviation occurs until the corresponding image is displayed on the head-mounted kit, which sometimes causes the user to feel dizzy, so-called "VR sickness (called Virtual Reality Sickness)". This VR sickness also occurs when the delay amount is constant, and it is particularly obvious in the case of so-called jitter caused by time changes due to the delay amount including the above-mentioned transmission delay.

[0005] Therefore, in order to reduce the impact of the time deviation from the above-mentioned user's action or the operation of the controller to the display of the corresponding image, for example, a technology for re-projecting the image in a device on the head-mounted kit side has been proposed (for example, refer to patent document 1).

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-106628 Summary of the invention

[0007] However, in the reprojection process, for example, the following processes are performed. One of them is to use the viewpoint position and line of sight direction at the time of image display to correct the image generated based on the viewpoint position and line of sight direction of the user at the time of image generation, thereby absorbing the time difference from the generation of the image to the display. In addition, other processes are as follows, that is, in the distribution of information streams, in the event of an error that the video information stream does not arrive on time, the last decrypted image is reused to perform reprojection processing on the last decrypted image in a manner consistent with the latest position / posture of the head-mounted kit, thereby coping with communication delays. That is, the reprojection process is to perform complex image processing in the device on the head-mounted kit side to correct the displayed image. Therefore, in the device on the head-mounted kit side, the processing load of the processor for the reprojection process is sometimes very large, a high-performance processor is required, and the device is expensive.

[0008] The present invention is proposed with an eye on the above-mentioned situation, and a technical solution is to provide a technology for reducing VR motion sickness without performing complicated image processing.

[0009] In order to solve the above-mentioned problem, the first embodiment of the present invention is a remote control system comprising: a user device connected to a display unit; and a remote operation device, which performs an action according to a control signal sent from the user device via a network, wherein the remote operation device comprises: a movable part capable of changing its posture; an image data acquisition unit, which acquires image data obtained by a camera provided on the movable part; a motion detection data acquisition unit, which acquires motion detection data representing the change of the posture of the movable part from a motion sensor; and a data sending unit, which sends the acquired image data and the motion detection data to the user device via the network. On the other hand, the user device includes: a data receiving unit, which receives the image data and the motion detection data sent from the remote working device via the network; a determination unit, which determines the period of the posture change of the movable part based on the received motion detection data; an image processing unit, which performs processing to reduce the immersion of the image data received during the period of the posture change of the movable part based on the determination result of the determination unit; and an output unit, which outputs the data after the processing to reduce the immersion to the display unit.

[0010] The second embodiment of the present invention is a remote working device, which has a movable part, and the movable part can perform actions and change postures according to a control signal sent from a user device connected to a display part via a network. The remote working device has: an image data acquisition unit, which acquires image data obtained by using a camera set on the movable part; a motion detection data acquisition unit, which acquires motion detection data representing changes in the posture of the movable part from a motion sensor; and a data sending unit, which sends the acquired image data and the motion detection data to the user device via the network.

[0011] The third embodiment of the present invention is an image processing device, which is provided in any one of a user device and a remote working device, the user device is connected to a display unit, and the remote working device has a movable part that can perform actions and change postures according to a control signal sent from the user device via a network, and the image processing device has: an image data acquisition unit, which acquires image data obtained by using a camera provided on the movable part; a motion detection data acquisition unit, which acquires motion detection data indicating the change in the posture of the movable part obtained by using a motion sensor; a determination unit, which determines the period of the posture change of the movable part based on the acquired motion detection data; an image processing unit, which performs a process of reducing the immersion of the image data acquired during the period of the posture change of the movable part based on the determination result of the determination unit; and an output unit, which outputs the data after the process of reducing the immersion to the display unit.

[0012] Effects of the Invention

[0013] According to the first embodiment of the present invention, the image data acquired in the remote working device and the motion detection data indicating the change in the posture of the remote working device are sent from the remote working device to the user device via the network. In contrast, in the user device, during the period when the posture change of the remote working device is determined based on the motion detection data, processing is performed to reduce the sense of immersion for the image data received from the remote working device during this period, and the processed data is output to the display unit for display. Therefore, even if there is a transmission delay in the image data sent from the remote working device, and even if there is a scanning delay in the display processing action of the user device, the adverse condition of VR motion sickness in the user caused by this influence can be reduced.

[0014] According to a second aspect of the present invention, in a remote working device, based on the motion detection data, it is determined that the remote working device is performing an action, and processing is performed to reduce the immersion of image data obtained by a camera during the period when the posture of the remote working device changes. Furthermore, the data subjected to the immersion reduction processing is sent from the remote working device to the user device.

[0015] Therefore, while the remote working device is performing an action, the image data processed in a manner that reduces the sense of immersion is sent from the remote working device to the user device and displayed on the display unit. Therefore, even if there is a transmission delay in the image data sent from the remote working device, and even if there is a scanning delay in the image display processing in the user device or the display unit, the adverse situation of VR motion sickness caused to the user due to the influence can be reduced.

[0016] Furthermore, according to the second aspect, since the remote operation device performs processing to reduce the immersion of the image, the processing load on the user device can be reduced, and a general-purpose user device that does not have a processing function for reducing the immersion can be used.

[0017] According to the third aspect of the present invention, in either the remote working device or the user device, processing is performed to reduce the sense of immersion for the image data obtained during the period when the posture of the remote working device changes, and the processed data is output from the user device to the display unit. Therefore, even if there is a transmission delay in the image data sent from the remote working device, and even if there is a scanning delay in the image display processing in the user device or the display unit, the adverse situation of VR motion sickness in the user caused by the influence can be reduced.

[0018] That is, according to each embodiment of the present invention, it is possible to provide a technology for reducing VR motion sickness without performing complicated image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a diagram showing the overall configuration of a remote control system according to the first embodiment of the present invention.

[0020] Figure 2 Yes means Figure 1 FIG. 1 is a block diagram of the hardware configuration of a robot used as a remote working device in the remote control system shown.

[0021] Figure 3 Yes means Figure 1 A block diagram of the hardware structure of a head-mounted display worn by a user in a remote control system is shown.

[0022] Figure 4 Yes means Figure 1FIG. 1 is a block diagram of the hardware structure of an information processing device used as a user device in the remote control system shown in FIG.

[0023] Figure 5 Yes means Figure 1 FIG. 1 is a block diagram of the software structure of an information processing device used as a user device in the remote control system shown in FIG.

[0024] Figure 6 Yes means Figure 5 A flowchart showing the processing order and processing contents of the information processing device shown.

[0025] Figure 7 This is a block diagram showing a software configuration of a robot used as a remote working device in a remote control system according to a second embodiment of the present invention.

[0026] Figure 8 Yes means Figure 7 A flowchart showing the processing order and content of the robot. DETAILED DESCRIPTION

[0027] Several embodiments of the present invention will be described below with reference to the accompanying drawings.

[0028] [First embodiment]

[0029] (Structure example)

[0030] (1) System

[0031] Figure 1 It is a diagram showing the overall configuration of a remote control system according to the first embodiment of the present invention.

[0032] The remote control system according to the first embodiment includes a head mounted display (HMD)

[0033] 1; an information processing device 2 that performs actions as a user device; and a remote operation device 3 that can communicate with the information processing device 2 via a network 4. In addition, here, the user device is described as including only the information processing device 2, but the user device may also include both the information processing device 2 and the HMD 1.

[0034] The network 4 is composed of, for example, a public IP (Internet Protocol) network such as the Internet and an access network for accessing the public IP network. The access network uses a LAN (Local Area Network), a wireless LAN, a public wired network, a public mobile communication network, and a CATV (Cable Television) network.

[0035] (2) Device

[0036] (2-1) Remote operation device 3

[0037] The remote working device 3 is composed of, for example, a humanoid robot that performs actions at a distance. In addition, the remote working device 3 is also referred to as a robot hereinafter. The robot 3 is composed of a main body, a head, an arm, and a leg. The head, the arm, and the leg are respectively configured to perform actions within a predetermined movable range using a drive unit having a servo mechanism, thereby causing various changes in posture.

[0038] A camera 34 as a photographing device is provided at the front of the head of the robot 3A, i.e., at a position corresponding to the face. In addition, a motion sensor is provided inside the head of the robot 3A to detect its motion. In addition, a microphone 36 is provided at a position corresponding to the side of the head of the robot 3A.

[0039] Figure 2 It is a block diagram showing the hardware configuration of the robot 3A.

[0040] The robot 3A includes, for example, a control unit 31. The control unit 31 includes a hardware processor such as a central processing unit (CPU). The control unit 31 is connected to a storage unit 32, a sensor interface (sensor I / F) 33, a drive interface (drive I / F) 37, and a communication interface (communication I / F) 39 via a bus 30.

[0041] The storage unit 32 uses a non-volatile memory that can be written and read at any time, such as a Solid State Drive (SSD), as a storage medium, and has a program storage area and a data storage area. Various applications / programs for realizing the actions of the robot 3A are stored in the program storage area. The data storage area is used to store various data acquired or generated during the action of the robot 3A. In addition, as a storage medium, ReadOnlyMemory (ROM), Random AccessMemory (RAM), etc. can also be used at the same time.

[0042] The camera 34, the motion sensor 35, and the microphone 36 are connected to the sensor I / F 33. The camera 34 is composed of, for example, a dual-camera camera, and captures the front of the robot 3A within a predetermined field of view (e.g., 130 degrees), and outputs the obtained stereoscopic image data to the sensor I / F 33. The motion sensor 35 is composed of, for example, a 6-axis acceleration sensor, and detects the 6-axis motion of the head of the robot 3A and outputs the detected data to the sensor I / F 33. In addition, the microphone 36 detects the sounds around the robot 3A and outputs the obtained voice data to the sensor I / F 33.

[0043] The driving unit 38 for driving the head, arms, and legs is connected to the driving I / F 37. The driving I / F 37 converts the control signal output from the control unit 31 into a driving signal, and outputs the converted driving signal to the driving unit 38 to operate the driving unit 38.

[0044] The communication I / F 39 has an interface corresponding to, for example, a wireless LAN, a short-range wireless data communication standard (such as Bluetooth (registered trademark)), or a public mobile communication network, and transmits data to and from the information processing device 2 via the network 4. In addition, the communication I / F 30 may have an interface corresponding to a wired network such as a public wired network, a wired LAN, or a CATV network.

[0045] The control unit 31 operates the driving unit 38 according to the remote control signal transmitted from the information processing device 2, so that the posture of the head, arms and legs of the robot 3A is changed. As the action to be controlled, for example, the head is rotated in the plane direction or the inclined direction to variably control the shooting direction of the camera 34. In addition, various actions are envisioned, that is, the legs are driven to make the robot 3A walk in the front-back direction or the left-right direction, or bend and stretch in the up-down direction. In addition, in addition to the humanoid robot 3A, the remote operation device may also be, for example, an industrial robot or a stand provided with only a monitoring camera.

[0046] In addition, the control unit 31 receives the stereoscopic image data obtained by the camera 34 via the sensor I / F 33. Then, the received stereoscopic image data is converted into data packets according to a predetermined format, and the data packets are sent from the communication I / F 39 to the information processing device 2. In addition, at this time, the voice data detected by the microphone 36 can also be multiplexed and sent to the stereoscopic image data.

[0047] Furthermore, the control unit 31 receives data indicating changes in the posture of the head detected by the motion sensor 35 via the sensor I / F 33. Furthermore, the control unit 31 performs the following processing, that is, adds time data indicating the detection time to the received detection data to generate motion detection data, and transmits the motion detection data from the communication I / F 39 to the information processing device 2. The time data is used to control the user's immersion in the stereoscopic image when the stereoscopic image data is displayed in the information processing device 2 described later.

[0048] (2-2) Head-mounted display (HMD)1

[0049] The HMD 1 is formed into, for example, a goggle type, and is detachably mounted on the user's head. Figure 3It is a block diagram showing the hardware configuration of the HMD 1 .

[0050] The HMD 1 is connected to a control unit 11 having a CPU, for example, via a bus 10, a storage unit 12, a display interface (display I / F) 13, a sensor interface (sensor I / F) 15, and a communication interface (communication I / F) 17. In addition, various switches, microphones, speakers, cameras, position sensors, and other devices may be connected to the control unit 11.

[0051] The storage unit 12 is a non-volatile memory that can be written and read at any time, such as an SSD, as a storage medium, and has a program storage area and a data storage area. Various applications / programs for realizing the operation of the HMD 1 are stored in the program storage area. The data storage area is used to store various data acquired or generated during the operation of the HMD 1. In addition, as a storage medium, ROM, RAM, etc. can also be used at the same time.

[0052] The display unit 14 is connected to the display I / F 13. The display unit 14 has two display panels composed of organic EL (Electro Luminescence) displays corresponding to the left and right eyes of the user in order to cope with virtual reality (VR) display. In addition, the display panel can be a single panel. In addition, the display unit 14 is not limited to an organic EL display, and can also be other types of displays such as a liquid crystal display (Liquid Cristal Display: LCD) and a display using 7 segments. The display I / F 13 displays the image data generated by the information processing device 2 described later on the display unit 14.

[0053] The motion sensor 16 is connected to the sensor I / F 15. The motion sensor 16 is composed of, for example, a 6-axis angular velocity sensor (gyro sensor), and is used to detect the motion of the HMD 1, that is, the motion of the user's head. In addition, preferably, the motion of the head as the detection object is, for example, a motion in the 6-axis direction, or only a motion in the 2-axis direction of the plane and the tilt. The sensor I / F 15 generates motion data representing the motion of the user's head based on the output signal of the motion sensor 16.

[0054] In addition to the motion sensor 16, a magnetic sensor, an acceleration sensor, a position sensor, an infrared sensor, a brightness sensor, a proximity sensor, a camera, etc. may be connected to the sensor I / F 15. In addition to the movement of the user's head, a sensor for detecting the movement of the user's line of sight may be connected to the sensor I / F 15. For example, the movement of the user's line of sight may be detected by using a camera that captures the user's eyeballs.

[0055] A wired interface using a signal cable such as a Universal Serial Bus (USB) cable is used for the communication I / F 17. Furthermore, the communication I / F 17 receives image data sent from the information processing device 2 under the control of the control unit 11, and transmits motion data generated by the sensor I / F 15 to the information processing device 2. In addition, a short-distance wireless data communication standard (such as Bluetooth (registered trademark)) can be used for the communication I / F 17.

[0056] In this example, HMD1 is described as a multifunctional device including control unit 11 and storage unit 12. However, HMD1 may be a standard or simplified device including only display unit 14, display I / F 13, motion sensor 16, and sensor I / F 15.

[0057] (2-3) Information Processing Device 2

[0058] Figure 4 and Figure 5 Each of them is a block diagram showing the configuration of hardware and software of the information processing device 2 used as a user device.

[0059] The information processing device 2 is composed of, for example, a portable information terminal such as a smartphone or a tablet terminal, or a notebook or stationary personal computer. The information processing device 2 has a control unit 21 having a hardware processor such as a CPU, and is connected to a storage unit 22, an input / output interface (input / output I / F) 23, and a communication interface (communication I / F) 24 via a bus 20.

[0060] The HMD 1 is connected to the input / output I / F 23 via the USB cable or wireless interface. In addition, an operation unit 256 such as a controller is connected to the input / output I / F 23. The operation unit 256 is used for the user to input the control target part, control direction and control amount when controlling the action of the robot 3A.

[0061] The communication I / F 24 has an interface corresponding to, for example, a wireless LAN, a short-range wireless data communication standard (such as Bluetooth (registered trademark)), or a public mobile communication network, and transmits data to and from the robot 3A via the network 4. In addition, the communication I / F 24 may have an interface corresponding to a wired network such as a public wired network, a wired LAN, or a CATV network.

[0062] The storage unit 22 is composed of a non-volatile memory such as HDD or SSD that can be written and read at any time, a non-volatile memory such as ROM, and a volatile memory such as RAM as a storage medium. A program storage area and a data storage area are set in the storage area. On the basis of middleware such as OS, applications / programs required to execute various control processes involved in the first embodiment of the present invention are stored in the program storage area.

[0063] The data storage area is provided with a video content storage unit 221. When displaying the stereoscopic video data transmitted from the robot 3A, the video content storage unit 221 is used as a buffer memory for performing a predetermined process on the stereoscopic video data.

[0064] As control processing functions for realizing the first embodiment of the present invention, the control unit 21 includes an operation data acquisition unit 211, an action control data generation unit 212, a remote control signal transmission unit 213, an image content reception unit 214, an image processing unit 215, and an image display control unit 216. The above control processing functions are all realized by causing the hardware processor of the control unit 21 to execute an application / program stored in the program storage area in the storage unit 22.

[0065] The operation data acquisition unit 211 receives the operation data inputted in the operation unit 25 via the input / output I / F 23. At the same time, the operation data acquisition unit 211 receives the motion detection data obtained by the motion sensor 16 of the HMD 1 via the input / output I / F 23. The operation data acquisition unit 211 delivers the received operation data and motion detection data to the motion control data generation unit 212.

[0066] The motion control data generation unit 212 generates motion control data for controlling the motion of the robot 3A based on the operation data and the motion detection data. The motion control data includes information indicating the controlled part, control direction, and control amount of the robot 3A.

[0067] The remote control signal transmission unit 213 performs processing for transmitting a remote control signal including the motion control data generated by the motion control data generation unit 212 from the communication I / F 24 to the robot 3A.

[0068] The video content receiving unit 214 receives the video content, that is, the stereoscopic video data, transmitted from the robot 3A via the network 4 via the communication I / F 24 , and performs a process of decompressing and decrypting the received stereoscopic video data and then storing it in the video content storage unit 221 .

[0069] The video processing unit 215 reads the stereoscopic video data from the video content storage unit 221 to generate display data for AR or VR display, and outputs the generated display data to the HMD 1 from the input / output I / F 23 .

[0070] The image display control unit 216 receives the motion detection data transmitted from the robot 3A via the communication I / F 24. Based on the received motion detection data, the robot 3A determines the period of posture change of the head, and performs processing for reducing the immersion of the display data generated by the image processing unit 215 during the period of posture change of the head.

[0071] (Action Example)

[0072] Next, an operation example of the remote control system configured as described above will be described. Figure 6 It is a flowchart showing the processing procedure and processing contents of the information processing device 2.

[0073] If the user starts the information processing device 2 after installing the HMD1 on his head, a connection is first made between the communication I / F17 of the HMD1 and the input / output I / F23 of the information processing device 2, and a communication link is formed between the communication I / F24 of the information processing device 2 and the communication I / F39 of the robot 3A via the network 4.

[0074] In this state, the user operates the operation unit 25 to make the robot 3A perform an action, and the operation data is input to the information processing device 2. In addition, even if the user makes his head perform an action, the action is detected by the motion sensor 16 and input to the information processing device 2. The information processing device 2 acquires the above-mentioned operation data and motion detection data in step S11 under the control of the operation data acquisition unit 211. Moreover, the information processing device 2 generates motion control data for controlling the action of the robot 3A based on the above-mentioned acquired data in step S12 under the control of the motion control data generation unit 212.

[0075] For example, the motion control data generating unit 212 identifies the control target part of the robot 3A and its control direction and control amount based on the operation data. As an example, the movement amount of the leg in the front-back direction or the left-right direction is identified. In addition, the change amount of the orientation of the head of the robot 3A is identified based on the motion detection data.

[0076] Under the control of the remote control signal transmission unit 213 , the information processing device 2 generates a remote control signal including the motion control data generated by the motion control data generation unit 212 in step S13 , and transmits the generated remote control signal from the communication I / F 24 to the robot 3A.

[0077] On the other hand, when the robot 3A receives a remote control signal sent from the information processing device 2 under the control of the control unit 31, the robot 3A extracts motion control data from the remote control signal, and based on the information specifying the control target part, control direction, and control amount contained in the motion control data, the robot 3A causes the corresponding drive unit 38 to perform motion using the drive I / F 37. For example, if the control target part is a leg or a head, the drive unit 38 corresponding to these parts is caused to perform motion. As a result, the robot 3A walks in the front-back direction or the left-right direction, or changes the direction of the head.

[0078] In addition, the control unit 31 of the robot 3A imports stereoscopic image data from the camera 34 mounted on the head via the sensor I / F 33 while causing the legs or the head to perform the motion. In addition, at the same time, the motion detection data of the head is imported from the motion sensor 35 via the sensor I / F 33. Then, the imported image data and motion detection data are sent to the information processing device 2 from the communication I / F 39. In addition, the voice data detected by the microphone 36 can be multiplexed and sent to the stereoscopic image data. In addition, the time data indicating the detection time is given to the detection data.

[0079] On the other hand, the information processing device 2 receives the stereoscopic image data transmitted from the robot 3A in step S14 under the control of the image content receiving unit 214. In step S14, for example, the received stereoscopic image data is decompressed and decrypted and then stored in the image content storage unit 221. Furthermore, the information processing device 2 reads the stereoscopic image data from the image content storage unit 221 under the control of the image processing unit 215, and performs information adding processing such as adding additional information as needed on the read stereoscopic image data to generate display data. Furthermore, the generated display data is output from the input / output I / F 23 to the HMD 1 in step S18. As a result, the stereoscopic image captured by the camera 34 of the robot 3A is displayed on the display unit 14 of the HMD 1 in substantially real time.

[0080] However, during the above-described display process of the stereoscopic image, the information processing device 2 performs a process of controlling the user's sense of immersion in the displayed stereoscopic image in the following manner.

[0081] That is, the information processing device 2 receives the motion detection data sent from the robot 3A in step S15 under the control of the image display control unit 216. And, in step S16, the period during which the head of the robot 3A performs motion, that is, the period during which the posture changes, is determined based on the received motion detection data. As a result of this determination, the period during which the posture of the robot 3A changes is detected. In this case, in step S17, the image display control unit 216 reduces the user's immersion during the period during which the head posture changes are detected for the display data generated by the image processing unit 215.

[0082] As a process to reduce the immersion, for example, it is considered to freeze or eliminate the display of the above-mentioned three-dimensional image. As an elimination process, for example, it is considered to replace the display data with white, black, gray or single-color image data of a specified color, or replace it with specified standby image data pre-stored in the storage unit 32. At this time, a message or mark, icon, or dot indicating that the robot 3A is performing an action may be overlapped with the above-mentioned single-color image data or standby image data.

[0083] In addition, as other processing to reduce the sense of immersion, for example, it is considered to make the display data semi-transparent or have a high pixel density, or to reduce the size of the display data and display it at an arbitrary position of the display unit 14 of the HMD 1. In this case, the display position of the display data is not limited to the center, and can be any position. In short, any method can be used as long as it is a display method that can reduce the user's sense of immersion in the displayed image.

[0084] Furthermore, the voice data transmitted from the robot 3A may be replaced or superimposed thereon, and a voice message, melody, or call indicating that the robot 3A is in operation may be output from the speaker built into the HMD 1 .

[0085] (Function / Effect)

[0086] As described above, in the remote control system according to the first embodiment, in the robot 3A, the change in the posture of a predetermined part such as the head, arm or leg is detected by the motion sensor 35, and data indicating the detected motion is sent to the information processing device 2 via the network 4 together with the stereoscopic image data obtained by the camera 34. On the other hand, in the information processing device 2, the display data including the stereoscopic image data sent from the robot 3A is output to the HMD 1 for display, and the posture change of the robot 3A is determined based on the motion detection data sent from the robot 3A at the same time, and the immersion feeling felt by the user for the display data is reduced during this period.

[0087] Therefore, while the robot 3A is performing an action, the display data processed in a manner that reduces the immersion is supplied to the HMD 1 and displayed. Therefore, even if there is a transmission delay in the image data sent from the robot 3A, and even if there is a scanning delay in the display processing action due to the information processing device 2 or the HMD 1, the adverse situation of VR motion sickness in the user caused by this influence can be reduced.

[0088] [Second embodiment]

[0089] In a second embodiment of the present invention, in a robot as a remote working device, it is determined whether the robot is in motion based on detection data of a motion sensor, and a process of reducing immersion is performed on image data obtained by a camera during a period when the robot's posture changes. Furthermore, the image data subjected to the immersion reduction process is transmitted from the robot to an information processing device as a user device.

[0090] (Structure example)

[0091] Figure 7 2 is a block diagram showing the software structure of the robot 3B as a remote operation device according to the second embodiment of the present invention. Figure 2 The same as above, so the illustration is omitted.

[0092] As the control processing unit involved in the second embodiment of the present invention, the control unit 31 of the robot 3B includes a remote control signal receiving unit 311, a motion control data extracting unit 312, a drive control unit 313, an image data acquiring unit 314, an action detection data acquiring unit 315, and an image processing unit 316. The above control processing units are all implemented by causing the hardware processor of the above control unit 31 to execute the program stored in the storage unit 32.

[0093] The remote control signal receiving unit 311 performs processing for receiving the remote control signal transmitted from the information processing device 2 via the communication I / F 39 and delivering the received remote control signal to the motion control data extracting unit 312 .

[0094] The motion control data extraction unit 312 extracts the motion control data from the remote control signal and transfers the extracted motion control data to the drive control unit 313 .

[0095] The drive control unit 313 selects the corresponding drive unit 38 based on the information specifying the control target part contained in the extracted motion control data, and generates a drive control signal for driving the selected drive unit 38 based on the information specifying the control direction and control amount contained in the motion control data. In addition, the generated drive control signal is output from the drive I / F 37 to the drive unit 38.

[0096] The image data acquisition unit 314 receives the stereoscopic image data captured by the camera 34 via the sensor I / F 33 and performs processing to temporarily store the stereoscopic image data in the image data storage unit 321 for image processing described later.

[0097] The motion detection data acquisition unit 315 imports data indicating the motion of the head of the robot 3B detected by the motion sensor 35 via the sensor I / F 33. Then, the image processing unit 316 is notified of information indicating the determination result while determining the posture change of the head of the robot 3B based on the imported motion detection data.

[0098] The image processing unit 316 performs processing such that, based on the information indicating the determination result notified from the motion detection data acquisition unit 315, the stereoscopic image data stored in the image data storage unit 321 is directly sent from the communication I / F 39 to the information processing device 2 during a period when the posture of the head does not change. On the other hand, the image processing unit 316 performs processing to reduce the immersion of the stereoscopic image data stored in the image data storage unit 321 during a period when the posture of the head changes, and sends the processed data from the communication I / F 39 to the information processing device 2.

[0099] (Action Example)

[0100] Next, the operation of the robot 3B configured as described above will be described. Figure 8 This is a flowchart showing the order and content of the control processing.

[0101] When a remote control signal including motion control data is transmitted from the information processing device 2, the robot 3B receives the remote control signal via the communication I / F 39 in step S21 under the control of the remote control signal receiving unit 311. Furthermore, under the control of the motion control data extracting unit 312, the robot 3B extracts the motion control data from the remote control signal in step S22.

[0102] Next, the robot 3B, under the control of the drive control unit 313, selects the corresponding drive unit 38 in step S23 based on the information on the control target part included in the motion control data. Furthermore, based on the information specifying the control direction and the control amount included in the motion control data, a drive control signal for driving the selected drive unit 38 is generated, and the generated drive control signal is output from the drive I / F 37 to the drive unit 38. As a result, the drive unit 38 is driven to cause the corresponding part to perform the motion.

[0103] For example, if the controlled part is a leg, the corresponding drive unit 38 is driven to make the robot 3B walk in the front-back direction, for example. In addition, if the controlled part is a head, the corresponding drive unit 38 is driven to change the orientation of the head in the planar direction or the inclined direction.

[0104] On the other hand, when the above-mentioned legs or head are performing actions, the control unit 31 of the robot 3B, under the control of the image data acquisition unit 314, imports stereoscopic image data from the camera 34 installed on the head via the sensor I / F33 in step S24, and temporarily stores the stereoscopic image data in the image data storage unit 321.

[0105] At the same time, the control unit 31 of the robot 3B, under the control of the motion detection data acquisition unit 315, imports motion detection data indicating the motion of the head of the robot 3B, that is, the change in posture, from the motion sensor 35 via the sensor I / F 33 in step S25. Based on the imported motion detection data, the period during which the head of the robot 3B performs motion is determined in step S26.

[0106] Next, the control unit 31 of the robot 3B performs the image data transmission process in the following manner based on the above-mentioned determination result under the control of the image processing unit 316. That is, when the determination result obtained in the above-mentioned step S26 indicates that the posture of the head of the robot 3B has not changed, the stereoscopic image data is read from the image data storage unit 321, and the stereoscopic image data is transmitted from the communication I / F 39 to the information processing device 2 in step S28.

[0107] On the other hand, as a result of the determination in step S26, while the posture of the head of the robot 3B changes, the image processing unit 316 performs processing to reduce the immersion in step S27 for the stereoscopic image data stored in the image data storage unit 321. The processed data is then transmitted from the communication I / F 39 to the information processing device 2.

[0108] Here, as a process for reducing the immersion, for example, the following process is adopted. For example, the above-mentioned three-dimensional image data is made still, or replaced with single-color image data of white, black, gray or a specified color. Alternatively, the above-mentioned three-dimensional image data can be replaced with a specified standby image data that is pre-stored in the storage unit 32. At this time, a message or a mark, icon, or dot indicating that the robot 3B is moving can be overlapped with the above-mentioned standby image data.

[0109] Furthermore, as other processing to reduce the sense of immersion, for example, processing may be performed to make the stereoscopic image data semi-transparent or with a coarse pixel density, or the frame size of the stereoscopic image may be reduced. In addition, when the frame size is reduced, the position of the stereoscopic image data is not limited to the center, but may be any position. In short, any method may be used as long as it is an image processing method that can reduce the user's sense of immersion in the image data sent from the robot 3B.

[0110] In addition, the control unit 31 can be used to assist in replacing the voice message, melody or chirping sound indicating that the robot 3B is in action with the sound around the robot 3B detected by the microphone 36 or overlapping them, and the processed voice data can be sent from the communication I / F 39 to the information processing device 2.

[0111] (Function / Effect)

[0112] As described above, in the second embodiment of the present invention, in the robot 3B, during a period when the posture change of the robot 3B is determined based on the motion detection data of the motion sensor 35, processing to reduce the immersion is performed on the stereoscopic image data obtained by using the camera 34 during the period when the posture change of the robot 3B occurs, and the image data subjected to the immersion reduction processing is sent from the robot 3B to the information processing device 2.

[0113] Therefore, during the period when the posture of the robot 3B changes, the image data processed in a manner that reduces the immersion is sent from the robot 3B to the information processing device 2 and displayed on the HMD 1. Therefore, even if there is a transmission delay in the image data sent from the robot 3B, and even if there is a scanning delay in the image display processing in the information processing device 2 or the HMD 1, the adverse situation of VR motion sickness in the user is reduced due to this influence.

[0114] In addition, according to the second embodiment, the robot 3B performs processing for reducing the immersion in the stereoscopic image, which has the following advantages: the processing load of the information processing device 2 can be reduced, and a general-purpose information processing device that does not have a processing function for reducing the immersion can be used.

[0115] [Other embodiments]

[0116] (1) In the first and second embodiments, when the robot 3A, 3B is detected walking forward and backward or left and right, or when the head direction is changed in a planar direction or in an inclined direction, the case where the immersion is reduced with respect to the stereoscopic image obtained by the camera 34 is described as an example. However, the present invention is not limited to this, and when the robot 3A, 3B is detected tilting the head left and right, etiquette movements, or bending and stretching movements with the legs, the immersion may be reduced with respect to the stereoscopic image obtained by the camera 34.

[0117] (2) In the first and second embodiments, the motion sensor 35 is used to detect the motion of the robot 3A, 3B when it moves according to the remote control signal sent from the information processing device 2, and the image data obtained during the detection of the motion is processed to reduce the immersion. However, the present invention is not limited to this, and the following processing may be performed, that is, the motion of the robot 3A, 3B is detected when it moves autonomously without the remote control signal from the information processing device 2, and when it moves according to the remote control signal sent from another information processing device or a controller attached to the robot, and the immersion is reduced for the image data obtained during the detection of the motion.

[0118] (3) In the first embodiment, data communication and image display control with the robot 3A are performed using the information processing device 2 provided separately from the HMD 1. However, when the HMD 1 has the function of the information processing device 2, that is, when an HMD integrated with the information processing device is used, the data communication and image display control with the robot 3A may be performed in the HMD 1.

[0119] (4) In addition, various improvements and modifications may be made to the type and structure of the remote operation device, the type and structure of the camera, the motion sensor, the structure of the information processing device, the processing to reduce the immersion, etc. without departing from the main purpose of the present invention. That is, when implementing the present invention, a specific structure corresponding to the embodiment may be appropriately adopted.

[0120] In short, the present invention is not directly limited to the above-mentioned embodiments, and the structural elements can be deformed and concretized in the implementation stage without departing from the scope of its purpose. In addition, various inventions can be formed by appropriate combinations of multiple structural elements disclosed in the above-mentioned embodiments. For example, several structural elements can be deleted from all the structural elements shown in each embodiment. In addition, the structural elements of different embodiments can be appropriately combined.

[0121] Description of the label

[0122] 1…Head-mounted display (HMD)

[0123] 2…Information processing device

[0124] 3A, 3B…Robots

[0125] 4…Network

[0126] 10, 20, 30…bus

[0127] 11, 21, 31…Control Unit

[0128] 12, 22, 32... Storage

[0129] 13…Display interface (display I / F)

[0130] 14…Display unit

[0131] 15, 33…Sensor interface (sensor I / F)

[0132] 16, 35…Motion sensor

[0133] 17, 24, 39…Communication interface (communication I / F)

[0134] 23…Input and output interface (input and output I / F)

[0135] 25…Operation Department

[0136] 34…Camera

[0137] 36…Microphone

[0138] 37…Drive interface (Drive I / F)

[0139] 38…Driver

[0140] 211…Operation data acquisition unit

[0141] 212…Motion control data generation unit

[0142] 213…Remote control signal sending unit

[0143] 214…Image content receiving unit

[0144] 215, 316…Image Processing Department

[0145] 216…Image display control unit

[0146] 221…Image content storage unit

[0147] 311…Remote control signal receiving unit

[0148] 312…Motion control data extraction unit

[0149] 313…Drive control unit

[0150] 314…Image data acquisition unit

[0151] 315…Motion detection data acquisition unit

[0152] 321…Image data storage unit

Claims

1. A remote control system comprising: a user device connected to a display unit; and a remote operation device that performs an operation according to a control signal sent from the user device via a network, wherein: The remote operation device comprises: A movable part capable of changing its posture; an image data acquisition unit that acquires image data output from a camera provided on the movable unit; a motion detection data acquisition unit that acquires motion detection data indicating a change in the posture of the movable unit from a motion sensor; and a data transmission unit configured to transmit the acquired image data and the motion detection data to the user device via the network, The user device comprises: a data receiving unit that receives the image data and the motion detection data transmitted from the remote operation device via the network; a determination unit that determines a period during which the posture of the movable unit changes based on the received motion detection data; an image processing unit that performs processing to reduce the immersive feeling for the image data received during a period in which the posture of the movable unit changes, based on the determination result of the determination unit; as well as an output unit configured to output the data processed to reduce the immersion to the display unit, The process of reducing the immersive feeling includes performing any one of a process of freezing an image, replacing the image with still image data of a single color, and rendering the image semi-transparent, on the image data received during a period in which the posture of the movable unit changes.

2. An image processing device, which is provided in any one of a user device and a remote working device, the user device is connected to a display unit, and the remote working device has a movable part that can perform an action and change its posture according to a control signal sent from the user device via a network, wherein: The image processing device comprises: an image data acquisition unit that acquires image data output from a camera provided on the movable unit; a motion detection data acquisition unit that acquires motion detection data indicating a change in the posture of the movable unit obtained by using a motion sensor; a determination unit that determines a period during which the posture of the movable unit changes based on the acquired motion detection data; an image processing unit that performs processing to reduce the immersive feeling for the image data acquired during a period when the posture of the movable unit changes based on the determination result of the determination unit; as well as an output unit configured to output the data processed to reduce the immersion to the display unit, The process of reducing the immersive feeling includes performing any one of a process of freezing an image, replacing the image with single-color still image data, and rendering the image semi-transparent, on the image data acquired during a period when the posture of the movable unit changes.

3. The image processing device according to claim 2, wherein: The image processing unit replaces the image data acquired during a period in which the posture of the movable unit changes with standby image data to which information on a status in which the remote working device is operating is added.

4. A storage medium storing a program, wherein: The program causes a processor included in the video processing device according to claim 2 or 3 to execute processing of each unit included in the video processing device.

Images