Remote operation device and its program

Abstract

By appropriately compressing images obtained at a distance, the visual confirmation of the user is improved. In a robot (3) at a distance, the action scene of the robot (3) is determined based on the feature quantity derived from the detection data of its position and action and the image data, and an image parameter or shooting mode corresponding to the determined action scene is selected. In addition, the image data is processed to adjust the selected image parameter, or the camera is processed to set the selected shooting mode, and the processed image data is transmitted to the information processing device (2) on the user side via a network (4) and displayed on the HMD (1).

Description

Technical Field

[0001] Embodiments of the present invention relate to, for example, a remote working device capable of communicating with a user device via a network, and a program used in the remote working device. Background Art

[0002] A system has been developed in which a user remotely controls the actions of a robot while viewing an image transmitted from a robot located at a distance via a network. This system is, for example, a head mounted display (HMD) (hereinafter referred to as a head mounted kit) mounted on the user's head. The system is configured such that the user remotely controls the actions of the robot by causing his or her body to perform actions or operating a controller while viewing an image captured by a robot located at a distance using the head mounted kit.

[0003] In addition, regarding this system, for example, by using augmented reality (AR) technology or virtual reality (VR) technology based on images transmitted from the robot, AR images or VR images are generated and displayed on a head-mounted kit, thereby enabling users to gain a high sense of immersion.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2019-106628 Summary of the invention

[0005] However, the current system has the following problems that need to be solved. That is, when transmitting the captured image via the network, in a simple system with fixed image compression conditions, it is affected by the communication environment. In particular, in a wireless environment, if the user moves to an environment with weak radio waves, the communication band that can be transmitted within a constant time is narrowed. Therefore, if a high-quality image is to be transmitted, for example, frames are lost and the image becomes incoherent, or noise overlaps, or transmission delays occur, resulting in deterioration of transmission quality.

[0006] To solve the above problem, methods such as Adaptive Bitrate Streaming are currently being considered. This method estimates the communication band and changes the compression of the image so that it falls within the above range. For example, the resolution of the image is reduced to reduce the data transmission capacity.

[0007] However, in this method, regardless of whether the spatial resolution or the temporal resolution needs to be actively ensured, compression processing is performed according to the algorithm of the Adaptive Bitrate Streaming implementation system that does not take the type of image into consideration. Therefore, in the case where a robot at a distance captures images of various scenes, such as in a remote control system, and a user views the images using an HMD, it is conceivable that the image quality required for viewing cannot be obtained, and it is not suitable for practical application.

[0008] The present invention has been proposed in view of the above situation, and aims to provide a technology capable of appropriately compressing an image acquired at a distance to improve the visual confirmation of the user.

[0009] In order to solve the above-mentioned problem, the first embodiment of the present invention is a remote working device capable of communicating with a user device connected to a display unit via a network, and is configured to include: an image data acquisition unit, which acquires image data of an object photographed by a camera from a camera; a state detection data acquisition unit, which acquires state detection data representing at least one of a position and an action of the remote working device detected by the sensor from a sensor; a determination unit, which determines an action scene of the remote working device based on the acquired image data and the state detection data; a selection unit, which selects and adjusts image parameters of an object according to the determined action scene; a control unit, which adjusts the selected image parameters with respect to the image data; and a sending unit, which sends the image data after the image parameters are adjusted to the user device via the network.

[0010] A second embodiment of the present invention is a remote working device capable of communicating with a user device connected to a display unit via a network, and is configured to include: an image data acquisition unit, which acquires image data of an object captured by a camera from a camera; a state detection data acquisition unit, which acquires state detection data representing at least one of a position and an action of the remote working device detected by the sensor from a sensor; a determination unit, which determines an action scene of the remote working device based on the acquired image data and the state detection data; a selection unit, which selects a shooting mode corresponding to the determined action scene; a control unit, which sets the selected shooting mode for the camera; and a sending unit, which sends the image data captured by the camera using the set shooting mode to the user device via the network.

[0011] Effects of the Invention

[0012] According to the first embodiment of the present invention, in a remote working device, the action scene of the remote working device is determined based on the position or movement of the remote working device and the captured image, and the image parameters of the adjustment object are selected according to the determined action scene. Moreover, the selected image parameters are adjusted for the image data, and the adjusted image data is sent to the user device. Therefore, for example, for each action scene such as a reading scene or a sports game watching scene, the image data is adjusted using the image parameters corresponding to the action scene and sent to the user device. Therefore, for each action scene, a remote image compressed with image parameters suitable for the scene is transmitted to the user device, thereby providing the user with a remote image with good audiovisual quality suitable for each action scene of the remote working device.

[0013] According to the second embodiment of the present invention, in a remote working device, the action scene of the remote working device is determined based on the position or movement of the remote working device and the captured image, and the shooting mode of the camera is selected according to the determined action scene. Moreover, the selected shooting mode is set for the camera, and the image data captured by the shooting mode is transmitted to the user device via the network. Therefore, for example, for each action scene such as a reading scene or a sports game watching scene, the image data captured by the shooting mode corresponding to the action scene is transmitted to the user device. Therefore, for each action scene, a remote image with a quality suitable for the scene is transmitted to the user device, thereby providing the user with a remote image with good audiovisual quality suitable for each action scene of the remote working device.

[0014] That is, according to each aspect of the present invention, it is possible to provide a technology for improving the visual recognition of the user by appropriately compressing an image acquired at a distance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a diagram showing the overall configuration of a remote control system according to one embodiment of the present invention.

[0016] Figure 2 Yes means Figure 1 A block diagram of the hardware structure of a head-mounted display worn by a user in a remote control system is shown.

[0017] Figure 3 Yes means Figure 1 FIG. 1 is a block diagram of the hardware structure of an information processing device used as a user device in the remote control system shown in FIG.

[0018] Figure 4 Yes means Figure 1FIG. 1 is a block diagram of the hardware configuration of a robot used as a remote working device in the remote control system shown.

[0019] Figure 5 Yes means Figure 1 A block diagram of the software structure of a robot used as a remote working device in the remote control system shown.

[0020] Figure 6 Yes means Figure 5 The flowchart shown is the robot-based processing order and processing content.

[0021] Figure 7 Yes means Figure 6 A flowchart showing the processing procedure and processing contents of the action scene determination processing and the image parameter selection processing in the processing procedure of the robot.

[0022] Figure 8 Yes means Figure 5 FIG. 1 is a diagram showing an example of action scene information stored in a storage unit shown in FIG.

[0023] Fig. 9 Yes means Figure 5 A diagram showing an example of sub-scene information stored in a storage unit shown.

[0024] Fig.10 Yes means Figure 5 FIG. 1 is a diagram showing an example of video parameter information stored in a storage unit shown in FIG. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0026] [One embodiment]

[0027] (Structure example)

[0028] (1) System

[0029] Figure 1 It is a diagram showing the overall configuration of a remote control system according to one embodiment of the present invention.

[0030] A remote control system according to an embodiment of the present invention includes: a head mounted display (HMD) 1; an information processing device 2 that performs operations as a user device; and a remote operation device 3 that can communicate with the information processing device 2 via a network 4. The remote operation device 3 is, for example, a humanoid robot. Hereinafter, in this embodiment, the remote operation device 3 is referred to as a robot.

[0031] In addition, in this embodiment, although the case where the user device includes only the information processing device 2 is described as an example, the user device may include both the information processing device 2 and the HMD 1 .

[0032] The network 4 is composed of a public IP (Internet Protocol) network such as the Internet and an access network for accessing the public IP network. The access network uses a LAN (Local Area Network), a wireless LAN, a public wired network, a public mobile communication network, and a CATV (Cable Television) network.

[0033] (2) Device

[0034] (2-1) Head-mounted display (HMD)1

[0035] The HMD 1 is formed in, for example, a goggle type, and is detachably worn on the head of the user. Figure 2 It is a block diagram showing the hardware configuration of the HMD 1 .

[0036] The HMD 1 has a control unit 11 having a hardware processor such as a central processing unit (CPU) and a storage unit 12, a display interface (display I / F) 13, a sensor interface (sensor I / F) 15, and a communication interface (communication I / F) 17 connected to it via a bus 10. In addition, various switches, microphones, speakers, cameras, position sensors, and other devices are connected to the control unit 11.

[0037] The storage unit 12 is a non-volatile memory that can be written and read at any time, such as a Solid State Drive (SSD) as a storage medium, and has a program storage area and a data storage area. Applications / programs for implementing the actions of the HMD 1 are stored in the program storage area. The data storage area is used to save various data acquired or generated during the operation of the HMD 1. In addition, as a storage medium, Read Only Memory (ROM), Random Access Memory (RAM), etc. can also be used at the same time.

[0038] The display unit 14 is connected to the display I / F 13. The display unit 14 has two display panels composed of organic EL (Electro Luminescence) displays corresponding to the left and right eyes of the user in order to cope with virtual reality (VR) display. In addition, the display panel can be a single panel. In addition, the display unit 14 is not limited to organic EL displays, and can also be other types of displays such as liquid crystal displays (Liquid Cristal Display: LCD) and displays using 7 segments. The display I / F 13 displays the image data generated by the information processing device 2 described later on the display unit 14.

[0039] The motion sensor 16 is connected to the sensor I / F 15. The motion sensor 16 is composed of, for example, a 6-axis angular velocity sensor (gyro sensor), and is used to detect the motion of the HMD 1, that is, the motion of the user's head. In addition, preferably, the motion of the head as the detection object is, for example, a motion in the 6-axis direction, or only a motion in the 2-axis direction of the plane and the tilt. The sensor I / F 15 generates motion detection data representing the motion of the user's head based on the output signal of the motion sensor 16.

[0040] In addition to the above-mentioned motion sensor 16, a magnetic sensor, an acceleration sensor, a position sensor, an infrared sensor, a brightness sensor, a proximity sensor, a camera, etc. may be connected to the sensor I / F 15. In addition, in addition to the movement of the user's head, a sensor for detecting the movement of the user's line of sight may be connected to the sensor I / F 15. For example, the movement of the user's line of sight may be detected by using a camera that captures the user's eyeballs.

[0041] A wired interface using a signal cable such as a Universal Serial Bus (USB) cable is used for the communication I / F 17. Furthermore, the communication I / F 17 receives image data sent from the information processing device 2 under the control of the control unit 11, and transmits motion detection data generated by the sensor I / F 15 to the information processing device 2. In addition, a short-distance wireless data communication standard (such as Bluetooth (registered trademark)) can be used for the communication I / F 17.

[0042] In this example, HMD1 is described as a multifunctional device including control unit 11 and storage unit 12. However, HMD1 may be a standard or simplified device including only display unit 14, display I / F 13, motion sensor 16, and sensor I / F 15.

[0043] (2-2) Information Processing Device 2

[0044] Figure 3 It is a block diagram showing the hardware configuration of the information processing device 2 used as a user device.

[0045] The information processing device 2 is composed of, for example, a portable information terminal such as a smartphone or a tablet terminal, or a notebook or stationary personal computer. The information processing device 2 has a control unit 21 having a hardware processor such as a CPU, and is connected to a storage unit 22, an input / output interface (input / output I / F) 23, and a communication interface (communication I / F) 24 via a bus 20.

[0046] The HMD 1 is connected to the input / output I / F 23 via the USB cable or wireless interface. In addition, an operation unit 256 such as a controller is connected to the input / output I / F 23. The operation unit 256 is used to input, for example, the control target part, control direction, and control amount when the user controls the action of the robot 3A.

[0047] The communication I / F 24 has an interface corresponding to, for example, a wireless LAN, a short-range wireless data communication standard (such as Bluetooth (registered trademark)), or a public mobile communication network, and transmits data to and from the robot 3A via the network 4. In addition, the communication I / F 24 may have an interface corresponding to a wired network such as a public wired network, a wired LAN, or a CATV network.

[0048] The storage unit 22 is composed of a non-volatile memory such as a Hard Disk Drive (HDD) or SSD that can be written and read at any time, a non-volatile memory such as a ROM, and a volatile memory such as a RAM as a storage medium. A program storage area and a data storage area are provided in the storage area. On the basis of middleware such as an Operating System (OS), the applications / programs required to execute various control processes involved in an embodiment of the present invention are stored in the program storage area. The data storage area is provided with: an image data storage area for temporarily storing the image data transmitted from the robot 3; and a storage area for operations for the control unit 21 to perform various processing operations.

[0049] As control processing functions for realizing an embodiment of the present invention, the control unit 21 includes a remote control signal transmission unit, an image data reception unit, and an image display control unit. The above control processing functions are all realized by causing the hardware processor of the control unit 21 to execute the application / program stored in the program storage area in the storage unit 22.

[0050] The remote control signal transmission unit generates a motion control signal for controlling the motion of the robot 3 based on the operation data inputted into the operation unit 25 and the motion detection data obtained by the motion sensor 16 of the HMD 1. The motion control signal includes information indicating the control target part, control direction, and control amount of the robot 3. Then, the remote control signal including the generated motion control signal is transmitted from the communication I / F 24 to the robot 3.

[0051] The image data receiving unit receives the image data transmitted from the robot 3 via the network 4 via the communication I / F 24. After decrypting the received image data, the image data is temporarily stored in the image data storage area in the storage unit 22.

[0052] The video display control unit performs video editing processing for AR display or VR display on the video data read from the video data storage area as necessary, and then outputs the video data from the input / output I / F 23 to the HMD 1 for display.

[0053] (2-3) Robot 3

[0054] For example Figure 1 As shown, the robot 3 has a main body, a head, an arm, and a leg. The head, the arm, and the leg perform actions within the movable ranges specified by the drive unit having a servo mechanism, respectively, so that the robot 3 can be changed into various postures. A camera 34 as a photographing device is provided at the front of the head of the robot 3, that is, at a portion corresponding to the face. In addition, a motion sensor is provided in the head of the robot 3 to detect its motion.

[0055] Figure 4 and Figure 5 These are block diagrams showing the hardware structure and software structure of the robot 3 respectively.

[0056] The robot 3 includes a control unit 31 . The control unit 31 includes a hardware processor such as a CPU. A storage unit 32 , a sensor interface (sensor I / F) 33 , a drive interface (drive I / F) 37 , and a communication interface (communication I / F) 39 are connected to the control unit 31 via a bus 30 .

[0057] The camera 34, the motion sensor 35, and the position sensor 36 are connected to the sensor I / F 33. The camera 34 includes, for example, a dual-camera camera capable of capturing a stereoscopic image and a panoramic camera capable of capturing a panoramic image. The dual-camera camera captures the front direction of the robot 3 at a predetermined field of view angle (e.g., 130 degrees), and outputs the obtained stereoscopic image data to the sensor I / F 33. The panoramic camera captures the range from the front direction of the robot 3 to both sides at a larger field of view angle (e.g., 180 degrees) than the dual-camera camera, and outputs the obtained panoramic image data to the sensor I / F 33.

[0058] The motion sensor 35 is composed of, for example, a 6-axis acceleration sensor, and detects the 6-axis motion of the head of the robot 3 and outputs the detection data to the sensor I / F 33. The position sensor 36 measures the position of the robot 3 using, for example, a Global Positioning System (GPS), and outputs the measured position data to the sensor I / F 33.

[0059] The driving unit 38 for driving the head, arms and legs is connected to the driving I / F 37. The driving I / F 37 converts the control signal output from the control unit 31 into a driving signal, and outputs the converted driving signal to the driving unit 38 to cause the driving unit 38 to operate.

[0060] The communication I / F 39 has an interface corresponding to, for example, a wireless LANA, a short-range wireless data communication standard (such as Bluetooth (registered trademark)), or a public mobile communication network, and transmits data to and from the information processing device 2 via the network 4. In addition, the communication I / F 30 may have an interface corresponding to a wired network such as a public wired network, a wired LAN, or a CATV network.

[0061] The storage unit 32 is composed of a nonvolatile memory such as an SSD that can be written and read at any time, a ROM, and a RAM, and has a program storage area and a data storage area. Based on middleware such as an OS, various applications / programs for realizing the actions of the robot 3 are stored in the program storage area.

[0062] The data storage area includes an image data storage unit 321 , an action scene storage unit 322 , and an image parameter storage unit 323 .

[0063] The image data storage unit 321 functions as a buffer memory for temporarily storing image data captured by the camera 34 in order to perform image processing described later.

[0064] The action scenario storage unit 322 includes: a first table storing information indicating a plurality of scenarios assumed as action scenarios of the robot 3; and a second table storing information indicating a plurality of sub-scenarios obtained by more finely classifying the action scenarios. In the first table, information indicating action scenarios assumed for the robot 3 is stored in association with a combination of the position and action of the robot 3 and the image of the camera 34. In addition, in the second table, information indicating a plurality of sub-scenarios further assumed based on the image is stored for each action scenario defined in the first table. The information indicating the action scenario and the information indicating the sub-scenario will be described below with examples.

[0065] In addition, the above-mentioned action scene information and its sub-scene information can be stored in a table in advance by an administrator, for example, they can be appropriately acquired from a scheduler of a user's terminal device or a system that predicts the actions of other users and stored in a table. In this way, the actions of each user can be appropriately estimated and the corresponding action scene information and sub-scene information can be set.

[0066] For example, if a meeting schedule is recorded in the scheduler, the meeting materials need to be read, so the image parameters can be adjusted to settings based on the reading. In addition, if a baseball game schedule is recorded in the scheduler, the image parameters can be adjusted to settings corresponding to watching sports games.

[0067] The video parameter storage unit 323 stores, for each of the above-mentioned action scenes, video parameters preset for the sub-scenes in association with the sub-scenes. The video parameters will be described below with examples.

[0068] As a processing function unit involved in one embodiment of the present invention, the control unit 31 includes a remote control signal receiving unit 311, a motion control data extracting unit 312, a drive control unit 313, an image data acquiring unit 314, a position / motion detection data acquiring unit 315, an action scene determining unit 316, an image control unit 317, an image processing unit 318, and an image sending unit 319. The above control processing units are all implemented by causing the hardware processor of the above control unit 31 to execute the program stored in the storage unit 32.

[0069] The remote control signal receiving unit 311 receives the remote control signal transmitted from the information processing device 2 via the network 4 via the communication I / F 39 , and delivers the received remote control signal to the motion control data extracting unit 312 .

[0070] The motion control data extraction unit 312 extracts the motion control signal from the remote control signal and transmits the extracted motion control signal to the drive control unit 313 .

[0071] The drive control unit 313 selects the corresponding drive unit 38 based on the information specifying the control target part contained in the extracted motion control signal, and generates a drive control signal for driving the selected drive unit 38 based on the information specifying the control direction and control amount contained in the motion control signal. Then, the drive control unit 313 outputs the generated drive control signal to the drive I / F 37. The drive I / F 37 generates a drive signal based on the drive control signal and gives it to the drive unit 38.

[0072] The image data acquisition unit 314 imports the stereoscopic image data or the panoramic image data captured by the camera 34 via the sensor I / F 33 and temporarily stores the stereoscopic image data or the panoramic image data in the image data storage unit 321 for image processing described later.

[0073] The position / motion detection data acquisition unit 315 performs processing for importing the position data detected by the position sensor 36 and the motion detection data indicating the motion of the head of the robot 3 detected by the motion sensor 35 via the sensor I / F 33 .

[0074] The action scene determination unit 316 extracts feature quantities from the position data and action detection data acquired by the position / action detection data acquisition unit 315 and the image data acquired by the image data acquisition unit 314. Then, the action scene determination unit 316 performs processing to determine the current action scene and its sub-scenes of the robot 3 based on the extracted feature quantities with reference to the action scene storage unit 322.

[0075] The video control unit 317 selects video parameters corresponding to the action scene and the sub-scene determined by the action scene determination unit 316 from the video parameter storage unit 323 , and gives the video parameters to the video processing unit 318 .

[0076] The video processing unit 318 reads the video data from the video data storage unit 321 and performs a process of adjusting the video parameters given from the video control unit 317 for the video data. An example of the video parameter adjustment process will be described in detail later.

[0077] The image transmission unit 319 performs a process of transmitting the image data subjected to the image processing by the image processing unit 318 from the communication I / F 39 to the information processing device 2 .

[0078] (Action Example)

[0079] Next, an operation example of the robot 3 configured as described above will be described. Figure 6This is a flowchart showing an example of the overall processing procedure and processing content by the robot 3.

[0080] When a remote control signal including a motion control signal is transmitted from the information processing device 2, the robot 3 receives the remote control signal via the communication I / F 39 in step S11 under the control of the remote control signal receiving unit 311. Furthermore, under the control of the motion control data extracting unit 312, the motion control signal is extracted from the remote control signal in step S12.

[0081] Next, under the control of the drive control unit 313, the robot 3 selects the corresponding drive unit 38 in step S13 based on the information specifying the control target part contained in the above-mentioned motion control signal. Moreover, based on the information specifying the control direction and the control amount contained in the above-mentioned motion control signal, a drive control signal for driving the selected drive unit 38 is generated, and the generated drive control signal is output to the drive I / F 37. As a result, the drive signal is given to the drive unit 38 from the drive I / F 37, and the drive unit 38 is driven to make the corresponding part perform the action.

[0082] For example, if the control target part is the head, the corresponding drive unit 38 is driven so that the direction of the head changes to, for example, a rolling direction or a tilting direction. In addition, if the control target part is the leg, the corresponding drive unit 38 is driven so that the robot 3 walks in, for example, the front-back direction. The other parts also perform actions in the same manner according to the drive control signal.

[0083] On the other hand, when the head or legs are performing actions, the control unit 31 of the robot 3, under the control of the image data acquisition unit 314, imports image data from the camera 34 installed on the head via the sensor I / F 33 in step S14, and temporarily stores the image data in the image data storage unit 321.

[0084] At the same time, the control unit 31 of the robot 3 imports the motion detection data indicating the change of the head motion of the robot 3 from the motion sensor 35 via the sensor I / F 33 in step S15 under the control of the position / motion detection data acquisition unit 315. In addition, in step S16, the position data indicating the current position of the robot 3 is imported from the position sensor 36 via the sensor I / F 33.

[0085] Next, the control unit 31 of the robot 3 , under the control of the action scene determination unit 316 and the video control unit 317 , performs the following process of determining the current action scene of the robot 3 and selecting video parameters in step S20 . Figure 7It is a flowchart showing the processing order and processing contents.

[0086] That is, the action scene determination unit 316 first determines the feature quantities based on the position data and action detection data acquired by the position / action detection data acquisition unit 315 and the image data acquired by the image data acquisition unit 314 in step S21. Then, the action scene storage unit 322 is searched based on the determined feature quantities to determine the action scene of the robot 3.

[0087] For example, currently the first table in the action scene storage unit 322 stores Figure 8 In this state, the action scene determination unit 316 first determines that the feature quantity representing the position of the robot 3 is "home" based on the position data and the map data stored in the storage unit 32. In addition, based on the motion detection data, it is determined that the feature quantity representing the motion of the robot 3 is "stationary state". And, the "book" reflected in the image data is determined as a feature quantity. In addition, the action scene determination unit 316 determines that the feature quantity representing the position of the robot 3 is "home" based on the position data and the map data stored in the storage unit 32. Figure 8 The action scene information shown determines that the current action scene of the robot 3 is "reading".

[0088] Similarly, if it is determined that the feature quantity of the current position of the robot 3 is "library", the feature quantity of the action of the robot 3 is "stationary state", and if the feature quantity extracted from the image is "book", the action scene determination unit 316 determines that the current position of the robot 3 is "library", the feature quantity of the action of the robot 3 is "stationary state", and ...". Figure 8 The action scene information shown in the figure also determines that the action scene of the robot 3 at this time is "reading". In addition, if the feature quantity of the current position of the robot 3 is determined to be "home" and the feature quantity of the action of the robot 3 is "action state", and if the feature quantity extracted from the image is "kitchen", the action scene determination unit 316 determines that the action scene of the robot 3 at this time is "cooking".

[0089] On the other hand, the feature quantity of the current position of the robot 3 is determined to be "sports center" based on the position data and the map data stored in the storage unit 32, and the feature quantity of the action of the robot 3 is determined to be "stationary state" based on the action detection data, and the feature quantity extracted from the image data is set to "field or court". In this case, the action scene determination unit 316 determines the current position of the robot 3 based on the position data and the map data stored in the storage unit 32. Figure 8 Based on the action scene information shown, the current action scene of the robot 3 is determined to be "watching a sports game".

[0090] If it is determined as "reading" in the above step S21, then in step S22, the action scene determination unit 316 determines the sub-scene further classified from the reading scene based on the above-mentioned feature value extracted from the image data and the sub-scene information stored in the second table of the action scene storage unit 322.

[0091] For example, currently Table 2 stores Fig. 9 In the sub-scene information exemplified in FIG, the image of the “book”, which is the feature quantity extracted from the image data, is set as the “monochrome subject”. In this case, the action scene determination unit 316 determines the sub-scene as a reading scene of a “literary book” such as a single volume or a library book. In contrast, the image of the “book”, which is the feature quantity extracted from the image data, is set as the “color subject”. In this case, the action scene determination unit 316 determines the sub-scene as a reading scene of a “reference book” such as an intaglio book, a picture book, or an illustrated book.

[0092] When the sub-scene is determined in the above-described manner, the control unit 31 of the robot 3 then refers to the image parameter storage unit 323 under the control of the image control unit 317 and selects an image parameter set corresponding to the sub-scene.

[0093] For example, currently in the image parameter storage unit 323 Fig.10 In this case, if the image control unit 317 sets the determined sub-scene to "literary book", the image control unit 317 selects "spatial resolution" and "brightness" as image parameters that are prioritized when reading the literary book based on the image in step S23. On the other hand, if the determined sub-scene is set to "reference book", the image control unit 317 selects "spatial resolution" and "hue" as image parameters that are prioritized when reading the reference book based on the image in step S24.

[0094] If the above-mentioned image parameters are selected, the control unit 31 of the robot 3, under the control of the image processing unit 318, performs a process of adjusting the above-mentioned selected image parameters for the image data read from the image data storage unit 321 in step S17. Then, the adjusted image data is transmitted from the communication I / F 39 to the information processing device 2 in step S18 under the control of the image transmission unit 319.

[0095] For example, when "spatial resolution" and "brightness" are selected as image parameters, the image processing unit 318 performs the following processing, that is, for example, image processing is performed in a manner such that the spatial resolution reaches a preset high resolution, and the brightness is further increased to a specified level. Furthermore, the image sending unit 319 sends the image data after the above processing to the information processing device 2. In contrast, when "spatial resolution" and "hue" are selected as image parameters, the image processing unit 318 performs the following processing, that is, for example, image processing is performed in a manner such that the spatial resolution reaches a preset high resolution, and the chromaticity of the color is further increased to a specified level. Furthermore, the image sending unit 319 sends the image data after the above image processing to the information processing device 2.

[0096] As a result, when the user uses the HMD 1 to read a book photographed by the robot 3, if it is a literary book, the user can clearly visually confirm the text through the image set to a high resolution and high brightness. On the other hand, when reading a reference book such as a picture book or illustrated book, the user can clearly visually confirm the pictures and photos through the image set to a high resolution and increased color.

[0097] Furthermore, as a method of adjusting the “brightness”, in addition to adjusting the image data, a method of adjusting the so-called focusing function of the camera can also be used.

[0098] On the other hand, if the action scene of the robot 3 is determined as "watching sports games" in the above-mentioned step S21, the action scene determination unit 316 then determines the sub-scene as its sub-classification based on the image data and the sub-scene information stored in the second table of the action scene storage unit 322 in step S25.

[0099] For example, if the feature quantity currently extracted from the image data is "outdoors", the action scene determination unit 316 determines that the sub-scene is watching a "field game" such as soccer or rugby. On the other hand, if the feature quantity extracted from the image data is "indoors", the action scene determination unit 316 determines that the sub-scene is watching a "court game" such as basketball or volleyball.

[0100] Furthermore, if the above sub-scene is determined, the control unit 31 of the robot 3 then refers to the image parameter storage unit 323 under the control of the image control unit 317 and selects the image parameters set corresponding to the above sub-scene.

[0101] For example, if the sub-scene is currently determined to be "field game", the image control unit 317 selects "temporal resolution" and "angle of view" as image parameters that are prioritized when viewing the field game through images in step S26. On the other hand, if the sub-scene is determined to be "stadium game", the image control unit 317 selects "temporal resolution" and "stereoscopic effect" as image parameters that are prioritized when viewing the stadium game through images in step S27.

[0102] Next, the control unit 31 of the robot 3 adjusts the selected image parameters for the image data in step S17 under the control of the image processing unit 318. The image data with the image parameters adjusted is transmitted from the communication I / F 39 to the information processing device 2 in step S18 under the control of the image transmitting unit 319.

[0103] For example, when "temporal resolution" and "angle of view" are selected as image parameters, the image processing unit 318 selectively reads panoramic image data with a high frame rate and a large angle of view from the image data storage unit 321. Then, the image sending unit 319 sends the panoramic image data to the information processing device 2. On the other hand, when "temporal resolution" and "stereoscopic effect" are selected as image parameters, the image processing unit 318 selectively reads stereoscopic image data with a high frame rate and stereoscopic effect from the image data storage unit 321. Then, the image sending unit 319 sends the stereoscopic image data to the information processing device 2.

[0104] Furthermore, the panoramic image and the stereoscopic image may be selected by switching the image data output from the camera 34 by issuing a switching instruction of the shooting mode to the camera 34. In addition, when the frame rate of the panoramic image alone is insufficient, the panoramic image and the stereoscopic image may be synthesized so that the coordinate positions are consistent. However, when watching a field game, the distance from the camera to the field is usually far, so even if the frame rate is high, the effect on the viewing and hearing is small.

[0105] As described above, when the user wants to watch the video of the field game shot by the robot 3 in the stadium or the like through the HMD 1, the user can watch the game in a wide range without missing any panoramic video with a large field of view. On the other hand, when the user wants to watch the video of the field game shot by the robot 3 in the arena or the like through the HMD 1, the user can watch the video with a sense of presence with a stereo video with a high frame rate.

[0106] (Function / Effect)

[0107] As described above, in one embodiment, based on the feature quantity derived from the position and motion detection data and the image data of the remote robot 3, the action scene of the robot 3 is determined, and the image parameters corresponding to the determined action scene are selected. Furthermore, the image data is processed to adjust the selected image parameters, and the processed image data is transmitted to the information processing device 2 on the user side via the network 4 and displayed on the HMD 1.

[0108] Therefore, the remote robot 3 can transmit the image data whose image parameters are adjusted according to the action scene of the robot 3 to the information processing device 2 on the user side. Therefore, the user can watch and listen to the remote image processed with the image parameters suitable for each action scene of the robot 3 controlled remotely by the user, thereby improving the visual confirmation of the remote image.

[0109] In addition, the robot 3 autonomously performs the adjustment processing for the image parameters of the image data. Therefore, it is not necessary for the user's device, that is, the information processing device 2 or HMD 1, to have a special processing function for adjusting the image parameters, so that the user can use a general-purpose information processing device 2 or HMD 1.

[0110] [Other embodiments]

[0111] (1) In the above-mentioned embodiment, an example is given in which an image parameter suitable for the scene is selected based on the result of the determination of the action scene of the robot 3, and the selected image parameter is adjusted for the image data. However, the present invention is not limited to this. For example, a shooting mode suitable for the scene may be selected based on the result of the determination of the action scene of the robot 3, and the selected shooting mode may be set for the camera, thereby transmitting the shot image data to the user device.

[0112] Typically, a camera has multiple shooting modes, for example, a mode for shooting with a lower temporal resolution (15 FPS FPS) and a higher spatial resolution (4K), a mode for shooting with a higher temporal resolution (120 FPS) and a lower spatial resolution (VGA), and so on.

[0113] Therefore, if the above-mentioned shooting mode is selectively set for the camera according to the action scene of the robot 3 and shooting is performed under this condition, image data with a temporal resolution or spatial resolution suitable for the action scene of the above-mentioned robot 3 can be sent to the user device, thereby achieving the same effect as the above-mentioned embodiment.

[0114] (2) In the above-mentioned embodiment, the case where the robot 3 is remotely controlled according to the remote control signal sent from the information processing device 2 is used as an example for explanation. However, the present invention is not limited to this, and when the robot 3 acts autonomously according to a pre-set program, the robot 3 may be configured as follows, that is, the robot 3 determines its own action scene, selects image parameters corresponding to the action scene, adjusts the selected image parameters for the image data, and sends the adjusted image data to the information processing device 2 on the user side.

[0115] (3) In one embodiment, a pre-prepared storage table storing sub-scene information and image parameter information is provided, and the action scene and sub-scene are determined and the image parameters are selected with reference to the storage table. However, the present invention is not limited to this. For example, machine learning such as deep learning can be used to select the best image parameters. This can be achieved, for example, in the following manner, that is, feature quantities are respectively extracted from the robot's position data, action detection data, and image data, and the extracted feature quantities are input into a learning model that has been learned, and the best image parameters are output from the learning model.

[0116] (4) In the above-described embodiment, data communication and image display control with the robot 3 are performed by the information processing device 2 provided separately from the HMD 1. However, when the HMD 1 has the function of the information processing device 2, that is, when an HMD integrated with the information processing device is used, data communication and image display control with the robot 3 may be performed by the HMD 1.

[0117] (5) The remote working device is not limited to a humanoid robot, and a movable remote camera or smart speaker fixedly installed indoors or outdoors may also be used. In addition, various modifications may be made to the type of remote working device, its structure and processing content, the type and number of shooting modes of the camera, the type and structure of the user device, the type of remote image, etc. without departing from the scope of the present invention.

[0118] The embodiments of the present invention have been described in detail above, but the aforementioned description is only an example of the present invention in all aspects. Of course, various improvements and deformations can be made without departing from the scope of the present invention. That is, when implementing the present invention, a specific structure corresponding to the embodiment can be appropriately adopted.

[0119] In a word, the present invention is not limited to the above-mentioned embodiments, and the structural elements can be deformed and embodied in the implementation stage without departing from the scope of its main purpose. In addition, various inventions can be formed by appropriate combinations of multiple structural elements disclosed in the above-mentioned embodiments. For example, several structural elements can be deleted from all the structural elements shown in the embodiments. In addition, the structural elements of different embodiments can be appropriately combined.

[0120] Description of the label

[0121] 1…Head-mounted display (HMD)

[0122] 2…Information processing device

[0123] 3…Robots

[0124] 4…Network

[0125] 10, 20, 30…bus

[0126] 11, 21, 31…Control Unit

[0127] 12, 22, 32... Storage

[0128] 13…Display interface (display I / F)

[0129] 14…Display unit

[0130] 15, 33…Sensor interface (sensor I / F)

[0131] 16, 35…Motion sensor

[0132] 17, 24, 39…Communication interface (communication I / F)

[0133] 23…Input and output interface (input and output I / F)

[0134] 25…Operation Department

[0135] 34…Camera

[0136] 35…Motion Sensor

[0137] 36…Position sensor

[0138] 37…Drive interface (Drive I / F)

[0139] 38…Driver

[0140] 311…Remote control signal receiving unit

[0141] 312…Motion control data extraction unit

[0142] 313…Drive control unit

[0143] 314…Image data acquisition unit

[0144] 315…Position / motion detection data acquisition unit

[0145] 316…Action Scene Judgment Department

[0146] 317…Image Control Department

[0147] 318…Image Processing Department

[0148] 319…Image transmission department

[0149] 321…Image data storage unit

[0150] 322…Action scene storage

[0151] 323…Image parameter storage unit

Claims

1. A remote operation device capable of communicating with a user device connected to a display unit via a network, wherein: The remote operation device comprises: an image data acquisition unit that acquires, from a camera, image data of an object photographed by the camera; a state detection data acquisition unit that acquires, from a sensor, state detection data indicating at least one of a position and a motion of the remote working device detected by the sensor; a determination unit that determines an action scene of the remote operation device based on the acquired image data and the state detection data; A selection unit, which selects image parameters of an adjustment object according to the determined action scene; a control unit, which adjusts the selected image parameters with respect to the image data; and a transmitting unit configured to transmit the image data after adjusting the image parameters to the user device via the network, The determination unit determines whether the action scene is a first scene in which a subject in a stationary state is mainly photographed by the camera based on the image data and the state detection data. When it is determined that the action scene is the first scene, the selection unit selects spatial resolution as the image parameter. The control unit performs a process of adjusting the selected spatial resolution for the image data. The determination unit determines whether the action scene is a second scene in which the camera mainly photographs a moving object based on the image data and the state detection data. When it is determined that the action scene is the second scene, the selection unit selects time resolution as the video parameter. The control unit performs a process of adjusting the selected temporal resolution for the image data.

2. A remote operation device capable of communicating with a user device connected to a display unit via a network, wherein: The remote operation device comprises: an image data acquisition unit that acquires, from a camera, image data of an object photographed by the camera; a state detection data acquisition unit that acquires, from a sensor, state detection data indicating at least one of a position and a motion of the remote working device detected by the sensor; a determination unit that determines an action scene of the remote operation device based on the acquired image data and the state detection data; a selection unit, which selects a shooting mode corresponding to the determined action scene; a control unit configured to set the selected shooting mode for the camera; as well as a transmitting unit configured to transmit the image data captured by the camera using the set shooting mode to the user device via the network, The determination unit determines whether the action scene is a first scene in which a subject in a stationary state is mainly photographed by the camera based on the image data and the state detection data. When it is determined that the action scene is the first scene, the selection unit selects spatial resolution as the shooting mode. The control unit performs a process of setting a shooting mode for obtaining the selected spatial resolution for the camera. The determination unit determines whether the action scene is a second scene in which the camera mainly photographs a moving object based on the image data and the state detection data. When it is determined that the action scene is the second scene, the selection unit selects time resolution as the shooting mode, The control unit performs a process of setting a photographing mode for obtaining the selected spatial resolution for the camera.

3. The remote operation device according to claim 1 or 2, wherein: When it is determined that the action scene is the first scene, the determination unit further determines, based on the image data, whether the color tone of the object is mainly monochrome. When it is determined that the color tone of the object is mainly monochrome, the selection unit further selects brightness as the image parameter. The control unit performs processing for further adjusting the selected brightness on the image data, or sets a shooting mode for obtaining the selected brightness on the camera.

4. The remote operation device according to claim 1 or 2, wherein: When it is determined that the action scene is the first scene, the determination unit further determines, based on the image data, whether the hue of the object is mainly color. When it is determined that the hue of the object is mainly color, the selection unit further selects hue as the image parameter or the shooting mode. The control performs a process of further adjusting the selected color tone for the image data or a process of setting a shooting mode for obtaining the selected color tone for the camera.

5. The remote operation device according to claim 1 or 2, wherein: When it is determined that the action scene is the second scene, the determination unit determines whether the subject is outdoors based on the image data. When it is determined that the object is outdoors, the selection unit further selects a field of view angle as the image parameter or the shooting mode. The control unit performs a process of further adjusting the selected angle of view with respect to the image data, or performs a process of setting a shooting mode for obtaining the selected angle of view with respect to the camera.

6. The remote operation device according to claim 1 or 2, wherein: When it is determined that the action scene is the second scene, the determination unit determines whether the subject is indoors based on the image data. When it is determined that the object is indoors, the selection unit further selects stereoscopic effect as the image parameter or the shooting mode. The control unit performs a process of further adjusting the selected stereoscopic effect on the image data, or performs a process of setting a shooting mode for obtaining the selected stereoscopic effect on the camera.

7. A computer-readable storage medium storing a program, wherein: The program causes a processor included in the remote working device according to any one of claims 1 to 6 to execute processing by the respective components included in the remote working device.

Images