Robot VR System

Abstract

To enable a robot VR system to generate a VR video in real time.SOLUTION: A video transmission device comprises: a video transmission unit which transmits a plurality of camera videos photographed by a mobile robot to a VR video generation unit which generates a VR video; and a transmission video control unit which reduces the bit rate of a partial region of the plurality of camera videos transmitted from the video transmission unit.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a robot VR system that generates VR video from multiple camera images taken by a mobile robot. [Background technology]

[0002] With the spread of 5G / FTTH and the growing demand for remote work due to the spread of infectious diseases, use cases in which operators remotely control mobile robots for monitoring and communication are becoming more common. In this use case, images from cameras installed on the mobile robots are transmitted over the network, and the operators operate the robots while viewing the images.

[0003] To enable more realistic operation, a robot VR system that displays VR images to the operator via a VR (Virtual Reality) device is being studied. In order to display VR images, it is necessary to stitch together in real time images taken by multiple cameras mounted on the mobile robot, generate a 360-degree VR image around the mobile robot, and display it on the VR device.

[0004] A real-time VR communication system has been proposed for generating VR video in real time from video captured by multiple cameras (see, for example, Patent Document 1). In Patent Document 1, VR video is generated on the service provider side.

[0005] However, mobile robots are small and cannot be equipped with computers with high processing power, making it impossible to generate VR images on mobile robots. Furthermore, due to the nature of mobile robots, they must connect to networks wirelessly. This means that it is not possible to secure the bandwidth required to transmit images from multiple cameras, and there is a risk that VR images will not be generated in real time due to delays or failures in the delivery of camera images. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-167699 Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, an object of the present disclosure is to enable the generation of VR video in real time in a robot VR system. [Means for solving the problem]

[0008] The video transmission device of the present disclosure includes: a transmission image control unit that acquires a plurality of camera images captured by a mobile robot and reduces the bit rate of at least one of the camera images; a video transmission unit that transmits the plurality of camera videos whose bit rates have been reduced by the transmission video control unit to a VR video generation unit that generates VR video using the plurality of camera videos; Equipped with.

[0009] The video transmission method of the present disclosure includes: a transmission image control unit that acquires a plurality of camera images taken by the mobile robot and reduces the bit rate of at least one of the plurality of camera images; A video transmission unit transmits the multiple camera videos, the bit rate of which has been reduced by the transmission video control unit, to a VR video generation unit that generates a VR video using the multiple camera videos.

[0010] The robot VR system of the present disclosure includes: A video transmission device according to the present disclosure; a VR video generation unit that generates a VR video using the plurality of camera videos transmitted from the video transmission device; Equipped with.

[0011] In the present disclosure, the transmission video control unit may reduce the bit rate of areas of the multiple camera images that are not within the field of view of the operator, based on posture information of the operator of the VR device that displays the VR video.

[0012] In the present disclosure, the mobile robot may further include a motor control unit that controls the orientation of the mobile robot based on posture information of an operator of a VR device that displays the VR video.

[0013] In the present disclosure, the video transmission device Acquire posture information of an operator of a VR device that displays the VR video; when the posture information of the operator has changed, a transmission video control unit is executed to reduce the bit rate of an area outside the field of view of the operator based on the posture information of the operator after the change; When the posture information of the operator continues for a certain period of time, a motor control unit may be executed to control the orientation of the mobile robot based on the posture information that has continued for the certain period of time.

[0014] The program of the present disclosure is a program for causing a computer to realize each functional unit of the video transmission device and VR video generation unit of the present disclosure, and is a program for causing a computer to execute each step of the method performed by the video transmission device of the present disclosure.

[0015] The above disclosures can be combined as much as possible. [Effects of the Invention]

[0016] According to the present disclosure, it is possible to generate VR video in real time in a robot VR system. [Brief explanation of the drawings]

[0017] [Figure 1] An example of the configuration of a real-time VR communication system is shown below. [Figure 2] An example of the configuration of a robot VR system is shown below. [Figure 3] This is an example of combining footage from multiple cameras into VR footage. [Figure 4] 1 illustrates an example embodiment of a robot VR system according to the present disclosure. [Figure 5] This is an example of combining footage from multiple cameras into VR footage. [Figure 6] 1 illustrates an example embodiment of a robot VR system according to the present disclosure. [Figure 7] This is an example of combining footage from multiple cameras into VR footage. [Figure 8] 1 illustrates an example embodiment of a robot VR system according to the present disclosure. [Figure 9] This is an example of combining footage from multiple cameras into VR footage. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings indicate the same components.

[0019] (Real-time VR communication system) An example of the configuration of a real-time VR communication system is shown in Figure 1. In the real-time VR communication system, the service provider is equipped with multiple cameras 51 that capture a partial range, and a VR video transmission device 100. The VR video transmission device 100 is equipped with a real-time stitching unit 123, a VR video output unit 124, and a video transmission unit 112.

[0020] The real-time stitching unit 123 generates VR video by synthesizing multiple camera images from each camera 51 in real time. The VR video output unit 124 outputs the VR video, and the video transmission unit 112 transmits the VR video. The VR video is transmitted via the network 41 and received by the video receiving unit 122. This makes it possible for the VR device 30 to display the VR video.

[0021] (Robot VR System) 2 shows an example of the configuration of a robot VR system. In this disclosure, an example is shown in which a mobile robot 50 is equipped with cameras 51A, 51B, and 51C. In the robot VR system, the mobile robot 50 is operated by a VR device 30. A method for controlling the mobile robot 50 by the VR device 30 at this time will not be described in this disclosure.

[0022] Because the mobile robot 50 is small, it cannot be equipped with a computer with high processing performance such as real-time stitching. Therefore, in the robot VR system, the mobile robot 50 is equipped with a camera image transmission device 10, and a VR image generation unit 20 is arranged outside the mobile robot 50. The camera image transmission device 10 functions as the image transmission device of the present disclosure. The VR image generation unit 20 can be arranged in any device outside the mobile robot 50, for example, in a server or PC near the VR device 30.

[0023] The camera video transmission device 10 includes a network interface (NW IF) 11 and a video transmission unit 12. The VR video generation unit 20 includes a network interface (NW IF) 21, a video reception unit 22, a real-time stitching unit 23, and a VR video output unit 24. The network interfaces 11 and 21 are connected via a network 41.

[0024] The video transmitting unit 12 transmits multiple camera videos via the network 41, the video receiving unit 22 receives the multiple camera videos, and the real-time stitching unit 23 performs stitching. In this embodiment, an example is shown in which the mobile robot 50 uses cameras 51A, 51B, and 51C to capture images in a 360-degree direction around the mobile robot 50. In this case, the real-time stitching unit 23 combines the camera videos PA, PB, and PC captured by the cameras 51A, 51B, and 51C, as shown in FIG. 3, to generate a VR video P360 of a 360-degree direction around the mobile robot 50.

[0025] Here, the stitching method in the real-time stitching unit 23 is arbitrary and is not limited in the present disclosure. For example, a general method can be used in which the matching portions of the camera images PA, PB, and PC are combined to make the seams invisible.

[0026] The cameras 51A, 51B, and 51C may be arranged in any desired manner, but in this embodiment, the cameras 51A, 51B, and 51C are arranged at 120-degree intervals in the horizontal direction, and the camera 51B captures images in the forward direction of the mobile robot 50. Note that, although this embodiment illustrates an example in which three cameras are used to capture camera images in a 360-degree horizontal direction, the number of cameras may be four or more. Camera images in a 360-degree vertical direction may also be captured and generated as VR images. Furthermore, the VR images generated by the real-time stitching unit 23 are not limited to a 360-degree direction, and may have a range corresponding to the display range of the VR device 30.

[0027] The VR video output unit 24 outputs the VR video P360 from the real-time stitching unit 23 to the VR device 30. Of course, in the robot VR system of this embodiment, it is also possible to further transmit the VR video from the VR video generation unit 20 to the VR device 30 via a network (not shown).

[0028] Due to its characteristics, the mobile robot 50 needs to use wireless communication. Therefore, in the robot VR system, there is a possibility that the bandwidth required for transmitting the multiple camera images PA, PB, and PC may not be secured.

[0029] Therefore, the robot VR system of the present disclosure has the following configuration. First configuration: The camera image transmitting device 10 includes a transmission image control unit that reduces the bit rate of a portion of the multiple camera images PA, PB, and PC captured by the mobile robot 50. Here, the transmission image control unit may reduce the bit rate of portions of the multiple camera images PA, PB, and PC that are not within the field of view of the operator of the VR device 30, based on the posture information received from the VR device 30. Second configuration: The camera image transmitting device 10 includes a motor control unit that controls the orientation of the mobile robot 50 based on the orientation information received from the VR device 30. Third configuration: The VR video generation unit 20 includes a real-time stitching unit 23 that controls the orientation when combining the multiple camera videos PA, PB, and PC based on the control information received from the motor control unit. Fourth configuration: The camera image transmitting device 10 receives posture information from the VR device 30, and includes a state control unit that operates the first and second configurations in response to changes in the posture information.

[0030] The robot VR system of the present disclosure includes a transmission video control unit (first configuration), which realizes a method for reducing the bit rate while maintaining the user's experience when transmitting multiple camera videos PA, PB, and PC. This enables the operator to operate the mobile robot 50 using the VR device 30 without impairing the operator's experience.

[0031] In the robot VR system, since the directions of both the mobile robot 50 and the VR device 30 are variable, the forward direction of the operator and the forward direction of the mobile robot 50 may not match. Therefore, the present disclosure uses a motor control unit (second configuration) and a real-time stitching unit (third configuration). This makes it possible to match the forward direction of the mobile robot 50 and the forward direction of the operator of the VR device 30 while simultaneously reducing the bit rate and maintaining the user experience.

[0032] Furthermore, in a robot VR system, it may take time for the mobile robot 50 to change direction due to motor constraints, and frequent changes in direction in the VR video may reduce the operability of the VR device 30. Therefore, the present disclosure uses a state control unit (fourth configuration).

[0033] In the fourth configuration, the state control unit operates the transmission image control unit in response to changes in the posture information, and operates the motor control unit if the posture information does not change for a certain period of time. If the posture information does not change for a certain period of time, the VR image generation unit 20 may operate the third configuration. This makes it possible to reduce the frequency with which the direction of the mobile robot 50 is changed, while simultaneously reducing the bit rate and maintaining the user experience, and to align the forward direction of the mobile robot 50 with the forward direction of the operator of the VR device 30.

[0034] The camera image transmitting device 10 and the VR image generating unit 20 of the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0035] (Example 1) An embodiment of the robot VR system of the present disclosure is shown in Figure 4. In the robot VR system of this embodiment, the VR device 30 includes a posture information output unit 33, and the camera image transmitting device 10 includes a transmission image control unit 13.

[0036] The posture information output unit 33 outputs posture information of the operator of the VR device 30. The posture information is any information that can detect the forward direction of the operator, and can be acquired by a gyro sensor or the like that is provided in a typical VR device 30. The VR device 30 transmits the posture information output from the posture information output unit 33 to the camera image transmission device 10. In the figure, an example is shown in which the VR image generation unit 20 transmits the posture information to the camera image transmission device 10, but this is not limiting.

[0037] The transmission video control unit 13 acquires the multiple camera videos PA1, PB1, and PC1, and reduces the bit rate of some areas of the multiple camera videos PA1, PB1, and PC1. The video transmission unit 12 transmits the camera videos PA1, PB1, and PC1 from the transmission video control unit 13.

[0038] FIG. 5 shows an example of the operation of the transmission video control unit 13. The transmission video control unit 13 determines the camera video PA1 that matches the viewpoint of the operator of the VR device 30 based on the posture information from the VR device 30. Here, the viewpoint of the operator is, for example, the center of the operator's field of view obtained from the direction ahead of the operator.

[0039] Then, based on the forward direction of the operator, the transmission video control unit 13 determines areas AA1, AB1, and AC1 that are out of the operator's field of view from the camera images PA1, PB1, and PC1. The areas that are out of the operator's field of view can be, for example, areas that are a certain distance or more away from the operator's viewpoint.

[0040] Then, the transmission video control unit 13 reduces the bit rate of the areas AA1, AB1, and AC1 of the camera images PA1, PB1, and PC1. Here, the transmission video control unit 13 may reduce the bit rate of the entire camera image PC1, or may reduce the bit rate of a portion of the area AA1 of the camera image PA1. Furthermore, any method for reducing the bit rate may be used, and examples include compressing the amount of data of at least one of the camera images PA1, PB1, and PC1 by reducing the number of pixels of the camera image or by sending only a portion of the image.

[0041] In this embodiment, the bit rate of part of the camera images PA1 and PB1 is reduced, and the overall bit rate of the camera image PC1 is reduced. Therefore, in this embodiment, the bit rate when transmitting the camera images PA1, PB1, and PC1 can be reduced.

[0042] In the VR video generation unit 20, the video receiving unit 22 receives the camera videos PA1, PB1, and PC1 transmitted via the network 41, and the real-time stitching unit 23 synthesizes the multiple camera videos PA1, PB1, and PC1 in real time. This generates the VR video P360 shown in FIG. 5. In this embodiment, as shown in FIG. 5, the bit rate of areas AL and AR in the VR video P360 is reduced, but these are areas that are not within the field of view of the operator of the VR device 30. Therefore, there is little impact on the user experience.

[0043] As described above, the robot VR system of this embodiment is equipped with a posture information output unit 33 and a transmission image control unit 13, thereby realizing a method that can reduce the bit rate while maintaining the user experience when transmitting multiple camera images PA1, PB1, and PC1 to compose VR images.

[0044] (Example 2) An embodiment of the robot VR system of the present disclosure is shown in Figure 6. In the robot VR system of this embodiment, the VR device 30 includes a posture information output unit 33, and the camera image transmitting device 10 includes a transmitted image control unit 13 and a motor control unit 14.

[0045] The motor control unit 14 controls the motor of the mobile robot 50 in accordance with the posture information from the VR device 30 , and aligns the forward direction of the mobile robot 50 with the forward direction of the operator of the VR device 30 .

[0046] Furthermore, the motor control unit 14 notifies the real-time stitching unit 23 of control information for the mobile robot 50. The control information includes any information indicating the forward direction of the mobile robot 50. As a result, the real-time stitching unit 23 generates the VR video P360 so that the camera video taken in the forward direction of the mobile robot 50 is displayed in the forward direction of the operator.

[0047] In this embodiment, the forward direction of the mobile robot 50 and the operator are aligned. Therefore, as shown in Fig. 7, the transmission video control unit 13 sets the bit rate of the camera video PB2 from the camera 51B capturing the image in the forward direction of the mobile robot 50 to a high rate, and sets the bit rate of the camera videos PA2 and PC2 from the cameras 51A and 51C arranged on the sides of the mobile robot 50 to a low rate. In this case, the transmission video control unit 13 may reduce the bit rate of only the areas AA2 and AC2 of the camera videos PA2 and PC2 that do not overlap with the camera video PB2, and may maintain the bit rate of the areas that overlap with the camera video PB2 without reducing it.

[0048] In this embodiment, the orientation of the mobile robot 50 coincides with the forward direction of the operator. Therefore, in the VR video generation unit 20, the real-time stitching unit 23 stitches the camera video PB2 from the camera 51B capturing the forward direction of the mobile robot 50 based on the control information so that it is displayed in the forward direction of the operator, and changes the angle of the VR video P360 to be output. This makes it possible to prevent the VR video on the VR device 30 from rotating when the direction of the mobile robot 50 changes.

[0049] As described above, the robot VR system of this embodiment simultaneously changes the direction of the mobile robot 50 and corrects the direction of the VR video P360 output from the real-time stitching unit 23, based on the posture information from the VR device 30. As a result, the robot VR system of this embodiment realizes a method of increasing the bit rate of the video in the area within the field of view of the operator of the VR device 30, while reducing the bit rate.

[0050] (Embodiment Example 3) An embodiment of the robot VR system of the present disclosure is shown in Figure 8. In the robot VR system of this embodiment, the VR device 30 includes a posture information output unit 33, and the camera image transmitting device 10 includes a transmitted image control unit 13, a motor control unit 14, and a state control unit 15.

[0051] The state control unit 15 detects a change in the viewpoint of the operator of the VR device 30 based on the posture information. When the state control unit 15 detects a change in the viewpoint of the operator, it instructs the transmission video control unit 13 to change the bit rate reduction area. Then, the camera video transmission device 10 performs the same operation as in the first embodiment. As a result, as shown in FIG. 9, the orientation of the mobile robot 50 does not change, and camera video with a reduced bit rate for the area outside the operator's field of view is transmitted from the camera video transmission device 10.

[0052] In this embodiment, while the operator's viewpoint continues to change, the state control unit 15 does not execute the motor control unit 14. This makes it possible to prevent a decrease in the operability of the VR device 30 due to frequent changes in the direction of the VR video.

[0053] When the state control unit 15 detects that the operator's viewpoint has been facing the same direction for a certain period of time, it causes the motor control unit 14 to change the direction of the mobile robot 50, as shown in FIG. 9, in the same manner as in the second embodiment, and causes the real-time stitching unit 23 to correct the direction of the VR image.

[0054] As described above, the robot VR system of this embodiment is equipped with a posture information output unit 33, a transmission image control unit 13, a motor control unit 14, and a state control unit 15, thereby reducing the bit rate of the portion of the camera image that is not within the field of view of the operator of the VR device 30, while simultaneously changing the direction of the mobile robot 50 and correcting the direction of the VR image output from the real-time stitching unit 23. [Industrial Applicability]

[0055] The present disclosure can be applied to the information and communications industry. [Explanation of symbols]

[0056] 10: Camera image transmission device 11, 111: Network interface (NW IF) 12, 112: Video transmission unit 13: Transmitted video control section 14: Motor control unit 15: State control section 50:Mobile robot 51, 51A, 51B, 51C: Camera 52: Motor 20:VR video generation section 21, 121: Network interface (NW IF) 22, 122: Video receiving unit 23: Real-time stitching section 24: VR video output section 30:VR device 33: Posture information output unit 41, 141: Network 100: VR video transmission device 123: Real-time stitching section 124: VR video output section

Claims

1. a transmission image control unit that acquires a plurality of camera images captured by a mobile robot and reduces the bit rate of at least one of the camera images; a video transmission unit that transmits the plurality of camera videos whose bit rates have been reduced by the transmission video control unit to a VR video generation unit that generates VR video using the plurality of camera videos; a state control unit that detects a change in the viewpoint of an operator of a VR device that displays the VR video, based on posture information including the viewpoint of the operator; a motor control unit that aligns the forward direction of the mobile robot with the forward direction of the operator based on the posture information; Equipped with the transmission video control unit reduces a bit rate of a region of the plurality of camera videos that is not within the field of view of the operator, based on a viewpoint of the operator; The state control unit When detecting a change in the operator's viewpoint, instruct the transmission video control unit to change a bit rate reduction area; If the posture information continues to change, the execution of the motor control unit is stopped; When it is detected that the operator's viewpoint is facing the same direction for a certain period of time, the motor control unit is executed. Video transmission device.

2. The video transmission device according to claim 1; a VR video generation unit that generates a VR video using the plurality of camera videos transmitted from the video transmission device; Equipped with the VR video generation unit acquires control information for the mobile robot, and generates the VR video based on the acquired control information so that a camera image taken in a forward direction of the mobile robot is displayed in a forward direction of an operator of a VR device that displays the VR video; Robot VR system.

3. a first step in which a transmission image control unit acquires a plurality of camera images captured by a mobile robot and reduces a bit rate of at least one of the camera images; a second step in which a video transmission unit transmits the plurality of camera videos whose bit rates have been reduced by the transmission video control unit to a VR video generation unit that generates a VR video using the plurality of camera videos; a third step in which a motor control unit aligns a forward direction of the mobile robot with a forward direction of an operator of a VR device that displays the VR video, based on posture information including a viewpoint of the operator of the VR device; Equipped with in the first step, when a change in the viewpoint of the operator is detected based on posture information including the viewpoint of the operator, the state control unit instructs the transmission video control unit to change a bit rate reduction region; In the third step, If the posture information continues to change, the execution of the motor control unit is stopped; When it is detected that the operator's viewpoint is facing the same direction for a certain period of time, the motor control unit is executed. Video transmission method.

4. A program for causing a computer to implement each functional unit of the video transmission device according to claim 1.

Images