Robot VR system

Abstract

To allow reduction in transmission rate in a robot VR system comprising a movable control object.SOLUTION: An image transmission device 300 comprises: a robot movement unit 140 that receives information on an operation of an image acquisition unit 110; a geometric transformation unit 120 that, on the basis of the operation information, performs geometric transformation of images acquired by the image photographing unit 110 and outputs resultant images as geometric transformation data; and a difference encoding unit 130A that calculates the difference between the images acquired by the image photographing unit 110 and the images obtained through the geometric transformation performed by the geometric transformation unit 120.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a robot VR system that generates VR video from camera images captured by a mobile robot. [Background technology]

[0002] Conventionally, when remotely controlling a controlled object such as a robot while watching images from a camera installed on the controlled object, there is a demand to operate the controlled object while watching VR images because the field of view is narrow with 2D images. Therefore, a robot VR system has been proposed in which the controlled object is equipped with a 360-degree camera and VR images are transmitted to the operator.

[0003] In such robot VR systems, high-definition VR images tend to have a higher bit rate than normal images, but since the network near the controlled object is wireless, there is a possibility that a sufficient bandwidth may not be available. Therefore, it has been proposed to reduce the bit rate of images by differentially encoding the images and transmitting them (see Non-Patent Documents 1 to 3).

[0004] In Non-Patent Documents 1 to 3, a video frame is divided into a number of regions, and each region is compared with the immediately preceding frame, and only the region of the video frame that has a difference is transmitted. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Ching-Yeh Chen, Shao-Yi Chien, Yu-Wen Huang, Tung-Chien Chen, Tu-Chih Wang and Liang-Gee Chen, “Analysis and architecture design of variable block-size motion estimation for H.264 / AVC,” in IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 53, no. 3, pp. 578-593, March 2006. [Non-Patent Document 2] Alireza Zare, Alireza Aminlou, Miska M. Hannuksela, and Moncef Gabbouj. 2016. HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications. In Proceedings of the 24th ACM international conference on Multimedia (MM '16). [Non-Patent Document 3] Park, Sang-Hyo, and Je-Won Kang. “Fast affine motion estimation for versatile video coding (VVC) encoding.” IEEE Access 7 (2019): 158075-158084. Summary of the Invention [Problem to be solved by the invention]

[0006] However, when the camera direction changes, the position of the object in the video frame changes according to the camera direction even if the same object is captured. In this way, when comparing the same area in the video frame of an image captured by a movable camera such as a mobile robot, differences occur in all areas, and all areas become the target of transmission, making it impossible to reduce the bit rate.

[0007] Therefore, an object of the present disclosure is to make it possible to reduce the transmission rate of video captured by a movable camera. [Means for solving the problem]

[0008] In order to solve the above problems, the video transmission device provided in the robot VR system disclosed herein employs a method of geometrically transforming a video frame based on operation information for the video acquisition unit, and transmitting the difference between the video frame after the geometric transformation and the actual captured video.

[0009] Specifically, the video transmission device of the present disclosure includes: an operation information receiving unit that receives operation information for the image acquisition unit; a geometric transformation unit that performs geometric transformation on the image acquired by the image acquisition unit based on the operation information and outputs the geometric transformation data; a difference calculation unit that calculates a difference between the image acquired by the image acquisition unit and the image geometrically transformed by the geometric transformation unit; Equipped with.

[0010] In addition, the video transmission method of the present disclosure includes: receiving operation information for an image acquisition unit; a step of geometrically transforming the image acquired by the image acquisition unit based on the operation information and outputting the transformed image as geometric transformation data; calculating a difference between the image acquired by the image acquisition unit and the geometric transformation data obtained by the geometric transformation; Includes.

[0011] According to the above configuration, even if the orientation of the image capture unit changes, the difference data can be reduced by taking the difference between the actually captured image and the geometric transformation data after the geometric transformation, thereby making it possible to reduce the transmission rate.

[0012] In addition, since the geometric transformation unit performs geometric transformation on the image based on the operation information, the geometric transformation process can be completed in one go, and the computational processing power required for the controlled object can be reduced.

[0013] In the aforementioned configuration, it is preferable that the geometric transformation section geometrically transforms the image in accordance with an amount or direction of movement of the image acquisition section based on the operation information.

[0014] According to such a configuration, it is possible to calculate the difference while appropriately reflecting the operation of the controlled object based on the operation information.

[0015] In the aforementioned configuration, it is preferable that the image acquisition unit is provided in a control target that receives the operation information, and acquires an image of the surroundings of the control target.

[0016] According to this configuration, the difference can be appropriately calculated based on the change in the image around the control target and the geometric transformation data.

[0017] In the above configuration, it is preferable that the difference calculation unit calculates a difference between a first image acquired by the image acquisition unit and geometric transformation data obtained by geometrically transforming a second image acquired by the image acquisition unit before the first image was acquired by the image acquisition unit.

[0018] According to such a configuration, the difference between the first video image and the geometric transformation data that are temporally successive is calculated, so that the difference can be calculated while appropriately reflecting the operation of the control target based on the operation information.

[0019] It is also preferable that the image forming apparatus further comprises a communication section that transmits the result of the calculation by the difference calculation section to an outside as difference data.

[0020] In addition, the robot VR system of the present disclosure includes: The above video transmitting device, a receiver including a receiving unit that receives the difference data transmitted from the communication unit, a second geometric transformation unit that performs geometric transformation on a video frame corresponding to the difference data received by the receiving unit and outputs geometric transformation data, and a video decoding unit that decodes video based on the geometric transformation data and the difference data output by the second geometric transformation unit; Equipped with.

[0021] According to the above configuration, the video can be suitably decoded based on the difference data.

[0022] The program of the present disclosure is a program for causing a computer to realize each functional unit of a video transmission device of the present disclosure, and is a program for causing a computer to execute each step of a method performed by a video transmission device of the present disclosure.

[0023] The above disclosures may be combined to the greatest extent possible. Effect of the Invention

[0024] According to the present disclosure, in a robot VR system equipped with a movable control target, the transmission rate can be reduced by appropriately considering the fluctuation in video content according to the movement of the control target. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a robot VR system. [Diagram 2] FIG. 13 is a diagram explaining the differential encoding process of the robot VR system. [Diagram 3] FIG. 13 is a diagram explaining the differential encoding process of the robot VR system. [Figure 4] FIG. 1 is a diagram illustrating a configuration example of a robot VR system according to an embodiment of the present disclosure. [Diagram 5] 1A and 1B are diagrams illustrating a movement direction of a mobile robot according to an embodiment of the present disclosure. [Figure 6] 13A to 13C are diagrams illustrating a geometric transformation process of the robot VR system according to an embodiment of the present disclosure. [Figure 7] FIG. 11 is a sequence diagram illustrating a process executed by a robot VR system according to an embodiment of the present disclosure. [Figure 8] 11 is a flowchart illustrating a process executed by a video transmission device according to an embodiment of the present disclosure. [Figure 9] 11 is a flowchart illustrating a process executed by a VR device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings are assumed to indicate the same components.

[0027] (Robot VR System) FIG. 1 shows an example of the configuration of a robot VR system. The robot VR system includes a mobile robot 10 and a VR device 20. The mobile robot 10 includes a video shooting unit 11. The video shooting unit 11 shoots video (video) at a predetermined frame rate. The mobile robot 10 transmits the video shot by the video shooting unit 11 to the VR device 20 as video data. The mobile robot 10 may generate VR video by synthesizing the videos shot by the video shooting unit 11 in real time and transmit the VR video to the VR device 20. The video data transmitted from the mobile robot 10 is received by the VR device 20 via a network.

[0028] In the robot VR system, the mobile robot 10 is operated by the VR device 20. Specifically, the VR device 20 transmits operation information for controlling the movement of the mobile robot 10 to the mobile robot 10. The method of controlling the mobile robot 10 by the VR device 20 at this time is omitted in this disclosure.

[0029] Here, high-definition VR images tend to have a higher bit rate than normal images. However, since the network near the mobile robot 10 to be controlled is wireless, there are cases where a sufficient bandwidth is not available.

[0030] In response to this, it is possible to reduce the bit rate of the video to be transmitted by using differential encoding. The video transmission device of the mobile robot 10 is equipped with a differential encoding unit that performs differential encoding processing. Here, the "differential encoding processing" refers to a process of calculating and encoding the difference between each video frame constituting the video data (frame video) and a video frame at the immediately preceding time. Any commonly known method can be used as the differential encoding method. For example, differential encoding may be performed in units of macroblocks or tiles. Note that information on the elapsed time (time stamp) is added to each video frame. The differential encoding unit executes the differential encoding process while checking the time stamp added to each video frame. Specifically, the differential encoding unit compares a video frame captured by the video shooting unit 11 at a certain time with a video frame captured by the video shooting unit 11 at the immediately preceding time while checking the time stamp, and calculates and encodes the difference excluding the identical parts. This will be described in detail below.

[0031] FIG. 2 is a diagram for explaining the differential encoding process of the robot VR system. FIG. 2 shows a case where the mobile robot 10 does not move and the position of the video shooting unit 11 does not change even if the time changes. The differential encoding unit of the mobile robot 10 compares a video frame captured by the video shooting unit 11 at time t+1 with a video frame captured by the video shooting unit 11 at the time t immediately before that, and obtains a difference D1 excluding the identical part C1. The mobile robot 10 transmits the difference D1 to the VR device 20 as video data. The VR device 20 combines the video frame captured by the video shooting unit 11 at time t with the difference D1, and decodes the video frame at time t+1. In the following description, the video data related to the difference between the video frames transmitted from the mobile robot 10 to the VR device 20 is called "difference data". Also, the video data decoded in the VR device 20 is called "decoded data".

[0032] Similarly, the differential encoding unit compares the video frame captured by the video shooting unit 11 at time t+2 with the video frame captured by the video shooting unit 11 at the immediately preceding time t+1, and obtains a difference D1 excluding the identical portion C1. The mobile robot 10 transmits the difference D2 to the VR device 20 as differential data. The VR device 20 combines the video frame at time t+1 generated above with the difference D2, and decodes the video frame at time t+2. In this way, by using differential encoding, it is only necessary to transmit the difference data corresponding to the difference between the video frames, and therefore the bit rate of the transmitted video can be reduced.

[0033] When the position of the video shooting unit 11 does not change, the background of the subject to be shot does not move, so by performing the differential encoding process described above, the difference between video frames can be reduced. However, in a robot VR system, the position of the video shooting unit 11 changes in response to the movement of the mobile robot 10, so it is not possible to reduce the difference between video frames and therefore the bit rate of the video to be transmitted.

[0034] FIG. 3 is a diagram for explaining the problem in the differential encoding process of the robot VR system. FIG. 3 shows a case where the mobile robot 10 moves and the position of the video shooting unit 11 changes with the change in time. As in the above, the differential encoding unit of the mobile robot 10 compares the video frame shot by the video shooting unit 11 at time t+1 with the video frame shot by the video shooting unit 11 at the time t immediately before that, and obtains the difference D1' excluding the identical portion C1'. In addition, the differential encoding unit compares the video frame shot by the video shooting unit 11 at time t+2 with the video frame shot by the video shooting unit 11 at the time t+1 immediately before that, and obtains the difference D2' excluding the identical portion C2'.

[0035] When the position of the video shooting unit 11 changes in accordance with the movement of the mobile robot 10, the background of the subject to be shot changes significantly. Since such background changes appear as differences D1' and D2' between video frames, simply executing a differential encoding process does not reduce the differences D1' and D2', and therefore the bit rate of the video to be transmitted cannot be reduced.

[0036] Therefore, the video transmission device in the robot VR system of the present disclosure has the following configuration. First configuration: the video transmission device includes a geometric transformation unit that performs geometric transformation on the video frame based on the operation information transmitted from the VR device. In the following description, the process of performing geometric transformation on the video frame is referred to as "geometric transformation process." Second configuration: the video transmission device includes a differential encoding unit that calculates and encodes the difference between a video frame captured at a certain time and a video frame captured immediately before that time and subjected to geometric transformation. In the following description, the process of calculating and encoding the difference between such a geometrically transformed video frame and an actually captured video frame is also referred to as the "difference encoding process."

[0037] According to the above configuration, even if the position of the image capturing unit changes in accordance with the movement of the mobile robot, the difference data can be reduced by taking the difference between the actually captured image and the video frame after the geometric transformation, thereby making it possible to reduce the transmission rate.

[0038] In addition, since the geometric transformation unit performs geometric transformation on the video frames based on the operation information, the geometric transformation process can be completed in one go, and the computational processing power required for the mobile robot can be reduced.

[0039] (Embodiment 1) A robot VR system according to an embodiment of the present disclosure will be described with reference to FIGS. 4 is a diagram illustrating a configuration example of a robot VR system according to an embodiment of the present disclosure. The robot VR system of this embodiment includes a mobile robot 100 instead of the mobile robot 10, and a VR device 200 instead of the VR device 20. The mobile robot 100 functions as a "control target."

[0040] The mobile robot 100 includes a video shooting unit 110 and a video transmission device 300. The video transmission device 300 includes a geometric transformation unit 120, a video processing unit 130, and a robot operation unit 140.

[0041] Specifically, the video transmission device 300 of the present disclosure includes: a robot operating unit 140 that receives operation information for the video acquisition unit 110; a geometric transformation unit that performs geometric transformation on the image captured by the image capture unit based on the operation information and outputs the transformed image as geometric transformation data; a differential encoding unit 130A that calculates the difference between the image captured by the image capture unit 110 and the image that has been geometrically transformed by the geometric transformation unit 120; Equipped with.

[0042] The robot operation unit 140 controls the posture and motion of the mobile robot 100 in response to the operation information from the VR device 200. The robot operation unit 140 also controls the posture and motion of each component of the mobile robot 100. Specifically, the robot operation unit 140 controls the orientation and direction of the video acquisition unit 110. In other words, the operation information also functions as control information for the video acquisition unit 110. In the following description, the posture change and motion of the mobile robot 100 are collectively referred to as "motion". For example, the robot operation unit 140 is a motor control unit for controlling the motor of the mobile robot 100. Note that there is a certain time lag between when the robot operation unit 140 receives the operation information and when the mobile robot 100 starts to move. In this embodiment, this time lag is about the same as the time it takes for the video capture unit 110 to capture one frame. The robot operation unit 140 functions as an "operation information receiving unit".

[0043] The video shooting unit 110 shoots video (moving image) at a predetermined frame rate. The video shooting unit 110 shoots 360 degrees around the mobile robot 100. That is, the video shooting unit 110 acquires video around the mobile robot 100. The video shooting unit 110 may be composed of multiple cameras. For example, the multiple cameras may be arranged at predetermined intervals in the circumferential direction of the mobile robot 100. The video shooting unit 110 may shoot not only a 360-degree range in the horizontal direction, but also a 360-degree range in the vertical direction. The position of the video shooting unit 110 changes according to the change in posture and movement of the mobile robot 100. The video shooting unit 110 sends the shot video to the video processing unit 130 and the geometric transformation unit 120 as video data. Note that the first video frame is a reference for decoding by the VR device 200, so it is transmitted from the video processing unit 130 to the outside without being subjected to geometric transformation processing or differential encoding processing, and is received by the VR device 200. The video shooting unit 110 functions as an "video acquisition unit."

[0044] The geometric transformation unit 120 performs geometric transformation on the video frames based on the operation information input to the geometric transformation unit 120 via the robot operation unit 140. Specifically, the geometric transformation unit 120 performs geometric transformation on the video frames according to the movement amount or movement direction of the mobile robot 100 based on the operation information. In other words, the geometric transformation unit 120 performs geometric transformation on the video frames according to the movement amount or movement direction of the image acquisition unit 110. The geometric transformation unit 120 may perform geometric transformation on the video frames according to a combination of the movement amount and the movement direction. In the following description, the process of performing geometric transformation on the video frames is called "geometric transformation process." The geometric transformation unit 120 can perform any geometric transformation by calculating the variation of a certain feature point in the corresponding video frame from the operation information and encoding the variation as a motion vector. The geometric transformation unit 120 sends the video frames after the geometric transformation to the image processing unit 130 as "geometric transformation data."

[0045] The video processing unit 130 includes a differential encoding unit 130A and a video transmission unit 130B. The differential encoding unit 130A performs differential encoding processing based on the video data received from the video shooting unit 110 and the geometric transformation data received from the geometric transformation unit 120. Specifically, the differential encoding unit 130A checks the timestamp of each video frame constituting the video data (video frame), calculates the difference between the video frame and a video frame (geometric transformation data) that was shot and geometrically transformed immediately before, and encodes the difference. Here, each video frame constituting the video data (video frame) acquired by the video shooting unit 110 is an example of a "first video". Also, for each video frame, a video frame constituting the video data acquired by the video shooting unit 110 immediately before is an example of a "second video".

[0046] Here again, any commonly known method can be used as the differential encoding method. For example, differential encoding may be performed in units of macroblocks or tiles. As described above, there is a time lag between when the robot operating unit 140 receives operation information and when the mobile robot 100 starts to move, which is approximately the same as the time it takes for the video shooting unit 110 to shoot one frame. Therefore, the video can be smoothly decoded in the VR device 200 based on the difference between each video frame and the video frame shot and geometrically transformed at the time immediately before that. The differential encoding unit 130A functions as a "difference calculation unit."

[0047] The video transmission unit 130B transmits to the outside the differential data differentially encoded by the differential encoding unit 130A. That is, the video transmission unit 130B functions as a communication unit that transmits to the outside the result of the difference calculation by the differential encoding unit 130A as differential data.

[0048] In this manner, in this embodiment, the difference between each video frame and the video frame that was captured immediately before and geometrically transformed is sent as difference data, so that the bit rate can be reduced even when the position of the video capture unit 110 changes with the movement of the mobile robot 100. Also, in this embodiment, the number of times that the mobile robot 10 needs to perform geometric transformation based on the operation information from the VR device 200 is only one.

[0049] If the robot operating unit 140 does not receive operation information and no operation information is input to the geometric transformation unit 120, there is no change in the posture or position of the mobile robot 100, and the position of the video shooting unit 110 also does not change. For this reason, the geometric transformation unit 120 sends the video data received from the video shooting unit 110 directly to the video processing unit 130 without performing geometric transformation. In this case, since there is no difference between each video frame, the differential encoding unit 130A outputs the differential data as null data.

[0050] The VR device 200 includes a video receiving unit 210, a geometric transformation unit 220, a video decoding unit 230, and an operation information communication unit 240. The VR device 200 functions as a "receiver." The geometric transformation unit 220 functions as a "second geometric transformation unit." Specifically, the robot VR system of the present disclosure includes: A VR device 200 including the above-mentioned video transmission device 300, a video reception unit 210 that receives the difference data transmitted from the video transmission unit 130B, a geometric transformation unit 220 that performs geometric transformation on a video frame corresponding to the difference data received by the video reception unit 210 and outputs the geometric transformation data, and a video decoding unit 230 that decodes a video based on the geometric transformation data and the difference data output by the geometric transformation unit 220; Equipped with.

[0051] The operation information communication unit 240 sends operation information for controlling the mobile robot 100 to the geometric transformation unit 220 in response to a user's input (operation) to a device directly connected to the VR device 200, and also transmits the operation information to the outside of the VR device 200. The operation information transmitted to the outside of the VR device 200 is received by the robot operation unit 140 of the mobile robot 100.

[0052] The video receiving unit 210 receives the differential data transmitted from the video transmitting unit 130B. The video receiving unit 210 sends the received differential data to the video decoding unit 230, and also sends it to the geometric transformation unit 220. The differential data includes information related to a timestamp. Note that it is optional whether or not the video receiving unit 210 sends the differential data to the geometric transformation unit 220, and it is not necessary for the video receiving unit 210 to send the differential data to the geometric transformation unit 220. The video receiving unit 210 functions as a "receiving unit".

[0053] The geometric transformation unit 220 receives the decoded data decoded by the video decoding unit 230 at any time. The decoded data includes information related to a timestamp. The geometric transformation unit 220 performs a geometric transformation process on the decoded data from the video decoding unit 230 based on the operation information received from the operation information communication unit 240. In this case, the geometric transformation unit 220 checks information related to the timestamp of the decoded data, and performs the geometric transformation process based on the operation information corresponding to the operation immediately after that. Note that the first video frame is sent from the video decoding unit 230 to the geometric transformation unit 220 without being subjected to the video decoding process.

[0054] The video decoding unit 230 receives the difference data from the video receiving unit 210. The video decoding unit 230 also receives the geometric transformation data from the geometric transformation unit 220. The video decoding unit 230 combines the difference data and the geometric transformation data while checking the timestamp to generate decoded data. The video decoding unit 230 manages the decoded data in chronological order. The decoded data decoded by the video decoding unit 230 is output as video to a device directly or indirectly connected to the VR device 200.

[0055] If the difference data is null data, there is no need to perform geometric transformation processing or video decoding processing in the VR device 200. In this case, the video decoding unit 230 determines that there is no difference from the immediately preceding video frame, and outputs the immediately preceding video frame again.

[0056] Next, the geometric transformation process and the differential encoding process in the mobile robot 100 will be described in detail with reference to FIGS. FIG. 5 is a diagram for explaining the movement direction of the mobile robot 100. In this embodiment, the mobile robot 100 can move in direction A (forward), direction B (left), direction C (backward), and direction D (right) in the figure in response to a user's operation. However, the movement direction of the mobile robot 100 is not limited to this. For example, the mobile robot 100 may be able to move to the front right, front left, rear right, and rear left. Furthermore, the movement of the mobile robot 100 is not limited to a straight line, and the mobile robot 100 may move in a curved line.

[0057] FIG. 6 is a diagram for explaining the geometric transformation process and the differential encoding process of the mobile robot 100. The geometric transformation unit 120 executes the geometric transformation process on the video frame captured by the video capture unit 110 at time t based on the operation information input from the robot operation unit 140. In the case of the situation shown in the figure, the operation information includes information indicating that the mobile robot 100 has moved in the C direction. Therefore, the geometric transformation unit 120 calculates the change of a certain feature point when the mobile robot 100 moves in the C direction, and encodes this change as a motion vector. The geometric transformation unit 120 sends the video frame after the geometric transformation to the video processing unit 130 as geometric transformation data.

[0058] The differential encoding unit 130A of the video processing unit 130 compares the video frame captured by the video shooting unit 110 at time t+1 with the video frame (geometric transformation data) captured at the immediately preceding time t and geometrically transformed by the geometric transformation unit 120, obtains the difference D0 excluding the identical portion C0, and encodes it.

[0059] The process executed by the robot VR system according to the embodiment of the present disclosure will be described with reference to the sequence diagram of Fig. 7. Note that in the diagram, S11 to S14 indicate the process executed by the video transmission device 300, and S21 to S23 indicate the process executed by the VR device 200.

[0060] Specifically, the video transmission method of the present disclosure includes: receiving operation information for the video acquisition unit 110; A step of geometrically transforming the image captured by the image capturing unit 110 based on the operation information and outputting the transformed data; Calculating a difference between an image captured by the image capture unit 110 and the geometrically transformed image; Includes.

[0061] First, at time t=0, the video capture unit 110 of the mobile robot 100 captures a video (video frame) (S11). At the stage of time t=0, the mobile robot 100 has not received any operation information, so there is no change in the posture or position of the mobile robot 100, and the position of the video capture unit 110 does not change either. In this case, there is no difference between each video frame, so the video transmission unit 130B transmits the video frame captured by the video capture unit 110 to the outside as is (S12). The video frame is received by the video receiving unit 210 of the VR device 200. The VR device 200 displays the video frame as a video on a device directly or indirectly connected to the VR device 200 (S21).

[0062] When an operation from the user is accepted, the operation information communication unit 240 sends the operation information to the geometric transformation unit 220 of the VR device 200 and also transmits it to the outside of the VR device 200 (S22). The geometric transformation unit 220 executes a geometric transformation process on the video frame captured at time t=0 based on the operation information (S23). The geometric transformation unit 220 sends the geometrically transformed video frame as geometric transformation data to the video decoding unit 230. Note that in the next and subsequent geometric transformation processes, the geometric transformation unit 220 performs the geometric transformation process on the decoded data sent from the video decoding unit 230.

[0063] On the other hand, the operation information transmitted to the outside of the VR device 200 is received by the robot operation unit 140 of the mobile robot 100. The mobile robot 100 operates based on the operation information (S13). In addition, the robot operation unit 140 sends the operation information to the geometric transformation unit 120.

[0064] The geometric transformation unit 120 performs a geometric transformation process on the video frame captured at time t=0 based on the operation information (S14). The geometric transformation unit 120 sends the geometrically transformed video frame to the image processing unit 130 as geometric transformation data.

[0065] The differential encoding unit 130A compares the video frame captured by the video shooting unit 110 at time t=1 with the video frame (geometric transformation data) captured at the immediately preceding time t and geometrically transformed by the geometric transformation unit 120, calculates the difference excluding identical portions, and encodes it (S15). The video transmission unit 130B transmits the difference data to the outside of the mobile robot 100 (S16).

[0066] The difference data from the video transmission unit 130B is received by the video receiving unit 210 of the VR device 200. The video receiving unit 210 sends the difference data to the video decoding unit 230. The video decoding unit 230 checks the timestamp and synthesizes the difference data with the geometric transformation data received from the geometric transformation unit 220 to generate decoded data. Thereafter, the mobile robot 100 and the VR device 200 repeat the same process while exchanging difference data and operation information.

[0067] Next, the processes executed by the video transmission device 300 and the VR device 200 will be described with reference to the flowcharts of FIGS. 8 is a flowchart illustrating the processing executed by video transmission device 300. It is assumed that at the start of the processing shown in this flowchart, robot operation unit 140 of video transmission device 300 has not yet received operation information.

[0068] The video capturing unit 110 of the mobile robot 100 captures video. The video captured by the video capturing unit 110 is transmitted to the outside via the video transmission unit 130B of the video processing unit 130 (step S101).

[0069] The geometric transformation unit 120 determines whether or not operation information has been received via the robot operation unit 140 (S102). If the geometric transformation unit 120 determines that operation information has not been received (step S102: No), the video transmitting device 300 repeats the processes of steps S101 and S102 until the geometric transformation unit 120 receives operation information.

[0070] On the other hand, when it is determined that operation information has been received (step S102: Yes), the geometric transformation unit 120 executes a geometric transformation process on the video frames captured by the video shooting unit 110 (S103). The geometric transformation unit 120 sends the geometrically transformed video frames to the video processing unit 130 as geometric transformation data.

[0071] The differential encoding unit 130A of the video processing unit 130 performs differential encoding processing based on the video frames received from the video shooting unit 110 and the geometric transformation data received from the geometric transformation unit 120 (step S104). The video transmission unit 130B transmits the differential data to the outside (step S105).

[0072] FIG. 9 is a flowchart illustrating the processing executed by the VR device 200. The video receiving unit 210 of the VR device 200 judges whether or not video data (video frames) have been received from the video transmitting unit 130B (step S201). If it is judged that video data has not been received (step S201: No), the video receiving unit 210 waits until video data is received.

[0073] On the other hand, when it is determined that the video data has been received (step S201: Yes), the VR device 200 displays the video data as a video on a device directly or indirectly connected to the VR device 200 (step S202).

[0074] The operation information communication unit 240 determines whether or not an operation from the user has been received (step S203). If it is determined that an operation from the user has not been received (step S203: Yes), the operation information communication unit 240 repeats the processes from steps S201 to S203 until an operation from the user is received.

[0075] On the other hand, when it is determined that an operation from the user has been accepted (step S203: No), the operation information communication unit 240 sends the operation information to the geometric transformation unit 220 of the VR device 200 and also transmits it to the outside of the VR device 200 (step S204).

[0076] The geometric transformation unit 220 constantly receives the decoded data decoded by the video decoding unit 230. The geometric transformation unit 220 executes a geometric transformation process on the decoded data from the video decoding unit 230, based on the operation information received from the operation information communication unit 240 (step S205).

[0077] Next, the video receiving unit 210 of the VR device 200 judges whether or not the differential data has been received from the video transmitting unit 130B (step S206). If it is judged that the differential data has not been received (step S206: No), the video receiving unit 210 waits until the differential data is received.

[0078] On the other hand, if it is determined that the differential data has been received (step S206: Yes), the video receiving unit 210 sends the differential data to the video decoding unit 230. The video decoding unit 230 checks the timestamp, synthesizes the differential data and the geometric transformation data, and outputs the decoded data (step S207).

[0079] The device of the present invention can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network. The program of the present disclosure is a program for causing a computer to realize each function of the device according to the present disclosure, and is a program for causing a computer to execute each procedure of the method executed by the device according to the present disclosure. [Industrial Applicability]

[0080] The present disclosure can be applied to the information and communications industry. [Explanation of symbols]

[0081] 10, 100: Mobile robot 11, 110: Videography Department 120: Geometric transformation section 130: Image processing section 130A: Differential encoding unit 130B: Video transmission section 20, 200: VR device 210: Video receiving unit 220: Geometric transformation section 230: Video decoding unit 240: Operation information communication unit 300: Video transmission device

Claims

1. an operation information receiving unit that receives operation information for the image acquisition unit; a geometric transformation unit that performs geometric transformation on the image acquired by the image acquisition unit based on the operation information and outputs the geometric transformation data; a difference calculation unit that calculates a difference between the image acquired by the image acquisition unit and the geometric transformation data; A video transmitting device comprising:

2. The geometric transformation unit performs geometric transformation on the image in accordance with a movement amount or a movement direction of the image acquisition unit based on the operation information. The video transmission device according to claim 1.

3. The image acquisition unit is provided in a control target that receives the operation information, and acquires an image of a surrounding area of ​​the control target. The video transmission device according to claim 1.

4. the difference calculation unit calculates a difference between a first image acquired by the image acquisition unit and the geometric transformation data obtained by geometrically transforming a second image acquired by the image acquisition unit before the first image was acquired by the image acquisition unit, by the geometric transformation unit; The video transmission device according to claim 1.

5. A communication unit that transmits a result of the calculation by the difference calculation unit as difference data to an external device. The video transmission device according to claim 1.

6. The video transmission device according to claim 5 , a receiver including a receiving unit that receives the difference data transmitted from the communication unit, a second geometric transformation unit that performs geometric transformation on a video frame corresponding to the difference data received by the receiving unit and outputs geometric transformation data, and a video decoding unit that decodes video based on the geometric transformation data and the difference data output by the second geometric transformation unit; A robot VR system comprising:

7. receiving operation information for an image acquisition unit; a step of geometrically transforming the image acquired by the image acquisition unit based on the operation information and outputting the transformed image as geometric transformation data; Calculating a difference between the image acquired by the image acquisition unit and the geometrically transformed image; A video transmission method comprising:

8. A program for causing a computer to realize each of the functional units of the video transmission device according to any one of claims 1 to 5.

Images