Robot VR system

Abstract

To allow reduction in the transmission rate of images picked up by a movable camera.SOLUTION: An image transmission device 300 comprises: a plurality of geometric transformation units 120 that perform geometric transformation of images acquired by an image photographing unit 110 as geometric transformation data, for every movement pattern of the image acquisition unit 110; and a difference encoding unit 130A that compares the differences between the images acquired by the image photographing unit 110 and a plurality of pieces of geometric transformation data obtained through the geometric transformation performed by the plurality of geometric transformation units 120 for every movement pattern, and encodes the difference between the images acquired by the image photographing unit 110 and the geometric transformation data according to the movement of the image acquisition unit 110.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a robot VR system that generates VR video from camera images captured by a mobile robot. [Background technology]

[0002] Conventionally, when remotely controlling a controlled object such as a robot while watching images from a camera installed on the controlled object, there is a demand to operate the controlled object while watching VR images because the field of view is narrow with 2D images. Therefore, a robot VR system has been proposed in which the controlled object is equipped with a 360-degree camera and VR images are transmitted to the operator.

[0003] In such robot VR systems, high-definition VR images tend to have a higher bit rate than normal images, but since the network near the controlled object is wireless, there is a possibility that a sufficient bandwidth may not be available. Therefore, it has been proposed to reduce the bit rate of images by differentially encoding the images and transmitting them (see Non-Patent Documents 1 to 3).

[0004] In Non-Patent Documents 1 to 3, a video frame is divided into a number of regions, and each region is compared with the immediately preceding frame, and only the region of the video frame that has a difference is transmitted. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Ching-Yeh Chen, Shao-Yi Chien, Yu-Wen Huang, Tung-Chien Chen, Tu-Chih Wang and Liang-Gee Chen, “Analysis and architecture design of variable block-size motion estimation for H.264 / AVC,” in IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 53, no. 3, pp. 578-593, March 2006. [Non-Patent Document 2] Alireza Zare, Alireza Aminlou, Miska M. Hannuksela, and Moncef Gabbouj. 2016. HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications. In Proceedings of the 24th ACM international conference on Multimedia (MM '16). [Non-Patent Document 3] Park, Sang-Hyo, and Je-Won Kang. “Fast affine motion estimation for versatile video coding (VVC) encoding.” IEEE Access 7 (2019): 158075-158084. Summary of the Invention [Problem to be solved by the invention]

[0006] However, when the camera direction changes, the position of the object in the video frame changes according to the camera direction even if the same object is captured. In this way, when comparing the same area in the video frame of an image captured by a movable camera such as a mobile robot, differences occur in all areas, and all areas become the target of transmission, making it impossible to reduce the bit rate.

[0007] Therefore, an object of the present disclosure is to make it possible to reduce the transmission rate of video captured by a movable camera. [Means for solving the problem]

[0008] In order to solve the above problems, the video transmission device provided in the robot VR system disclosed herein employs a method of geometrically transforming a video frame for each movement pattern of the video acquisition unit, and transmitting the difference between the video frame after the geometric transformation and the actual captured video.

[0009] Specifically, the video transmission device of the present disclosure includes: a geometric transformation unit that performs geometric transformation on the image acquired by the image acquisition unit for each movement pattern of the image acquisition unit and outputs the result as geometric transformation data; a difference calculation unit that compares a difference between the image acquired by the image acquisition unit and each of a plurality of pieces of geometric transformation data obtained by performing geometric transformation for each of the movement patterns by the geometric transformation unit, and encodes a difference between the image and the geometric transformation data corresponding to the movement of the control object; Equipped with.

[0010] In addition, the video transmission method of the present disclosure includes: a step of geometrically transforming the image acquired by the image acquisition unit for each movement pattern of the image acquisition unit and outputting the transformed image as geometric transformation data; comparing a difference between the image acquired by the image acquisition unit and each of a plurality of pieces of geometric transformation data obtained by performing geometric transformation for each of the movement patterns, and encoding the difference between the image and the geometric transformation data corresponding to the movement of the image acquisition unit; Includes.

[0011] According to the above configuration, for each video frame, the difference between each of the multiple geometric transformation data captured immediately before and geometrically transformed for each movement pattern is compared. Then, the difference between the video captured by the video capture unit and the geometric transformation data corresponding to the movement of the video capture unit is encoded, so that the bit rate can be reduced even when the position of the video capture unit changes.

[0012] Furthermore, since the geometric transformation section can start the geometric transformation process without waiting for reception of operation information for the image acquisition section, a processing delay can be prevented.

[0013] In the aforementioned configuration, it is preferable that the geometric transformation unit geometrically transforms the image according to an amount of movement or a direction of movement of the image acquisition unit for each movement pattern of the image acquisition unit.

[0014] According to such a configuration, it is possible to calculate the difference while appropriately reflecting the operation of the controlled object based on the operation information.

[0015] In the above configuration, it is preferable that the image acquisition section is provided in a control target that receives operation information, moves based on the operation information, and acquires an image of the surroundings of the control target.

[0016] According to this configuration, the difference can be appropriately calculated based on the change in the image around the control target and the geometric transformation data.

[0017] It is preferable that the difference calculation unit compares a first image acquired by the image acquisition unit with each of a plurality of geometric transformation data obtained by geometrically transforming a second image acquired by the image acquisition unit before the first image was acquired by the image acquisition unit, and calculates a difference between the images and the geometric transformation data corresponding to the movement of the control object.

[0018] According to such a configuration, the difference between the first video image and the geometric transformation data that are temporally successive is calculated, so that the difference can be calculated while appropriately reflecting the operation of the control target based on the operation information.

[0019] It is also preferable that the image forming apparatus further comprises a communication section that transmits the result of the calculation by the difference calculation section to an outside as difference data.

[0020] In addition, the robot VR system of the present disclosure includes: The above video transmitting device, a receiver including a receiving unit that receives the difference data transmitted from the communication unit, a second geometric transformation unit that performs geometric transformation on a video frame corresponding to the difference data received by the receiving unit and outputs geometric transformation data, and a video decoding unit that decodes video based on the geometric transformation data and the difference data output by the second geometric transformation unit; Equipped with.

[0021] According to the above configuration, the video can be suitably decoded based on the difference data.

[0022] The program of the present disclosure is a program for causing a computer to realize each functional unit of a video transmission device of the present disclosure, and is a program for causing a computer to execute each step of a method performed by a video transmission device of the present disclosure.

[0023] The above disclosures may be combined to the greatest extent possible. Effect of the Invention

[0024] According to the present disclosure, it is possible to reduce the transmission rate of video captured by a movable camera. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a robot VR system. [Diagram 2]FIG. 13 is a diagram explaining the differential encoding process of the robot VR system. [Diagram 3] FIG. 13 is a diagram explaining the differential encoding process of the robot VR system. [Figure 4] FIG. 1 is a diagram illustrating a configuration example of a robot VR system according to an embodiment of the present disclosure. [Diagram 5] 1A and 1B are diagrams illustrating a movement direction of a mobile robot according to an embodiment of the present disclosure. [Figure 6] 13A to 13C are diagrams illustrating a geometric transformation process of the robot VR system according to an embodiment of the present disclosure. [Figure 7] FIG. 11 is a sequence diagram illustrating a process executed by a robot VR system according to an embodiment of the present disclosure. [Figure 8] 11 is a flowchart illustrating a process executed by a video transmission device according to an embodiment of the present disclosure. [Figure 9] 11 is a flowchart illustrating a process executed by a VR device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings are assumed to indicate the same components.

[0027] (Robot VR System) FIG. 1 shows an example of the configuration of a robot VR system. The robot VR system includes a mobile robot 10 and a VR device 20. The mobile robot 10 includes a video shooting unit 11. The video shooting unit 11 shoots video (video) at a predetermined frame rate. The mobile robot 10 transmits the video shot by the video shooting unit 11 to the VR device 20 as video data. The mobile robot 10 may generate VR video by synthesizing the videos shot by the video shooting unit 11 in real time and transmit the VR video to the VR device 20. The video data transmitted from the mobile robot 10 is received by the VR device 20 via a network.

[0028] In the robot VR system, the mobile robot 10 is operated by the VR device 20. Specifically, the VR device 20 transmits operation information for controlling the movement of the mobile robot 10 to the mobile robot 10. The method of controlling the mobile robot 10 by the VR device 20 at this time is omitted in this disclosure.

[0029] Here, high-definition VR images tend to have a higher bit rate than normal images. However, since the network near the mobile robot 10 to be controlled is wireless, there are cases where a sufficient bandwidth is not available.

[0030] In response to this, it is possible to reduce the bit rate of the video to be transmitted by using differential encoding. The video transmission device of the mobile robot 10 is equipped with a differential encoding unit that performs differential encoding processing. Here, the "differential encoding processing" refers to a process of calculating and encoding the difference between each video frame constituting the video data (frame video) and a video frame at the immediately preceding time. Any commonly known method can be used as the differential encoding method. For example, differential encoding may be performed in units of macroblocks or tiles. Note that information on the elapsed time (time stamp) is added to each video frame. The differential encoding unit executes the differential encoding process while checking the time stamp added to each video frame. Specifically, the differential encoding unit compares a video frame captured by the video shooting unit 11 at a certain time with a video frame captured by the video shooting unit 11 at the immediately preceding time while checking the time stamp, and calculates and encodes the difference excluding the identical parts. This will be described in detail below.

[0031] FIG. 2 is a diagram for explaining the differential encoding process of the robot VR system. FIG. 2 shows a case where the mobile robot 10 does not move and the position of the video shooting unit 11 does not change even if the time changes. The differential encoding unit of the mobile robot 10 compares a video frame captured by the video shooting unit 11 at time t+1 with a video frame captured by the video shooting unit 11 at the time t immediately before that, and obtains a difference D1 excluding the identical part C1. The mobile robot 10 transmits the difference D1 to the VR device 20 as video data. The VR device 20 combines the video frame captured by the video shooting unit 11 at time t with the difference D1, and decodes the video frame at time t+1. In the following description, the video data related to the difference between the video frames transmitted from the mobile robot 10 to the VR device 20 is called "difference data". Also, the video data decoded in the VR device 20 is called "decoded data".

[0032] Similarly, the differential encoding unit compares the video frame captured by the video shooting unit 11 at time t+2 with the video frame captured by the video shooting unit 11 at the immediately preceding time t+1, and obtains a difference D1 excluding the identical portion C1. The mobile robot 10 transmits the difference D2 to the VR device 20 as differential data. The VR device 20 combines the video frame at time t+1 generated above with the difference D2, and decodes the video frame at time t+2. In this way, by using differential encoding, it is only necessary to transmit the difference data corresponding to the difference between the video frames, and therefore the bit rate of the transmitted video can be reduced.

[0033] When the position of the video shooting unit 11 does not change, the background of the subject to be shot does not move, so by performing the differential encoding process described above, the difference between video frames can be reduced. However, in a robot VR system, the position of the video shooting unit 11 changes in response to the movement of the mobile robot 10, so it is not possible to reduce the difference between video frames and therefore the bit rate of the video to be transmitted.

[0034] FIG. 3 is a diagram for explaining the problem in the differential encoding process of the robot VR system. FIG. 3 shows a case where the mobile robot 10 moves and the position of the video shooting unit 11 changes with the change in time. As in the above, the differential encoding unit of the mobile robot 10 compares the video frame shot by the video shooting unit 11 at time t+1 with the video frame shot by the video shooting unit 11 at the time t immediately before that, and obtains the difference D1' excluding the identical portion C1'. In addition, the differential encoding unit compares the video frame shot by the video shooting unit 11 at time t+2 with the video frame shot by the video shooting unit 11 at the time t+1 immediately before that, and obtains the difference D2' excluding the identical portion C2'.

[0035] When the position of the video shooting unit 11 changes in accordance with the movement of the mobile robot 10, the background of the subject to be shot changes significantly. Since such background changes appear as differences D1' and D2' between video frames, simply executing a differential encoding process does not reduce the differences D1' and D2', and therefore the bit rate of the video to be transmitted cannot be reduced.

[0036] Therefore, the robot VR system of the present disclosure has the following configuration. First configuration: The mobile robot includes a plurality of geometric transformation units that perform geometric transformation on video frames so as to correspond to all possible movement patterns of the mobile robot. In the following description, the process of performing geometric transformation on video frames is referred to as "geometric transformation process." Second configuration: The mobile robot includes a differential encoding unit that calculates and compares the difference between a video frame captured at a certain time and each of a plurality of pieces of geometric transformation data captured at the time immediately prior to that time and subjected to geometric transformation processing for each movement pattern. The differential encoding unit compares the difference for each movement pattern and outputs the smallest difference as "minimum difference data." In the following description, the process of calculating and encoding the difference between such a geometrically transformed video frame and an actually captured video frame is also referred to as "differential encoding processing."

[0037] According to the above configuration, for each video frame, the difference between each of the plurality of geometric transformation data captured at the immediately preceding time and subjected to the geometric transformation process for each movement pattern is calculated.Then, the smallest difference is sent as the minimum difference data, so that the bit rate can be reduced even when the position of the video capture unit changes.

[0038] Furthermore, since the geometric transformation unit can start the geometric transformation process without waiting for reception of operation information in the mobile robot, processing delays can be prevented.

[0039] (Embodiment 1) A robot VR system according to an embodiment of the present disclosure will be described with reference to FIGS. 4 is a diagram illustrating a configuration example of a robot VR system according to an embodiment of the present disclosure. The robot VR system of this embodiment includes a mobile robot 100 instead of the mobile robot 10, and a VR device 200 instead of the VR device 20. The mobile robot 100 functions as a "control target."

[0040] The mobile robot 100 includes a video shooting unit 110 and a video transmission device 300. The video transmission device 300 includes a plurality of geometric transformation units 120 and a differential encoding unit 130A.

[0041] Specifically, the video transmission device 300 of the present disclosure includes: a plurality of geometric transformation units 120 that perform geometric transformation on the images captured by the image capturing unit 110 for each movement pattern of the image capturing unit 110 and output the geometric transformation data; a difference encoding unit 130A that compares the difference between the image captured by the image capturing unit 110 and each of a plurality of pieces of geometric transformation data that have been geometrically transformed for each movement pattern by a plurality of geometric transformation units 120, and encodes the difference between the image captured by the image capturing unit 110 and the geometric transformation data corresponding to the movement of the mobile robot 100; Equipped with.

[0042] Although not shown in the figure, the mobile robot 100 is equipped with a robot operation unit that controls the posture and movement of the mobile robot 100 in response to operation information from the VR device 200. In addition, the robot operation unit 140 controls the posture and movement of each component of the mobile robot 100. Specifically, the robot operation unit 140 controls the orientation and direction of the video acquisition unit 110. In other words, the operation information also functions as control information for the video acquisition unit 110. In the following description, the change in posture and movement of the mobile robot 100 are collectively referred to as "movement." For example, the robot operation unit is a motor control unit for controlling the motor of the mobile robot 100.

[0043] The video shooting unit 110 shoots video (moving image) at a predetermined frame rate. The video shooting unit 110 shoots 360-degree images around the mobile robot 100. That is, the video shooting unit 110 acquires video of the surroundings of the mobile robot 100. The video shooting unit 110 may be composed of a plurality of cameras. For example, the plurality of cameras may be arranged at a predetermined interval in the circumferential direction of the mobile robot 100. Furthermore, the video shooting unit 110 may not only shoot a 360-degree range in the horizontal direction, but may also shoot a 360-degree range in the vertical direction. The position of the video shooting unit 110 changes according to the change in posture and the movement of the mobile robot 100. The video shooting unit 110 sends the shot video to the video processing unit 130 and the plurality of geometric transformation units 120 as video data. In addition, since the first video frame is used as a reference for decoding by the VR device 200, it is not subjected to geometric transformation processing or differential encoding processing, and is transmitted from the video processing unit 130 to the outside and received by the VR device 200. The video shooting unit 110 functions as a "video acquisition unit."

[0044] The multiple geometric transformation units 120 perform geometric transformation on the video frames. Specifically, the multiple geometric transformation units 120 perform geometric transformation on the video frames according to the movement amount or movement direction of the mobile robot 100 for each movement pattern of the mobile robot 100. In other words, the multiple geometric transformation units 120 perform geometric transformation on the video frames according to the movement amount or movement direction of the image acquisition unit 110 for each movement pattern of the image acquisition unit 110. The multiple geometric transformation units 120 may perform geometric transformation on the video frames according to a combination of the movement amount and the movement direction. In the following description, the process in which the multiple geometric transformation units 120 perform geometric transformation on the video frames is called "geometric transformation process." Each of the multiple geometric transformation units 120 can perform any geometric transformation by calculating the variation of a certain feature point in the corresponding video frame from the operation information and encoding the variation as a motion vector. The multiple geometric conversion units 120 are provided for each possible movement pattern of the mobile robot 100 (image acquisition unit 110). For example, four multiple geometric conversion units are provided corresponding to the forward / backward / left / right movements of the mobile robot 100 (image acquisition unit 110). Each of the multiple geometric conversion units 120 performs a geometric conversion process on the assumption that the mobile robot 100 (image acquisition unit 110) operates in the movement pattern corresponding to each of them. In other words, the multiple geometric conversion units 120 perform a geometric conversion process on the video frame so as to correspond to all possible movement patterns of the mobile robot 100 (image acquisition unit 110). Each of the multiple geometric conversion units 120 sends the geometrically converted video frame to the image processing unit 130 as "geometric conversion data".

[0045] The video processing unit 130 includes a differential encoding unit 130A and a video transmission unit 130B. The differential encoding unit 130A performs differential encoding processing based on the video data received from the video shooting unit 110 and the geometric transformation data received from the plurality of geometric transformation units 120. Specifically, the differential encoding unit 130A checks the timestamp of each video frame constituting the video data (video frame) and calculates the difference between the video frame and each of the plurality of geometric transformation data captured at the immediately preceding time and subjected to the geometric transformation processing for each movement pattern. Then, the differential encoding unit 130A compares the difference for each movement pattern, encodes the smallest difference, and outputs it as "minimum difference data". Here, each video frame constituting the video data (video frame) acquired by the video shooting unit 110 is an example of the "first video". A video frame constituting the video data acquired by the video shooting unit 110 immediately before is an example of the "second video".

[0046] Again, any commonly known method can be used as the differential encoding method. For example, the differential encoding may be performed on a macroblock-by-macroblock or tile-by-tile basis. The differential encoding unit 130A functions as a "difference calculation unit."

[0047] The video transmission unit 130B transmits to the outside the minimum differential data differentially encoded by the differential encoding unit 130A. The video transmission unit 130B functions as a communication unit that transmits to the outside the result of the difference calculation by the differential encoding unit 130A as minimum differential data.

[0048] In this manner, in this embodiment, for each video frame, the difference between each of the multiple pieces of geometric transformation data captured immediately before and subjected to geometric transformation processing is calculated. Then, the smallest difference is sent as the minimum difference data, so that the bit rate can be reduced even when the position of the video capture unit 110 changes with the movement of the mobile robot 100. Also, in this embodiment, the geometric transformation processing can be started without waiting for the reception of operation information in the mobile robot 100, so that processing delays can be prevented.

[0049] When the mobile robot 100 is not moving, there is no difference between the video frames, and therefore the differential encoding unit 130A outputs the minimum differential data as null data.

[0050] The VR device 200 includes a video receiving unit 210, a geometric transformation unit 220, a video decoding unit 230, and an operation information communication unit 240. The VR device 200 functions as a "receiver." The geometric transformation unit 220 functions as a "second geometric transformation unit."

[0051] Specifically, the robot VR system of the present disclosure includes: A VR device 200 including the above-mentioned video transmission device 300, a video reception unit 210 that receives the difference data transmitted from the video transmission unit 130B, a geometric transformation unit 220 that performs geometric transformation on a video frame corresponding to the difference data received by the video reception unit 210 and outputs the geometric transformation data, and a video decoding unit 230 that decodes a video based on the geometric transformation data and the difference data output by the geometric transformation unit 220; Equipped with.

[0052] The operation information communication unit 240 sends operation information for controlling the mobile robot 100 to the geometric transformation unit 220 in response to a user's input (operation) to a device directly connected to the VR device 200, and also transmits the operation information to the outside of the VR device 200. The operation information transmitted to the outside of the VR device 200 is received by the robot operation unit of the mobile robot 100.

[0053] Video receiving unit 210 receives the minimum difference data transmitted from video transmitting unit 130B. Video receiving unit 210 sends the received difference data to video decoding unit 230, and also sends it to geometric transformation unit 220. The minimum difference data includes information related to a timestamp. Note that it is optional whether or not video receiving unit 210 sends the minimum difference data to geometric transformation unit 220, and it is not necessary to send the difference data to geometric transformation unit 220.

[0054] The geometric transformation unit 220 receives the decoded data decoded by the video decoding unit 230 at any time. The decoded data includes information related to a timestamp. The geometric transformation unit 220 performs a geometric transformation process on the decoded data from the video decoding unit 230 based on the operation information received from the operation information communication unit 240. In this case, the geometric transformation unit 220 checks information related to the timestamp of the decoded data, and performs the geometric transformation process based on the operation information corresponding to the operation immediately after that. Note that the first video frame is sent from the video decoding unit 230 to the geometric transformation unit 220 without being subjected to the video decoding process.

[0055] The video decoding unit 230 receives the minimum difference data from the video receiving unit 210. The video decoding unit 230 also receives the geometric transformation data from the geometric transformation unit 220. The video decoding unit 230 combines the minimum difference data and the geometric transformation data while checking the timestamp to generate decoded data. The video decoding unit 230 manages the decoded data in chronological order. The decoded data decoded by the video decoding unit 230 is output as video to a device directly or indirectly connected to the VR device 200.

[0056] If the minimum difference data is null data, there is no need to perform geometric transformation processing or video decoding processing in the VR device 200. In this case, the video decoding unit 230 determines that there is no difference from the immediately preceding video frame, and outputs the immediately preceding video frame again.

[0057] Next, the geometric transformation process and the differential encoding process in the mobile robot 100 will be described in detail with reference to FIGS. FIG. 5 is a diagram for explaining the movement direction of the mobile robot 100. In this embodiment, the mobile robot 100 can move in the A direction (forward), B direction (leftward), C direction (backward), and D direction (rightward) in the figure in response to a user's operation. However, the movement direction of the mobile robot 100 is not limited to this. For example, the mobile robot 100 may be able to move right front, left front, right rear, and left rear. In addition, the movement of the mobile robot 100 is not limited to a linear one, and the mobile robot 100 may move in a curved line. In this embodiment, a plurality of geometric conversion units 120 are provided so as to correspond to such various movement patterns. That is, in this embodiment, at least four geometric conversion units 120 are provided so as to correspond to the four movement patterns of the mobile robot 100, namely, forward, backward, left, and right.

[0058] FIG. 6 is a diagram for explaining the geometric transformation process and the differential encoding process of the mobile robot 100. The diagram shows an example in which the mobile robot 100 moves in the C direction (backward). The multiple geometric transformation units 120 perform the geometric transformation process on the video frame captured by the video capture unit 110 at time t. Specifically, each of the multiple geometric transformation units 120 assumes that the mobile robot 100 moves in a direction corresponding to each of the multiple geometric transformation units 120. Then, each of the multiple geometric transformation units 120 calculates the transformation of a certain feature point and encodes this transformation as a motion vector. Each of the multiple geometric transformation units 120 sends the video frame after the geometric transformation to the video processing unit 130 as geometric transformation data.

[0059] The differential encoding unit 130A of the video processing unit 130 calculates the difference between the video frame captured by the video shooting unit 110 at time t+1 and each of the multiple geometric transformation data captured at the immediately preceding time t and subjected to geometric transformation processing for each of the four movement patterns of forward, backward, left and right. Then, the differential encoding unit 130A compares the difference for each movement pattern. In the illustrated case, since the mobile robot 100 moves in the C direction, the difference D0 between the video frame at time t+1 and the geometric transformation data corresponding to the movement of the mobile robot 100 in the C direction (backward) is the smallest. In other words, the identical portion C0 between the video frame at time t+1 and the geometric transformation data corresponding to the movement of the mobile robot 100 in the C direction (backward) is the largest. The differential encoding unit 130A encodes the difference D0 excluding the identical portion C0 and outputs it as the minimum differential data.

[0060] The process executed by the robot VR system according to the embodiment of the present disclosure will be described with reference to the sequence diagram of Fig. 7. Note that, in the figure, S11 to S16 indicate the process executed by the mobile robot 100, and S21 to S24 indicate the process executed by the VR device 200.

[0061] Specifically, the video transmission method of the present disclosure includes: A step of geometrically transforming the image acquired by the image capturing unit 110 for each movement pattern of the image acquiring unit 110 and outputting the transformed image as geometric transformation data; A step of comparing a difference between the image acquired by the image capturing unit 110 and each of a plurality of pieces of geometric transformation data geometrically transformed for each movement pattern by the geometric transformation unit, and encoding the difference between the image acquired by the image capturing unit 110 and the geometric transformation data corresponding to the movement of the image acquiring unit; Includes.

[0062] First, at time t=0, the video capture unit 110 of the mobile robot 100 captures a video (video frame) (S11). At the stage of time t=0, the mobile robot 100 has not received any operation information, so there is no change in the posture or position of the mobile robot 100, and the position of the video capture unit 110 does not change either. In this case, there is no difference between each video frame, so the video transmission unit 130B transmits the video frame captured by the video capture unit 110 to the outside as is (S12). The video frame is received by the video receiving unit 210 of the VR device 200. The VR device 200 displays the video frame as a video on a device directly or indirectly connected to the VR device 200 (S21).

[0063] When an operation from the user is accepted, the operation information communication unit 240 sends the operation information to the geometric transformation unit 220 of the VR device 200 and also transmits it to the outside of the VR device 200 (S22). The geometric transformation unit 220 executes a geometric transformation process on the video frame captured at time t=0 based on the operation information (S23). The geometric transformation unit 220 sends the geometrically transformed video frame as geometric transformation data to the video decoding unit 230. Note that in the next and subsequent geometric transformation processes, the geometric transformation unit 220 performs the geometric transformation process on the decoded data sent from the video decoding unit 230.

[0064] On the other hand, the operation information transmitted to the outside of the VR device 200 is received by a robot operation unit of the mobile robot 100. The mobile robot 100 operates based on the operation information (S10).

[0065] Each of the plurality of geometric transformation units 120 performs a geometric transformation process assuming that the mobile robot 100 has moved in a movement pattern corresponding to each of the plurality of geometric transformation units 120 (S13). Each of the plurality of geometric transformation units 120 transmits the geometrically transformed video frame to the image processing unit 130 as geometric transformation data.

[0066] The differential encoding unit 130A calculates the difference between the video frame captured by the video capture unit at time t=1 and each of the multiple pieces of geometric transformation data captured at the immediately preceding time t and subjected to geometric transformation processing for each movement pattern.The differential encoding unit 130A then compares the differences for each movement pattern, and calculates (selects) and encodes the smallest difference (S14, S15).The video transmission unit 130B transmits the smallest difference data to the outside of the mobile robot 100 (S16).

[0067] The minimum difference data from the video transmission unit 130B is received by the video receiving unit 210 of the VR device 200. The video receiving unit 210 sends the minimum difference data to the video decoding unit 230. The video decoding unit 230 checks the timestamp and synthesizes the minimum difference data with the geometric transformation data received from the geometric transformation unit 220 to generate decoded data. Thereafter, the mobile robot 100 and the VR device 200 exchange minimum difference data and repeat the same process.

[0068] Next, the processes executed by the video transmission device 300 and the VR device 200 will be described with reference to the flowcharts of FIGS. FIG. 8 is a flowchart illustrating the processing executed by the mobile robot 100.

[0069] The video capturing unit 110 of the mobile robot 100 captures video. The video captured by the video capturing unit 110 is transmitted to the outside via the video transmission unit 130B of the video processing unit 130 (step S101).

[0070] Each of the plurality of geometric transformation units 120 performs a geometric transformation process assuming that the mobile robot 100 moves according to a movement pattern corresponding to each of the plurality of geometric transformation units 120 (step S102). Each of the plurality of geometric transformation units 120 transmits the geometrically transformed video frame to the image processing unit 130 as geometric transformation data.

[0071] The differential encoding unit 130A of the video processing unit 130 calculates (selects) and encodes the smallest difference based on the video frame received from the video shooting unit 110 and the multiple pieces of geometric transformation data received from the multiple geometric transformation units 120 (step S103). The video transmission unit 130B transmits the minimum difference data to the outside of the mobile robot 100 (step S104).

[0072] FIG. 9 is a flowchart illustrating the processing executed by the VR device 200. The video receiving unit 210 of the VR device 200 judges whether or not video data (video frames) have been received from the video transmitting unit 130B (step S201). If it is judged that video data has not been received (step S201: No), the video receiving unit 210 waits until video data is received.

[0073] On the other hand, when it is determined that the video data has been received (step S201: Yes), the VR device 200 displays the video data as a video on a device directly or indirectly connected to the VR device 200 (step S202).

[0074] The operation information communication unit 240 determines whether or not an operation from the user has been received (step S203). If it is determined that an operation from the user has not been received (step S203: Yes), the operation information communication unit 240 repeats the processes from steps S201 to S203 until an operation from the user is received.

[0075] On the other hand, when it is determined that an operation from the user has been accepted (step S203: No), the operation information communication unit 240 sends the operation information to the geometric transformation unit 220 of the VR device 200.

[0076] The geometric transformation unit 220 constantly receives the decoded data decoded by the video decoding unit 230. The geometric transformation unit 220 executes a geometric transformation process on the decoded data from the video decoding unit 230, based on the operation information received from the operation information communication unit 240 (step S204).

[0077] Next, the video receiving unit 210 of the VR device 200 determines whether or not the minimum difference data has been received from the video transmitting unit 130B (step S205). If it is determined that the minimum difference data has not been received (step S205: No), the video receiving unit 210 waits until the minimum difference data is received.

[0078] On the other hand, when it is determined that the minimum difference data has been received (step S205: Yes), the video receiving unit 201 sends the minimum difference data to the video decoding unit 230. The video decoding unit 230 checks the timestamp, synthesizes the minimum difference data and the geometric transformation data, and outputs the decoded data (step S206).

[0079] The device of the present invention can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network. The program of the present disclosure is a program for causing a computer to realize each function of the device according to the present disclosure, and is a program for causing a computer to execute each procedure of the method executed by the device according to the present disclosure. [Industrial Applicability]

[0080] The present disclosure can be applied to the information and communications industry. [Explanation of symbols]

[0081] 10, 100: Mobile robot 110: Videography Department 120: Geometric transformation section 130: Image processing section 130A: Differential encoding unit 130B: Video transmission section 20, 200: VR device 210: Video receiving unit 220: Geometric transformation section 230: Video decoding unit 240: Operation information communication unit 300: Video transmission device

Claims

1. a geometric transformation unit that performs geometric transformation on the image acquired by the image acquisition unit for each movement pattern of the image acquisition unit and outputs the result as geometric transformation data; a difference calculation unit that compares a difference between the image acquired by the image acquisition unit and each of a plurality of pieces of geometric transformation data obtained by performing geometric transformation for each of the movement patterns by the geometric transformation unit, and encodes a difference between the image and the geometric transformation data corresponding to the movement of the image acquisition unit; A video transmitting device comprising:

2. the geometric transformation unit performs geometric transformation on the image in accordance with a movement amount or a movement direction of the image acquisition unit for each of the movement patterns of the image acquisition unit; 2. The video transmission device according to claim 1.

3. The image acquisition unit is provided in a control target that receives operation information, moves in response to the operation information, and acquires an image of the surroundings of the control target.

2. The video transmission device according to claim 1.

4. the difference calculation unit compares a first image acquired by the image acquisition unit with each of a plurality of pieces of geometric transformation data obtained by geometrically transforming a second image acquired by the image acquisition unit before the first image was acquired by the image acquisition unit, and calculates a difference between the images and the geometric transformation data corresponding to the movement of the control object.

2. The video transmission device according to claim 1.

5. A communication unit that transmits a result of the calculation by the difference calculation unit as minimum difference data to an external device. The video transmission device according to claim 1.

6. The video transmission device according to claim 5 , a receiver including a receiving unit that receives the difference data transmitted from the communication unit, a second geometric transformation unit that performs geometric transformation on a video frame corresponding to the difference data received by the receiving unit and outputs geometric transformation data, and a video decoding unit that decodes video based on the geometric transformation data and the difference data output by the second geometric transformation unit; A robot VR system comprising:

7. a step of geometrically transforming the image acquired by the image acquisition unit for each movement pattern of the image acquisition unit and outputting the transformed image as geometric transformation data; comparing a difference between the image acquired by the image acquisition unit and each of a plurality of pieces of geometric transformation data obtained by performing geometric transformation for each of the movement patterns, and encoding the difference between the image and the geometric transformation data corresponding to the movement of the image acquisition unit; A video transmission method comprising:

8. A program for causing a computer to realize each of the functional units of the video transmission device according to any one of claims 1 to 5.

Images