Video transmission system

Abstract

To provide a video transmission technique that can ensure video continuity at a receiving side.SOLUTION: A communication device is connected to a receiving-side device via N (N is an integer greater than or equal to 2) communication lines. The communication device transmits video to the receiving-side device. More precisely, the communication device spatially divides one frame of image into N segmented images. The communication device then transmits the N segmented images to the receiving-side device in parallel via each of the N communication lines.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a technology for transmitting video.

Background Art

[0002] Patent Document 1 discloses a video data transmission device. The video data transmission device separately encodes a basic frame and a differential frame of video. Then, the video data transmission device preferentially transmits the basic frame and transmits the differential frame when there is a margin in the communication bandwidth.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the technology disclosed in Patent Document 1, when there is no margin in the communication bandwidth, only the basic frame is transmitted. In that case, the continuity and real-time property of the video on the reception side deteriorate. This is not preferable from the viewpoint of the use of the video on the reception side.

[0005] One object of the present disclosure is to provide a video transmission technology capable of ensuring the continuity of video on the reception side.

Means for Solving the Problems

[0006] A first aspect relates to a communication device that transmits video to a reception-side device connected via N communication lines (N is an integer of 2 or more). The communication device includes a controller. The controller spatially divides an image of one frame into N divided images. Then, the controller transmits the N divided images to the reception-side device in parallel via each of the N communication lines.

[0007] The second perspective relates to video transmission systems. The video transmission system includes a communication device and a receiving device relating to the first aspect. The receiving device generates a reconstructed image by combining one or more segmented images received via N communication lines within the reception period for one frame. [Effects of the Invention]

[0008] According to this disclosure, it becomes possible to ensure the continuity and real-time nature of video at the receiving end while suppressing communication costs. [Brief explanation of the drawing]

[0009] [Figure 1] This is a conceptual diagram illustrating the outline of a video transmission system according to an embodiment. [Figure 2] This is a conceptual diagram illustrating the segmented image transmission according to the embodiment. [Figure 3] This is a conceptual diagram illustrating the segmented image transmission according to the embodiment. [Figure 4] This is a block diagram showing an example configuration of a video transmission system according to an embodiment. [Figure 5] This flowchart shows an example of processing by the first communication device (transmitting device) according to the embodiment. [Figure 6] This flowchart shows an example of processing by the second communication device (receiving device) according to the embodiment. [Modes for carrying out the invention]

[0010] Embodiments of this disclosure will be described with reference to the attached drawings.

[0011] 1. Overview of the video transmission system Figure 1 is a conceptual diagram illustrating the outline of the video transmission system 1 according to this embodiment. The video transmission system 1 includes a first communication device 10 (transmitter) and a second communication device 20 (receiver). The first communication device 10 and the second communication device 20 are connected to each other via a communication network.

[0012] The first communication device 10 acquires the video to be transmitted. The video consists of multiple image frames (IMGs). The first communication device 10 transmits the video to the second communication device 20. The second communication device 20 receives the video transmitted from the first communication device 10. The video received by the second communication device 20 is used for various purposes. For example, the video is presented to the user by being displayed on a display device or the like.

[0013] For example, the first communication device 10 is a mobile body 100. Examples of the mobile body 100 include vehicles, robots, flying objects, etc. A vehicle may be an autonomous vehicle. Examples of robots include logistics robots, work robots, etc. Examples of flying objects include airplanes, drones, etc. The second communication device 20 is an external device located outside the mobile body 100.

[0014] As an example, consider the case where the mobile device 100 is the target of remote support by a remote operator O. Remote support is a concept that includes remote operation, remote monitoring, remote decision-making, etc. The remote operator terminal 200 is a terminal operated by the remote operator O to provide remote support to the mobile device 100. The mobile device 100 corresponds to the first communication device 10, and the remote operator terminal 200 corresponds to the second communication device 20, and the mobile device 100 and the remote operator terminal 200 communicate with each other. The mobile device 100 is equipped with a camera C. Camera C acquires video (image IMG) showing the surrounding situation of the mobile device 100. The mobile device 100 transmits the video (image IMG) obtained by camera C to the remote operator terminal 200. The remote operator terminal 200 receives the video (image IMG) from the mobile device 100 and displays the received video (image IMG) on a display device. The remote operator O observes the video (image IMG) displayed on the display device to understand the surrounding environment of the mobile device 100 and provides remote support to the mobile device 100. Instructions from the remote operator are sent to the mobile device 100 from the remote operator terminal 200. The mobile device 100 operates according to the instructions from the remote operator O.

[0015] According to this embodiment, the first communication device 10 and the second communication device 20 are connected via multiple communication lines L-1 to LN, and communication can be performed using multiple communication lines L-1 to LN. Here, N is an integer of 2 or more. A communication line can also be referred to as a communication carrier. The communication carrier and communication method may differ for each communication line L. Examples of communication methods include the standard cellular method provided by a Mobile Network Operator (MNO), the low-cost cellular method provided by a Mobile Virtual Network Operator (MVNO), and the Wireless LAN (Local Area Network) method.

[0016] The first communication device 10 can transmit the image IMG to the second communication device 20 via N communication lines L-1 to L-N. Even if some of the communication lines are interrupted, it is possible to transmit the image IMG via other available communication lines. Therefore, the robustness of video transmission is improved. Also, the continuity and real-time nature of the video on the receiving side are ensured. These are preferable from the perspective of video utilization on the receiving side.

[0017] There is room for further consideration regarding how to transmit one frame of the image IMG using the N communication lines L-1 to L-N.

[0018] First, consider "redundant image transmission" as a comparative example. In redundant image transmission, the first communication device 10 duplicates the entire image IMG of one frame and transmits N identical images IMG via N communication lines L-1 to L-N. The second communication device 20 receives the same image IMG from the N communication lines L-1 to L-N. Even if some of the communication lines are interrupted, the image IMG is transmitted to the second communication device 20 via other communication lines. Therefore, the above-described technical effects are obtained. However, the second communication device 20 adopts the image IMG received first and discards the image IMG received later. The transmission of the discarded image IMG is consequently wasted. Therefore, the demerit of redundant image transmission is that the communication cost increases.

[0019] Therefore, the present embodiment provides a video transmission technology that can ensure the continuity and real-time nature of the video on the receiving side while suppressing the communication cost. That is the "divided image transmission" described below.

[0020] 2. Divided Image Transmission FIG. 2 is a conceptual diagram for explaining the divided image transmission according to the present embodiment. In the example shown in FIG. 2, N is 4. Hereinafter, the processing regarding one frame of the image IMG constituting the video will be described.

[0021] The first communication device 10 spatially divides one frame of image IMG into N segmented images SIMG-1 to SIMG-N. Conversely, by combining these N segmented images SIMG-1 to SIMG-N, the original image IMG is restored. The first communication device 10 assigns identification information to each segmented image SIMG-i (i=1 to N). The identification information includes the position of the segmented image SIMG-i in the original image IMG.

[0022] In the example shown in Figure 2, each segmented image SIMG-i (i=1 to N) is widely distributed within the original image IMG without local bias. To realize such distributed segmented images SIMG-i, the first communication device 10 may divide the image IMG into multiple matrix-like unit regions UA and perform image segmentation processing for each unit region UA. In this case, the first communication device 10 divides each unit region UA ​​into first unit segmented images USIMG-1 to the Nth unit segmented images USIMG-N. The first communication device 10 then generates the i-th segmented image SIMG-i as a set of the i-th unit segmented images USIMG-i in each of the multiple unit regions UA.

[0023] The segmented images SIMG-1 to SIMG-N may be uniformly distributed within the image IMG. The segmented images SIMG-1 to SIMG-N may have the same arrangement pattern except for their position.

[0024] The first communication device 10 assigns the N divided images SIMG-1 to SIMG-N generated in this way to each of the N communication lines L-1 to LN. That is, the first communication device 10 assigns the divided image SIMG-i to communication line Li. Then, the first communication device 10 transmits the N divided images SIMG-1 to SIMG-N to the second communication device 20 in parallel through each of the N communication lines L-1 to LN.

[0025] The second communication device 20 receives one or more segmented images SIMG via N communication lines L-1 to LN within the reception period for one frame. The second communication device 20 then generates a reconstructed image IMGR by combining the one or more received segmented images SIMG. At this time, the second communication device 20 determines the position of the segmented image SIMG-i in the original image IMG based on the identification information contained in the received segmented image SIMG-i. The position of the segmented image SIMG-i in the original image IMG is used directly as the position of the segmented image SIMG-i in the reconstructed image IMGR.

[0026] In the example shown in Figure 2, all four communication lines L-1 to L-4 are functioning normally, and no communication errors such as communication interruptions occur. The second communication device 20 receives four segmented images SIMG-1 to SIMG-4 in parallel via each of the four communication lines L-1 to L-4 within the reception period for one frame. The second communication device 20 then generates a reconstructed image IMGR by combining the four received segmented images SIMG-1 to SIMG-4. The reconstructed image IMGR matches the original image IMG. The reconstructed image IMGR is displayed, for example, on a display device.

[0027] Figure 3 shows a scenario where a communication error, such as a communication interruption, occurs in some of the communication lines L. For example, a communication error occurs in communication line L-1, which is used to transmit the segmented image SIMG-1. The second communication device 20 receives segmented images SIMG-2 to SIMG-4 via communication lines L-2 to L-4, respectively, within the reception period for one frame, but does not receive segmented image SIMG-1 from communication line L-1. For convenience, the segmented image SIMG-i received within the reception period for one frame will be referred to below as the received segmented image RIMG-i. The second communication device 20 generates the reconstructed image IMGR by combining at least the received segmented images RMG-2 to RIMG-4.

[0028] For segmented images SIMG-1 that were not received within the reception period for one frame, the previously received segmented image PIMG-1 from communication line L-1 may be used as a substitute. In this case, the second communication device 20 generates a reconstructed image IMGR by combining the received segmented images RMG-2 to RIMG-4 with the previously received segmented image PIMG-1.

[0029] In the example shown in Figure 3, the reconstructed image IMGR does not perfectly match the original image IMG. However, at least for the portion of the received segmented images RIMG-2 to RIMG-4, the reconstructed image IMGR matches the original image IMG. Even when such a reconstructed image IMGR is displayed on a display device, it does not look significantly inferior. It can be said that the original image IMG is sufficiently reproduced to a level where there are no problems with image recognition.

[0030] The important point is that the entire image data (IMG) is not completely interrupted, but rather that at least a portion of the real-time image data reaches the second communication device 20. In other words, the segmented image transmission according to this embodiment ensures the continuity and real-time nature of the video at the receiving end. This is desirable from the viewpoint of how the video can be used at the receiving end.

[0031] Furthermore, according to this embodiment, the original image IMG is divided into N segmented images SIMG-1 to SIMG-N, and the N segmented images SIMG-1 to SIMG-N are transmitted via each of the N communication lines L-1 to LN. Unlike the comparative example described above (redundant image transmission), the same image IMG is not transmitted via the N communication lines L-1 to LN. Therefore, duplicate image IMGs are not discarded by the receiving second communication device 20. In other words, there is no image transmission that ends up being wasted. Compared to the case of redundant image transmission, the amount of transmitted data is significantly reduced, and therefore communication costs are also significantly reduced.

[0032] Thus, the segmented image transmission method according to this embodiment makes it possible to suppress communication costs while ensuring the continuity and real-time nature of the video at the receiving end.

[0033] As shown in the examples in Figures 2 and 3, the image IMG may be divided into multiple matrix-like unit regions UA, and image segmentation processing may be performed for each unit region UA. In this case, each segmented image SIMG-i will be widely distributed within the image IMG without local bias. Therefore, even if some segmented images RIMG-i received from some communication lines Li are missing, the appearance of the reconstructed image IMGR will be closer to the original image IMG. In other words, the visual reproduction of the reconstructed image IMGR will be higher. From this perspective, the segmented images SIMG-1 to SIMG-N may be uniformly distributed within the image IMG. The segmented images SIMG-1 to SIMG-N may have the same arrangement pattern except for their position.

[0034] The data size (area) of each segmented image SIMG-1 to SIMG-N does not have to be the same. For example, the data size (area) of each segmented image SIMG-1 to SIMG-N may be set based on the communication quality of communication lines L1 to LN. In other words, the segmentation ratio of segmented images SIMG-1 to SIMG-N may be set according to the communication quality of communication lines L1 to LN. Examples of communication quality include communication speed (throughput), delay, jitter, etc. In this case, the image segmentation process is performed so that a larger amount of data is allocated to the communication line Li with higher communication quality. By considering the communication quality (performance) of each communication line L in this way, it becomes possible to transmit segmented images more efficiently.

[0035] In general terms, the following applies: The first communication device 10 divides the image IMG into N segmented images SIMG-1 to SIMG-N based on the communication quality of each of the N communication lines L-1 to LN. Here, we consider a first communication line La with relatively high communication quality and a second communication line Lb with relatively low communication quality. The first segmented image SIMG-a is the segmented image SIMG assigned to the first communication line La, and the second segmented image SIMG-b is the segmented image SIMG assigned to the second communication line Lb. In this case, the data size of the first segmented image SIMG-a is larger than the data size of the second segmented image SIMG-b.

[0036] The first communication device 10 applies the above-described segmented image transmission to at least some of the multiple image IMGs (frames) that constitute the video to be transmitted. This will at least provide some effect. The first communication device 10 may also apply the above-described segmented image transmission to all of the image IMGs (frames) that constitute the video to be transmitted.

[0037] 3. Configuration Examples and Processing Examples Figure 4 is a block diagram showing an example configuration of the video transmission system 1 according to this embodiment.

[0038] The transmitting first communication device 10 includes a first controller 11. The first controller 11 includes, as functional blocks, an image acquisition unit 12, an image splitting unit 13, an encoder 14, and a transmission unit 15. The first controller 11 may be composed of a processor, memory, various circuits, a transceiver, etc. The processor performs various information processing. The memory stores various information necessary for processing by the processor. The processor may execute a computer program. The computer program may be recorded on a computer-readable recording medium.

[0039] The receiving second communication device 20 includes a second controller 21. The second controller 21 includes, as functional blocks, a receiving unit 22, a decoder 23, a buffer 24, and an image synthesis unit 25. The second controller 21 may be composed of a processor, memory, various circuits, a transceiver, etc. The processor performs various information processing. The memory stores various information necessary for processing by the processor. The processor may execute a computer program. The computer program may be recorded on a computer-readable recording medium.

[0040] Figure 5 is a flowchart showing an example of processing by the first communication device 10 on the transmitting side. Figure 6 is a flowchart showing an example of processing by the second communication device 20 on the receiving side.

[0041] In step S12, the image acquisition unit 12 acquires the image IMG included in the video to be transmitted, frame by frame.

[0042] In step S13, the image splitting unit 13 performs an image splitting process that spatially divides one frame of image IMG into N split images SIMG-1 to SIMG-N. This image splitting process is as described in Section 2 above. The image splitting unit 13 may also divide the image IMG into a matrix of multiple unit regions UA and perform the image splitting process for each unit region UA. In this case, each split image SIMG-i will be widely distributed within the image IMG without local bias.

[0043] In step S14, the encoder 14 encodes the segmented images SIMG-1 to SIMG-N. The encoder 14 may include N encoders and perform the encoding of the N segmented images SIMG-1 to SIMG-N in parallel.

[0044] In step S15, the transmission unit 15 transmits the encoded segmented images SIMG-1 to SIMG-N to the second communication device 20 in parallel via the respective communication lines L-1 to LN.

[0045] In step S22, the receiving unit 22 receives segmented images SIMG-1 to SIMG-N via communication lines L-1 to LN, respectively. The received segmented image RIMG-i is the segmented image SIMG-i received within the reception period for one frame.

[0046] In step S23, the decoder 23 decodes the received segmented image RIMG-i. The decoder 23 may include N decoders and perform the decoding of N received segmented images RIMG-1 to RIMG-N in parallel.

[0047] In step S24, buffer 24 stores the decoded received segmented image RIMG-i.

[0048] In step S25, the image synthesis unit 25 determines whether all N received segmented images RIMG-1 to RIMG-N are available within the reception period for one frame. If all N received segmented images RIMG-1 to RIMG-N are available (step S25; Yes), the process proceeds to step S26. On the other hand, if there are no received segmented images RIMG-m from some of the communication lines Lm (step S25; No), the process proceeds to step S27.

[0049] In step S26, the image synthesis unit 25 generates a reconstructed image IMGR by synthesizing N received segmented images RIMG-1 to RIMG-N. The reconstructed image IMGR is displayed, for example, on a display device.

[0050] In step S27, the image synthesis unit 25 retrieves the previously received segmented image RIMG-m from the communication line Lm from the buffer 24. The previously received segmented image PIMG-m is the same as the received segmented image RIMG-m retrieved from the buffer 24. The image synthesis unit 25 then generates a reconstructed image IMGR by combining one or more received segmented images RIMG-i with the previously received segmented image PIMG-m. The reconstructed image IMGR is displayed, for example, on a display device.

[0051] 4. Variations As shown in Figure 4, the video transmission system 1 may further include a communication quality acquisition unit 30. The communication quality acquisition unit 30 acquires communication quality information for each of the communication lines L1 to LN. Examples of communication quality include communication speed (throughput), delay, jitter, etc. The communication speed may be an actual value or a predicted value. For example, the communication quality acquisition unit 30 acquires communication quality information for each communication line L based on at least one of the data transmission result from the transmission unit 15 and the data reception result at the reception unit 22. For example, the communication quality acquisition unit 30 is included in the first controller 11 on the transmission side. In that case, the data reception result at the reception unit 22 may be fed back to the first controller 11. As another example, the communication quality acquisition unit 30 may be included in the second controller 21 on the reception side.

[0052] The image splitting unit 13 acquires communication quality information for each communication line L from the communication quality acquisition unit 30. Then, the image splitting unit 13 divides the image IMG into N split images SIMG-1 to SIMG-N based on the communication quality of each communication line L. For example, consider a first communication line La with relatively high communication quality and a second communication line Lb with relatively low communication quality. The first split image SIMG-a is the split image SIMG assigned to the first communication line La, and the second split image SIMG-b is the split image SIMG assigned to the second communication line Lb. In this case, the data amount of the first split image SIMG-a is larger than the data amount of the second split image SIMG-b. That is, the splitting ratio of split images SIMG-1 to SIMG-N is set according to the communication quality. This makes it possible to transmit split images more efficiently. [Explanation of symbols]

[0053] 1…Video transmission system, 10…First communication device, 11…First controller, 20…Second communication device, 21…Second controller, L…Communication line, IMG…Image, IMGR…Restored image, SIMG…Segmented image, RIMG…Received segmented image, USIMG…Unit segmented image

Claims

1. A transmitting device that transmits video, A receiving device connected to the transmitting device via N communication lines (where N is an integer greater than or equal to 2) Equipped with, The transmitting device includes a controller, The aforementioned controller, The image of one frame is spatially divided into N segmented images, The N divided images are transmitted in parallel to the receiving device via each of the N communication lines. It is configured in such a way, The receiving device generates a reconstructed image by combining one or more segmented images received via the N communication lines within the reception period for one frame. The received segmented image is a segmented image received by the receiving device within the reception period for the frame, If, within the reception period for the aforementioned frame, there are no received segmented images from some of the N communication lines, the receiving device generates the reconstructed image by combining the received segmented image with the segmented image previously received from some of the communication lines. Video transmission system.

2. A video transmission system according to claim 1, The aforementioned N divided images include the i-th divided image, where i is 1 to N. The aforementioned controller, The image of the aforementioned one frame is divided into a matrix of multiple unit regions, Each of the aforementioned plurality of unit regions is spatially divided into a first unit division image and an Nth unit division image, The set of i-th unit division images in each of the plurality of unit regions is generated as the i-th division image. Video transmission system.

3. A video transmission system according to claim 1, The controller is further configured to divide the image of one frame into the N divided images based on the communication quality of each of the N communication lines. The N communication lines include a first communication line and a second communication line having lower communication quality than the first communication line. The N divided images include a first divided image assigned to the first communication line and a second divided image assigned to the second communication line. The data size of the first divided image is larger than the data size of the second divided image. Video transmission system.

4. The video transmission system according to Claim 1, The receiving device includes a buffer for storing the received segmented images, If, within the reception period for the aforementioned frame, there are no received segmented images from some of the N communication lines, the receiving device retrieves the previously received segmented image from the some communication lines from the buffer. Video transmission system.

5. A video transmission system according to any one of claims 1 to 4, The transmitting device is a mobile object that is the target of remote support by a remote operator. The receiving device is a terminal operated by the remote operator to provide remote support to the mobile object. The receiving device displays the restored image on a display device so that the remote operator can view the restored image. Video transmission system.

Images