System, receiving device, transmitting device, method, program, and recording medium recording program, for moving image communication

Abstract

To provide a system, device, method, program, and recording medium on which the program is recorded, which are capable of reproducing transmitted moving images in real time without delay and improving the reproducibility of the moving images.SOLUTION: In a moving image communication system 1, an image transmitting device 10 divides each still image frame of a moving image composed of a plurality of still image frames into a plurality of blocks, encodes the plurality of blocks for each block independently, and transmits coded blocks, which are the blocks that have been encoded, using a connectionless communication method. An image receiving device 40 reconstructs a still image frame by arranging, at the corresponding positions, decoded blocks generated by decoding the coded blocks of one still image frame received during a coded block reception period corresponding to one still image frame for each of the transmitted still image frames.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system, a receiving device, a transmitting device, a method, a program, and a recording medium on which a program is recorded, for video communication. [Background technology]

[0002] In recent years, technology for streaming live video images from a camera via networks such as the Internet, Ethernet, LAN, etc. has become widespread. Such technology is used, for example, in video conference systems, surveillance camera systems, and remote control systems.

[0003] Japanese Patent Application Laid-Open Publication No. 2008-193510 discloses the following technology for transmitting moving images from a camera in real time. Specifically, moving images captured by a camera are encoded and transmitted using UDP (User Datagram Protocol) / IP protocol. Then, (1) on the receiving side, if an error such as packet loss occurs, a request is made to retransmit the packet in which the error was detected, and an image in which the error has been corrected based on the packet transmitted in response to the transmission request is decoded and displayed. Also, (2) when real-time performance is important, moving images including errors are displayed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-193510 Summary of the Invention [Problem to be solved by the invention]

[0005] In surveillance camera systems and remote control systems, it is necessary to know what is currently happening at the site, so it is required to be able to check on-site images in real time without delay.

[0006] Regarding remote control systems, for example, when an unmanned mobile object such as an unmanned aircraft or vehicle equipped with a camera is remotely controlled by viewing video images transmitted from the camera via wireless LAN, even the slightest delay will result in a time difference between the actual movement of the unmanned mobile object and the movement of the unmanned mobile object in the video images viewed by the operator, and in the worst case scenario, it may become impossible to avoid the unmanned mobile object colliding with an obstacle or crashing.

[0007] Furthermore, if the communication conditions between the video transmitting device and the video receiving device deteriorate (for example, due to line failure or deterioration in radio wave conditions), the video seen by the operator may stop (the so-called "image freeze" phenomenon occurs), and the operator may be viewing the displayed video without realizing that it is an image from a short time ago.In such cases, the operator will not be able to recognize the deterioration in the communication conditions without delay, and will be unable to immediately stop the unmanned vehicle, which would normally be necessary in response to a significant deterioration in the communication conditions or a situation where communication is impossible.

[0008] In the technology of the above-mentioned Cited Document 1, in case (1), when communication conditions are poor, packet retransmission requests overlap, resulting in packet delays. In case (2), although received packets are real-time, some packets may be lost. In both cases (1) and (2), if a packet is not received in a timely manner, the entire frame of that packet may not be decoded. Furthermore, for frames that could not be decoded in whole or in part due to packets not being received in a timely manner, a previous frame that could be decoded is displayed instead. Therefore, as mentioned above, the displayed video may stop, or the previous image may remain displayed.

[0009] Therefore, one of the objects of the present invention is to provide a system, a receiving device, a transmitting device, a method, a program, and a recording medium on which the program is recorded, which can reproduce transmitted moving images in real time without delay and improve the reproducibility of the moving images.

[0010] Another object of the present invention is to provide a system, a receiving device, a transmitting device, a method, a program, and a recording medium on which a program is recorded, which are capable of grasping the situation in real time without delay, including deterioration of communication conditions. [Means for solving the problem]

[0011] One aspect of the present invention provides a moving image communication method that includes dividing each still image frame of a moving image composed of a plurality of still image frames into a plurality of blocks, independently encoding each of the plurality of blocks, and transmitting the encoded blocks, which are the encoded blocks, using a connectionless communication method; and for each of the still image frames, reconstructing the still image frame by arranging, at the corresponding position, a decoded block generated by decoding the encoded block of one still image frame received during an encoded block reception period corresponding to the still image frame, wherein the encoded block reception period is the period from the start time of the current frame to the time elapsed when all encoded blocks of one frame are expected to be received plus a margin.

[0012] Reconstructing the still image frames may include, for each of the still image frames, placing a decoded block generated by decoding the coding block of one of the still image frames received during a coding block reception period corresponding to the one of the still image frames at a corresponding position, and reconstructing the still image frame with the remaining portion as an image corresponding to a no-image signal.

[0013] One aspect of the present invention provides an image receiving method in which each still image frame of a moving image made up of a plurality of still image frames is divided into a plurality of blocks, and the plurality of blocks are coded independently for each block and transmitted using a connectionless communication method, the method including, for each of the still image frames, decoding the coded block of one still image frame received during a coding block reception period corresponding to one of the still image frames to generate a decoded block, and arranging the generated decoded block at a corresponding position to reconstruct the still image frame, wherein the coding block reception period is the period from the start time of the current frame to the time when all coded blocks of one frame are expected to be received plus a margin.

[0014] Reconstructing the still image frames may include, for each of the still image frames, placing a decoded block generated by decoding the coding block of one of the still image frames received during a coding block reception period corresponding to the one of the still image frames at a corresponding position, and reconstructing the still image frame with the remaining portion as an image corresponding to a no-image signal.

[0015] Reconstructing the still image frames may include arranging, for each of the still image frames, a decoded block generated by decoding the coded block of the still image frame received during a coded block reception period corresponding to the still image frame, at a corresponding position, and for the remaining portion, for each of past still image frames from the frame immediately preceding the still image frame to a still image frame that is a predetermined number of frames before the still image frame, among the decoded blocks generated by decoding the coded block of the received past still image frame, at a position corresponding to the remaining portion, with priority given to the decoded block of a still image frame that is temporally closer to the still image frame, to reconstruct the still image frames.

[0016] The next frame start time can be set to (1) a time corresponding to the time at which one frame time has elapsed since the reception time of the coding block of the still image frame that was first received within the coding block reception period, if the coding block that was first transmitted within the still image frame is received within the coding block reception period; or (2) a time at which the difference between one frame time and the expected time required to receive the coding block from the coding block that was first transmitted within the still image frame to the coding block that is earliest in transmission order among the coding blocks of the still image frame that were received within the coding block reception period, if the coding block that was first transmitted within the still image frame is not received and a coding block that was transmitted within the still image frame other than the coding block that was first transmitted within the still image frame is received within the coding block reception period, from the reception time of the coding block of the still image frame that was first received within the coding block reception period, if the next frame start time is set to (1) a time corresponding to the time at which one frame time has elapsed since the reception time of the coding block of the still image frame that was first received within the coding block reception period, if the coding block that was first transmitted within the still image frame is not received and a coding block that was transmitted within the still image frame other than the coding block that was first transmitted within the still image frame is received within the coding block reception period, if the next frame start time is set to (2) a time at which the difference between one frame time and the expected time required to receive the coding block from the coding block that was first transmitted within the still image frame to the coding block that is earliest in transmission order among the coding blocks of the still image frame that were received within the coding block reception period, from the reception time of the coding block of the still image frame that was first ...

[0017] The next frame start time may be set to the time when one frame time has elapsed from the current frame start time if all of the coding blocks of one of the still image frames have not been received within the coding block reception period.

[0018] The plurality of coded blocks may be transmitted in an order in which the order of the blocks within the still image frame is rearranged.

[0019] The plurality of coded blocks may be transmitted in an order in which the order of positions of the blocks within the still image frame is randomly rearranged.

[0020] The moving image encoding method may be a Motion JPEG-based method.

[0021] The margin time can be the difference between the one frame time and the sum of the expected time to receive all coded blocks of the one frame and the rendering time of the one frame when all blocks of the one frame are decoded.

[0022] The estimated time for receiving all the coded blocks of one frame, the estimated time for receiving the coded blocks from the first transmitted block in one still image frame to the earliest transmitted coded block among the coded blocks of one still image frame received within the coded block reception period, and / or the drawing time may be determined based on corresponding actual measured values ​​and updated at predetermined timings during reception of the moving image.

[0023] The estimated time to receive all the coded blocks of one frame, which is determined based on the corresponding actual measurement values, the estimated time to receive the coded blocks from the first transmitted block in one still image frame to the earliest transmitted coded block among the coded blocks of one still image frame received within the coded block reception period, and / or the value of the drawing time, may be updated at a predetermined timing during the reception of the moving image.

[0024] One aspect of the present invention provides an image transmission method that includes dividing each still image frame of a moving image composed of a plurality of still image frames into a plurality of blocks, independently encoding each of the plurality of blocks, and transmitting the encoded blocks, which are the encoded blocks, using a connectionless communication method.

[0025] One aspect of the present invention provides a program for causing a computer to execute the above method.

[0026] One aspect of the present invention provides a computer-readable recording medium on which the above program is recorded.

[0027] One aspect of the present invention provides a moving image communication system comprising an image division unit that divides each still image frame of a moving image composed of a plurality of still image frames into a plurality of blocks; an encoding unit that encodes the plurality of blocks independently for each block; a transmitting unit that transmits the encoded blocks, which are encoding blocks, using a connectionless communication method; a decoding unit that, for each of the still image frames, decodes the encoding block of one still image frame received during an encoding block reception period corresponding to the still image frame to generate a decoded block; and a frame reconstruction unit that places the generated decoded block at a corresponding position to reconstruct a still image frame, wherein the encoding block reception period is the period from the start time of the current frame to the expected time for receiving all encoding blocks of one frame plus a margin time.

[0028] The frame reconstruction unit may arrange the generated decoded blocks at corresponding positions and reconstruct a still image frame by treating the remaining portion as an image corresponding to a non-image signal.

[0029] One aspect of the present invention provides an image receiving device in which each still image frame of a moving image made up of a plurality of still image frames is divided into a plurality of blocks, and the plurality of blocks are coded independently for each block and transmitted using a connectionless communication method, the image receiving device comprising: a decoding unit that decodes, for each of the still image frames, the coded block of one still image frame received during a coding block reception period corresponding to the still image frame to generate a decoded block; and a frame reconstruction unit that places the generated decoded block at a corresponding position to reconstruct a still image frame, the coding block reception period being the period from the start time of the current frame to the expected time for receiving all coded blocks of one frame plus a margin time.

[0030] The frame reconstruction unit may arrange the generated decoded blocks at corresponding positions and reconstruct a still image frame by treating the remaining portion as an image corresponding to a non-image signal.

[0031] The frame reconstruction unit may be configured to, for each of the still image frames, place a decoded block generated by decoding the coded block of the still image frame received during a coded block reception period corresponding to the still image frame at a corresponding position, and for the remaining portion, for each of past still image frames from the frame immediately preceding the still image frame to a still image frame that is a predetermined number of frames before the still image frame, place the decoded block at a position corresponding to the remaining portion, giving priority to the decoded block of a still image frame that is temporally closer to the still image frame, to reconstruct the still image frame.

[0032] The image receiving device may further include a frame start time setting unit that (1) sets the next frame start time to a time corresponding to the time at which one frame time has elapsed since the reception time of the coding block of the one still image frame that was first received within the coding block reception period, if the coding block that was first transmitted within the one still image frame is received within the coding block reception period, or (2) sets the next frame start time to a time at which the difference between one frame time and the expected time required to receive from the time at which the coding block in the one still image frame was first received to the coding block that was received within the coding block reception period, if the coding block that was first transmitted within the one still image frame is not received and a coding block transmitted within the one still image frame other than the coding block that was first transmitted within the one still image frame, has elapsed since the reception time of the coding block in the one still image frame that was first received within the coding block reception period.

[0033] The frame start time setting unit may set the next frame start time to a time when one frame time has elapsed from the current frame start time if all of the coding blocks of one of the still image frames are not received within the coding block reception period.

[0034] The plurality of coded blocks may be transmitted in an order in which the order of the blocks within the still image frame is rearranged.

[0035] The plurality of coded blocks may be transmitted in an order in which the order of positions of the blocks within the still image frame is randomly rearranged.

[0036] One aspect of the present invention provides an image transmission device comprising an image division unit that divides each still image frame of a moving image composed of a plurality of still image frames into a plurality of blocks, an encoding unit that encodes the plurality of blocks independently for each block, and a transmission unit that transmits the encoded blocks, which are encoded blocks, using a connectionless communication method.

[0037] One aspect of the present invention provides a remote control device including the image transmission device and an imaging device that supplies a moving image to the image transmission device.

[0038] The remote control device may be a mobile object.

[0039] The moving body may be an unmanned moving body.

[0040] In this specification and claims, "encoding independently" refers to encoding such that an image can be decoded using only the encoded data. Therefore, an encoding method that uses differences between units to be encoded is not an encoding method that performs independent encoding. For example, an encoding method that uses differences between frames when the units to be encoded are frames, or an encoding method that uses differences between blocks when the units to be encoded are blocks into which frames are divided, as in the present invention, are not encoding methods that perform independent encoding. For example, in the JPEG2000 format, each frame is encoded independently, but each tile that makes up a frame is not encoded independently. This is because each tile cannot be decoded without main header information. [Effects of the Invention]

[0041] According to the present invention having the above configuration, it is possible to provide a system, a receiving device, a transmitting device, a method, a program, and a recording medium on which a program is recorded, which can reproduce transmitted moving images in real time without delay and improve the reproducibility of the moving images.

[0042] Furthermore, according to the present invention having the above configuration, it is possible to provide a system, a receiving device, a transmitting device, a method, a program, and a recording medium on which a program is recorded, which are capable of grasping the situation in real time without delay, including deterioration of communication conditions, etc. [Brief explanation of the drawings]

[0043] [Figure 1] 1 is a diagram showing the overall configuration of a video communication system according to an embodiment of the present invention; [Figure 2] 1 is a block diagram showing a functional configuration of an image transmitting device according to an embodiment of the present invention; [Figure 3] 1 is a block diagram showing a functional configuration of an image receiving device according to an embodiment of the present invention; [Figure 4] 1 is a diagram showing a hardware configuration of an image transmission device 10 according to an embodiment of the present invention. [Figure 5] 10 is a flowchart illustrating an example of an image transmission process according to an embodiment of the present invention. [Figure 6] FIG. 10 is a diagram showing an example of block position numbers and block transmission numbers of coding blocks in one still image frame. [Figure 7] FIG. 2 illustrates an example of a packet data format. [Figure 8] 10 is a flowchart illustrating an example of an image receiving process according to an embodiment of the present invention. [Figure 9] 10 is a flowchart illustrating an example of details of a process for setting a next frame start time. [Figure 10A] FIG. 1 illustrates an example of a reconstructed still image frame. [Figure 10B] FIG. 10 illustrates another example of a reconstructed still image frame. [Figure 11] FIG. 10 is a diagram showing an example of the relationship between the time elements involved in setting the start time of the next frame when the coding block transmitted first in one still image frame is received within the coding block reception period. [Figure 12]This figure shows an example of the relationship between each time element related to setting the next frame start time when the coding block transmitted first in a still image frame is not received within the coding block reception period, and a coding block transmitted in the still image frame other than the coding block transmitted first in the still image frame is received within the coding block reception period. [Figure 13] FIG. 10 is a diagram showing an example of the relationship between time elements related to setting the start time of the next frame when all the coded blocks of one still image frame are not received within a coded block reception period. DETAILED DESCRIPTION OF THE INVENTION

[0044] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0045] 1 is a diagram showing the overall configuration of a video communication system according to an embodiment of the present invention. The video communication system 1 includes an image transmitting device 10, an imaging device 20, a network 30, an image receiving device 40, and a display device 50. The image transmitting device 10 and the image receiving device 40 are connected via the network 30. Each of the image transmitting device 10, the imaging device 20, the image receiving device 40, and the display device 50 does not have to be configured as a single physical device, and may be configured as multiple physical devices.

[0046] The imaging device 20 captures an image of a subject, acquires moving image data, and supplies the moving image data to the image transmission device 10. The moving image supplied to the image transmission device 10 can be made up of a plurality of still image frames.

[0047] The network 30 may be, for example, the Internet, a telephone network, a wired / wireless public network such as a satellite communication network, or any of various wired / wireless LANs and WANs including Ethernet (registered trademark).

[0048] As the image receiving device 40, in addition to a dedicated device, any appropriate device such as a PC, a smartphone, or a tablet terminal can be used.

[0049] The display device 50 displays the moving image data sent from the image receiving device 40. The display device 50 can be, for example, a CRT display, a liquid crystal display, or an organic EL display.

[0050] FIG. 2 is a block diagram showing the functional configuration of the image transmitting device according to the embodiment of the present invention.

[0051] The image transmitting device 10 includes an image dividing unit 101 , an encoding unit 103 , an encoded block storage unit 105 , a block order changing unit 107 , a packet generating unit 109 , and a transmitting unit 111 .

[0052] The image dividing unit 101 divides each of the still image frames of a moving image made up of a plurality of still image frames into a plurality of blocks.

[0053] The encoding unit 103 generates coded blocks by independently encoding each of the multiple blocks into which each still image frame is divided by the image dividing unit 101. The encoding unit 103 also stores the generated coded blocks in the coded block storage unit 105 in association with the frame number and the block position number.

[0054] The coded block storage unit 105 stores the coded blocks generated by the coding unit 103 .

[0055] The block order changing unit 107 randomly rearranges the positional order of the coding blocks in each still image frame stored in the coding block storage unit 105. The block order changing unit 107 also assigns block transmission numbers in order from the top of the coding blocks whose order has been changed, and stores them in the coding block storage unit 105 in association with the coding blocks.

[0056] The packet generation unit 109 reads out the coding block and the associated frame number, block transmission number, and block position number from the coding block memory unit 105, and generates a packet for a connectionless communication method by attaching a header including the frame number, block transmission number, and block position number to the coding block.

[0057] The transmitter 111 transmits the coded blocks using a connectionless (non-procedure) communication method.

[0058] FIG. 3 is a block diagram showing the functional configuration of the image receiving device according to the embodiment of the present invention.

[0059] The image receiving device 40 includes a receiving unit 401 , a header detecting unit 403 , a decoding unit 405 , a decoded block storage unit 407 , a frame reconstructing unit 409 , and a frame start time setting unit 411 .

[0060] The receiving unit 401 receives packets of coded blocks transmitted from the image transmitting device 10 .

[0061] The header detection unit 403 detects the frame number, block transmission number, and block position number from the packet header, and supplies them to the decoding unit 405 and frame start time setting unit 411 .

[0062] The decoding unit 405 generates a decoded block by decoding the coded block of each still image frame received during a coded block reception period corresponding to the still image frame, and stores the decoded block in the decoded block storage unit 407 together with the corresponding frame number, block transmission number, and block position number.

[0063] The decoded block storage unit 407 stores the decoded blocks together with the corresponding frame numbers, block transmission numbers, and block position numbers.

[0064] The frame reconstruction unit 409 arranges the decoded blocks generated by the decoding unit 405 at the corresponding positions and reconstructs still image frames by treating the remaining portions as images equivalent to a non-image signal. The frame reconstruction unit 409 also sends the reconstructed still image frames to the display device 50 in order.

[0065] The video storage unit 413 stores the still image frames reconstructed by the frame reconstruction unit 409 .

[0066] 4 is a diagram showing the hardware configuration of an image transmitting device 10 according to an embodiment of the present invention. The image transmitting device 10 includes a CPU 10a, a RAM 10b, a ROM 10c, an external memory 10d, an input unit 10e, an output unit 10f, and a communication unit 10g. The RAM 10b, the ROM 10c, the external memory 10d, the input unit 10e, the output unit 10f, and the communication unit 10g are connected to the CPU 10a via a system bus 10h.

[0067] The CPU 10a comprehensively controls each device connected to the system bus 10h.

[0068] The ROM 10c and the external memory 10d store the BIOS and OS, which are control programs for the CPU 10a, as well as various programs and data required to realize the functions executed by the computer.

[0069] The RAM 10b functions as the CPU's main memory, work area, etc. The CPU 10a loads programs and the like required for executing processes from the ROM 10c or the external memory 10d into the RAM 10b, and executes the loaded programs to perform various operations.

[0070] The external memory 10d is configured by, for example, a flash memory, a hard disk, a DVD-RAM, a USB memory, or the like.

[0071] The input unit 10e receives operation instructions, etc. from a user, etc. The input unit 10e is configured with input devices, such as input buttons, a keyboard, a pointing device, a wireless remote control, and a microphone, for example.

[0072] The output unit 10f outputs data processed by the CPU 10a and data stored in the RAM 10b, ROM 10c, and external memory 10d. The output unit 10f is configured by output devices such as a CRT display, LCD, organic EL panel, printer, and speaker.

[0073] The communication unit 10g is an interface for connecting and communicating with external devices via a network or directly, and is configured from an interface such as a serial interface or a LAN interface.

[0074] The hardware configuration of the image receiving device 40 is also similar.

[0075] Each part of the image transmitting device 10 shown in FIG. 1 is realized by various programs stored in ROM or external memory using the CPU, RAM, ROM, external memory, input unit, output unit, communication unit, etc. as resources.

[0076] Based on the above system configuration, an example of video communication processing of the video communication system according to the embodiment of the present invention will be described below with reference to Figs.

[0077] <Image transmission process> FIG. 5 is a flowchart illustrating an example of an image transmission process according to an embodiment of the present invention.

[0078] First, the image division unit 101 of the image transmission device 10 divides each still image frame constituting a moving image supplied from the imaging device 20 into a plurality of blocks (S101). Then, the encoding unit 103 independently encodes each divided block to generate an encoded block (S103). This encoding of the moving image can be performed, for example, using a Motion JPEG-based method. That is, in the Motion JPEG method, each still image frame constituting a moving image is independently encoded / decoded in JPEG format and continuously displayed to form a moving image. In this embodiment, based on the Motion JPEG method, each still image frame is further divided into a plurality of blocks, each of which is independently encoded in JPEG format, and the still image frames are reconstructed and continuously displayed to form a moving image. The method for encoding the moving image is not limited to the Motion JPEG-based method, and any other appropriate method can be used as long as each still image frame constituting a moving image is further divided into a plurality of blocks and each of which is independently encoded in JPEG format. Here, "encoding independently" refers to encoding such that an image can be decoded using only the encoded data. Therefore, an encoding method that uses differences between units to be encoded is not an encoding method that performs independent encoding. For example, when the units to be encoded are frames, an encoding method that uses differences between frames, or when the units to be encoded are blocks into which frames are divided, as in the present invention, an encoding method that uses differences between those blocks, is not an encoding method that performs independent encoding. Also, for example, in the JPEG2000 format, each frame is encoded independently, but each tile that makes up a frame is not encoded independently. This is because each tile cannot be decoded without main header information.

[0079] The encoding unit 103 associates the encoding blocks with frame numbers and block position numbers and stores them in the encoding block storage unit 105 (S105). Here, the frame numbers are consecutive numbers of the still image frames to which the encoding blocks belong. The block position numbers are assigned in the order of the positions of the encoding blocks in the still image frames, and indicate the positions of the encoding blocks within each still image frame. In the example shown in FIG. 6, the block position numbers are assigned as 1 for the block in the upper left corner of the frame, and are assigned as 1, 2, . . . , 30 in order from left to right for each row from the top row to the bottom row. However, the order in which the block position numbers are assigned is not limited to this, and any other appropriate order can be used, such as assigning the block position numbers as 1 for the block in the upper left corner of the frame and assigning the numbers as 1, 2, . . . , 30 in order from top to bottom for each column from the left to the right.

[0080] The block order change unit 107 randomly rearranges the position order of the coding blocks in each still image frame stored in the coding block storage unit 105 (S107). Specifically, the order of the coding blocks based on their block position numbers is randomly changed, and block transmission numbers are assigned in order from the top of the reordered coding block as 1, 2, ..., 30, and stored in the coding block storage unit 105 in association with the coding block. Here, the order of the coding blocks may be changed according to a predetermined rule other than random change, which is expected to have the effect of dispersing parts that are not reproduced when a still image frame is reconstructed, as described below. This block order change process may be omitted.

[0081] The packet generation unit 109 reads out the coded blocks and their associated frame numbers, block transmission numbers, and block position numbers from the coded block storage unit 105, and generates RTP (Real-time Transport Protocol) packets 7 using UDP by attaching a header 71 including a frame number 711, a block transmission number 713, and a block position number 715 to the coded block data 73 (S109). Fig. 7 shows an example of the data format of the packets 7. Here, the packets are not limited to RTP packets using UDP, and can be packets for any other appropriate connectionless communication method.

[0082] Then, the transmitting unit 111 transmits the packets generated by the packet generating unit 109 to the image receiving device 40 via the network 30 in the order of the frame numbers and the block transmission numbers (S111).

[0083] In the above embodiment, the coded blocks are transmitted in the order in which the positional order of the blocks in a still image frame is rearranged, that is, the positional order of the coded blocks is rearranged after encoding each block, and the coded blocks are transmitted. However, this is not limited to this, and any other appropriate configuration can be used. For example, the positional order of each block may be rearranged before encoding each block, and then the coded blocks may be transmitted. Furthermore, in such a configuration, the coded blocks may be transmitted each time a block rearranged before encoding is coded. With such a configuration, the coded blocks are transmitted more quickly, thereby reducing delay time.

[0084] <Image reception processing> Fig. 8 is a flowchart illustrating an example of image reception processing according to an embodiment of the present invention, and Fig. 9 is a flowchart illustrating an example of details of processing for setting the next frame start time.

[0085] The receiving unit 401 of the image receiving device 40 receives a packet transmitted from the image transmitting device 10 (S201). The header detecting unit 403 detects the frame number, block transmission number, and block position number from the header of the packet, and supplies them to the decoding unit 405 and the frame start time setting unit 411 (S203).

[0086] The decoding unit 405 decodes coded blocks constituting a corresponding frame among packets received from the image transmitting device 10 during a coded block reception period, which is the period from the frame start time set by the frame start time setting unit 411 (described later) to the expected time for receiving all coded blocks of one frame (the expected time for receiving all coded blocks constituting one still image frame) plus a margin time, and stores the decoded blocks together with the corresponding frame number, block transmission number, and block position number in the decoded block storage unit 407 (S205). The decoding unit 405 discards packets received outside the coded block reception period. Note that if all coded blocks of one still image frame are received before the coded block reception period has elapsed, the decoding unit 405 may proceed to the next frame reconstruction process (S207) after decoding all coded blocks, without waiting for the coded block reception period to elapse.

[0087] The frame reconstruction unit 409 reads out the decoded blocks stored in the decoded block storage unit 407 and arranges the decoded blocks at positions corresponding to the block position numbers. If there are any coded blocks that were not received within the coded block reception period, the frame reconstruction unit 409 reconstructs a still image frame by displaying the areas at positions corresponding to the block position numbers of the coded blocks that were not received within the coded block reception period as images corresponding to no-image signals (e.g., black images) (S207). If all coded blocks are not received within the coded block reception period, the entire frame becomes an image corresponding to no-image signals. The image corresponding to no-image signals is not limited to black images, but may be a white image or any other appropriate image. FIG. 10A shows an example of a reconstructed frame when all coded blocks are received within the coded block reception period. FIG. 10B shows an example of a reconstructed frame when some coded blocks are not received within the coded block reception period due to poor communication conditions or the like.

[0088] Then, the frame reconstruction unit 409 sequentially sends the reconstructed still image frames to the display device 50 (S209). At this time, the frame reconstruction unit 409 may store the reconstructed still image frames in the video storage unit 413 of the image receiving device 40 and / or an external storage device.

[0089] The display device 50 displays the still image frames sent from the frame reconstruction unit 409 in order to display a moving image.

[0090] In the above embodiment, the decoding unit 405 discarded packets received outside the coding block reception period, but the decoding unit 405 may receive and decode packets received outside the coding block reception period without discarding them, and store them in the decoding block 407. After the coding block reception period has elapsed, the frame reconstruction unit 409 may reconstruct a still image frame based on the decoded blocks stored in the decoding block 407, which are obtained by decoding the coding blocks that constitute the corresponding frame of the packets received from the image transmitting device 10 during the coding block reception period.

[0091] In the frame reconstruction process, instead of a configuration in which a still image frame is reconstructed by treating the region corresponding to the block position number of a coding block not received within a coding block reception period as an image equivalent to a non-image signal, a configuration may be adopted in which, for each of past still image frames from the frame immediately before the still image frame to the still image frame a predetermined number before the still image frame to be reconstructed, the region corresponding to the block position number of a coding block not received within the coding block reception period is used, and the decoded blocks generated by decoding the coding blocks of received past still image frames are arranged in positions corresponding to the block position numbers of coding blocks not received within the coding block reception period corresponding to the still image frame to be reconstructed, with priority given to the decoded blocks of still image frames temporally closer to the still image frame to be reconstructed. In this case, the decoding unit 405 receives and decodes packets received outside the coding block reception period without discarding them, and stores the decoded blocks in the decoded block storage unit 407.

[0092] For example, in the reconstructed frame of FIG. 10B , the still image frame is reconstructed with a black image corresponding to a non-image signal in the regions corresponding to block position numbers other than 4, 6, 15, 19, 22, and 27. In this black image region, a decoded block corresponding to this black image region is placed among the decoded blocks of past still image frames, from the frame immediately preceding the still image frame to the still image frame a predetermined number of frames before the still image frame to be reconstructed. If decoded blocks of multiple still image frames exist at the same position, priority is given to the decoded block of the still image frame temporally closer to the still image frame to be reconstructed. The number of past still image frames can be set to a small number that does not impair real-time performance depending on the scene. Since past still image frames often contain decoded blocks at positions different from the decoded blocks of the still image frame to be reconstructed, part or all of the black image region is displayed using the decoded blocks of past still image frames. As described above, the number of past still image frames is a small number that does not impair real-time performance, and therefore the visibility of the image can be improved without impairing real-time performance.

[0093] The number of past still image frames can be determined by considering the balance between the required real-timeness and image visibility. To achieve high real-timeness, the number of past still image frames should be reduced. In other words, the highest real-timeness is achieved by using only one past still image frame. On the other hand, to achieve high image visibility, the number of past still image frames should be increased, but this reduces the real-timeness accordingly. Therefore, image visibility can be improved by reducing the real-timeness to a level that still ensures the required real-timeness. In smooth audiovisual communication, a delay of approximately 200 ms is acceptable. Furthermore, in remote control systems, it has been reported that users gradually become accustomed to delays in environments with a delay of 120 to 360 ms, making operation easier, but that operation becomes extremely difficult above 480 ms (Shinichi Hamasaki et al., "Implementation and Evaluation of a Low-Latency Decorator for Delayed Video in Remote Control Systems," Human-Agent Interaction Symposium 2008, 2008). It has also been reported that the acceptable delay in video during remote control of a remote-controlled automated driving system is 800 ms at a speed of 10 km / h (Kazuhiro Mizushima et al., "Evaluation of the Impact of Video Delay on Steering Operations During Remote Control of a Remote-Controlled Autonomous Driving System," Transactions of the Society of Automotive Engineers of Japan, May 2019, Vol. 50, No. 3). In the case of drone operation, as described below, assuming a drone flying at 1.5 m / s and a stopping operation approximately 2 m before the target arrival point, the stopping operation must be performed 2 ÷ 1.5 ≒ 1.33 s beforehand. If the pilot's reaction time is 0.9 s, subtracting this from 1.33 s yields an acceptable delay of approximately 400 ms. Based on the above, the number of past still image frames can be set to a number of frames equivalent to 800 ms or less, preferably 400 ms or less, and more preferably 200 ms or less.

[0094] Meanwhile, the frame start time setting unit 411 sets the next frame start time (S211). A specific method for setting the frame start time will be described below.

[0095] The frame start time setting unit 411 monitors the block number supplied from the header detection unit 403. Then, (1) if the coding block with block number 1, which is the coding block transmitted first in one still image frame, is received within the coding block reception period tbr corresponding to that one still image frame (S301: Yes), the frame start time setting unit 411 sets the next frame start time to the time corresponding to the time when one frame time has elapsed since the reception time of the coding block of that one still image frame which was received first within the coding block reception period (S303). That is, as shown in FIG. 11, (next frame start time nTfr)=(reception time Tr of the first encoded block of the still image frame received within the encoded block reception period)+(one frame time tfr) The next frame start time is set by the above formula. Here, since the coded block of one still image frame that is first received within the coded block reception period is usually the coded block that is first transmitted within the still image frame, the time corresponding to the time at which one frame time has elapsed since the reception time of the coded block of one still image frame that is first received within the coded block reception period may be set as the time at which one frame time has elapsed since the reception time of the coded block of block number 1, which is the coded block that is first transmitted within a frame. The time corresponding to the time at which one frame time has elapsed since the reception time of the coded block of one still image frame that is first received within the coded block reception period is not limited to this, and may be any other appropriate time.

[0096] (2) If the coded block transmitted first in one still image frame is not received within the coded block reception period corresponding to the one still image frame, and if a coded block transmitted in the one still image frame other than the coded block transmitted first in the one still image frame is received within the coded block reception period corresponding to the one still image frame (S305: Yes), the next frame start time is set to the time when the difference between one frame time and the estimated time required to receive all the coded blocks of the one still image frame, from the time when the coded block in the one still image frame was first received, up to the coded block with the earliest transmission order, received within the coded block reception period (S307). That is, as shown in FIG. (next frame start time nTfr) = (reception time Tr of the first received coded block) + (one frame time tfr) - (estimated time tpe to receive the coded block from the first transmitted block to the coded block with the earliest transmission order among the received coded blocks) and sets the next frame start time by the above formula. Here, the earliest coded block in transmission order among the coded blocks of a still image frame received within a coded block reception period is usually the coded block in the still image frame that was received first. Therefore, the estimated time required to receive the coded block from the first coded block transmitted in a still image frame to the coded block that was received first within the coded block reception period may be the estimated time required to receive the coded block from the first coded block transmitted in a still image frame to the coded block in the still image frame that was received first. The estimated time required to receive the coded block from the first coded block transmitted in a still image frame to the coded block that was received first within the coded block reception period is not limited to this and may be any other appropriate time.

[0097] (3) If all the coded blocks of one still image frame are not received within the coded block reception period corresponding to the still image frame (S305: No), the next frame start time is set to the time when one frame time has elapsed from the current frame start time (S309). That is, as shown in FIG. (next frame start time nTfr) = (current frame start time Tfr) + (1 frame time tfr) The next frame start time is set by

[0098] Although the "frame time tfr," "estimated time tae to receive all coded blocks in one frame," "estimated time tpe to receive the coded block from the first transmitted block in one still image frame to the earliest transmitted coded block among the coded blocks of that still image frame received within the coded block reception period," and "margin time tmg" may use specified values, the coded block reception period can be set more appropriately by using values ​​based on actual measurements in the system actually used. An example of using values ​​based on actual measurements will be described below, in which a video captured by the image capturing device 20 is transmitted from a drone equipped with an image transmitting device 10 and an image capturing device 20 via wireless LAN to a pilot's PC equipped with an image receiving device 40 and a display device 50 and displayed.

[0099] First, in a situation where the radio wave conditions are good and all coded blocks can be received (for example, before the start of a flight), actual measurements are taken, and each value is calculated and set based on the measurements.

[0100] The "frame time tfr" is the average frame interval, which can be, for example, the average of the measured time between the reception times of the first transmitted coded blocks of adjacent frames.

[0101] For the "estimated time to receive all coded blocks of one frame, tae," the average time to receive all coded blocks of one frame, which is the average of the actual measured values ​​of the time required for a PC to receive all coded blocks of one frame, can be used.

[0102] For the "estimated time tpe required to receive the coded block from the first transmitted in one still image frame to the earliest transmitted coded block among the coded blocks of that still image frame received within the coded block reception period," the "average block interval tbla" obtained by dividing the above-mentioned "average reception time of all coded blocks in one frame" by the total number of blocks N in one frame, and tbla × (k-1) can be used, where k is the block transmission number of the first received coded block.

[0103] The "margin time tmg" can be calculated by (1 frame time tfr) - [(estimated time to receive all coded blocks of 1 frame tae) + (drawing time tdr)]. The "drawing time tdr" mainly depends on the processing performance of the PC and the performance of the rendering software, but the average drawing time, which is the average of the actual measured values ​​of the time it takes for the PC to draw a frame reconstructed based on the decoded blocks of all coded blocks received for 1 frame, can be used.

[0104] After the flight starts, the values ​​are recalculated and updated based on the actual measurements at a predetermined timing, and the values ​​are reset. The actual measurements used for the recalculation can exclude values ​​when the radio wave conditions are poor.

[0105] Although steps are presented sequentially in the description of the above embodiments, these steps may, in some instances, be performed in parallel and / or in a different order than that described herein. Also, various steps may be combined into fewer steps, divided into additional steps, and / or eliminated based on the desired implementation.

[0106] According to this embodiment, a frame is divided into multiple blocks, and each divided block is coded independently. This reduces the data capacity of packets, thereby reducing the packet error rate. In addition, the inability to reproduce an image due to a packet not being received due to packet loss or delay can be handled in block units smaller than frame units. In other words, since a portion of a frame can be reproduced, the transmitted video can be reproduced without delay on the receiving side, and the reproducibility of the video can be improved.

[0107] Furthermore, according to this embodiment, the situation at the site, including deterioration of the communication situation, can be grasped without delay.

[0108] Furthermore, according to this embodiment, the start time of the next frame is set based on the reception time of the earliest-transmitted encoding block among the encoding blocks of the still image frame received within the encoding block reception period. This allows synchronization to be achieved according to the communication conditions, device performance and processing conditions (e.g., instability in the frame rate of the imaging device, instability in the processing speed or operation of the image transmitting device, instability in the processing speed or operation of the image receiving device), etc., and the encoding block reception period can be set more appropriately, allowing more encoding blocks to be received and improving the reproducibility of images on the receiving side.

[0109] Furthermore, according to this embodiment, the values ​​of "one frame time tfr," "estimated time tae to receive all coded blocks in one frame," "estimated time tpe to receive from the first coded block transmitted in one still image frame to the earliest coded block transmitted within the coded block reception period," and "margin time tmg" are determined based on corresponding actual measurement values, thereby enabling the coded block reception period to be set more appropriately and further improving image reproducibility on the receiving side. Furthermore, these values ​​based on actual measurement values ​​are updated at predetermined timing during reception of a moving image, enabling the coded block reception period to be set even more appropriately and further improving image reproducibility on the receiving side.

[0110] Furthermore, according to this embodiment, the positional order of the divided blocks within a still image frame is randomly rearranged before the encoded blocks are transmitted, so that in the event of packet delays or burst packet losses, etc., the parts that are not reproduced on the receiving side can be prevented from concentrating in a specific part of the image, and the parts that are not reproduced can be dispersed, improving visibility.

[0111] Although the above example shows an example in which the imaging device and the image transmission device are mounted on a drone, the present invention is not limited to this. In particular, the above effects can be more effectively achieved by mounting the imaging device and the image transmission device on an unmanned mobile body such as an unmanned vehicle other than a drone, a mobile body other than an unmanned mobile body, or a remote-controlled device other than a mobile body.

[0112] While the present invention has been described with respect to several embodiments for illustrative purposes, it will be apparent to those skilled in the art that the present invention is not limited thereto and that various changes and modifications in form and detail may be made therein without departing from the scope and spirit of the invention. [Explanation of symbols]

[0113] 1. Video communication system 10 Image transmission device 101 Image division unit 103 Encoding section 105 Encoding block memory unit 107 Block Order Change Unit 109 Packet Generation Unit 111 Transmitter 10a CPU 10b RAM 10c ROM 10d External Memory 10e Input section 10f output section 10g Communication Unit 10h system bus 20 Imaging device 30 Network 40 Image receiving device 401 Receiving unit 403 Header Detector 405 Decoding Unit 407 Decoding block memory unit 409 Frame Reconstruction Unit 411 Frame start time setting section 50 Display device 6 still image frames 7 packets

Claims

1. Dividing each still image frame of a moving image composed of a plurality of still image frames into a plurality of blocks, encoding each of the plurality of blocks independently, and transmitting the encoded blocks by a connectionless communication method; for each of the still image frames, reconstructing the still image frame by arranging, at a corresponding position, a decoded block generated by decoding the coded block of the one still image frame, which is received during a coded block reception period corresponding to the one still image frame; Including, The video communication method, wherein the coded block reception period is a period from the start time of the current frame to the time elapsed that is equal to the estimated time for receiving all coded blocks of one frame plus a margin time.

2. 2. The moving image communication method according to claim 1, wherein reconstructing the still image frames includes, for each of the still image frames, placing a decoded block generated by decoding the coded block of one of the still image frames received during a coded block reception period corresponding to the one of the still image frames at a corresponding position, and reconstructing the still image frame with the remaining portion as an image equivalent to a no-image signal.

3. a coding block for each of the still image frames of a moving image composed of a plurality of still image frames, the coding block being divided into a plurality of blocks, the plurality of blocks being coded independently for each block, and the coding block being transmitted by a connectionless communication method, the coding block of the one still image frame being received in a coding block receiving period corresponding to the one still image frame, and generating a decoded block; arranging the generated decoded blocks at corresponding positions to reconstruct a still image frame; Including, The image receiving method, wherein the coded block reception period is a period from a current frame start time to a time period equal to an estimated time for receiving all coded blocks of one frame plus a margin.

4. 4. The image receiving method according to claim 3, wherein reconstructing the still image frames includes arranging, for each of the still image frames, a decoded block generated by decoding the coded block of one of the still image frames received during a coded block reception period corresponding to the one of the still image frames at a corresponding position, and reconstructing the still image frame with the remaining portion as an image corresponding to a no-image signal.

5. 4. The image receiving method according to claim 3, wherein reconstructing the still image frames includes arranging, for each of the still image frames, a decoded block generated by decoding the coded block of the one still image frame received during a coded block reception period corresponding to the one still image frame, at a corresponding position, and for the remaining portion, for each of past still image frames from the frame immediately preceding the one still image frame to a still image frame that is a predetermined number of frames before the one still image frame, among the decoded blocks generated by decoding the coded block of the received past still image frame, at a position corresponding to the remaining portion, with priority given to the decoded block of a still image frame that is temporally closer to the one still image frame, to reconstruct the still image frames.

6. The next frame start time is (1) if the coded block that is first transmitted in the one still image frame is received within the coded block reception period, the time is set to a time corresponding to the time when one frame time has elapsed since the reception time of the coded block of the one still image frame that is first received within the coded block reception period; (2) In the image receiving method according to any one of claims 3 to 5, when the coding block transmitted first within the still image frame is not received and a coding block transmitted within the still image frame other than the coding block transmitted first within the still image frame is received within the coding block reception period, the image receiving time is set to the time at which the difference between the one frame time and the expected time required to receive the coding block from the coding block transmitted first within the still image frame to the coding block with the earliest transmission order among the coding blocks of the still image frame received within the coding block reception period has elapsed from the time at which the coding block within the still image frame was first received.

7. An image receiving method according to any one of claims 3 to 6, wherein the next frame start time is set to the time when one frame time has elapsed from the current frame start time if all the coding blocks of one of the still image frames are not received within the coding block reception period.

8. 8. The image receiving method according to claim 3, wherein the plurality of coded blocks are transmitted in an order in which the positional order of the blocks within the still image frame is rearranged.

9. The image receiving method according to claim 8 , wherein the plurality of coded blocks are transmitted in an order in which the order of positions of the blocks within the still image frame is randomly rearranged.

10. 10. The image receiving method according to claim 3, wherein the encoding method for the moving image is a Motion JPEG-based method.

11. An image receiving method according to any one of claims 3 to 10, wherein the margin time is the difference between the one frame time and the sum of the estimated time for receiving all coded blocks of the one frame and the drawing time of the one frame when all blocks of the one frame are decoded.

12. An image receiving method according to any one of claims 6 to 11, wherein the estimated time to receive all coded blocks of one frame, the estimated time to receive the coded block from the first coded block transmitted in one still image frame to the coded block with the earliest transmission order among the coded blocks of one still image frame received within the coded block reception period, and / or the drawing time are determined based on corresponding actual measured values ​​and updated at a predetermined timing during reception of the moving image.

13. The image receiving method of claim 12, wherein the estimated time to receive all coded blocks of one frame, determined based on corresponding actual measurement values, the estimated time to receive from the coded block that was first transmitted in one still image frame to the coded block that was received within the coded block reception period and that was transmitted earliest among the coded blocks of one still image frame, and / or the value of the drawing time are updated at predetermined timings during reception of the moving image.

14. Dividing each still image frame of a moving image composed of a plurality of still image frames into a plurality of blocks; encoding the plurality of blocks independently for each block, and transmitting the encoded blocks by a connectionless communication method; An image transmission method including:

15. A program for causing a computer to execute the method according to any one of claims 1 to 14.

16. A computer-readable recording medium on which the program according to claim 15 is recorded.

17. an image dividing unit that divides each still image frame of a moving image composed of a plurality of still image frames into a plurality of blocks; an encoding unit that encodes the plurality of blocks independently for each block; a transmitting unit that transmits coded blocks by a connectionless communication method; a decoding unit that decodes, for each of the still image frames, the coded blocks of the still image frame received in a coded block reception period corresponding to the still image frame to generate decoded blocks; a frame reconstruction unit that reconstructs a still image frame by arranging the generated decoded blocks at corresponding positions; Equipped with In a video communication system, the coded block reception period is a period from a current frame start time to a time period equal to an estimated time for receiving all coded blocks of one frame plus a margin.

18. 18. The moving image communication system according to claim 17, wherein the frame reconstruction unit arranges the generated decoded blocks at corresponding positions and reconstructs a still image frame by treating the remaining portion as an image corresponding to a non-image signal.

19. a decoding unit that decodes the coded block of one still image frame received during a coded block reception period corresponding to one still image frame to generate a decoded block, the coded block being transmitted by a connectionless communication method, wherein each still image frame of a moving image is divided into a plurality of blocks, and the plurality of blocks are coded independently for each still image frame; and a frame reconstruction unit that reconstructs a still image frame by arranging the generated decoded blocks at corresponding positions; Equipped with The image receiving device, wherein the coded block reception period is a period from the start time of the current frame to the time elapsed that is equal to the estimated time for receiving all coded blocks of one frame plus a margin time.

20. 20. The image receiving device according to claim 19, wherein the frame reconstruction unit arranges the generated decoded blocks at corresponding positions and reconstructs a still image frame by treating the remaining portion as an image equivalent to a non-image signal.

21. 20. The image receiving device of claim 19, wherein the frame reconstruction unit, for each of the still image frames, places a decoded block generated by decoding the coded block of the one still image frame received during a coded block reception period corresponding to the one still image frame at a corresponding position, and for the remaining portion, for each of past still image frames from the frame immediately preceding the one still image frame to a still image frame that is a predetermined number of frames before the one still image frame, places the decoded block generated by decoding the coded block of the received past still image frame at a position corresponding to the remaining portion, with priority given to the decoded block of a still image frame that is temporally closer to the one still image frame, to reconstruct the still image frame.

22. Next frame start time, (1) if the coded block that is first transmitted in the one still image frame is received within the coded block reception period, set the time to a time corresponding to the time when one frame time has elapsed since the reception time of the coded block of the one still image frame that is first received within the coded block reception period; (2) If the coded block transmitted first in the one still image frame is not received and a coded block transmitted in the one still image frame other than the coded block transmitted first in the one still image frame is received within the coded block reception period, the time is set to the time at which a difference between the one frame time and the estimated time required to receive the coded block from the coded block transmitted first in the one frame to the coded block with the earliest transmission order among the coded blocks of the one still image frame received within the coded block reception period has elapsed from the time at which the coded block in the one still image frame was first received.

22. The image receiving device according to claim 19, further comprising a frame start time setting unit.

23. An image receiving device as described in any one of claims 19 to 22, wherein the frame start time setting unit sets the next frame start time to a time when one frame time has elapsed from the current frame start time if all of the coding blocks of one of the still image frames are not received within the coding block reception period.

24. 24. The image receiving device according to claim 19, wherein the plurality of coded blocks are transmitted in an order in which the positional order of the blocks within the still image frame is rearranged.

25. The image receiving device according to claim 24, wherein the plurality of coded blocks are transmitted in an order in which the order of positions of the blocks within the still image frame is randomly rearranged.

26. an image dividing unit that divides each still image frame of a moving image composed of a plurality of still image frames into a plurality of blocks; an encoding unit that encodes the plurality of blocks independently for each block; a transmitting unit that transmits coded blocks by a connectionless communication method; An image transmitting device comprising:

27. 27. A remote control device comprising: the image transmitting device according to claim 26; and an imaging device that supplies a moving image to said image transmitting device.

28. 28. The remote control device according to claim 27, wherein the remote control device is a mobile object.

29. The remote control device according to claim 28, wherein the moving body is an unmanned moving body.

Images