Information processing device, method, and program

Abstract

An information processing device according to one embodiment comprises: a calculation unit that calculates a time lag, which is the difference between a delay amount of video constituting a moving image and a delay amount of audio constituting the moving image, on the basis of a first processing delay which is a time delay due to processing pertaining to the video and processing pertaining to the audio, including packetizing processing of the video and the audio by a transmission device for the same, a transmission delay which is a time delay due to transmission of the video and transmission of the audio from the transmission device to a reception device for the same, and a second processing delay which is a time delay due to processing pertaining to the video and processing pertaining to the audio by the reception device; and a control unit that performs control for waiting to output one among the video and the audio, on the basis of a relationship between the time lag and a permissible range for the time lag, when the time lag calculated by the calculation unit is outside the permissible range.

Description

Information processing device, method and program

[0001] FIELD Embodiments of the present invention relate to an information processing device, method, and program.

[0002] When transmitting the images (hereafter sometimes referred to as video) and audio that make up a moving image, delays always occur. This delay is the time difference that occurs between the sending and receiving sides, and includes two types of delays: a processing delay that occurs when performing the processing required for transmission, and a transmission delay that occurs when data travels along the transmission path.

[0003] Furthermore, when transmitting video, a time lag also occurs between the video and audio. This lag can occur because the video data and audio data of the video are compressed and packetized separately and then transmitted and received separately. Because video data and audio data have different characteristics, they have different processing and transmission speeds. When integrating both image data and audio data into a video on the receiving side, it is necessary to wait for the arrival of the data that arrives later than the data that arrived earlier before processing the integration.

[0004] In other words, video delays are caused by two factors: the time difference between the sending and receiving sides, and the time difference between the video and audio. Therefore, various measures are required to minimize the delays.

[0005] Japanese Patent Application Publication No. 2023-118163

[0006] "Delay Time Adjustment Device: EDA-1000 / 2000 / 2100" <https: / / www.for-a.co.jp / products / eda1000_2000 / >

[0007] Here, a commonly considered strategy for reducing delays that occur in processing and transmission is to strengthen the equipment. For example, delays can be reduced by strengthening the equipment, such as by improving the specifications of the server that processes the video or by widening the bandwidth used for transmission.

[0008] However, there is a problem in that upgrading the facilities is very costly, and there is also a problem in that even if the facilities are upgraded on a large scale, it is not possible to completely eliminate delays in processing and transmission.

[0009] The present invention has been made in light of the above circumstances, and an object of the present invention is to provide an information processing device, method, and program that are capable of appropriately processing moving images.

[0010] An information processing device according to one embodiment of the present invention comprises a calculation unit that calculates a delay time, which is the difference between the delay amount of the video and the delay amount of the audio, based on a first processing delay, which is the delay time caused by video processing and audio processing, including the processing of packetizing the video and audio by a transmitting device of the video and audio that constitute a moving image, a transmission delay, which is the delay time caused by transmitting the video and audio from the transmitting device to a receiving device of the video and audio, and a second processing delay, which is the delay time caused by the processing of the video and audio by the receiving device; and a control unit that performs control to wait for output of one of the video and the audio based on the relationship between the delay time and the allowable range when the delay time calculated by the calculation unit is outside an allowable range.

[0011] An information processing method according to one embodiment of the present invention is a method performed by an information processing device, and includes: calculating, by a calculation unit of the information processing device, a delay time which is the difference between the delay amount of the video and the delay amount of the audio based on a first processing delay which is the delay time caused by video processing and audio processing, including the processing of packetizing the video and audio by a transmitting device of the video and audio that constitute a moving image; a transmission delay which is the delay time caused by transmitting the video and audio from the transmitting device to a receiving device of the video and audio; and a second processing delay which is the delay time caused by the processing of the video and audio by the receiving device; and controlling, by a control unit of the information processing device, to wait for output of one of the video and the audio based on the relationship between the delay time and the allowable range when the delay time calculated by the calculation unit is outside an allowable range.

[0012] According to the present invention, moving images can be processed appropriately.

[0013] Fig. 1 is a diagram showing an application example of a video and audio transmission system according to an embodiment of the present invention. Fig. 2 is a flowchart showing an example of a procedure of processing operations by the video and audio transmission system. Fig. 3 is a block diagram showing an example of the hardware configuration of a video and audio transmission device according to an embodiment of the present invention.

[0014] An embodiment of the present invention will now be described. FIG. 1 is a diagram showing an application example of a video and audio transmission system according to an embodiment of the present invention. As shown in FIG. 1, the video and audio transmission system according to an embodiment of the present invention includes a video and audio transmitting device 100 and a video and audio receiving device 200, which are information processing devices. The video and audio transmitting device 100 includes an input unit 11 and a transmission unit 12. The video and audio receiving device 200 includes a receiving unit 21, a delay amount / deviation calculation unit 22, a delay control unit 23, and an output unit 24. Details of the processing of each unit will be described later. Furthermore, the calculations related to the processing delay and transmission delay described below may be performed by a separate device.

[0015] In this embodiment, video and audio delays are dynamically controlled on a packet-by-packet basis, making it possible to present moving images within a range that does not cause the user to feel uncomfortable.

[0016] 2 is a flowchart showing an example of a procedure for processing operations by the video and audio transmission system. (Step 1-1) The input unit 11 of the video and audio transmitting device 100 assigns a timestamp Ta, which is the time at which video A data (sometimes referred to as video data A) acquired by an external camera c is acquired, to the data. The input unit 11 also assigns a timestamp Tb, which is the time at which audio B data (sometimes referred to as audio data B) acquired by an external microphone mi is acquired. At the time of assignment, it is assumed that there is no discrepancy between the timestamps Ta and Tb.

[0017] (Step 1-2) The transmitting unit 12 compresses and packetizes the video data A and the audio data B, respectively, and transmits them to the video and audio receiving device 200. At this point, a processing delay Dp(a1) occurs due to the compression and packetization process for the video data A, and a processing delay Dp(b1) occurs due to the compression and packetization process for the audio data B. The transmitting unit 12 also transmits information indicating these processing delays to the video and audio receiving device 200.

[0018] (Step 2-1) The receiving unit 21 of the video and audio receiving device 200 receives packets transmitted from the video and audio transmitting device 100, and extracts video data A and audio data B from the received packets. At this point, a processing delay Dp(a2) occurs for the video data A due to the process of extracting the video data A from the packets, and a processing delay Dp(b2) occurs for the audio data B due to the process of extracting the audio data B from the packets. Furthermore, a transmission delay Dt(a1) occurs in the transmission of the video data A from the video and audio transmitting device 100 to the video and audio receiving device 200, and a transmission delay Dt(b1) occurs in the transmission of the audio data B from the video and audio transmitting device 100 to the video and audio receiving device 200.

[0019] (Step 2-2) The delay / shift calculation unit 22 measures the current time Tc, and calculates the delay amount Da, which is the delay time of the video, based on the timestamp Ta assigned to the extracted video data A and the current time Tc, as shown in the following (1): Da = Tc - Ta ... (1) As another example of calculating the video delay amount Da, the delay / shift calculation unit 22 calculates the video delay amount Da based on the processing delays Dp(a1) and Dp(a2) of the extracted video data A and the transmission delay Dt(a1) of the video data A, as shown in the following (2): Da = Dp(a1) + Dp(a2) + Dt(a1) ... (2)

[0020] The delay / shift calculation unit 22 also calculates the delay Db, which is the delay time of the audio, based on the timestamp Tb and the current time Tc attached to the extracted audio data B, as shown in the following (3): Db=Tc−Tb (3)

[0021] As another example of calculating the audio delay Db, the delay / deviation calculation unit 22 calculates the audio delay Db based on the processing delays Dp(b1) and Dp(b2) of the extracted audio data B and the transmission delay Dt(b1) of the audio data B, as shown in the following (4): Db = Dp(b1) + Dp(b2) + Dt(b1) (4)

[0022] At this time, the delay / shift calculation unit 22 also calculates the delay shift D, which is the synchronization shift between the video and audio in the moving image, based on the data of the video delay Da and the audio delay Db, as shown in the following (5). The calculation result is passed to the delay control unit 23 together with the extracted video data A and audio data B. D=Da-Db (5)

[0023] (Step 2-3) The delay control unit 23 compares the delay deviation amount D calculated in Step 2-2 with a preset "tolerance range (sometimes referred to as the tolerance range) Dth for video and audio synchronization deviation," and performs processing of a pattern according to the following (a), (b), or (c) depending on the result of this comparison. The tolerance range for synchronization deviation is a range set as a range of video and audio delay deviation amount D that the user will not feel uncomfortable within. The range Dth is assumed to be the range from the minimum value "Dth(min)," which is the lower limit of the tolerance range of delay deviation amount D, to the maximum value "Dth(max)," which is the upper limit of the tolerance range of delay deviation amount D.

[0024] (a) When the following (6) is true, that is, when the delay deviation amount D is within the allowable range (YES in Step 2-3a1), the delay control unit 23 passes the extracted video data A and audio data B to the output unit 24 as is (Step 2-3a2). Dth(min)≦D≦Dth(max) (6)

[0025] (b) When the following (7) is true, i.e., when the delay deviation amount D is outside the allowable range of the delay deviation amount (NO in Step 2-3a1) and when the audio data B acquired by the video and audio transmitting device 100 precedes the video data A acquired by the same device (YES in Step 2-3b1), the delay control unit 23 waits for the time indicated in the following (8) before outputting the extracted audio data B, and when this time has elapsed, passes the extracted video data A and audio data B to the output unit 24 (Step 2-3b2). That is, when the delay deviation amount D is outside the allowable range and the video data A acquired by the video and audio transmitting device 100 lags behind the audio data B acquired at the same timing, control is performed to wait for the output of the audio data B for a time that is the difference between the delay deviation amount D and a value closest to the delay deviation amount D between the upper and lower limits of the allowable range, in this case the upper limit value, and when the delay deviation amount D falls within the allowable range as a result of this waiting, the video data A and audio data B are output.

[0026] Dth(max)<D (7) D-Dth(max): (delay deviation amount-upper limit of delay deviation amount tolerance) (8)

[0027] For example, when the delay deviation amount D is "+100 [ms]" and the upper limit of the tolerance is "+90 [ms]", the output of the audio data B is delayed for 10 [ms], which is the time shown in the following (9): +100 [ms] - (+90 [ms]) = 10 [ms] ... (9)

[0028] (c) When the following (10) is true, i.e., when the delay deviation amount D is outside the allowable range of the delay deviation amount (NO in Step 2-3a1) and when the audio data B, which has also been acquired by the video and audio transmitting device 100, arrives after the video data A, which has also been acquired by the video and audio transmitting device 100 (NO in Step 2-3b1), the delay control unit 23 waits for the time indicated in the following (11) before outputting the extracted video data A, and when this time has elapsed, passes the extracted video data A and audio data B to the output unit 24 (Step 2-3b3). That is, when the delay deviation amount D is outside the allowable range and the audio data B, which has been acquired by the video and audio transmitting device 100, is delayed relative to the video data A, which has been acquired at the same timing, control is performed to wait for the output of the video data A for a time that is the difference between the delay deviation amount D and a value closest to the delay deviation amount D, either the upper or lower limit of the allowable range (here, the lower limit), and when the delay deviation amount D falls within the allowable range as a result of this waiting, the video data A and audio data B are output.

[0029] D<Dth(min) (10) Dth(min)-D: (lower limit of delay deviation tolerance-delay deviation) (11)

[0030] For example, if the delay deviation is "-200 [ms]" and the lower limit of the tolerance is "-185 [ms]", the output of video data A is delayed for 15 [ms], which is the time shown in the following (12): (-185 [ms]) - (-200 [ms]) = 15 [ms] (12)

[0031] In the above (b), the output of the received video data A is not put on hold, and in the above (c), the output of the received audio data B is not put on hold.

[0032] (Step 2-4) The output unit 24 outputs the video data A passed from the delay control unit 23 to the external monitor mo, causing the video to be played on this monitor mo, and outputs the audio data B passed from the delay control unit 23 to the external speaker s, causing the audio to be played on this speaker s.

[0033] The above-mentioned allowable range may be changed depending on the content. In this embodiment, processing is performed for each packet of video data or each packet of audio data, so dynamic delay control can be realized, and output can always be in an optimal state compared to delay control that is fixed by, for example, an offset.

[0034] In conventional methods, delay control is performed based on the difference between processing delay and transmission delay. As a result, even if some data arrives early, the system waits for the latest data to arrive and only outputs it once all data is ready, which impairs the real-time nature of data output.

[0035] In contrast, in this embodiment, an acceptable limit is set in advance as a range within which the user will not feel uncomfortable, and data delay control is performed based on this acceptable limit before data output, thereby improving the real-time nature of data output.

[0036] 3 is a block diagram showing an example of the hardware configuration of a video and audio transmitting device according to an embodiment of the present invention. In the example shown in FIG. 3, the video and audio transmitting device 100 according to the above embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115. The same applies to the video and audio receiving device 200 shown in FIG. 1.

[0037] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0038] An input device 500 and an output device 600, which are attached to the video and audio transmitting apparatus 100 and used by a user or the like, are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 500, such as a keyboard, a touch panel, a touchpad, or a mouse, and can output output data to an output device 600, which includes a display device using a liquid crystal or an organic electroluminescence (EL) display, for display. The input device 500 and the output device 600 may be devices built into the video and audio transmitting apparatus 100, or may be input devices and output devices of other information terminals that can communicate with the video and audio transmitting apparatus 100 via a network.

[0039] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0040] The data memory 112 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during various processing steps.

[0041] The video / audio transmitting device 100 according to one embodiment of the present invention can be configured as a data processing device having the units shown in FIG. 1 as software-based processing function units.

[0042] The information storage unit used as a work memory or the like by each unit of the video and audio transmitting device 100 can be configured by using the data memory 112 shown in Fig. 3. However, these configured storage areas are not essential components within the video and audio transmitting device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0043] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0044] The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables or data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0045] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0046] REFERENCE SIGNS LIST 100: Audio and video transmitting device 11: Input section 12: Transmitting section 200: Audio and video receiving device 21: Receiving section 22: Delay amount / shift calculating section 23: Delay control section 24: Output section

Claims

1. An information processing device comprising: a calculation unit that calculates a delay time that is the difference between the delay amount of the video and the delay amount of the audio based on a first processing delay, which is the delay time caused by video processing and audio processing, including the processing of packetizing the video and audio by a transmitting device of the video and audio that constitute a moving image; a transmission delay, which is the delay time caused by transmitting the video and audio from the transmitting device to a receiving device of the video and audio, and a second processing delay, which is the delay time caused by the processing of the video and audio by the receiving device; and a control unit that controls to wait for output of one of the video and the audio based on the relationship between the delay time and the allowable range when the delay time calculated by the calculation unit is outside an allowable range.

2. The information processing device of claim 1, wherein the control unit, when the delay time calculated by the calculation unit is outside the allowable range and the video is delayed relative to the audio, controls to wait for the output of the audio for a time period equal to the difference between the delay time and the value closest to the delay time among the upper and lower limits of the allowable range, and when the delay time calculated by the calculation unit is outside the allowable range and the audio is delayed relative to the video, controls to wait for the output of the video for a time period equal to the difference between the delay time and the value closest to the delay time among the upper and lower limits of the allowable range.

3. A method performed by an information processing device, comprising: calculating, by a calculation unit of the information processing device, a delay time that is the difference between the delay amount of the video and the delay amount of the audio, based on a first processing delay that is the delay time caused by video processing and audio processing, including the processing of packetizing the video and audio by a transmitting device of the video and audio that constitute a moving image, a transmission delay that is the delay time caused by transmitting the video and audio from the transmitting device to a receiving device of the video and audio, and a second processing delay that is the delay time caused by the processing of the video and audio by the receiving device; and performing control by a control unit of the information processing device to wait for output of one of the video and the audio, based on the relationship between the delay time and the allowable range, when the delay time calculated by the calculation unit is outside an allowable range.

4. An information processing program that causes a processor to function as each part of the information processing device according to claim 1 or 2.

Images