Video and audio synthesis device, method and program

Abstract

The purpose of the present disclosure is to shorten a delay time until sound signals are output.　According to the present disclosure, video signals and sound signals, which form a plurality of videos with voices, are acquired, the video signals and the sound signals are separated from the plurality of video signals with voices, a synthesis process for the video signals is performed separately from a process for the sound signals, and the sound signals are output without waiting for the completion of the synthesis of the video signals.

Description

[Technical field]

[0001] This relates to a video and audio synthesis system that synthesizes and outputs a single image and audio from multiple video and audio input signals. [Background technology]

[0002] In recent years, many video devices have come into use. The images produced by these devices use a wide variety of pixel counts (resolutions) and frame rates. Although the video signals of these video devices differ in terms of physical signals, control signals, etc., depending on the standard, one screen is transmitted in a time period equal to one frame rate.

[0003] These images can be used in a variety of ways, such as video conferencing, where multiple cameras are displayed on a monitor with fewer monitors than the number of cameras. In such cases, multiple video / audio signals can be split and displayed on a single screen, or a reduced version of a video screen can be displayed in one video screen and fitted into the other, and the corresponding audio signals can be synthesized at the same time as the images are synthesized.

[0004] Normally, the timing of a video signal is not synchronized, and the timing of other video signals to be mixed is different, so the signals are temporarily buffered in memory before being mixed. As a result, delays occur in the output of the mixed screen and in the audio signal.

[0005] If we were to imagine a video conference in which an ensemble played in a remote location were to be held using this type of image / audio synthesis, the delay involved in this synthesis would greatly impair the feasibility of such a system. For example, in a piece of music with 120 beats per minute (hereafter referred to as 120 BPM (Beat Per Minute)), the duration of one beat is 60 / 120 seconds = 500 milliseconds. If we were to match this with an accuracy of 5%, then the delay between when the image was captured by the camera and when it was displayed would need to be kept to 500 x 0.05 = 25 milliseconds or less.

[0006] In reality, the process from capturing an image with a camera to displaying it must include other delays, such as the image processing time in the camera, the display time on the monitor, and the time involved in transmission, in addition to the processing involved in synthesis. The accompanying audio signal also includes a delay. As a result, with conventional technology, it is difficult to collaborate in applications where timing is important, such as playing an ensemble while watching each other's videos in remote locations. However, if only one of the video or audio signals has low delay, collaborative work in applications where timing is important may be possible.

[0007] Therefore, for collaborative work that requires strict low latency, we provide a system that synthesizes multiple screens / multiple audio signals from multiple locations, etc., and reduces the latency from the video input / audio signal input of asynchronous video to the audio output in particular. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] VESA and Industry Standards and Guidelines for Computer Display Monitor Timing (DMT), Version 1.0, Rev. 13, February 8, 2013 Summary of the Invention [Problem to be solved by the invention]

[0009] An object of the present disclosure is to reduce the delay time until an audio signal is output. [Means for solving the problem]

[0010] The video and audio synthesizer of the present disclosure comprises: Acquiring video signals and audio signals constituting a plurality of audio-accompanied videos; Separating a video signal and an audio signal from the plurality of audio-accompanying video signals; The synthesis of the video signal is performed separately from the processing of the audio signal, To output an audio signal without waiting for completion of a synthesis process of a video signal.

[0011] The video and audio synthesis method of the present disclosure includes: The video and audio synthesis device Acquiring video signals and audio signals constituting a plurality of audio-accompanied videos; Separating a video signal and an audio signal from the plurality of audio-accompanying video signals; The synthesis of the video signal is performed separately from the processing of the audio signal, To output an audio signal without waiting for completion of a synthesis process of a video signal.

[0012] The program of the present disclosure is a program for causing a computer to realize each functional unit of the video and audio synthesis device of the present disclosure, and is a program for causing a computer to execute each step of the method performed by the video and audio synthesis device of the present disclosure. Effect of the Invention

[0013] The present disclosure can reduce the delay time until an audio signal is output. [Brief description of the drawings]

[0014] [Figure 1] 3 shows an example of screen information included in a video signal. [Diagram 2] An example of screen composition is shown below. [Diagram 3] 1 illustrates an example of a video and audio synthesis method related to the present disclosure. [Figure 4] 1 illustrates an example of a video and audio synthesis method according to the present disclosure. [Diagram 5] 1 illustrates an example of a video and audio synthesis method according to the present disclosure. [Figure 6] 1 shows an example of the configuration of a video and audio synthesis device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings are assumed to indicate the same components.

[0016] Fig. 1 shows an example of screen information contained in a video signal. The screen information is transmitted by scanning the screen horizontally, one scanning line 21 at a time, and then scanning the lower scanning lines 21 in sequence. This scanning includes scanning of the display screen 24, blanking portion 22, and overhead information / signals such as border portion 23. The blanking portion 22 may contain information other than video information, such as control information and audio information. (See, for example, Non-Patent Document 1, Chapter 3.)

[0017] FIG. 2 shows an example of combining video signals. In the present disclosure, as an example, four video signals are input to a video and audio synthesizer, which combines them into one video signal and outputs it. In a video signal, one screen is transmitted using a time period equal to one frame rate. For example, if the video signal has 60 frames per second, one screen of the video signal is transmitted in 1 / 60 seconds, or approximately 16.7 milliseconds (hereinafter, 60 fps (Frames per Second)). One screen of information contained in a video signal at each time point is called a "video frame", one screen of information of each video signal input to the video and audio synthesizer is called an "input frame", and one screen of information combined and output from the video and audio synthesizer is called an "output frame". In the present disclosure, the "input frame" and "output frame" may include an audio signal.

[0018] For example, consider a case where a video and audio synthesizer inputs video and audio signals with four different timings, synthesizes them into a single output frame, and outputs the input frame by reading all of the input frames, synthesizing them, and outputting them, as shown in Fig. 3. In this case, if the frame time is T_f and the synthesis processing time is T_p, the output frame into which the video and audio signals are synthesized will be delayed by a maximum of 2T_f+T_p from the time of input of the first input frame, input 1. For example, considering a 60 fps video, this means that the synthesized video may contain a delay of more than two frame times, i.e., more than 33.3 milliseconds.

[0019] Here, in this disclosure, input video x,k is the k-th video frame of input x, and audio x,k is the audio signal included in the k-th video frame of input x. Also, output signal video k,k,k,k is the k-th output frame synthesized from video 1,k, video 2,k, video 3,k, and video 4,k.

[0020] The present disclosure relates to a system that inputs video signals and audio signals constituting multiple asynchronous audio-accompanied videos and synthesizes those video signals and audio signals, and is characterized by separating the audio signal from the input signals, synthesizing the video signal separately from the audio signal, and synthesizing and superimposing the audio signal on the video signal without waiting for the synthesis of the original video signal.

[0021] An example of synthesis according to the present disclosure is shown in Figure 4. As shown in the figure, audio 1,k, audio 2,k, audio 3,k, and audio 4,k should be synthesized as audio into video k,k,k,k, which is an image synthesized from the original video. However, the present disclosure separates only the audio signal and synthesizes and superimposes it into the previous video frame as much as possible.

[0022] For example, as shown in Fig. 4, when each input frame of the audio signal can be output without delay, the audio synthesizer outputs the output frame of the audio signal without delay. In this case, when the input time of the input frame of the video 1,k and the audio 1,k coincides with the output time of the output frame of the video k-2, k-2, k-2, k-2, the audio synthesizer superimposes the audio 1,k on the video k-2, k-2, k-2, k-2. When the input frames of the video 2,k and the audio 2,k, the video 3,k and the audio 3,k, the video 4,k and the audio 4,k are input while the output frames of the video k-2, k-2, k-2, k-2, k-2 and the video k-1, k-1, k-1, k-1, k-1 are output, the audio synthesizer superimposes the audio 2,k, the audio 3,k, and the audio 4,k separately on the video k-2, k-2, k-2, k-2 and the video k-1, k-1, k-1, k-1. This allows the delay of the audio signal to be significantly reduced. Note that, although the example of Fig. 4 shows an example in which each input frame of the audio signal can be output without delay, the present disclosure is not limited to this, and the audio signal can be output at any timing prior to the output timing of the video signal input at the same time.

[0023] Also, for example, the output audio signal at time t1 becomes data of audio 1,k, audio 2,k, video 3,k and audio 3,k, video 4,k and audio 4,k, and each of these can be superimposed on video k-2,k-2,k-2,k-2 as different audio channels, or a part or all of the audio signals can be synthesized at any intensity and then superimposed on video k-2,k-2,k-2,k-2 as the same audio channel. By separating the channels, it becomes possible for the audiovisual device that receives the output signal and presents the video and audio to control the audio intensity for each input signal. Also, by synthesizing the channels, it is possible to avoid restrictions on the number of channels of audio signals multiplexed onto a video signal.

[0024] In addition, in the example of Fig. 4, an example is shown in which each input frame of the audio signal is output without being synthesized, but the present disclosure is not limited to this, and synthesis processing may also be performed on the audio signal. For example, as shown in Fig. 5, audio 1,k, audio 2,k, audio 3,k, and audio 4,k may be synthesized so as to match the timing of the output frame of audio 4,k that arrives last. This allows the video signal and the audio signal to be treated as one frame each.

[0025] 6 shows an example of the configuration of a video and audio synthesis device according to this embodiment. A video and audio synthesis device 10 according to this embodiment includes a separation unit 101, an up / down converter 102, a buffer 103, and a superimposition unit 104. Although the diagram shows four inputs and one output, any number of inputs and outputs may be used.

[0026] Reference numeral 101 denotes a functional unit capable of separating an input signal into a video signal and an audio signal. The separated signals are input to 104 described below, or may be output separately to the outside. An up-down converter 102 enlarges or reduces the number of pixels to an arbitrary size. For example, the number of pixels of the input 1 is enlarged or reduced so as to match the size of the screen shown in FIG. A buffer 103 stores each input frame. The input of 102 can be buffered and output in any order. Reference numeral 104 denotes a pixel synthesis, audio synthesis, and superimposition unit, which reads out the pixels of each input from 103, synthesizes the entire screen, and further synthesizes an audio signal, and then instantly superimposes the video signal as an audio signal regardless of the original video on which the audio signal was superimposed, and outputs the result. The superimposing unit 104 may add an arbitrary control signal to the blanking portion of the screen. If any of the audio signals separated by the separating unit 101 does not require synthesis processing, the video and audio synthesizing device 10 may output the audio signal without going through the superimposing unit 104.

[0027] The video and audio synthesis device of the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0028] Although the above embodiment shows an example of four inputs and one screen divided into four, the present disclosure is not limited to this and can be applied to any input.

[0029] (Effects of the present disclosure) By separating the asynchronous video / audio input signals, synthesizing the audio signal without waiting for completion of the synthesis process of the video signal before separation, and superimposing the audio signal on the video signal that has already been synthesized, the delay time until the audio signal is output after synthesis can be shortened. This will enable collaborative work that requires strict low latency in a system that synthesizes multiple screens at multiple locations. [Industrial Applicability]

[0030] The present disclosure can be applied to the information and communications industry, which distributes video content and game content, as well as the film, advertising, and game industries involved in video production. [Explanation of symbols]

[0031] 10: Audio-visual synthesis device 21: Scan line 22: Blanking section 23: Border part 24:Display screen 101: Separation part 102: Up / down converter 103: Buffer 104: Superimposition section

Claims

1. Acquiring video signals and audio signals constituting a plurality of audio-accompanied videos; Separating a video signal and an audio signal from the plurality of audio-accompanying video signals; The synthesis of the video signal is performed separately from the processing of the audio signal, Outputting an audio signal without waiting for completion of the synthesis process of the video signal; synthesizing the audio signals included in the plurality of audio-accompanied videos in accordance with the last audio signal of the plurality of audio-accompanied videos in one frame; Audio-visual synthesis device.

2. Starting synthesis of an audio signal corresponding to an acquired video signal with a video signal acquired prior to the acquired video signal without waiting for the start of synthesis of a video frame corresponding to the acquired video signal.

2. The video and audio synthesis apparatus according to claim 1.

3. The video and audio synthesis device Acquiring video signals and audio signals constituting a plurality of audio-accompanied videos; Separating a video signal and an audio signal from the plurality of audio-accompanying video signals; The synthesis of the video signal is performed separately from the processing of the audio signal, Outputting an audio signal without waiting for completion of the synthesis process of the video signal; synthesizing the audio signals included in the plurality of audio-accompanied videos in accordance with the last audio signal of the plurality of audio-accompanied videos in one frame; A method for synthesizing audio and video.

4. Starting synthesis of an audio signal corresponding to an acquired video signal with a video signal acquired prior to the acquired video signal without waiting for the start of synthesis of a video frame corresponding to the acquired video signal. The video / audio synthesis method according to claim 3 .

5. 3. A program for causing a computer to realize each of the functional units of the video / audio synthesizing apparatus according to claim 1.

Images