Apparatus, method and program for synthesizing video signals

Abstract

The purpose of the present disclosure is to reduce the delay in time from the video input of an asynchronous video to the composite video output thereof.　The present disclosure provides a device and method for compositing a plurality of video signals, which have been input asynchronously, into video signals displayed on one screen. The one screen is formed by a plurality of sub-screens greater in number than the plurality of video signals, the plurality of video signals are arranged on sub-screens among the plurality of sub-screens such as where the output delay of the respective video signals becomes lower, and the plurality of video signals are composited.

Description

[Technical field]

[0001] The present disclosure relates to a video synthesizer that synthesizes a plurality of video input signals into one screen and outputs the screen. [Background technology]

[0002] In recent years, many video devices have come into use. The images on these many video devices use a wide variety of pixel counts (resolutions) and frame rates. Although the video signals of these video devices differ in physical signals, control signals, etc. depending on the standard, one screen is transmitted in a time period equal to one half of the frame rate. For example, if the video signal is 60 frames per second, one screen of video is transmitted in 1 / 60th of a second, or approximately 16.8 milliseconds (hereinafter referred to as 60 fps (Frames per Second)).

[0003] These images can be used in a variety of ways, such as video conferencing, where multiple cameras are displayed on a monitor with fewer cameras than the number of cameras. In such cases, multiple images can be split and displayed on a single screen, or one image can be displayed in a reduced size and fitted onto another image, resulting in screen composition.

[0004] Normally, the timing of a video signal is not synchronized, and the timing of other video signals to be mixed is different, so the signals are temporarily buffered in memory before being mixed. As a result, there is a delay in the output of the mixed image.

[0005] If we were to imagine a video conference in which an ensemble playing music in a remote location were to be held using this type of screen synthesis, the delay involved in this synthesis would greatly impair the feasibility of such a system. For example, in a piece of music with 120 beats per second (hereafter referred to as 120 BPM (Beat Per Minute)), the duration of one beat is 60 / 120 seconds = 500 milliseconds. If we were to need to match this with an accuracy of 5%, then the delay between when the image is captured by the camera and when it is displayed would need to be kept to 500 x 0.05 = 25 milliseconds or less.

[0006] In reality, the time it takes to capture an image with a camera and display it requires other delays besides the processing involved in composition, such as the time it takes to process the image on the camera, the time it takes to display it on the monitor, the time it takes to transmit it, etc. As a result, with conventional technology, it is difficult to collaborate in situations where timing is important, such as when people in remote locations play together while viewing each other's videos.

[0007] Therefore, for collaborative work that requires strict low latency, it is necessary to provide a system that can reduce the latency from the time when asynchronous video is input to the time when the composite video is output in a system that composites multiple screens from multiple locations. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] VESA and Industry Standards and Guidelines for Computer Display Monitor Timing (DMT), Version 1.0, Rev. 13, February 8, 2013 Summary of the Invention [Problem to be solved by the invention]

[0009] An object of the present disclosure is to reduce the delay from the video input of asynchronous videos to the output of a composite video. [Means for solving the problem]

[0010] The apparatus and method of the present disclosure include: A device for synthesizing a plurality of asynchronously input video signals into a video signal to be displayed on one screen, comprising: the one screen is composed of a plurality of sub-screens, the number of which is greater than the number of video signals; The plurality of video signals are arranged on sub-screens among the plurality of sub-screens such that an output delay of each video signal is reduced, and the plurality of video signals are synthesized.

[0011] The device of the present invention can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network. The program of the present disclosure is a program for realizing a computer as each functional unit of the device according to the present disclosure, and is a program for causing a computer to execute each step of the method executed by the device according to the present disclosure. Effect of the Invention

[0012] The present disclosure can reduce the delay from the video input of asynchronous videos to the output of a composite video. [Brief description of the drawings]

[0013] [Figure 1] 3 shows an example of screen information included in a video signal. [Diagram 2] 1 illustrates an example system configuration of the present disclosure. [Diagram 3] Here is an example of compositing four input frames into one output frame. [Figure 4] An example of a sub-screen obtained by dividing one screen is shown. [Diagram 5] 2 shows an example of an input frame and an output frame. [Figure 6] An example of layout on a sub screen is shown below. [Figure 7] 2 shows an example of an input frame and an output frame. [Figure 8] An example of layout on a sub screen is shown below. [Figure 9] 1 shows an example of the configuration of a video synthesizer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings are assumed to indicate the same components.

[0015] 1 shows an example of screen information contained in a video signal. The screen information is transmitted by scanning the screen horizontally one scanning line 21 at a time, and then scanning the lower scanning lines 21 in sequence. This scanning includes overhead information / signals such as a display screen 24, a blanking portion 22, and a border portion 23. The blanking portion 22 may contain information other than video information, such as control information and audio information (for example, see Non-Patent Document 1).

[0016] FIG. 2 shows an example of a system configuration according to the present disclosure. In the present disclosure, as an example, four video signals V1 to V4 are input to the video synthesizer 10, which synthesizes the video signals to be displayed on one screen 20 and outputs the video signals. A video signal transmits one screen over a period of time equal to one frame rate. For example, if the video signal has 60 frames per second, the video signal for one screen is transmitted over 1 / 60 seconds, or approximately 16.7 milliseconds (hereinafter, 60 fps (Frames per Second)). The information of one screen at each time point contained in the video signal is called a "frame," the information of one screen of each video signal input to the video synthesizer 10 is called an "input frame," and the information of one synthesized screen output from the video synthesizer 10 is called an "output frame."

[0017] Fig. 3 shows an example in which four images with different timings are input, synthesized into one screen, and output. Consider a case in which the image synthesizing device 10 reads all input image screens, synthesizes them, and outputs them. In this case, if the frame time is T_f and the synthesis processing time is T_p, the output frame will be delayed by a maximum of 2T_f+T_p from the input time of the first input frame. For example, when considering a 60 fps image, this means that the synthesized image may contain a delay of more than two frame times, i.e., more than 33.3 milliseconds.

[0018] FIG. 4 shows an example of the screen 20 of this embodiment. In this embodiment, an example is shown in which the screen 20 is divided into nine sub-screens of 3×3. In this disclosure, one screen 20 shown in FIG. 2 is composed of five or more sub-screens, which is more than the multiple video signals V1 to V4. Among the divided screens, the horizontal screens arranged on the same scanning line are regarded as one group, and the group is called a "sub-screen group." For example, when the video synthesizing device 10 synthesizes four video signals of inputs 1 to 4, the sub-screens D1-1, D1-2, and D1-3 arranged in the horizontal direction are regarded as a sub-screen group G1, the sub-screens D2-1, D2-2, and D2-3 arranged in the horizontal direction are regarded as a sub-screen group G2, and the sub-screens D3-1, D3-2, and D3-3 arranged in the horizontal direction are regarded as a sub-screen group G3. That is, the data of the output frame is output first from the sub-screen group G1 arranged at the top, followed by the sub-screen groups G2 and G3.

[0019] The present disclosure is a system that inputs multiple asynchronous videos and synthesizes those images, characterized in that the images are arranged from the top to the bottom of the screen 20 in order of earliest input timing to minimize output delay. In particular, the system is characterized in that the number of output split screens is greater than the number of input frames, and there may be areas of the screen 20 that are not used to output input frames.

[0020] 5 and 6 show an example of screen composition according to the present disclosure. In these figures, four input frames are arranged in ascending order of input timing, and the output timing of an output frame obtained by compositing these frames is shown. In this example, there are sub-screen groups G1 to G3 from the top, and each sub-screen group can output up to three screens. Any number of sub-screen groups can be set, and any number of split screens (n>=1) can also be set within each sub-screen group. These can also be dynamically changed according to the number of input frames.

[0021] 5 and 6, inputs 1 and 2 are output to sub-screen group G1 because data input can be completed by time t4 when output of sub-screen group G1 is completed. For example, as shown in Fig. 6, inputs 1 and 2 can be arranged from the left of sub-screen group screen G1. However, this arrangement is arbitrary within the same sub-screen group G1. In this example, sub-screen D1-3 on the rightmost side of sub-screen group G1 is blank and does not display anything.

[0022] Input 3 is output to the sub-screen group G2 because data input can be completed by time t5 when output of the sub-screen group G2 is completed. For example, as shown in Fig. 6, it can be arranged on the leftmost sub-screen D2-1 of the sub-screen group screen G2. However, this arrangement is arbitrary within the same sub-screen group G2. In this example, the central and rightmost sub-screens D2-2 and D2-3 of the sub-screen group G2 are blank.

[0023] Input 4 is output to sub-screen group G3 because data input can be completed by time t6 when output of sub-screen group G3 is completed. For example, as shown in Fig. 6, it can be arranged on the leftmost sub-screen D3-1 of sub-screen group screen G3. However, this arrangement is arbitrary within the same sub-screen group G3. In this example, the central and rightmost sub-screens D3-2 and D3-3 of sub-screen group G3 are blank.

[0024] By arranging the screens in this way, it is possible to output a composite screen with the shortest delay for each input frame. The more sub-screen groups are divided, the finer the alignment can be.

[0025] If it is difficult to arrange screens in the sub-screen group that provides the shortest delay, it is also possible to arrange screens in sub-screen groups that provide shorter delays in stages. For example, as shown in Fig. 7, if the frames of inputs 1 to 4 all have the same input timing, only up to three screens can be arranged in sub-screen group G1. In such a case, as shown in Fig. 8, one can be arranged in the nearby sub-screen group G2 and output. In the figure, only input 4 is arranged in sub-screen group G2. This makes it possible to reduce the average delay.

[0026] When the clocks of the input video and the output video are out of sync, even if the screen frame rate is the same, the timing of each input frame and output frame gradually changes with each screen frame. The method disclosed herein arranges a video signal that is in time for the output timing of a sub-screen group on one of the sub-screens included in the sub-screen group, so that even in response to such changes, the screen arrangement can be changed each time to minimize the delay.

[0027] 9 shows an example of the configuration of an image synthesizer 10 according to this embodiment. The image synthesizer 10 according to this embodiment includes a detection unit 101, a crossbar switch 102, an up / down converter 103, a buffer 104, and a pixel synthesis unit 105. Although the figure shows four inputs and one output, any number of inputs and outputs may be used.

[0028] Reference numeral 101 denotes a functional unit that detects the input order within a frame time for N inputs. Reference numeral 102 denotes a crossbar switch, which has the function of sorting and outputting the input sequence from 101 in the order of the detection result. Reference numeral 103 denotes an up-down converter that enlarges or reduces the number of pixels to an arbitrary size. 102 and 103 may be connected inversely to the inputs (a, b, c, d, ...). In other words, the inputs a, b, c, d may be scaled by 103, and then 102 may rearrange and output them in the order of the inputs. A buffer 104 can buffer the input of the buffer 103 or 102 and output the input in any order. 105 is a pixel synthesis unit. It reads pixel data from 104 in the order of output from the entire screen, synthesizes them, and outputs them. The timing is as described above. 105 may add an arbitrary control signal to the blanking portion of the screen.

[0029] The image synthesizing device 10 of the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0030] (Effects of the present disclosure) The system according to the present disclosure can reduce the delay time until the output after synthesis for asynchronous video input signals. This enables collaborative work with strict low-latency requirements in a system that synthesizes multiple screens from multiple locations, and in particular with strict low-latency requirements for specific inputs.

[0031] (Key Points of This Disclosure) In a system that synthesizes and displays images from multiple locations, low latency synthesis processing is necessary for collaborative work such as ensemble performances, which require strict low latency requirements. The present disclosure is a system that inputs multiple asynchronous images and synthesizes these images, arranging them from the top to the bottom of the screen 20 in order of earliest input timing to minimize output latency. This enables collaborative work with strict low latency requirements in a system that synthesizes multiple screens from multiple locations, etc. [Industrial Applicability]

[0032] The present disclosure can be applied to the information and communications industry. [Explanation of symbols]

[0033] 10: Image synthesis device 20: Screen 21: Scan line 22: Blanking section 23: Border part 24:Display screen 101: Detection unit 102: Crossbar switch 103: Up / down converter 104: Buffer 105: Pixel synthesis unit

Claims

1. An apparatus for synthesizing a plurality of asynchronously input video signals into a video signal to be displayed on one screen, comprising: The one screen is composed of a plurality of sub-screens, the number of which is greater than the number of video signals, a functional unit for outputting a video signal included in the plurality of video signals for each sub-screen group arranged on the same scanning line of the one screen, the functional unit arranges a video signal that is in time for an output timing of the sub-screen group on any one of the sub-screens included in the sub-screen group, and combines the video signals of the sub-screen group. Device.

2. The functional unit arranges the plurality of video signals from an upper part to a lower part of the plurality of sub-screens in order of an input timing of the video signals.

2. The apparatus of claim 1.

3. A method for synthesizing a plurality of asynchronously input video signals into a video signal to be displayed on one screen, comprising the steps of: The one screen is composed of a plurality of sub-screens, the number of which is greater than the number of video signals, a step of outputting a video signal included in the plurality of video signals for each sub-screen group arranged on the same scanning line of the one screen, In the above-mentioned procedure, a video signal that is in time for the output timing of the sub-screen group is arranged on any one of the sub-screens included in the sub-screen group, and the video signals of the sub-screen group are synthesized. method.

4. A program for causing a computer to realize each of the functional units of the device according to claim 1 or 2.

Images