Apparatus, method and program for synthesizing video signals

Abstract

The purpose of this disclosure is to reduce the delay from when asynchronous videos are input to when a combined video of the asynchronous videos is output.　This disclosure provides a device that combines asynchronously-input multiple video signals into a video signal to be displayed on a single screen. When the input of input frames of one of the multiple video signals has not been completed yet, the device generates the video signal to be displayed on the single screen by using data of past input frames of the one of the multiple video signals instead of data the input of which has not been completed.

Description

[Technical field]

[0001] This relates to a video synthesis system that synthesizes and outputs a single screen from multiple video input signals. [Background technology]

[0002] In recent years, many video devices have come into use. The images produced by these many video devices use a wide variety of pixel counts (resolutions) and frame rates. Although the video signals of these video devices differ in physical signals, control signals, etc. depending on the standard, one screen is transmitted in a time period equal to one half of the frame rate. For example, if the video signal is 60 frames per second, one screen of video is transmitted in 1 / 60th of a second, or approximately 16.7 milliseconds (hereinafter referred to as 60 fps (Frames per Second)).

[0003] These images can be used in a variety of ways, such as video conferencing, where multiple cameras are displayed on a monitor with fewer cameras than the number of cameras. In such cases, multiple images can be split and displayed on a single screen, or one image can be displayed in a reduced size and fitted onto another image, resulting in screen composition.

[0004] Normally, the timing of a video signal is not synchronized, and the timing of other video signals to be mixed is different, so the signals are temporarily buffered in memory before being mixed. As a result, there is a delay in the output of the mixed image.

[0005] If we were to imagine a video conference in which an ensemble playing music in a remote location were to be held using this type of screen synthesis, the delay involved in this synthesis would make it very difficult to implement. For example, in a piece of music with 120 beats per second (hereafter referred to as 120 BPM (Beat Per Minute)), the duration of one beat is 60 / 120 seconds = 500 milliseconds. If we were to match this with an accuracy of 5%, then the delay between when the image is captured by the camera and when it is displayed would need to be kept to 500 x 0.05 = 25 milliseconds or less.

[0006] In reality, the time it takes to capture an image with a camera and display it requires other delays besides the processing involved in composition, such as the time it takes to process the image on the camera, the time it takes to display it on the monitor, the time it takes to transmit it, etc. As a result, with conventional technology, it is difficult to collaborate in situations where timing is important, such as when people in remote locations play together while viewing each other's videos.

[0007] Therefore, for collaborative work that requires strict low latency, it is necessary to provide a system that can reduce the latency from the time when asynchronous video is input to the time when the composite video is output in a system that composites multiple screens from multiple locations. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] VESA and Industry Standards and Guidelines for Computer Display Monitor Timing (DMT), Version 1.0, Rev. 13, February 8, 2013 Summary of the Invention [Problem to be solved by the invention]

[0009] An object of the present disclosure is to reduce the delay from the video input of asynchronous videos to the output of a composite video. [Means for solving the problem]

[0010] In order to achieve the above objective, the present disclosure inputs multiple asynchronous videos, and performs output synthesis without waiting for completion of input of one screen, and supplements any shortfall in input data with past frame data, synthesizing and outputting the screens from the input data as quickly as possible at a rate faster than the input frame rate.

[0011] Specifically, the apparatus and method of the present disclosure include: A device for synthesizing a plurality of asynchronously input video signals into a video signal to be displayed on one screen, comprising: When input of an input frame is not completed for any of the plurality of video signals, Using data of a previous input frame of the video signal instead of the data that has not yet been input, The video signals to be displayed on the one screen are synthesized.

[0012] Specifically, the program of the present disclosure is a program for causing a computer to realize each functional unit of an apparatus according to the present disclosure, and is a program for causing a computer to execute each step of a method performed by an apparatus according to the present disclosure. Effect of the Invention

[0013] The present disclosure can reduce the delay from the video input of asynchronous videos to the output of a composite video. [Brief description of the drawings]

[0014] [Figure 1] 3 shows an example of screen information included in a video signal. [Diagram 2] 1 illustrates an example system configuration of the present disclosure. [Diagram 3] Here is an example of compositing four input frames into one output frame. [Figure 4] An example of the timing when four input frames are combined into an output frame is shown below. [Diagram 5] An example of the video signal output at time t5 is shown. [Figure 6] An example of the timing when four input frames are combined into an output frame is shown below. [Figure 7] An example of the video signal output at time t5 is shown. [Figure 8] 1 shows an example of the configuration of a video synthesizer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings are assumed to indicate the same components.

[0016] An example of screen information included in a video signal is shown in Fig. 1. The screen information is transmitted by scanning the screen horizontally one scanning line 21 at a time, and then scanning the lower scanning lines 21 in sequence. This scanning includes overhead information / signals such as a display screen 24, a blanking portion 22, and a border portion 23. The blanking portion 22 may also include information other than video information, such as control information and audio information.

[0017] FIG. 2 shows an example of a system configuration according to the present disclosure. In the present disclosure, as an example, four video signals V1 to V4 are input to the image synthesizer 10, which synthesizes and outputs a video signal to be displayed on one screen 20. The image synthesizer 10 outputs the video signal VA from input 1 and the video signal VB from input 2 to the upper part of the screen 20, and outputs the video signal VC from input 3 and the video signal VD from input 4 to the lower part of the screen 20.

[0018] A video signal transmits one screen over a time period equal to one half of the frame rate. For example, if a video signal has 60 frames per second, one screen of the video signal is transmitted over 1 / 60th of a second, or approximately 16.7 milliseconds (hereinafter referred to as 60 fps (Frames per Second)). The information about one screen at each point in time contained in the video signal is called a "frame," the information about one screen of each video signal input to the video synthesizer 10 is called an "input frame," and the information about one synthesized screen output from the video synthesizer 10 is called an "output frame."

[0019] Fig. 3 shows an example in which four images with different timings are input, synthesized into a video signal to be displayed on one screen, and then output. Consider a case in which the image synthesizing device 10 reads all input video screens, synthesizes them, and outputs them. In this case, if the frame time is T_f and the synthesis processing time is T_p, the output frame will be delayed by a maximum of 2T_f+T_p from the input time of the first input frame. For example, when considering a 60 fps video, this means that the synthesized video may contain a delay of more than two frame times, i.e., more than 33.3 milliseconds.

[0020] The present disclosure is a system that inputs multiple asynchronous videos and synthesizes those images, and is characterized in that the output is a synthesized screen at a rate faster than the input frame rate. In this case, in the present disclosure, the missing input data is supplemented with data of past input frames that have already been input. Below, with reference to Figures 4 and 5, an example of reducing a four-input screen to 1 / 4 and synthesizing it into a four-split screen as shown in Figure 2 will be specifically described.

[0021] 4 shows an example of the timing at which input frames input from input 1 to input 4 are synthesized into an output frame. The horizontal axis indicates the passage of time, and the vertical axis indicates the degree of completion of one frame's worth of data from the start of input of input frame data, starting from the horizontal axis for each input, upward, with the end point on the arrow side indicating the completion of data input. Video synthesizer 10 synthesizes input frames of an arbitrary frame rate, and outputs output frames at times t1 to t5.

[0022] For example, input frames of video signals VA1 to VA4 are input from input 1 at times t1 to t5, input frames of video signals VB1 and VB2 are input from input 2 at times t2 and t4, input frames of video signals VC1 and VC2 are input from input 3 at times t2 and t4, and input frames of video signals VD1 and VD2 are input from input 4 at times t1 and t4. For inputs 2 and 3, output frames are output at twice the frame rate of the input frames, and for input 4, output frames are output at three times the frame rate of the input frames.

[0023] 5 shows an example of a video signal being synthesized into an output frame that begins to be output at time t5. At this time, the data of the input frame that has been input to the video synthesizer 10 up until t5 can be output. For the video signal VC input from input 3, only half of the video signal VC2 has been input at time t5. In this case, the usable data that has been input from the video signal VC2 is used for the output frame, and the previous video signal VC1 is used in place of the unusable data from the video signal VC2.

[0024] With regard to the video signal VD input from input 4, at time t5, only 1 / 3 of the video signal VD2 has been input. In this case, the usable data already input from the video signal VD2 is used for the output frame, and the previous video signal VD1 is used in place of the unusable data from the video signal VD2.

[0025] As for the video signal VB input from input 2, at time t5, only half of the video signal VC2 is input. In this case, the video signal VB2 is not used, and only the previous video signal VB1 is used.

[0026] In the present disclosure, synthesis can be performed not only based on the input-completed data at the start of output of the output frame, but also based on the data that has been inputted up until the time of individual data output of the output frame.

[0027] Figure 6 shows the timing of the output frame that starts to be output at time t5 and completes at t6, added by dashed lines, when combining based on the data that has been input up until the individual data output of the output frame in Figure 4. The data corresponding to input 1 and input 2 is at the top of the screen, so it is output from t5 to the midpoint between t5 and t6, and the data corresponding to input 3 and input 4 is at the top of the screen, so it is output from the midpoint between t5 and t6 to t6.

[0028] 7 shows an example of a video signal that starts to be output at time t5 and is synthesized into an output frame that is completely output at t6. At this time, the data of the input frame that has been input to the video synthesizer 10 up until the time of output can be output. As for the video signal VC input from input 3, only half of the video signal VC2 has been input at time t5, but since the input is completed by the time output is completed, in this case, only the data of the video signal VC2 is used for the output frame.

[0029] With regard to the video signal VD input from input 4, at time t5 only 1 / 3 of the video signal VD2 has been input, but input has been completed up to 3 / 5 of the video signal VD2, where the output frame indicated by the dashed line overtakes the input. In this case, the input usable data of the video signal VD2 is used for the output frame, and thereafter the previous video signal VD1 is used in place of the unusable data of the video signal VD2.

[0030] As for the video signal VB input from input 2, at time t5, only half of the video signal VC2 is input. In this case, the video signal VB2 is not used, and only the previous video signal VB1 is used.

[0031] The difference between the outputs of inputs 2 to 4 is due to the difference in settings. For example, if a flag is attached to the video signal VB to prohibit the use of part of the frame data, the video signal VC2 is not used in the output frame output at time t5, as in the case of the video signal VB.

[0032] In the present disclosure, it is not necessary for all outputs to have a rate faster than the input frame rate, and there may be a video signal such as video signal VA from input 1 where the input frame rate and output frame rate are the same.

[0033] 8 shows an example of the configuration of an image synthesizer 10 according to this embodiment. The image synthesizer 10 according to this embodiment includes a detection unit 101, a crossbar switch 102, an up / down converter 103, a buffer 104, and a pixel synthesis unit 105. Although the figure shows four inputs and one output, any number of inputs and outputs may be used.

[0034] Reference numeral 101 denotes a functional unit that detects the input order within a frame time for N inputs. Reference numeral 102 denotes a crossbar switch, which has the function of sorting and outputting the input sequence from 101 in the order of the detection result. Reference numeral 103 denotes an up-down converter that enlarges or reduces the number of pixels to an arbitrary size. 102 and 103 may be connected inversely to the inputs (a, b, c, d, ...). In other words, the inputs a, b, c, d may be scaled by 103, and then 102 may rearrange and output them in the order of the inputs. A buffer 104 can buffer the input of the buffer 103 or 102 and output the input in any order. 105 is a pixel synthesis unit. It reads pixel data from 104 in the order of output from the entire screen of the output, synthesizes it, and outputs it. This synthesis is as described above. 105 may add an arbitrary control signal to the blanking portion of the screen.

[0035] The image synthesizing device 10 of the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0036] (Key Points of This Disclosure) As described above, the present disclosure is a system that inputs multiple asynchronous videos and synthesizes those images, and outputs the images by synthesizing them at a rate faster than the input frame rate. Here, the output synthesis does not wait for the completion of input for one screen, but outputs the input data as soon as possible, and the missing input data is supplemented with past frame data. In this way, the present disclosure can reduce the delay time until the output after synthesis for asynchronous video input signals. Therefore, in a system that synthesizes multiple screens from multiple locations, etc., collaborative work with strict low-latency requirements, and especially strict low-latency requirements for specific inputs, becomes possible. [Industrial Applicability]

[0037] The present disclosure can be applied to the information and communications industry. [Explanation of symbols]

[0038] 10: Image composition device 20: Screen 21: Scan line 22: Blanking section 23: Border part 24:Display screen 101: Detection unit 102: Crossbar switch 103: Up / down converter 104: Buffer 105: Pixel synthesis unit

Claims

1. An apparatus for synthesizing a plurality of asynchronously input video signals into a video signal to be displayed on one screen, comprising: a functional unit that, when input of an input frame is not completed for any one of the plurality of video signals, constructs the input frame by complementing a missing portion of data that is not completed input in the input frame from a past input frame of the video signal, and synthesizes the video signals to be displayed on the one screen; Device.

2. At least one of the plurality of video signals has a different frame rate, the functional unit synthesizes the video signals to be displayed on the one screen at a rate faster than an input frame rate of the plurality of video signals.

2. The apparatus of claim 1.

3. A method for synthesizing a plurality of asynchronously input video signals into a video signal to be displayed on one screen, comprising the steps of: when input of an input frame is not complete for any of the plurality of video signals, compensating for a missing portion of the input frame, the input frame being not complete, from a previous input frame of the video signal, thereby forming the input frame, and synthesizing the video signals to be displayed on the one screen; method.

4. A program for causing a computer to realize each of the functional units of the device according to claim 1 or 2.

Images