Apparatus, method and program for synthesizing video signals

Abstract

The purpose of the present disclosure is to shorten the delay time until output of specific video input when a plurality of non-synchronous videos are input and the images thereof are combined.　The present invention is a device for combining, into a video signal to be displayed on one screen, a plurality of video signals input non-synchronously, wherein the plurality of video signals are combined so as to shorten the delay of a video signal of pivot input among the plurality of video signals.

Description

[Technical field]

[0001] This relates to a video synthesis technology that synthesizes and outputs a single screen from multiple video input signals. [Background technology]

[0002] In recent years, many video devices have come into use. The images on these many video devices use a wide variety of pixel counts (resolutions) and frame rates. Although the video signals of these video devices differ in physical signals, control signals, etc. depending on the standard, one screen is transmitted in a time period equal to one half of the frame rate. For example, if the video signal is 60 frames per second, one screen of video is transmitted in 1 / 60th of a second, or approximately 16.8 milliseconds (hereinafter referred to as 60 fps (Frames per Second)).

[0003] These images can be used in a variety of ways, such as video conferencing, where multiple cameras are displayed on a monitor with fewer cameras than the number of cameras. In such cases, multiple images can be split and displayed on a single screen, or one image can be displayed in a reduced size and fitted onto another image, resulting in screen composition.

[0004] Normally, the timing of a video signal is not synchronized, and the timing of other video signals to be mixed is different, so the signals are temporarily buffered in memory before being mixed. As a result, there is a delay in the output of the mixed image.

[0005] If we were to imagine a video conference in which an ensemble playing music in a remote location were to be held using this type of screen synthesis, the delay involved in this synthesis would make it very difficult to implement. For example, in a piece of music with 120 beats per second (hereafter referred to as 120 BPM (Beat Per Minute)), the duration of one beat is 60 / 120 seconds = 500 milliseconds. If we were to match this with an accuracy of 5%, then the delay between when the image is captured by the camera and when it is displayed would need to be kept to 500 x 0.05 = 25 milliseconds or less.

[0006] In reality, the time it takes to capture an image with a camera and display it requires other delays besides the processing involved in composition, such as the time it takes to process the image on the camera, the time it takes to display it on the monitor, the time it takes to transmit it, etc. As a result, with conventional technology, it is difficult to collaborate in situations where timing is important, such as when people in remote locations play together while viewing each other's videos.

[0007] Furthermore, in an ensemble where there is a conductor or other person instructing the timing, tempo, and articulation, it is particularly important that the video of that person has low latency.

[0008] Therefore, for collaborative work that requires strict low latency, it is necessary to provide a system that can reduce the latency from the video input of asynchronous video to the output of the composite video in a system that composites multiple screens from multiple locations, etc. In particular, it is necessary to provide a system that can reduce the latency from the time of a specific video input to the time of the composite video output. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] VESA and Industry Standards and Guidelines for Computer Display Monitor Timing (DMT), Version 1.0, Rev. 13, February 8, 2013 Summary of the Invention [Problem to be solved by the invention]

[0010] An object of the present disclosure is to reduce the delay time until a specific video input is output when multiple asynchronous videos are input and the images are synthesized. [Means for solving the problem]

[0011] The apparatus and method of the present disclosure include: A device for synthesizing a plurality of asynchronously input video signals into a video signal to be displayed on one screen, comprising: The plurality of video signals are synthesized so that the delay of the video signal of the set pivot input among the plurality of video signals is shortened.

[0012] The program of the present disclosure is a program for causing a computer to realize each functional unit of an apparatus of the present disclosure, and is a program for causing a computer to execute each step of a method performed by an apparatus of the present disclosure. Effect of the Invention

[0013] According to the present disclosure, when a plurality of asynchronous videos are input and these images are synthesized, it is possible to reduce the delay time until a specific video input is output. [Brief description of the drawings]

[0014] [Figure 1] 3 shows an example of screen information included in a video signal. [Diagram 2] 1 illustrates an example system configuration of the present disclosure. [Diagram 3] Here is an example of compositing four input frames into one output frame. [Figure 4] 1 shows an example of a sub-screen group. [Diagram 5] 1 illustrates an example of a video synthesis method according to the present disclosure. [Figure 6] 1 illustrates an example of a video synthesis method according to the present disclosure. [Figure 7] 1 shows an example of the configuration of a video synthesizer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings are assumed to indicate the same components.

[0016] An example of screen information included in a video signal is shown in Fig. 1. The screen information is transmitted by scanning the screen horizontally one scanning line 21 at a time, and then scanning the lower scanning lines 21 in sequence. This scanning includes overhead information / signals such as a display screen 24, a blanking portion 22, and a border portion 23. The blanking portion 22 may also include information other than video information, such as control information and audio information.

[0017] FIG. 2 shows an example of a system configuration according to the present disclosure. In the present disclosure, as an example, four video signals V1 to V4 are input to the video synthesizer 10, which synthesizes the video signals to be displayed on one screen 20 and outputs the video signals. A video signal transmits one screen over a period of time equal to one frame rate. For example, if the video signal has 60 frames per second, the video signal for one screen is transmitted over 1 / 60 seconds, or approximately 16.7 milliseconds (hereinafter, 60 fps (Frames per Second)). The information of one screen at each time point contained in the video signal is called a "frame," the information of one screen of each video signal input to the video synthesizer 10 is called an "input frame," and the information of one synthesized screen output from the video synthesizer 10 is called an "output frame."

[0018] Fig. 3 shows an example in which four images with different timings are input, synthesized into one screen, and output. Consider a case in which the image synthesizing device 10 reads all input image screens, synthesizes them, and outputs them. In this case, if the frame time is T_f and the synthesis processing time is T_p, the output frame will be delayed by a maximum of 2T_f+T_p from the input time of the first input frame. For example, when considering a 60 fps image, this means that the synthesized image may contain a delay of more than two frame times, i.e., more than 33.3 milliseconds.

[0019] In the present disclosure, among the divided screens, the screens arranged horizontally on the same scanning line are regarded as one group, and the group is referred to as a "sub-screen group." For example, when the video synthesizing device 10 synthesizes four video signals of inputs 1 to 4, as shown in Fig. 4, the sub-screens D1-1 and D1-2 arranged horizontally are regarded as a sub-screen group G1, and the sub-screens D2-1 and D2-2 arranged horizontally are regarded as a sub-screen group G2. In other words, the data of the output frame is output first from the sub-screen group G1 arranged at the top, followed by the sub-screen group G2.

[0020] In addition, in this disclosure, an input that should have the shortest delay from input to output (hereinafter referred to as a "pivot input") is set. This disclosure is a system that inputs multiple asynchronous videos and synthesizes those images, and optimizes the screen layout and output timing with this pivot input as the center.

[0021] In the present disclosure, data can be output to the output frames in sequence without waiting for the completion of input of the input frames. In other words, excluding overhead, the delay from input to output can be minimized when the completion of input of an input frame coincides with the completion of output of the screen of the subgroup to which that input belongs.

[0022] Specifically, in the present disclosure, a sub-screen group for a pivot input is selected so that the delay until the pivot input is output is short. Although the sub-images in the same sub-screen group where the pivot input is output are arbitrary, the present disclosure shows an example in which the pivot input is arranged so that the input timing is the latest.

[0023] After determining the sub-screen group to which the pivot input is to be output, a sub-screen group for the input other than the pivot input is selected. At this time, the sub-screen group assignment and the frame order of the input other than the pivot input are selected so as to minimize the average delay and maximum delay until the output of the input other than the pivot input.

[0024] Below, with reference to Figures 5 and 6, we will explain an example in which input a is a pivot input, the composite screen is divided into four, and the top two sub-screens D1-1 and D1-2 are sub-screen group G1, and the bottom two sub-screens D2-1 and D2-2 are sub-screen group G2.

[0025] For example, if input a, which is a pivot input, is output to sub-screen group G2 and the output is the shortest, excluding processing overhead, the composite output is performed so that the output of sub-screen group G2 is completed when the input of input a is completed. For example, the output of sub-screen group G2 is performed to coincide with the completion of input of the K+1 frame of input a.

[0026] For inputs b, c, and d other than input a to sub-screen group G2, inputs that have been completed earlier than frame K+1 of input a are selected. In the examples of Figures 5 and 6, frames of input b before K, frames of input c before K, and frames of input d before K can be selected.

[0027] Here, for inputs of sub-screen groups other than the sub-screen group to which the pivot input belongs, inputs and input frames that will arrive in time for the output of the sub-screen group to be output are selected based on the timing at which the output of the sub-screen group G2 to which the pivot input belongs can be output with the minimum delay.

[0028] For example, K frames of input d, which has the smallest input delay difference from K+1 of input a, can be selected for sub-screen group G2. In this case, the inputs to sub-screen group G1 are input b and input c. In FIG. 5, the Kth frames can be selected for the inputs to sub-screen group G1. On the other hand, in FIG. 6, the completion of input of K frames of input c occurs after the completion of output of sub-screen group G1. In such a case, the image synthesizer 10 selects input c, which is the K-1 frame, and outputs sub-screen group G1.

[0029] 5 and 6 show an example in which the input a is set as a pivot input and the pivot input is output to the sub-screen group G2, but the present disclosure is not limited thereto. When the input a is set as a pivot input and the pivot input is output to the sub-screen group G2, the sub-screen settings of the inputs b, c, and d may select a combination that minimizes the average delay or maximum delay of the inputs b, c, and d. For example, a combination that minimizes the delay of an input that should be minimized next to the pivot input may also be selected.

[0030] Furthermore, the sub-screen group that outputs the pivot input is not limited to the sub-screen group G2 displayed below the screen, but can also be the output of the sub-screen group G1. By evaluating all of the selectable sub-screen groups and their frames of each input including the pivot input, including the sub-screen that outputs the pivot input, it is possible to output a combination that minimizes the average delay or maximum delay until each input is output.

[0031] For example, the image synthesis device 10 compares the average delay of all of the inputs a to d when the output of the pivot input is set to the sub-screen group G2 with the average delay of all of the inputs a to d when the output of the pivot input is set to the sub-screen group G1, and if the average delay of all of the inputs a to d is smaller when the output of the pivot input is set to the sub-screen group G1, the image synthesis device 10 sets the output of the pivot input to the sub-screen group G1.

[0032] 7 shows an example of the configuration of an image synthesizer 10 according to this embodiment. The image synthesizer 10 according to this embodiment includes a detection unit 101, a crossbar switch 102, an up / down converter 103, a buffer 104, and a pixel synthesis unit 105. Although the figure shows four inputs and one output, any number N of inputs and outputs may be used. Also, the screen does not have to be equally divided.

[0033] Reference numeral 101 denotes a functional unit that detects the input order within a frame time for N inputs. Reference numeral 102 denotes a crossbar switch, which has the function of sorting and outputting the input sequence from 101 in the order of the detection result. Reference numeral 103 denotes an up-down converter that enlarges or reduces the number of pixels to an arbitrary size. 102 and 103 may be connected inversely to the inputs (a, b, c, d, ...). In other words, the inputs a, b, c, d may be scaled by 103, and then 102 may rearrange and output them in the order of the inputs. A buffer 104 can buffer the input of the buffer 103 or 102 and output the input in any order. 105 is a pixel synthesis unit. Of the entire screen to be output, pixel data is read from 104 in the order of output, synthesized, and output. This synthesis and output sub-screen are as described above. 105 may add an arbitrary control signal to the blanking portion of the screen.

[0034] The image synthesizing device 10 of the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0035] The pivot input can be set arbitrarily, for example, by an external instruction or based on information included in the video signal. For example, it can be set based on a flag indicating a priority included in the video signal, or based on the result of image processing. For example, a conductor is determined by image processing, and an image showing the conductor in a large size is set as the pivot input. Also, an object such as a person that moves a lot is determined by image processing, and an image showing the object that moves a lot is set as the pivot input.

[0036] The pivot input can be switched at any time. For example, if a moving target changes due to image processing, the pivot input is switched to the new moving target. This allows the output to be adjusted to match the input that should have the shortest delay.

[0037] (Effects of the present disclosure) The present disclosure can minimize the delay time until the output after synthesis for a specific input, while shortening the delay time until the output after synthesis for other inputs. This enables collaborative work with strict low-latency requirements in a system that synthesizes multiple screens at multiple locations, and in particular with stricter low-latency requirements for specific inputs.

[0038] (Key Points of This Disclosure) In a system that synthesizes and displays video from multiple locations, in collaborative work that requires strict low latency, such as in an ensemble, and in conditions where low latency is especially important, such as in a conductor, low latency is required in the synthesis process from one input to the output, but low latency is also necessary for the synthesis process for other inputs. The present disclosure relates to a system that inputs multiple asynchronous videos and synthesizes those images, and by arranging pivot inputs so that the input timing is the latest within the same sub-screen group, it is possible to reduce the delay time until the output after synthesis. This enables collaborative work with strict low-latency requirements in a system that synthesizes multiple screens from multiple locations, etc. [Industrial Applicability]

[0039] The present disclosure can be applied to the information and communications industry. [Explanation of symbols]

[0040] 10: Image composition device 20: Screen 21: Scan line 22: Blanking section 23: Border part 24:Display screen 101: Detection unit 102: Crossbar switch 103: Up / down converter 104: Buffer 105: Pixel synthesis unit

Claims

1. An apparatus for synthesizing a plurality of asynchronously input video signals into a video signal to be displayed on one screen, comprising: controlling output positions of the plurality of video signals in the one screen in accordance with a difference between a synthesis output completion time point of the pivot input video signal and an input time point of a video signal other than the pivot input so that a delay of the pivot input video signal among the plurality of video signals is minimized, thereby synthesizing the plurality of video signals; Device.

2. a video signal other than the pivot input that is to be mixed with the video signal of the pivot input is a video signal that was input before the pivot input, outputting an input having the smallest input delay difference from the pivot input to a sub-screen located on the same scanning line as the sub-screen to which the pivot input belongs, among the sub-screens obtained by dividing the one screen; 2. The apparatus of claim 1.

3. The pivot input video signal is output to a sub-screen that is scanned last in the one screen.

3. The apparatus of claim 2.

4. outputting a video signal included in the plurality of video signals for each sub-screen group, the sub-screen group being a set of sub-screens that constitute a part of one screen and are arranged on the same scanning line of the screen; selecting an input that is available in time for output for each of the sub-screen groups, and synthesizing the plurality of video signals for each of the sub-screen groups; 4. Apparatus according to any one of claims 1 to 3.

5. A method for synthesizing a plurality of asynchronously input video signals into a video signal to be displayed on one screen, comprising the steps of: controlling output positions of the plurality of video signals in the one screen in accordance with a difference between a synthesis output completion time point of the pivot input video signal and an input time point of a video signal other than the pivot input so that a delay of the pivot input video signal among the plurality of video signals is minimized, thereby synthesizing the plurality of video signals; method.

6. A program for implementing a computer as each functional unit of a device that synthesizes a plurality of asynchronously input video signals into a video signal to be displayed on a single screen, comprising: controlling output positions of the plurality of video signals in the one screen in accordance with a difference between a synthesis output completion time point and an input time point of a video signal other than the pivot input so that a delay of a video signal of a set pivot input among the plurality of video signals is minimized, thereby synthesizing the plurality of video signals; program.

Images