Apparatus, method and program for synthesizing video signals
The video synthesis device addresses asynchronous video signal delays by synchronizing input timing for efficient compositing, facilitating low-latency collaborative work in video conferencing.
Patent Information
- Application Number
- JP2022570805
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-12-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-12-22
AI Technical Summary
Existing video compositing systems experience significant delays due to asynchronous video signals, which hinder collaborative work requiring low latency, such as ensemble performances in remote video conferences.
A video synthesis device that arranges and synthesizes multiple asynchronous video signals based on their input timing, starting the compositing process in order of earliest input, and outputs the combined frames in a timely manner.
Reduces the delay time in outputting composite video signals, enabling low-latency collaborative work across multiple locations.
Smart Images

Figure 0007589753000001 
Figure 0007589753000002 
Figure 0007589753000003
Abstract
Description
[Technical field]
[0001] This relates to a video synthesis system that synthesizes and outputs a single screen from multiple video input signals. [Background technology]
[0002] In recent years, many video devices have come into use. The images produced by these devices use a wide variety of pixel counts (resolutions) and frame rates. Although the video signals of these video devices differ in terms of physical signals, control signals, etc., depending on the standard, one screen is transmitted in a time period equal to one frame rate.
[0003] These images can be used in a variety of ways, such as video conferencing, where multiple cameras are displayed on a monitor with fewer cameras than the number of cameras. In such cases, multiple images can be split and displayed on a single screen, or one image can be displayed in a reduced size and fitted onto another image, resulting in screen composition.
[0004] Normally, the timing of a video signal is not synchronized, and the timing of other video signals to be mixed is different, so the signals are temporarily buffered in memory before being mixed. As a result, there is a delay in the output of the mixed screen.
[0005] If we were to imagine a video conference in which an ensemble playing music in a remote location were to be held using this type of screen synthesis, the delay involved in this synthesis would make it very difficult to implement. For example, in a piece of music with 120 beats per second (hereafter referred to as 120 BPM (Beat Per Minute)), the duration of one beat is 60 / 120 seconds = 500 milliseconds. If we were to match this with an accuracy of 5%, then the delay between when the image is captured by the camera and when it is displayed would need to be kept to 500 x 0.05 = 25 milliseconds or less.
[0006] In reality, the process from taking a picture with a camera to displaying it requires other delays besides the process related to composition, such as the time it takes to process the image on the camera, the time it takes to display on the monitor, the time it takes to transmit, etc. As a result, with conventional technology, it has been difficult to collaborate in situations where timing is important, such as when playing an ensemble while watching each other's videos from a distance.
[0007] Therefore, for collaborative work that requires strict low latency, it is necessary to provide a system that combines multiple video signals from multiple locations, etc., and that reduces the latency between the input of asynchronous video signals and the output of the combined video signal. [Prior art documents] [Non-patent literature]
[0008] [Non-Patent Document 1] VESA and Industry Standards and Guidelines for Computer Display Monitor Timing (DMT), Version 1.0, Rev. 13, February 8, 2013 Summary of the Invention [Problem to be solved by the invention]
[0009] An object of the present disclosure is to reduce the delay time until a composite video is output. [Means for solving the problem]
[0010] The present disclosure performs a synthesis process in which a screen is arranged from the top in order of earliest input timing from a plurality of input video signals in a device that synthesizes and displays a plurality of asynchronous videos.
[0011] The image synthesizing device of the present disclosure comprises: Detecting input timing of each input frame constituting a plurality of video signals; When a set number of the video signals are input, a synthesis process of the set number of the video signals is started in sequence; The multiple video signals are combined into one video signal to generate an output frame.
[0012] The video compositing method of the present disclosure includes: The image synthesizer Detecting input timing of each input frame constituting a plurality of video signals; When a set number of the video signals are input, a synthesis process of the set number of the video signals is started in sequence; The multiple video signals are combined into one video signal to generate an output frame.
[0013] The program of the present disclosure is a program for causing a computer to realize each functional unit of an apparatus of the present disclosure, and is a program for causing a computer to execute each step of a method performed by an apparatus of the present disclosure. Effect of the Invention
[0014] The present disclosure can reduce the delay time until a composite image is output. [Brief description of the drawings]
[0015] [Figure 1] 3 shows an example of screen information included in a video signal. [Diagram 2] An example of screen composition is shown below. [Diagram 3] 1 illustrates an example of a video synthesis method related to the present disclosure. [Figure 4] 1 illustrates an example of a video synthesis method according to the present disclosure. [Diagram 5] 1 illustrates an example of a video synthesis method according to the present disclosure. [Figure 6] 1 illustrates an example of a video synthesis method according to the present disclosure. [Figure 7] 1 illustrates an example of a video synthesis method according to the present disclosure. [Figure 8] 1 shows an example of the configuration of an image synthesizing device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings are assumed to indicate the same components.
[0017] Fig. 1 shows an example of screen information contained in a video signal. The screen information is transmitted by scanning the screen horizontally, one scanning line 21 at a time, and then scanning the lower scanning lines 21 in sequence. This scanning includes scanning of the display screen 24, blanking portion 22, and overhead information / signals such as border portion 23. The blanking portion 22 may contain information other than video information, such as control information and audio information. (For example, see Non-Patent Document 1, Chapter 3.)
[0018] FIG. 2 shows an example of video signal composition. In this disclosure, as an example, four video signals are input to a video composition device, which then composes and outputs one video signal. A video signal transmits one screen over a period of time equal to one frame rate. For example, if a video signal has 60 frames per second, one screen of the video signal is transmitted over 1 / 60th of a second, or approximately 16.7 milliseconds (hereinafter, 60 fps (Frames per Second)). The information of one screen at each time point contained in the video signal is called a "frame," the information of one screen of each video signal input to the video composition device is called an "input frame," and the information of the composed one screen output from the video composition device is called an "output frame."
[0019] For example, consider a case where a video synthesizer reads all input frames, synthesizes them into one output frame, and outputs it, as shown in Fig. 3. In this case, if the frame time of each input frame is T_f and the synthesis processing time is T_p, the output of the output frame will be delayed by a maximum of 2T_f+T_p from the start of input of the first input 1. If the frame rate is 60 fps, a delay of at least 33.3 milliseconds will occur.
[0020] The device and method disclosed herein are a system that inputs multiple asynchronous video images and synthesizes those images, and is characterized by starting the synthesis process in order of the earliest input timing and placing them at the top of the screen.
[0021] In this embodiment, a case where there are four inputs and four split one screen output, with each input being shifted by 1 / 4 T_f time, will be described. The input timing is set as inputs 1, 2, 3, and 4 in order of earliest input timing. In this case, of the four video signals, the video signals of inputs 1 and 2 are displayed on the upper part of the display screen 24, and the remaining video signals of inputs 3 and 4 are displayed on the lower part of the display screen 24. Therefore, in this embodiment, when two of the four video signals are input, a synthesis process of the two video signals is started. For simplification, the blanking portion 22 and border portion 23 of the video signals are excluded, and only the signal of the display screen 24 will be described.
[0022] A first synthesis example of the present disclosure is shown in Fig. 4. When input of input 2 is completed, synthesis process (1) starts, and output to the upper part of the display screen 24 starts. Next, when the input of input 4 is completed, the synthesis process (2) begins, and output to the lower part of the display screen 24 begins. In this case, the maximum delay time from the start of input 1 to the start of output on the upper display screen 24 is (5 / 4T_f+T_p). This makes it possible to reduce the output delay by 3 / 4 T_f, compared to the example shown in Figure 3. For example, when the frame rate is 60 fps, the delay is 21 ms+T_p.
[0023] Next, we discuss the case where the inputs are not offset by 1 / 4 T_f time. 5, when the time difference T_in2toin4 between the ends of the input frames of input 2 and input 4 is longer than T_f / 2, the start of the composition process (1) is awaited for at least T_in2toin4-T_f / 2 time after the completion of input 2 so that the output of the lower display screen 24, which combines inputs 1 and 2, is completed in time for the output of the upper display screen 24. Alternatively, the output of the output frame of the upper screen may be awaited for T_in2toin4-T_f / 2 time after the composition process (1) is performed.
[0024] 6, if the time difference T_in2toin4 between the ends of the input frames of input 2 and input 4 is shorter than T_f / 2, the start of synthesis process (2) is awaited for a time T_f / 2-T_in2toin4 after input of input 4 is completed so that the output of the lower display screen 24 is in time to follow the output of the upper display screen 24. Alternatively, the output of the output frame for the lower screen may be awaited for a time T_f / 2-T_in2toin4 after execution of synthesis process (2).
[0025] In this embodiment, an example is shown in which four video signals are synthesized into one video signal of two upper screens and two lower screens as shown in FIG. 2 and output. For this reason, in the examples of FIG. 5 and FIG. 6, the time difference at the end of the input frame is set to the time difference between input 2 and input 4, and the comparison target is compared with T_f / 2. However, the time difference at the end of the input frame and the comparison target can be any number determined according to the number of video signals to be synthesized and the arrangement of the screens. For example, when six video signals are synthesized into one video signal of two upper screens, two middle screens, and three lower screens, the time difference at the end of the input frame may be set to the time difference between input 4 and input 6, and the comparison target may be T_f / 3.
[0026] In addition, since an actual video signal has overhead portions such as the blanking and border portions mentioned above, the values T_f / 2 and T_f / 3 to be compared are values for the signal for the display screen 24 and need to be corrected according to the overhead portions.
[0027] With reference to Fig. 7, the case where the above method is applied to pipeline processing will be described. The pipelined synthesis processing time is T_pp (Time of Pipelined Processing). Here, the pipelined synthesis processing time indicates only the initial overhead time for the pipeline (the time required for the overall processing before passing to the next stage processing, including data reading, etc.), and the synthesis processing is executed continuously in accordance with the input or output. The actual time of the pipelined synthesis processing is the time required to process one unit of data in the pipeline processing until the output, which is the subsequent stage processing. In this case, the processing can be started so that the output of the output frame is completed at the time point of the end time of the input of the video signal + T_pp.
[0028] The following describes a case where there are four inputs, one screen divided into four, and each input frame is offset by 1 / 4 T_f. Inputs 1, 2, 3, and 4 are listed in descending order of input timing. For simplicity, blanking portion 22 and border portion 23 of the video signal are excluded, and only the signal of display screen 24 is described.
[0029] The synthesis process (1) is started so that the time T_pp has elapsed since the input completion time T_2E of input 2 coincides with the output completion time T_UE of the upper display screen 24, and output to the upper part of the display screen 24 is started. The synthesis process (2) is started so that the time T_pp has elapsed since the input completion time T_4E of input 4 coincides with the output completion time T_DE of the lower display screen 24, and output to the lower part of the display screen 24 is started. In this case, the maximum delay time from the start of input 1 to the start of output on the upper display screen 24 is (3 / 4 T_f+T_pp). This makes it possible to reduce the output delay more than in the example shown in Fig. 3. For example, when the frame rate is 60 fps, the delay is 12.5 ms+T_pp.
[0030] Next, we discuss the case where the inputs are not offset by 1 / 4 T_f time. If the time difference T_in2toin4 between the ends of the input frames of input 2 and input 4 is longer than T_f / 2, the start of the synthesis process (1) is awaited for at least T_in2toin4-T_f / 2 time after completion of input 2 so that the output of the lower display screen 24, which combines inputs 1 and 2, can be completed in time to follow the output of the upper display screen 24. Alternatively, the output of the output frame may be awaited for T_in2toin4-T_f / 2 time after performing synthesis process (1).
[0031] If the time difference T_in2toin4 between the ends of the input frames of input 2 and input 4 is shorter than T_f / 2, the start of synthesis process (2) is awaited for a time T_f / 2-T_in2toin4 after input of input 4 is completed so that the output to the lower display screen 24 is timed to follow the output to the upper display screen 24. Alternatively, the output of the output frame may be awaited for a time T_f / 2-T_in2toin4 after execution of synthesis process (2).
[0032] In this embodiment, an example is shown in which four video signals are synthesized into one video signal of two upper screens and two lower screens as shown in FIG. 2 and output. For this reason, in the above example, the time difference at the end of the input frame is set to the time difference between input 2 and input 4, and the comparison target is compared with T_f / 2. However, the time difference at the end of the input frame and the comparison target can be any number determined according to the number of video signals to be synthesized and the arrangement of the screens. For example, when six video signals are synthesized into one video signal of two upper screens, two middle screens, and three lower screens, the time difference at the end of the input frame may be set to the time difference between input 4 and input 6, and the comparison target may be T_f / 3.
[0033] In addition, since an actual video signal has overhead portions such as the blanking and border portions mentioned above, the values T_f / 2 and T_f / 3 to be compared are values for the signal for the display screen 24 and need to be corrected according to the overhead portions.
[0034] An example of a system configuration according to this embodiment is shown in Fig. 8. The image synthesizer 10 according to this embodiment includes a detection unit 101, a crossbar switch 102, an up / down converter 103, a buffer 104, and a pixel synthesis unit 105. Although the figure shows four inputs and one output, any number of inputs and outputs may be used.
[0035] Reference numeral 101 denotes a functional unit that detects the input order within a frame time for N input frames. For example, the functional unit detects input timings at which input frames 1, 2, 3, and 4 shown in Fig. 4 and Fig. 5 are input, and determines the order of inputs 1, 2, 3, and 4 using the input timings. A crossbar switch 102 has a function of sorting and outputting the input frames in the order of the detection result of the input order from 101. For example, the input frames are sorted in the order of inputs 1, 2, 3, and 4 shown in FIGS. An up-down converter 103 enlarges or reduces the number of pixels to an arbitrary size. For example, the number of pixels of the input 1 is enlarged or reduced so as to match the size of the screen shown in FIG. 102 and 103 may be connected inversely to the inputs (a, b, c, d, ...). In other words, inputs a, b, c, d may be scaled in 103, and then inputs 1, 2, 3, 4 may be rearranged and output in that order in 102. A buffer 104 stores each input frame. The input of 103 or 102 can be buffered and output in any order. Reference numeral 105 denotes a pixel synthesis unit. The pixel synthesis unit 105 reads pixel data from 104 in the order of output from the entire screen of the output, synthesizes the data to generate an output frame, and outputs the output frame. As a result, an image in which four video signals are synthesized, as shown in FIG. 2, is displayed on the screen. The timing is as described above. The pixel synthesis unit 105 may add an arbitrary control signal to the blanking portion 22 of the screen.
[0036] The device of the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.
[0037] In the above embodiment, an example of 4 inputs and 4-split 1 screen is shown, but the present disclosure is not limited to this and can be applied to any input. Also, in the above embodiment, an example in which the frame time T_f of inputs 1 to 4 is the same is mainly shown, but the present disclosure can be applied to inputs 1 to 4 with different frame times T_f.
[0038] (Effects of the present disclosure) By compositing and outputting the images from the top in order of the earliest input timing of asynchronous video input signals, it is possible to reduce the delay time until the composite output. This enables collaborative work with strict low-latency requirements in a system that composites multiple screens from multiple locations, etc. [Industrial Applicability]
[0039] The present disclosure can be applied to the information and communications industry, which distributes video content and game content, as well as the film, advertising, and game industries involved in video production. [Explanation of symbols]
[0040] 10: Image composition device 21: Scan line 22: Blanking section 23: Border part 24:Display screen 101: Detection unit 102: Crossbar switch 103: Up / down converter 104: Buffer 105: Pixel synthesis unit
Claims
1. A method for detecting input timing of each input frame constituting a plurality of asynchronous, mutually different video signals, When a set number of the video signals are input, a synthesis process of the set number of video signals is started in sequence, so that the screens of the plurality of video signals are arranged on a synthesis screen divided into a plurality of areas according to the number of the plurality of video signals in the order of the earliest input timing, starting from the area having the earliest timing for scanning a scanning line, on the synthesis screen; generating an output frame by combining the plurality of video signals into one video signal; Image synthesis device.
2. Detecting input timing of each input frame constituting a plurality of video signals; When a set number of the video signals are input, a synthesis process of the set number of the video signals is started in sequence; generating an output frame by combining the plurality of video signals into one video signal; comparing a time difference between a time point when the input of the set number of the video signals among the plurality of video signals is completed and a time point when the input of the last video signal of the plurality of video signals is completed with a time determined according to either or both of the number of the plurality of video signals and an arrangement of a composite screen, and adjusting a timing of a composite process; Image synthesis device.
3. If the time difference is longer than a time determined according to either or both of the number of the plurality of video signals or the layout of the composite screen, adjusting the timing of the composite process of the set number of video signals; When the time difference is shorter than a time determined according to either or both of the number of the plurality of video signals and the layout of the composite screen, a timing of the composite process of the remaining video signals among the plurality of video signals is adjusted.
3. The image synthesizing apparatus according to claim 2.
4. The image synthesizer Detecting input timing of each input frame constituting a plurality of asynchronous, mutually different video signals; When a set number of the video signals are input, a synthesis process of the set number of video signals is started in sequence, so that the screens of the plurality of video signals are arranged on a synthesis screen divided into a plurality of areas according to the number of the plurality of video signals in the order of the earliest input timing, starting from the area having the earliest timing for scanning a scanning line, on the synthesis screen; generating an output frame by combining the plurality of video signals into one video signal; Video compositing method.
5. The image synthesizer Detecting input timing of each input frame constituting a plurality of video signals; When a set number of the video signals are input, a synthesis process of the set number of the video signals is started in sequence; generating an output frame by combining the plurality of video signals into one video signal; comparing a time difference between a time point when the input of the set number of the video signals among the plurality of video signals is completed and a time point when the input of the last video signal of the plurality of video signals is completed with a time determined according to either or both of the number of the plurality of video signals and an arrangement of a composite screen, and adjusting a timing of a composite process; Video compositing method.
6. The image synthesizer, If the time difference is longer than a time determined according to either or both of the number of the plurality of video signals or the layout of the composite screen, adjusting the timing of the composite process of the set number of video signals; When the time difference is shorter than a time determined according to either or both of the number of the plurality of video signals and the layout of the composite screen, a timing of the composite process of the remaining video signals among the plurality of video signals is adjusted. The video synthesis method according to claim 5 .
7. A program for causing a computer to realize each of the functional units of the image synthesizing apparatus according to any one of claims 1 to 3.
Citation Information
Patent Citations
Signal transmitting device, signal transmitting method, signal receiving device and signal receiving method
JP2009130639A
Video composition device
JP2009265319A
Signal transmitter, signal transmission method, signal receiver, signal receiving method, and signal transmission system
JP2011044826A
Image processing system
JP2015165628A
Image transmission apparatus, image reception apparatus, image transmission method, and image transmission program
JP2018182410A
