Video synthesis device

Abstract

The present disclosure relates to a video synthesis device or a video synthesis method, the video synthesis device comprising: an input unit to which a video signal is inputted; and a synthesis unit which generates an output video by performing screen synthesis of a plurality of video signals inputted to the input unit, wherein the synthesis unit can perform the screen synthesis of the video signals inputted to the input unit at different times, and the times at which the video signals that can be subjected to the image synthesis in the synthesis unit were inputted to the input unit differ according to regions in the output video.

Description

Video composition equipment

[0001] The present disclosure relates to a video synthesizer that synthesizes a plurality of video signals.

[0002] In recent years, many video devices have come into use. The images produced by these video devices have a variety of pixel counts (resolutions), frame rates, and the like. Video is used in a variety of ways, such as in video conferencing, where video signals from multiple cameras are displayed on a display device with fewer than the number of cameras. In such cases, a single display screen on the display device is divided into multiple regions, or a reduced version of a certain image is fitted onto a portion of the display screen of another image, thereby performing screen composition to generate and output an output image.

[0003] Conventionally, it has been assumed that video signals input at the same time are composited onto the screen. On the other hand, there is also a known interpolation method in which, if a video signal to be displayed is not ready in time, the missing video signal is replaced with a video signal that has already been received. However, this interpolation method has the problem of causing tearing, where the boundary between the video signal from the previous time is discontinuous and visible to the eye.

[0004] VESA and Industry Standards and Guidelines for Computer Display Monitor Timing (DMT), Version 1.0, Rev. 13, February 8, 2013

[0005] Therefore, an object of the present disclosure is to reduce the delay between the input of a video signal and the output of an output video.

[0006] There are areas of the output video that do not change much, and tearing is unlikely to occur even if the received video signal is substituted for such areas. Therefore, the present disclosure employs a complementation method that substitutes the received video signal for areas of the output video that are unlikely to cause tearing.

[0007] The image synthesis device of the present disclosure comprises an input unit to which an image signal is input, and a synthesis unit that generates an output image by performing screen synthesis of the multiple image signals input to the input unit, wherein the synthesis unit is capable of performing the screen synthesis of image signals input to the input unit at different times, and the time at which the image signals that can be screen synthesized by the synthesis unit are input to the input unit varies depending on the area in the output image.

[0008] The video synthesis method of the present disclosure comprises the steps of: inputting video signals to an input unit; and a synthesis unit performing screen synthesis of the multiple video signals input to the input unit to generate an output video; wherein the synthesis unit is capable of performing the screen synthesis of video signals input to the input unit at different times; and the times at which the video signals capable of performing the screen synthesis in the synthesis unit were input to the input unit vary depending on the areas in the output video.

[0009] In a region where the video signals capable of performing the screen composition in the composition unit have been input to the input unit for a short time, the most recent video signals must be used for the screen composition. On the other hand, in a region where the video signals capable of performing the screen composition in the composition unit have been input to the input unit for a long time, video signals previously input to the input unit can be used. Therefore, in a region where the video signals capable of performing the screen composition in the composition unit have been input to the input unit for a long time, a complementation method can be applied in which previously received video signals are substituted. As a result, the video composition device and video composition method disclosed herein can reduce the delay between the input of video signals and the output of an output video.

[0010] Specifically, when a video signal of a predetermined substitution-prohibited area in the output image is not input, the synthesis unit waits for a video signal to be input to the input unit and then performs the screen synthesis, and when a video signal of a substitution-allowed area other than the substitution-prohibited area in the output image is not input, the synthesis unit may perform the screen synthesis using a video signal that was previously input to the input unit.

[0011] The video signal of the present disclosure may include setting information for defining a substitution-prohibited area in a video signal to be displayed on one screen, in which the video signal cannot be substituted with a video signal of a different time on the screen. In this case, the combining unit can set the substitution-prohibited area in accordance with the setting information.

[0012] The image synthesizing device of the present disclosure may include a setting unit that sets the substitution prohibition area based on the image signal input to the input unit.

[0013] The input unit may receive a plurality of asynchronous video signals in parallel, and in this case, the combining unit may perform the screen combining of the plurality of video signals input in parallel.

[0014] The program of the present disclosure is a program for causing a computer to realize each function of the image synthesis device of the present disclosure, and is a program for causing a computer to execute each procedure of the method performed by the image synthesis device of the present disclosure.

[0015] The above disclosures can be combined as much as possible.

[0016] According to the present disclosure, it is possible to reduce the delay between when a video signal is input and when an output video is output.

[0017] 1 shows an example of information included in a video signal. FIG. 1 is an example embodiment of a video synthesis device according to the present disclosure. FIG. 2 is an example screen display in which one screen is divided into multiple regions. FIG. 3 shows an example of input / output timing of a video signal. FIG. 4 shows an example of input / output timing of a video signal. FIG. 5 is an example of a synthesis screen for input 1. FIG. 6 is an explanatory diagram of a substitution-prohibited area according to the present disclosure. FIG. 7 shows an example of input / output timing of a video signal. FIG. 8 is an example screen display when the present disclosure is not used. FIG. 9 shows an example of input / output timing of a video signal. FIG. 10 is an example screen display when the present disclosure is used. FIG. 11 shows an example embodiment of a video synthesis device according to the present disclosure.

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings indicate the same components.

[0019] (First embodiment) Fig. 1 shows an example of a video signal. Although the video signal of a video device differs depending on the standard in terms of physical signals, control signals, etc., one screen is transmitted using a time period equal to one half of the frame rate. For example, if the video signal has 60 frames per second, one screen of the video signal is transmitted over 1 / 60th of a second, or approximately 16.7 milliseconds (hereinafter referred to as 60 fps (Frames per Second)).

[0020] This screen information is scanned horizontally, line by line, moving sequentially down to the next line. This scanning includes overhead information / signals such as the display screen, blanking areas, and border areas. The blanking areas may also contain information other than video, such as control information and audio information (see, for example, Non-Patent Document 1).

[0021] Second Embodiment Fig. 2 shows an example configuration of an image synthesizing device according to the present disclosure. The image synthesizing device 10 of this embodiment includes an input unit 11 and a synthesizing unit 12. A video signal is input to the input unit 11. The synthesizing unit 12 synthesizes the screen of the multiple video signals input to the input unit 11 to generate an output video.

[0022] In this embodiment, an example is shown in which the input unit 11 has terminals I1 to I4 and four video signals are input. Hereinafter, in this disclosure, the video signal input from terminal I1 will be referred to as input 1, the video signal input from terminal I2 as input 2, the video signal input from terminal I3 as input 3, and the video signal input from terminal I4 as input 4. The combining unit 12 generates an output video by combining the four video signals on a screen.

[0023] An example of a display screen of this embodiment is shown in Figure 3. In this example, a single display screen is divided into partial screens S1 to S4, and input 1 is output to partial screen S1, input 2 is output to partial screen S2, input 3 is output to partial screen S3, and input 4 is output to partial screen S4.

[0024] 4 shows an example of the input / output timing of video signals in this embodiment. Normally, video signal timing is not synchronized, so inputs 1 to 4 are input to the input unit 11 at different times. For this reason, the synthesis unit 12 waits until all of inputs 1 to 4, which were input at the same time, are buffered in a memory or the like before starting the synthesis process to create one output video. For this reason, a delay occurs between the input of asynchronous video signals and the output of the output video in the video synthesis device 10.

[0025] For example, the frame time of one video signal is T ｆ , the synthesis processing time is T ｐ Then, the output video will be 2T at most after the first video signal is input. ｆ +T ｐ For example, in the case of a 60 fps video, a delay of more than two frame times, i.e., 33.3 milliseconds or more, may occur.

[0026] In reality, in addition to the processing involved in compositing images, there are other delays involved between capturing an image with a camera and displaying it on a display device, such as the time it takes for the image to be processed by the camera, the time it takes for the image to be displayed on the display device, and the time it takes for transmission. This makes it difficult to collaborate in situations where timing is important, such as when people in remote locations play musical instruments while viewing each other's videos.

[0027] Third Embodiment If the synthesis process of the video signals is started without waiting for all video signals to be collected, the delay from the input of the video signals to the output of the output video can be reduced. Therefore, the synthesis unit 12 of this embodiment starts the synthesis process before all of the input 1 is input to the input unit 11.

[0028] 5 shows an example of the input / output timing of the video signal in this embodiment. The synthesis unit 12 in this embodiment is capable of synthesizing the screen of video signals input to the input unit 11 at different times. For example, at a certain time T ｋ If the input of the video signal for part of the input 1 of the inputs 1 to 4 is not in time, the combining unit 12 ｋ－１ The video signal at time T ｋ As a result, as shown in FIG. 6, the area S1-1 of the partial screen S1 of the input 1 is synthesized as a video signal of time T ｋ The video signal of time T ｋ－１ The video signal is displayed.

[0029] In this way, the composition unit 12 of the present embodiment performs screen composition of the partial screen S1 using video signals input to the input unit 11 at different times. This reduces the delay from when the video signals are input until when the output video is output. However, at time T ｋ The video signal and time T ｋ－１ The video signal in the second area is different from the video signal in the first area. Therefore, tearing may occur at the boundary between the first area and the second area. If tearing occurs, the physical sensation is greatly impaired, making collaborative work difficult.

[0030] (Fourth embodiment) There are areas in the output video that do not change much, and tearing is unlikely to occur in such areas even if they are replaced with a video signal that has already been received. Therefore, in this embodiment, substitution-prohibited areas and substitution-permitted areas are set in the synthesis unit 12. Substitution-prohibited areas are areas included in the screen that cannot be replaced with a video signal of a different time. Substitution-permitted areas are areas included in the screen that can be replaced with a video signal of a different time. Thus, in this embodiment, the times at which video signals that can be subjected to screen synthesis in the synthesis unit 12 are input to the input unit 11 vary depending on the areas in the output video.

[0031] The image synthesizing device 10 of this embodiment executes the image synthesizing method of the present disclosure, which includes the steps of: inputting image signals to an input unit 11; and generating an output image by a synthesizing unit 12 performing screen synthesis of the multiple image signals input to the input unit 11, wherein the synthesizing unit 12 is capable of performing screen synthesis of the image signals input to the input unit 11 at different times, and the times at which the image signals capable of being screen synthesized by the synthesizing unit 12 were input to the input unit 11 vary depending on the areas in the output image.

[0032] In this embodiment, as shown in Figure 7, a substitution-prohibited area S4-0 is provided in the partial screen S4, and substitution is permitted only in the remaining substitution-permitted area S4-1. When the video signal S4-1 of the substitution-permitted area is not input, the synthesis unit 12 synthesizes the time T ｋ－１ On the other hand, when the video signal of the substitution prohibited area S4-0 is not input, the synthesis unit 12 synthesizes the screen using the video signal of the substitution prohibited area S4-0. ｋ The image synthesis is performed after the input of the video signal.

[0033] By setting a substitution-prohibited area S4-0 in an area that is attracting attention and experiencing rapid movement, tearing is prevented, suppressing degradation of the user's experience, and low latency is achieved by partially allowing substitution. The method for setting the substitution-permitted area and substitution-prohibited area is arbitrary, and examples thereof include the following: (i) A predetermined area of ​​the partial screen S4 of the input 4 is set as the substitution-prohibited area S4-0, and other areas are set as substitution-permitted areas. (ii) The video signal may include setting information that defines the substitution-prohibited area. In this case, the synthesis unit 12 can set the substitution-prohibited area in accordance with the setting information. (iii) The input unit 11 may be provided with a setting unit (not shown) that sets the substitution-prohibited area based on the video signal input to the input unit 11. In this case, the synthesis unit 12 can set the substitution-prohibited area in accordance with the setting of the setting unit (not shown).

[0034] Substitution prohibition areas can also be set for the other partial screens S1 to S3 at the same time as partial screen S4. Here, the number of substitution prohibition areas on partial screen S4 may be one, or may be two or more. Furthermore, the position of the substitution prohibition area on partial screen S4 may be set to a common position on each of partial screens S1 to S4, such as the center of partial screen S4, or may be different.

[0035] In this embodiment, as in the third embodiment, an example will be described in which four video signals are input, each video signal is reduced to 1 / 4 of its original size, and a four-split display is performed. The input video signal and the output video have the same video size and frame rate, and an substitution prohibition area is set in partial screen S4 that displays the video signal of input 4.

[0036] In the following description, for ease of understanding, the time T ｋ The following description focuses on input 4. However, before and after input 4, video signals are continuously input according to the frame rate. Inputs 1 to 3 are also input and output as output video.

[0037] The first condition of the present disclosure is shown in FIG. ｏｕｔ＿ｋ <T ｉｎ４＿ｋ However, T ｉｎ４＿ｋ is time T ｋ is the start time of input 4 to the input unit 11, and T ｏｕｔ＿ｋ is time T ｋ Input 4 is the output start time of the composite output video.

[0038] Under the first condition, at time T ｋ The input to input 4 is not in time for the output of ｋ As shown in FIG. 9, the output image of ｋ－１ In this case, the output start time T ｏｕｔ＿ｋ and time T ｋ－１ The input start time T ｉｎ４＿ｋ－１ The time difference (hereinafter referred to as delay time) between ｏｕｔ＿ｋ -T ｉｎ４＿ｋ +3Tｆ / 2.

[0039] FIG. 10 shows the second condition of the present disclosure. Second condition: (T ｉｎ４＿ｋ +T ｄｕｒａｔｉｏｎ ) < (T ｏｕｔ＿ｋ +T ｆ / 2+T ｄｕｒａｔｉｏｎ / 2) However, T ｄｕｒａｔｉｏｎ is time T ｋ The input start time T ｉｎ４＿ｋ This is the time from when the substitution prohibition area S4-0 ends to when the substitution prohibition area S4-0 ends.

[0040] If the second condition is satisfied in addition to the first condition, the video synthesizing device of this embodiment outputs the T of input 4 from the beginning. ｄｕｒａｔｉｏｎ Until time T ｋ After that, the video signal is output at time T ｋ－１ As a result, as shown in FIG. 11, the area S4-1 including the substitution prohibited area is output with a video signal of time T ｋ The image signal of the past time T is displayed in the area S4-2 that does not include the substitution prohibited area. ｋ－１ The video signal is displayed.

[0041] Therefore, according to this embodiment, new data for one frame is displayed in the substitution prohibition area, thereby shortening the delay time. ｋ The input start time T ｉｎ４＿ｋ From T ｄｕｒａｔｉｏｎ The delay time of ｏｕｔ＿ｋ -T ｉｎ４＿ｋ +T ｆ / 2.

[0042] 12 shows an embodiment of the image synthesizing device 10 of the present disclosure. The image synthesizing device 10 of this embodiment includes an input unit 11, an input control unit 101, a crossbar switch 102, a converter 103, a buffer 104, and a pixel synthesizing unit 105. The input control unit 101, the crossbar switch 102, the converter 103, the buffer 104, and the pixel synthesizing unit 105 function as a synthesizing unit 12. The image synthesizing device 10 of the present invention can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0043] The input unit 11 receives N video signals. In this embodiment, an example of M=4 is shown, and video signals of input a, input b, input c, and input d are input. The input control unit 101 is a functional unit that detects the input order within a frame time for the N inputs. The crossbar switch 102 has a function of sorting and outputting the inputs in the order detected by the input control unit 101. As a result, the inputs are sorted into input 1 to input 4. The converter 103 scales the number of pixels to any size. The buffer 104 buffers the data output from the converter 103 and can output it in any order.

[0044] The pixel synthesis unit 105 reads pixel data from the buffer 104 in the order of output of the entire screen, and outputs control signals to each pixel in the order of the scan lines. This timing is as described in the first embodiment. The pixel synthesis unit 105 may add any control signal to the blanking portion of the screen.

[0045] The image synthesis device 10 of this embodiment is equipped with an input control unit 101, a crossbar switch 102, and a converter 103, and is therefore capable of synthesizing the video signals of asynchronous inputs a, b, c, and d into a single output image having partial screens S1 to S4.

[0046] The image synthesizing device 10 of this embodiment includes the buffer 104 and the pixel synthesizing unit 105, and is therefore capable of displaying N input image signals on one screen.

[0047] The crossbar switch 102 and the converter 103 may be connected in reverse to the inputs (a, b, c, d, ...). That is, the converter 103 may enlarge or reduce the video signals input from the inputs a, b, c, and d, and then the crossbar switch 102 may rearrange and output the signals in the order of input.

[0048] Sixth Embodiment In the above-described embodiment, an example was shown in which the output frame rate and the input frame rate were the same, but the output frame rate and the input frame rate may be different. In particular, by making the output frame rate faster than the input frame rate, it is possible to reduce the average delay time.

[0049] Furthermore, in the above-described embodiment, there are four inputs and one output, but any number of inputs and outputs may be used.

[0050] Furthermore, although the above-described embodiment shows an example of combining four video signals with different timings into one output video, the present disclosure is not limited to this and can be applied to any form of generating an output video for displaying multiple cameras on a display device with fewer images than the number of cameras, such as reducing and fitting one image into part of another image.

[0051] As described above, the image synthesizing device 10 of the present disclosure provides a substitution-prohibited area and allows substitution only for the substitution-permitted area outside the substitution-prohibited area, thereby reducing the delay time until the output of the synthesized image without compromising the user experience. This enables collaborative work with strict low-latency requirements in a system that synthesizes multiple screens from multiple locations, etc.

[0052] 10: Video synthesizer 11: Input unit 12: Synthesizer 101: Input control unit 102: Crossbar switch 103: Converter 104: Buffer 105: Pixel synthesizer

Claims

1. A video synthesis device comprising: an input unit to which a video signal is input; and a synthesis unit that generates an output video by performing screen synthesis of a plurality of video signals input to the input unit, wherein the synthesis unit is capable of performing the screen synthesis of video signals input to the input unit at different times, and the times at which video signals capable of performing the screen synthesis in the synthesis unit were input to the input unit vary depending on the area in the output video.

2. The image synthesis device of claim 1, wherein the synthesis unit, when a video signal of a predetermined substitution-prohibited area in the output image is not input, waits for a video signal to be input to the input unit and then performs the screen synthesis; and, when a video signal of a substitution-allowed area other than the substitution-prohibited area in the output image is not input, performs the screen synthesis using a video signal previously input to the input unit.

3. The image synthesis device according to claim 2, wherein the image signal includes setting information that defines the substitution-prohibited area, and the synthesis unit sets the substitution-prohibited area in accordance with the setting information.

4. The image synthesizing device according to claim 2, further comprising a setting section which sets the substitution prohibited area based on the image signal input to the input section.

5. The video synthesis device according to any one of claims 1 to 4, wherein the input section receives a plurality of asynchronous video signals in parallel, and the synthesis section performs the screen synthesis of the plurality of video signals input in parallel.

6. A video synthesis method comprising: a step of inputting a video signal to an input unit; and a step of a synthesis unit generating an output video by performing screen synthesis of the multiple video signals input to the input unit, wherein the synthesis unit is capable of performing the screen synthesis of video signals input to the input unit at different times, and the time at which the video signals capable of performing the screen synthesis in the synthesis unit were input to the input unit varies depending on the area in the output video.

7. A video signal to be displayed on one screen, the video signal including setting information that defines a substitution-prohibited area of ​​the screen that cannot be substituted with a video signal of a different time.

Images