Video stage performance system and video stage performance provision method

Abstract

[Problem] To eliminate problems caused by timing signal mismatches, among a plurality of devices that are caused to perform playback in sync, when providing a video stage performance. [Solution] The present invention comprises: a main element playback unit that outputs a synchronization signal while playing back a main element, which is a main playback element; and a synchronization element playback unit that plays back a synchronization element, which is an element to be synchronized, in accordance with the synchronization signal, and operates in accordance with a reference timing signal input from the outside. All elements are stored in a state associated with time information in accordance with time series arrangement information that indicates the state in which the elements are arranged in chronological order, and as a result, the synchronization element played back in accordance with the synchronization signal is synchronized with the main element played back by the main element playback unit.

Description

Video stage performance system and method for providing video stage performance

[0001] The present invention relates to a video stage performance system and a method for providing a video stage performance.

[0002] In recent years, fictional characters have been marketed as virtual idols. Due to their nature, performances by these virtual idols are provided via video, such as streaming distribution over the Internet or video software such as DVDs, but sometimes pseudo-stage performances are provided that make full use of video technology.

[0003] When providing a stage performance using video and audio to listeners, it is essential that the video and audio are perceived by the listeners in synchronization with the intended timing. As a method for synchronizing video and audio, a method of superimposing a time code on the audio track of video data has been proposed (see, for example, Patent Document 1).

[0004] In the technology disclosed in Patent Document 1, an audio signal superimposed with a time code is recorded together with video as main content, and the audio signal associated with the time code is recorded as data separate from the main content. Then, when the main content is played back, the audio signal superimposed with the time code is demodulated to generate a time code, and the generated time code is synchronized with the time code associated with the audio signal to play back the audio.

[0005] Furthermore, a method of using a time code recorded for each frame of a video has been proposed as a method of synchronizing a video with other elements when providing the video to a large audience (see, for example, Patent Document 2). The technology disclosed in Patent Document 2 uses LTC (Linear Time Code) as an example of a time code, and also discloses a method of synchronizing seat motion with the video in a movie theater or the like.

[0006] Furthermore, a system has been proposed for improving the ease of handling video files to be played when providing such video stages (see, for example, Patent Document 3). Patent Document 3 defines time-series arrangement information that indicates the chronological arrangement of all music videos played on stage. While video data for each song is stored as a separate file, the video and time information for each piece of video data are associated according to the time-series arrangement information, and other synchronization elements such as audio are also associated with time information according to the time-series arrangement information, resulting in final synchronized playback.

[0007] JP 2012-114779 A JP 2015-514443 A Japanese Patent No. 6913874 A

[0008] In Patent Document 3, a main video is played back as a master element, a time code is output in response to the playback, and other elements such as audio and lighting are played back as slave elements based on the time code, thereby achieving synchronization. This is because, assuming that the resolution of the main video is FHD (Full High Definition: 1920 x 1080 pixels) or 4K (3840 x 2160 pixels) and the frame rate is 60 fps (frames per second), it is not realistic in terms of the processing capacity of the equipment to execute a process of decoding the video at a timing corresponding to the received time code at a cycle of 60 fps, so the video is the master and the other elements are slaves.

[0009] Therefore, improvements in equipment performance have made it possible to use video as a slave, that is, to decode video of FHD or higher resolution according to the input time code at a cycle of 60 fps or more. On an actual stage, a person called a manipulator may operate a DAW (Digital Audio Workstation) to adjust the timing of the start of the next song while watching the audience's reaction, and in such cases it is preferable to use the DAW's sound playback as the master and the video as the slave.

[0010] However, at 60 fps, the timing for each frame is extremely strict at approximately 0.017 seconds, and any discrepancy in timing between devices can cause problems with the playback of video according to the time code. This discrepancy in timing between devices means that although each device is equipped with a function to generate a timing signal according to a specific fps value, that timing is not necessarily completely accurate.

[0011] In video stage performances, where elements played on multiple devices must be synchronized, errors accumulate as playback progresses, especially at higher frame rates, resulting in visual glitches such as skipped frames that make the video appear discontinuous, or overlapping frames that make the video appear to freeze for a moment.

[0012] It is expected that video resolution and frame rates will continue to increase in the future, and this problem is likely to become more pronounced. However, this problem is not limited to when the element played as a slave is video, but will also be a problem when synchronizing a high-frame-rate element as a slave.

[0013] The present invention has been made in response to the above-mentioned problems, and aims to eliminate problems caused by misalignment of timing signals among multiple devices that are played back in sync when providing a video stage performance.

[0014] In order to solve the above problem, one aspect of the present invention is a video stage performance system that provides a stage performance to an audience by playing back video, and includes a main element playback unit that plays back a main element, which is a main playback element, and outputs a synchronization signal including time information corresponding to the playback timing of the main element, and a synchronization element playback unit that plays back synchronization elements, which are elements to be synchronized with the main playback element, in accordance with the synchronization signal, wherein the synchronization element playback unit operates in accordance with a reference timing signal that is an externally input signal and indicates an interval of one frame corresponding to a frame rate, and the main elements in the main element playback unit and the synchronization elements in the synchronization element playback unit are associated with playback elements and time information in accordance with time series arrangement information that indicates the state in which the elements played back in the stage performance are arranged in chronological order, and the synchronization signal output by the main element playback unit corresponds to the time information associated with the playback element, thereby the synchronization element played back in accordance with the synchronization signal is synchronized with the main element played back by the main element playback unit.

[0015] According to the present invention, when providing a stage performance using video, it is possible to eliminate problems caused by misalignment of timing signals among a plurality of devices that are to be played back in sync.

[0016] FIG. 1 is a diagram showing the overall configuration of a video stage performance control system according to an embodiment of the present invention. FIG. 2 is a diagram showing the hardware configuration of information devices included in the system according to an embodiment of the present invention. FIG. 3 is a diagram showing the data structure of time-series arrangement information according to an embodiment of the present invention. FIG. 4 is a block diagram showing the functional configuration of a DAW terminal according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of a GUI of a DAW application according to an embodiment of the present invention. FIG. 6 is a block diagram showing the functional configuration of a video player according to an embodiment of the present invention. FIG. 7 is a diagram showing an example of a GUI of a media server application according to an embodiment of the present invention. FIG. 8 is a diagram conceptually showing the data structure of a video file according to an embodiment of the present invention. FIG. 9 is a diagram showing an example of a defect due to an error in frame intervals that is a target of an embodiment of the present invention. FIG. 10 is a flowchart showing the operation of a media server application for each frame according to an embodiment of the present invention. FIG. 11 is a diagram showing information stored and used by an LTC I / F control unit according to an embodiment of the present invention. FIG. 12 is a flowchart showing the operation of an LTC I / F control unit according to an embodiment of the present invention.

[0017] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, a configuration for synchronizing high frame rate elements as slaves in a system for providing stage performances using video will be described.

[0018] Fig. 1 is a diagram showing the overall configuration of a video stage performance system 1 according to this embodiment. As shown in Fig. 1, the video stage performance system 1 according to this embodiment includes a DAW (Digital Audio Workstation) terminal 100, an LTC (Linear Time Code) splitter / shaper 200, a PA (Public Address) system 300, in-ear monitors for live bands 301, speakers 302, video synchronization systems 400a and 400b (hereinafter collectively referred to as video synchronization system 400), a synchronization reference signal generator 430, a video switcher 440, a display device 450, a preview device 460, a background video terminal 500, a background display device 510, a lighting control device 600, and a lighting device 610.

[0019] The DAW terminal 100 is configured as a PC on which DAW software is installed. The DAW terminal 100 is operated by an operator called a manipulator when a video stage is provided, and outputs audio signals to be played on the stage to the PA system 300, and also outputs LTC signals synchronized with the audio timing to the LTC divider and shaper 200.

[0020] That is, in the system according to this embodiment, the DAW terminal 100 plays back audio as a master element, and other elements such as video play back as slave elements in synchronization with the audio played back by the DAW terminal 100. In other words, in this embodiment, the audio played back by the DAW terminal 100 is the main playback element (main element), and the DAW terminal 100 functions as a main element playback unit. The LTC dividing and shaping unit 200 is a synchronization signal shaping unit that divides and shapes the LTC signal input from the DAW terminal 100 and outputs it to each slave element.

[0021] The PA system 300 is a system including audio equipment such as a mixer that distributes audio signals to audio systems such as live band in-ear monitors 301 and speakers 302. The live band in-ear monitors 301 are worn by the musicians who actually play the instruments during a stage performance, and output audio for the musicians to keep tempo. The speakers 302 output audio for the audience to hear at the venue where the stage performance is being held.

[0022] The video synchronization system 400 is a system that plays back video in accordance with the LTC signal input from the LTC divider / shaper 200, and includes a video player 410 and a video converter 420. The video synchronization system 400 is duplicated as a video synchronization system 400a and a video synchronization system 400b, each of which includes a video player 410 and a video converter 420.

[0023] The synchronous reference signal generator 430 generates a synchronous reference signal that serves as a reference for the frame rate and inputs it to the video player 410 and the video converter 420. As a result, the video player 410 and the video converter 420 play back video in accordance with the synchronous reference signal generated by the synchronous reference signal generator 430, rather than a synchronous signal that is independently generated internally. In other words, the synchronous reference signal is used as a reference timing signal that indicates the interval of one frame according to the frame rate. Since the video player 410 according to this embodiment plays back video at a frame rate of 60 fps (frames per second), the synchronous reference signal generator 430 generates and outputs a synchronous reference signal corresponding to 60 fps.

[0024] Video player 410 plays back the video file stored therein in accordance with the LTC signal, and outputs the video signal to video converter 420. Video converter 420 converts the video signal input from video player 410 into a professional video signal for display on a video stage, and outputs the signal to video switcher 440 and preview device 460.

[0025] In principle, the video switcher 440 selects the video signal input from the video converter 420 of the video synchronization system 400a, which is the main system, and outputs it to the display device 450, but if the video signal input from the video converter 420 of the video synchronization system 400a is interrupted, the video switcher 440 automatically switches to the video signal input from the video converter 420 of the video synchronization system 400b, which is the subsystem. The display device 450 is a display device that displays images for viewing by the audience at the venue of the stage performance, and is realized by a large display, projector, etc.

[0026] The preview device 460 is a display device that allows staff members who provide the video stage, such as manipulators who operate the DAW terminal 100, to check the video, and displays the video according to the video signal input from the video converter 420.

[0027] The background video terminal 500 is a terminal having the function of playing back background video, and is realized by a PC on which dedicated software is installed. The background video terminal 500 outputs a video signal of the background video using the installed software function in synchronization with the LTC signal input from the LTC dividing and shaping device 200. The video signal output by the background video terminal 500 is output to a background display device 510. The background display device 510 is a display device that displays video for viewing by the audience at the venue of a stage performance, and is realized by a large display, projector, etc.

[0028] The lighting control device 600 is a device having a function of controlling the lighting devices 610 installed in the venue of the video stage. The lighting control device 600 controls the operation of the lighting devices 610 in synchronization with the LTC signal input from the LTC dividing and shaping device 200. With this configuration, the video played by the video synchronization system 400, the background video played by the background video terminal 500, and the lighting controlled by the lighting control device 600 are controlled in synchronization with the music played by the DAW terminal 100.

[0029] In this manner, in this embodiment, the images, background images, lighting effects, etc. (hereinafter collectively referred to as slave elements) played by the image synchronization system 400, background image terminal 500, and lighting control device 600 (hereinafter collectively referred to as slave element reproducers) are synchronization elements that are synchronized with the main element, audio, and each of the playback devices is a synchronization element playback unit.

[0030] As described above, each slave element reproducer operates in synchronization with the LTC signal input via the LTC dividing and shaping device 200, but the timing is manually adjusted according to the output of the display device 450, background display device 510, and lighting device 610. In other words, the slave elements can reproduce and output video and lighting according to delay or advance settings manually set by the operator, based on synchronization with the LTC signal input from the LTC dividing and shaping device 200. This makes it possible to accommodate discrepancies in synchronized elements due to differences in the signal transmission environment of each venue where a video stage performance is provided.

[0031] 2 is a diagram showing the hardware configuration of information devices such as the DAW terminal 100 included in the system according to this embodiment. These information devices can be realized by the hardware configuration of general information processing devices, and as shown in FIG. 2, a CPU (Central Processing Unit) 10, a RAM (Random Access Memory) 20, a ROM (Read Only Memory) 30, a HDD (Hard Disk Drive) 40, and an I / F 50 are connected via a bus 80. An LCD (Liquid Crystal Display) 60 and an operation unit 70 are also connected to the I / F 50.

[0032] The CPU 10 is a computing means that controls the operation of the entire information device. The RAM 20 is a volatile storage medium that allows high-speed reading and writing of information and is used as a work area when the CPU 10 processes information. The ROM 30 is a read-only nonvolatile storage medium that stores programs such as firmware. The HDD 40 is a nonvolatile storage medium that allows reading and writing of information and stores an OS (Operating System), various control programs, application programs, etc.

[0033] The I / F 50 connects the bus 80 to various hardware components, networks, etc., and controls them. The LCD 60 is a visual user interface that allows the user to check the status of the information device. The operation unit 70 is a user interface that allows the user to input information to the information device, including a keyboard, a mouse, various hard buttons, a touch panel, etc. Of the configuration shown in Figure 2, the user interfaces such as the LCD 60 and operation unit 70 can be omitted depending on the type of device.

[0034] 2, an HDD is used as the non-volatile storage medium, but other types of storage media such as a solid-state drive (SSD) may be used depending on the configuration of each device. The I / F 50 includes a graphics processing unit (GPU) for connecting a display device such as an LCD 60. In the video player 410, the video decoding function of the GPU enables playback processing of high-resolution, high-frame-rate video in accordance with the LTC input from an external device.

[0035] In this hardware configuration, programs stored in storage media such as ROM 30, HDD 40, or an optical disk (not shown) are read into RAM 20, and the CPU 10 performs calculations in accordance with these programs, thereby forming a software control unit. The combination of the software control unit configured in this way and the hardware forms functional blocks that realize the functions of each device that makes up the video stage performance system of this embodiment.

[0036] In such a system, the gist of this embodiment is the use of a synchronization reference signal generated by a synchronization reference signal generator 430. This system synchronizes the DAW terminal 100, operated by a manipulator that controls the playback timing of music on stage, as the master, with video and other elements as slaves. As shown in FIG. 1 , the system includes the DAW terminal 100 and multiple slave elements, but naturally, each device is separate and often from different manufacturers. As a result, there is a possibility that errors may exist in the frame rate synchronization signals generated internally in each device.

[0037] Many of the images shown to the audience on stage have a high frame rate of 60 fps or more. When such high-frame-rate slave elements are played back in accordance with an externally input LTC signal, errors in the frame rate synchronization signal accumulate as playback progresses. As a result, for example, video frame skipping can cause video glitches, momentary freezes, or unnatural-looking movement. One of the gist of this embodiment is that the synchronization reference signal generator 430 can resolve these issues.

[0038] In the system according to this embodiment, when a video stage performance is provided, time-series arrangement information is used that indicates the time-series arrangement of the content, such as music and video, to be performed on that stage. The audio played by the DAW terminal 100, the video played by the video player 410, the background video played by the background video terminal 500, and the lighting effects controlled by the lighting control device 600 are all assembled according to this time-series arrangement information. Therefore, in this embodiment, the slaves operate according to the LTC signal output by the master DAW terminal 100, thereby achieving synchronization of each element.

[0039] 3 is a diagram showing the data structure of the chronological arrangement information according to this embodiment. As shown in FIG. 3, the chronological arrangement information according to this embodiment includes information on "set list number," "start timing," "content ID," and "content length."

[0040] "Set List No." is a value indicating the order of songs in the stage performance. "Start Timing" is a value indicating the start timing of each song in chronological order. "Content ID" is information that identifies each song, and is the file name or song name of the sample data provided to the audio or video operator along with chronological arrangement information.

[0041] "Content length" is a value indicating the length of each piece of music. As shown in FIG. 3, the time code, which is information indicating time and timing such as "start timing" and "content length," is information in a format indicating hours, minutes, seconds, and the number of frames, such as "01:00:0*-** / 60." This format is effective for displaying video and audio in chronological order in units of frames, according to the video frame rate.

[0042] The frame rate of video assumed by the system according to this embodiment is 60 fps. Therefore, the numerator of "** / 60" in the time information shown in the format "01:00:0*-** / 60" takes a value from 0 to 59, and when it reaches 60 it returns to 0 and the number of seconds increases by 1. In other words, the "** / 60" portion is a value that indicates the timing within one second according to the frame rate.

[0043] Because such information is included, by sharing the time series arrangement information, it becomes possible to share a time code indicating synchronization timing for those who generate each synchronization element such as audio, video, background video, lighting effects, etc. As a result, it becomes possible to associate the time code of each slave element with the LTC signal that is output in parallel with the playback of audio by the DAW terminal 100, which is the master element.

[0044] Therefore, on the day of the stage performance, the manipulator who is the operator operates the DAW terminal 100 at any timing to play back audio, and the LTC signal output from the DAW terminal 100 in accordance with this playback timing is input to the video synchronization system 400 that plays back the slave elements, etc., whereby video etc. corresponding to the time code indicated by the LTC signal is played back and synchronized with the audio played back by the DAW terminal 100. This eliminates the need for the operator to perform complex operations such as manually managing the correspondence between the video files to be played and the setting files for the timing of lighting control, etc., and makes it easier to handle materials.

[0045] Next, the functions of the DAW terminal 100 will be described with reference to Fig. 4. As shown in Fig. 4, the DAW terminal 100 according to this embodiment includes a control unit 110, an audio I / F 120, and an LTC I / F 130 in addition to the LCD 60 and operation unit 70 described in Fig. 2. The control unit 110 also includes a DAW application 111, an operation control unit 112, a display control unit 113, an audio I / F control unit 114, and an LTC I / F control unit 115.

[0046] The audio I / F 120 is hardware that outputs an audio signal when the DAW terminal 100 plays back audio using the function of the DAW application 111. As the audio I / F 120, a general audio interface with a terminal compatible with an RCA cable, a mini cable, a phone cable, an XLR cable, etc. can be used.

[0047] The LTC I / F 130 is hardware that outputs an LTC signal when the DAW terminal 100 plays back the LTC signal using the function of the DAW application 111. The DAW terminal 100 according to this embodiment generates an LTC signal as one track of multiple audio tracks managed by the DAW application 111.

[0048] Therefore, the same audio signal as that of the other audio tracks is used as the LTC signal, and the same type of LTC I / F as the audio I / F 120 can be used as the LTC I / F 130. However, by providing the audio I / F 120 and the LTC I / F 130 as separate interfaces, signal noise due to interference between the signals can be avoided, and the audio I / F 120 can achieve improved sound quality, while the LTC I / F 130 can achieve improved accuracy of the LTC signal.

[0049] The control unit 110 is configured by a combination of software and hardware, and is a control unit that controls the entire DAW terminal 100. The operation control unit 112 acquires signals of user operations on the operation unit 70, and transmits them to software modules running on the DAW terminal 100, such as the DAW application 111. The display control unit 113 displays on the LCD 60 a GUI (Graphical User Interface) of software running on the DAW terminal 100, such as the OS of the DAW terminal 100 and the DAW application 111.

[0050] The DAW application 111 is software that provides various audio-related functions, and in this embodiment, performs playback processing of various types of audio, including audio provided to the audience during a stage performance. The DAW application 111 according to this embodiment also outputs an LTC signal in response to audio playback, for synchronizing a slave element with the audio played by the DAW terminal 100.

[0051] The audio I / F control unit 114 controls the audio I / F 120 to output an audio signal in accordance with audio playback processing by the DAW application 111. The audio I / F control unit 114 is configured as driver software for controlling the audio I / F 120, which is configured as hardware that outputs an audio signal.

[0052] The LTC I / F control unit 115 controls the LTC I / F 130 to output the LTC signal in accordance with the output processing of the LTC signal by the DAW application 111. As described above, the LTC signal according to this embodiment is an audio signal, being one of multiple audio tracks managed by the DAW application 111. Therefore, the LTC I / F control unit 115, like the audio I / F control unit 114, is configured as driver software for controlling the LTC I / F 130, which is an audio interface.

[0053] Fig. 5 is a diagram showing an example of a GUI of the DAW application 111. The DAW application 111 is software that integrates audio-related functions, and as shown in Fig. 5, multiple audio tracks are arranged in chronological order. Also, as shown in Fig. 5, the DAW application 111 according to this embodiment imports the chronological arrangement information described in Fig. 3 to visually display the period in which songs and content are arranged in the chronological arrangement information. In Fig. 5, the period in which songs are arranged in the chronological arrangement information is indicated by a shaded area.

[0054] 5, an "LTC TRACK" is provided at the end of each track. As described above, an audio signal is used as the LTC signal in the DAW terminal 100 according to this embodiment. That is, the DAW application 111 outputs an audio signal indicating a time code according to a specific format as one of the audio tracks. Each slave element device that receives this LTC signal can recognize and decode the audio signal according to the same format, thereby recognizing the timing corresponding to the LTC signal.

[0055] By using an audio playback device such as the DAW terminal 100 as a master element playback device, it becomes easy to use an LTC signal, which is generally realized by an audio signal, as a synchronization signal for synchronizing slave elements. As a result, the synchronization signal output by the master element playback device can be input to the slave element playback device without converting the signal format, thereby simplifying the configuration.

[0056] The PA system 300 according to this embodiment has multiple channels corresponding to the audio tracks of the DAW terminal 100, and can output audio to individual output destinations such as speakers 302 and live band in-ear monitors 301 according to the channels. As described in FIG. 5 , multiple audio tracks are associated with a timeline in the DAW application 111. One of these multiple audio tracks is a clock sound that is used by musicians who actually play musical instruments in a stage performance to keep tempo. This clock sound is output only to the channel of the PA system 300 that corresponds to the live band in-ear monitors 301.

[0057] Furthermore, in the above example, an example has been described in which audio corresponding to each content is arranged in time series in the DAW application 111 according to the imported time series arrangement information, but this is not limiting, and the reverse may also be possible depending on the production environment. That is, a state in which each content is arranged in time series in the DAW application 111 may first be constructed, and the time series arrangement information may be generated based on that result, as shown in Figure 3. The time series arrangement information generated in this manner can be shared with producers who construct slave elements such as video, background video, and lighting, thereby achieving the same effect.

[0058] Next, the functional configuration of the video player 410 included in the video synchronization system 400 will be described with reference to Fig. 6. As shown in Fig. 6, the video player 410 according to this embodiment includes a control unit 411, a network I / F 416, an AV I / F 417, a REF I / F 418, and an LTC I / F 419. The control unit 411 also includes a media server application 412, a network control unit 413, a REF I / F control unit 414, and an LTC I / F control unit 415.

[0059] The control unit 411 is configured by loading software for configuring each functional block into the RAM 20 and operating the CPU 10 in accordance with that software. The media server application 412 is application software that handles video functions based on the LTC signal and synchronization reference signal in the video player 410, and outputs a played video signal via an AV I / F realized by an HDMI (High-Definition Multimedia Interface) (registered trademark) terminal or the like.

[0060] The network control unit 413 controls a network I / F 416 configured by an Ethernet terminal, etc. The REF I / F control unit 414 controls a REF I / F 418, which is an interface for receiving a synchronization reference signal, and transmits the input synchronization reference signal to the media server application 412.

[0061] The LTC I / F control unit 415 controls the LTC I / F 419, which is an interface for receiving LTC signals, and transmits the input LTC signals to the media server application 412. As described above, the LTC signals according to this embodiment are audio signals, and the LTC I / F 419 uses a terminal for transmitting audio, such as an RCA terminal.

[0062] Next, the GUI of the media server application 412 (hereinafter referred to as the media server GUI) that allows the operator to operate the video player 410 will be described with reference to Fig. 7. The video player 410 according to this embodiment is operated as a media server, and its operation is performed via a LAN or the Internet via a network I / F 416. Therefore, the media server GUI is not installed in the video player 410, but in another PC or the like that manages and operates the video player 410, and communicates with the video player 410 via a LAN, the Internet, or the like to acquire information and operate.

[0063] As shown in Fig. 7, the media server GUI has a column at the bottom that displays information about video frames in chronological order, and a column at the top that displays an enlarged image of the video frame at the currently selected timing in the chronological order information. As shown in Fig. 7, "01:00:00" to "01:05:00", the left-right direction represents the chronological order. The media server GUI also imports the chronological arrangement information described in Fig. 4, thereby visually displaying the period in which songs and content are arranged in the chronological arrangement information.

[0064] In the video track display area, images of video frames corresponding to the time series in the left-right direction are displayed. As described above, the frame rate of the video in this embodiment is 60 fps, and since it is not realistic to display all video frames, the frames are thinned out and displayed at a predetermined frequency.

[0065] 7, only a single track is shown as an example of a video track, but multiple video tracks may be displayed overlapping each other. In video stage performances, CG character images are mainly used, but other visually displayed images such as hand-drawn animation images and filmed live-action images can also be applied.

[0066] 7, the time series starts from "01:00:00", such as "01:00:00" to "01:05:00". This is in accordance with the conventions in the video production industry, but in order to obtain the effect of the present patent, it is acceptable to start from "00:00:00".

[0067] 8 is a diagram showing the data structure of video data managed by the media server application 412 in the video player 410. As shown in FIG. 8, the video data according to this embodiment is managed by associating "video" with "time code" in chronological order according to a predetermined frame rate. As described above, the frame rate according to this embodiment is 60 fps.

[0068] Here, the problem of fps mismatch between devices, which is the subject of this system, will be described with reference to FIG. 9 . FIG. 9 shows the 60 fps timing of the LTC signal output by the DAW terminal 100 in the upper part, and the 60 fps timing based on the clock generated internally by the video player 410 in the lower part. As shown in the figure, although both are based on timing corresponding to 60 fps, there may be an error in the interval corresponding to one frame. The example in FIG. 9 shows a case where the interval between one frame in the video player 410 is slightly shorter than the interval between one frame in the DAW terminal 100.

[0069] 9, the 23rd frame on the video player 410 corresponds to the 22nd frame on the DAW terminal 100, resulting in a one-frame shift. If different devices are operated in synchronization with such a one-frame shift, the same frame will be played back multiple times on the video player, causing the video to appear to freeze for a moment.

[0070] For example, the LTC signal input at the start timing of the 24th frame on the video player 410 side corresponds to the 23rd frame on the DAW terminal 100 side, so the video player 410 plays back video corresponding to the 23rd frame. Then, the LTC signal input at the start timing of the next frame, the 25th frame on the video player 410 side, still corresponds to the 23rd frame on the DAW terminal 100 side, so the video player 410 again plays back video corresponding to the 23rd frame.

[0071] 9 , if the interval between frames on the video player 410 is slightly longer than the interval between frames on the DAW terminal 100, this will appear as a skip of frames rather than an overlap of frames, resulting in the video appearing to have skipped movements.

[0072] In contrast to this, the system according to this embodiment operates by determining the interval of one frame according to the synchronization reference signal input from the synchronization reference signal generator 430, rather than the timing of the frame interval generated internally by the device that plays the slave elements, such as the video player 410, and therefore can avoid the problem shown in FIG. 9.

[0073] Fig. 10 is a flowchart showing the operation per frame of the media server application 412. As shown in Fig. 10, the media server application 412 acquires a synchronization reference signal via the REF I / F control unit 414 and recognizes the start timing of one frame (S1001), thereby starting the operation per frame.

[0074] The media server application 412, which has started operating per frame, acquires the LTC signal input via the LTC I / F control unit 415 (S1002), decodes it, and recognizes the time code (S1003). Having recognized the time code, the media server application 412 reads out the video data associated with the time code, as shown in Fig. 8, performs decoding processing to output it as a video signal in HDMI (registered trademark) format (S1004), and outputs it as a video signal (S1005).

[0075] This processing completes one frame of operation of the media server application 412. By repeating this processing at 60 fps, that is, every 1 / 60 seconds, video is played back in synchronization with the audio played back by the DAW terminal 100 that outputs the LTC signal.

[0076] Next, we will explain the LTC signal recognition process performed by the LTC I / F control unit 415 in the video player 410. As mentioned above, the LTC signal according to this embodiment is an analog audio signal. Therefore, there is a non-zero possibility that noise may be superimposed depending on the quality of the cable transmitting the signal or the surrounding environment, resulting in a change in the time code indicated by the signal and frames skipping rather than being sequential.

[0077] Such defects last for a few frames at most, or less than 1 / 10 of a second, but have an impact on the displayed image that can be recognized as a defect. Therefore, while it is preferable to ignore skips of a few frames and recognize them as consecutive frames when playing the image, there are cases where the skipped frames are correct depending on the operation of the manipulator between songs or the adjustment of the playback order of songs, and in such cases the image must be played according to the skipped frames.

[0078] The LTC I / F control unit 415 according to this embodiment has a frame skip determination function that responds to such requests. Fig. 11 is a diagram showing information that the LTC I / F control unit 415 holds for frame skip determination. As shown in Fig. 11, the LTC I / F control unit 415 holds information on "fps setting value," "latest recognized value," "previous output value," and "previous recognized value." Each piece of information will be described in detail in the explanation of operation.

[0079] 12 is a flowchart showing the operation of the LTC I / F control unit 415 according to this embodiment regarding frame skip determination. As shown in FIG. 11, when the LTC I / F control unit 415 receives an LTC signal via the LTC I / F 419, it decodes the LTC signal to recognize the time code (S1101), and stores the information as the "latest recognized value" in a storage medium such as the RAM 20 or the HDD 40 to hold it.

[0080] The LTC I / F control unit 415 then checks whether the "latest recognized value" is a time code indicating the frame next to the "previous output value" (S1202). The time code according to this embodiment is in the format of hour, minute, second, and number of frames. Therefore, in the process of S1202, it is possible to directly check whether the frames are progressing in chronological order.

[0081] On the other hand, for example, in the case where the time code is in a format in which the number of seconds is indicated up to the decimal point, the LTC I / F control unit 415 determines an expected change value for one frame based on the "fps setting value," calculates an expected time code based on that value and the "previous output value," and compares it with the "latest recognized value" to make the determination in S1202.

[0082] The expected change value here is the number of seconds equivalent to one frame of the "fps setting value." In this case, the determination in S1202 does not necessarily have to be a strict one that determines an exact match, but rather it may be sufficient to determine whether or not the value is within the expected range, allowing for a certain degree of error. In any case, the process of S1202 only needs to detect that the input LTC signal has changed in a way that differs from continuous time-series changes, and is executed appropriately depending on the information format of the time code.

[0083] If the result of the determination in S1202 is that the "latest recognized value" indicates the frame next to the "previous output value" (S1202 / YES), the LTC I / F control unit 415 recognizes that the input LTC signal is correct, resets a skip determination counter for determining frame skips (S1203), outputs the time code of the "latest recognized value" (S1204), and repeats the processing from S1201. At this time, the "previous output value" and the "previous recognized value" are updated by the "latest recognized value."

[0084] On the other hand, if the "latest recognized value" is not a time code indicating the frame next to the "previously output value" (S1202 / NO), the LTC I / F control unit 415 performs processing to determine whether the jump in the LTC signal is unintentional due to noise or intentional due to the operation of the manipulator. Therefore, the LTC I / F control unit 415 checks whether the "latest recognized value" is a time code indicating the frame next to the "previously recognized value" (S1205). The processing of S1205 is similar to the processing of S1202, but is performed by comparing it with the "previously recognized value," and is also executed appropriately depending on the format of the time code.

[0085] If the result of this determination is that the "latest recognized value" is the time code indicating the frame next to the "previously recognized value" (S1205 / YES), the LTC I / F control unit 415 recognizes that the skip in the input LTC signal may be intentional, and increments the skip determination counter (S1206). After incrementing the value of the skip determination counter, the LTC I / F control unit 415 compares the value with a predetermined threshold value (S1207).

[0086] As a result, if the value of the skip determination counter is less than the threshold value (S1207 / YES), the LTC I / F control unit 415 determines that the unexpected change in the LTC signal determined in S1202 is still within the realm of possibility of being temporary and unintended.The LTC I / F control unit 415 then stores the time code recognized by the processing of S1201 as the "previously recognized value," and stores the time code indicating the frame next to the "previously output value" as a new "previously output value," outputs that time code (S1208), and repeats the processing from S1201.

[0087] On the other hand, if the value of the skip determination counter is equal to or greater than the threshold value (S1207 / NO), the LTC I / F control unit 415 determines that the unexpected change in the LTC signal determined in S1202 is a legitimately intended change, such as an operation by a manipulator or a change in the track order, and resets the skip determination counter (S1203), performs the processing of S1204, and repeats the processing from S1201.

[0088] Furthermore, if the result of the determination in S1205 is that the "latest recognized value" is not a value indicating the frame next to the "previously recognized value" (S1205 / NO), the LTC I / F control unit 415 determines that the LTC signal is changing irregularly, resets the skip determination counter (S1209), performs the processing of S1208, and repeats the processing from S1201.

[0089] The LTC I / F control unit 415 repeats the process shown in Fig. 12 at a frame rate determined by the "fps setting value." Therefore, the determination in S1207 in Fig. 12 is equivalent to determining whether, after an unexpected change in the time code value indicated by the LTC signal received by the LTC I / F control unit 415, the change in the value after the change continues for a predetermined number of frames, as expected based on the fps. This predetermined number of frames is the threshold value in S1407. This threshold value is set appropriately based on the expected noise level in the environment and the actual interval between songs, but is, for example, one second, which is 60 for 60 fps.

[0090] 10 and 12, the video played by the video player 410 is synchronized with the audio played by the DAW terminal 100. The background video terminal 500 and the lighting control device 600 also operate based on the same functions as the video player 410, so that the background video and time-series lighting effects are synchronized with the audio played by the DAW terminal 100.

[0091] As described above, the video synchronization system 400 according to this embodiment operates based on an externally input synchronization reference signal rather than on the timing of internally generated frames when playing back a slave element synchronized with a master element. This makes it possible to avoid playback problems caused by an error even if there is an error in the timing of frame intervals between a device playing back the master element and a device playing back the slave element.

[0092] In the above embodiment, an example has been described in which the synchronization reference signal generator 430 supplies a synchronization reference signal to the video player 410, background video terminal 500, and lighting control device 600, which are playback devices of the slave elements. This is just one example, and the synchronization reference signal needs to be supplied only to devices that have an error in the timing of frames generated internally compared to the DAW terminal 100, which is the playback device of the master element. Therefore, the synchronization reference signal may be supplied only to some of the slave elements, rather than to all of the slave elements.

[0093] However, this system is most effective when there are errors in the frame intervals of slave elements with high frame rates, such as 60 fps or higher, particularly in images that are easily visually recognizable. Therefore, the effectiveness of this system can be maximized by supplying a synchronization reference signal to a slave element that plays back high frame rate images.

[0094] 1 assumes that the frame interval of the synchronization reference signal generated by the synchronization reference signal generator 430 matches the frame interval of the DAW terminal 100, which is the master element playback device, or that the error is so small that the accumulated error can be ignored even during long periods of continuous playback. Such a premise can be easily realized by selecting the model of the synchronization reference signal generator 430 and setting the synchronization reference signal to be output.

[0095] However, the present invention is not limited to this, and a synchronization reference signal may also be supplied to the DAW terminal 100, which is the master element playback device, and the DAW terminal 100 may also determine and operate the frame interval based on the synchronization reference signal. This eliminates the need to assume that the frame interval error between the master element playback device and the synchronization reference signal generator 430 is extremely small, allowing for greater flexibility in configuration.

[0096] In the above embodiment, the slave elements synchronized with the audio played by the DAW terminal 100, which is the master element, are described as being the video played by the video player 410, the background video played by the background video terminal 500, and the lighting effects controlled by the lighting control device 600. However, this is just one example, and any elements that are synchronized and played back in a stage performance can be similarly applied. Other elements include special effects such as confetti, camera work, and special effects such as seat vibration.

[0097] In addition, as described with reference to FIG. 3, the content set in the time series arrangement information according to this embodiment is, for example, the songs included in the set list to be performed in a video stage performance. However, in a stage performance presented to an audience by playing back video, the MC (microphone commentary) between songs is also provided by video. Therefore, the content set in the time series arrangement information includes not only the songs but also the video of the MC. In such a case, the devices responsible for playing back the master and slave elements prepare and store the audio, video, background video, etc. corresponding to the MC according to the time series arrangement information.

[0098] 1 Video stage performance system 10 CPU 20 RAM 30 ROM 40 HDD 50 I / F 60 LCD 70 Operation unit 80 Bus 100 DAW terminal 110 Control unit 111 DAW application 112 Operation control unit 113 Display control unit 114 Audio I / F control unit 115 LTC I / F control unit 120 Audio I / F 130 LTC I / F 200 LTC splitter / shaper 300 PA system 301 Live band in-ear monitor 302 Speaker 400, 400a, 400b Video synchronization system 411 Control unit 412 Media server application 413 Network control unit 414 REF I / F control unit 415 LTC I / F control unit 416 Network I / F 417 AV I / F 418 REF I / F 419 LTC I / F 410 Video player 420 Video converter 430 Synchronous reference signal generator 440 Video switcher 450 Display device 460 Preview device 500 Background video terminal 510 Display device 600 Lighting control device 610 Lighting device

Claims

1. A video stage performance system that provides a stage performance to an audience by playing back video, comprising: a main element playback unit that plays back a main element, which is the main playback element, and outputs a synchronization signal including time information corresponding to the playback timing of the main element; and a synchronization element playback unit that plays back synchronization elements, which are elements to be synchronized with the main playback element, in accordance with the synchronization signal; wherein the synchronization element playback unit operates in accordance with a reference timing signal that is an externally input signal and indicates an interval of unit frames corresponding to a frame rate; the main elements in the main element playback unit and the synchronization elements in the synchronization element playback unit are associated with playback elements and time information in accordance with time series arrangement information that indicates the state in which elements played back in the stage performance are arranged in chronological order; and the synchronization signal output by the main element playback unit corresponds to the time information associated with the playback element, thereby synchronizing the synchronization element played back in accordance with the synchronization signal with the main element played back by the main element playback unit.

2. The video stage performance system according to claim 1, wherein the main element playback unit is a device that plays back audio and outputs the synchronization signal as one track of a plurality of audio tracks.

3. A video stage performance system according to claim 2, wherein said main element reproduction section outputs the audio to be heard by the audience and said synchronization signal from different terminals.

4. A video stage performance system according to claim 1, wherein said main element reproduction section operates in accordance with said reference timing signal input to said synchronous element reproduction section.

5. A video stage performance system according to claim 1, wherein among the plurality of synchronization element playback units, the one that operates in accordance with the reference timing signal is a device that plays back video.

6. A video stage performance system as claimed in claim 1, characterized in that it includes a synchronization signal shaping unit which divides and shapes the synchronization signal output from the main element reproduction unit and outputs it to a plurality of synchronization element reproduction units.

7. The video stage performance system described in claim 1, characterized in that the synchronization element playback unit plays back the synchronization element using time information obtained by adding a time series change value to the time information before the change as a synchronization signal when the time information indicated by the input synchronization signal changes differently from a continuous time series change, and plays back the synchronization element using the time information after the change as a synchronization signal when the time information indicated by the input synchronization signal changes differently from a continuous time series change and the time information after the change continues to change in a continuous time series for a predetermined period.

8. A method for providing a video stage performance in which a stage performance is presented to an audience by playing back a video, comprising: outputting a synchronization signal containing time information according to the playback timing of a main element, which is the main playback element, along with the playback of the main element; recognizing frame intervals according to a reference timing signal which is an externally input signal and indicates the interval of unit frames according to a frame rate, and playing back a synchronization element, which is an element to be synchronized with the main playback element, according to the synchronization signal; the main element and the synchronization element are associated with the playback element and time information according to time series arrangement information which indicates the state in which the elements played in the stage performance are arranged in time series; and the synchronization signal corresponds to the time information associated with the playback element, thereby synchronizing the synchronization element played back according to the synchronization signal with the main element.

Images