VIDEO PLAYBACK SYSTEM, MEDIA PLAYBACK SYSTEM, METHOD FOR SYNCHRONOUSLY PLAYING BACK A VIDEO DATA STREAM OF AN AUDIO-VIDEO DATA STREAM, AND COMPUTER-READABLE STORAGE MEDIUM

Abstract

The present invention relates in particular to a video playback system (10) adapted to play back an audio-video data stream (61, 66) having an audio data stream (61) including a time code signal (62, 67) and a video data stream, the video playback system (10) comprising a synchronization unit (20) and a number of head-mounted displays (50, 52, 53), in particular augmented reality headsets, the synchronization unit (20) receiving the audio data stream (61, 66) from an audio data source (30) and transmitting the audio data stream (61, 66) to a sound system (30). (40), converting time code signals (62, 67) of the audio data streams (61, 66) into timestamp signals (25), and transmitting the timestamp signals (25) to a plurality of head-mounted displays (50, 52, 53), each of the head-mounted displays (50, 52, 53) configured to receive the timestamp signals (25), play a video data stream, and synchronize the playback of the video data stream with the audio data streams (61, 66) using the timestamp signals (25).

Description

Detailed Description of the Invention

[0001] [explanation] The present invention relates to a video playback system configured to play a video data stream of an audiovisual data stream having an audio data stream including a time code signal and a video data stream. The present invention also relates to a media playback system configured to play an audiovisual data stream having an audio data stream including a time code signal and a video data stream. The present invention also relates to a method for synchronously playing a video data stream of an audiovisual data stream having an audio data stream including a time code signal and a video data stream. Finally, the present invention relates to a computer-readable storage medium comprising instructions, which, when executed by a computer, cause a media playback system to perform a method for synchronously playing video data streams.

[0002] Augmented reality (XR) allows users to experience a world other than the one we normally perceive with our natural senses. This experience is enhanced by computers and wearable devices. The term XR is typically used as a general term for any environment that blends the real and the virtual, such as augmented reality (AR), mixed reality (MR), or virtual reality (VR).

[0003] XR hardware and software applications are usually designed for a single user. Typically, video data is displayed on a head-mounted display and audio data is played back through personal headphones that are either directly integrated with or connected to the head-mounted display. XR applications often use binaural sound, giving the listener the sensation of being in a room with a separate sound source.

[0004] The use of head-mounted displays and headphones can create an isolating or even claustrophobic experience for users of XR devices, which often limits usage time and reduces the user's enjoyment of the entertainment experience.

[0005] Additionally, the head-mounted display and headphones need to be rigorously calibrated so that the images displayed by the head-mounted display and the audio played back by the headphones are both precisely synchronized with the user's head movements. In some settings, the XR device requires continuous computational effort to (re)compute the appropriate audio signals corresponding to the user's head orientation.

[0006] Different sound analysis methods are known for synchronizing the audio and video playback of a single audiovisual data stream on separate devices, e.g. a display or screen for the video and headphones or a sound system for the audio.

[0007] U.S. Patent No. 10,178,487 discloses a method for presenting binaural audio. In response to an audio synchronization signal presented to a mobile device, audio information is played back from the mobile device in synchronization with the display of a movie on a theater screen. The audio synchronization signal may be a high frequency signal emitted from speakers in the theater.

[0008] EP 2628047 B1 discloses another system and method for providing alternative audio for combined video and audio, in particular for movies and television programmes, in which a position code is embedded in the audio signal as an audio watermark.

[0009] The object of the present invention is to improve the sound quality and user experience of XR devices.

[0010] This object is solved by a video playback system configured to play a video data stream of an audio-video data stream having an audio data stream including a time code signal and a video data stream, the video playback system comprising a synchronization unit and a plurality of head-mounted displays, in particular augmented reality headsets, characterized in that the synchronization unit is configured to receive the audio data stream from an audio data source, transmit the audio data stream to a sound system, convert the time code signal of the audio data stream into a timestamp signal and transmit the timestamp signal to the plurality of head-mounted displays, each of which is configured to receive the timestamp signal, play the video data stream and synchronize the playback of the video data stream with the audio data stream using the timestamp signal.

[0011] The video playback system is based on the following consideration: the playback of an audiovisual data stream is performed by two separate devices, each best suited to the task, e.g., one device for playing the audio data stream and another device for playing the video data stream. This creates a need to synchronize these two devices. According to an aspect of the invention, this is achieved by intercepting the audio data stream using a synchronization unit before it is played back by a sound system. A digitally accessible timestamp signal is obtained from a time code signal contained in the audio data stream. The timestamp signal is then transmitted to a number of head-mounted displays acting as display devices for the video data stream, of which the video data stream is one component.

[0012] An advantage of a video playback system is that it can be combined with a personal, i.e., single-user, head-mounted display and a sound system, such as a surround sound system, that provides audio to multiple users simultaneously. Each user can experience a full three-dimensional XR field of view through the head-mounted display in addition to the full-range audio experience provided by a powerful speaker system. Despite significant advances in headphone design, the sound of modern headphones still falls short of the sound experience of a full sound system; for example, a full-size subwoofer reproduces bass frequencies better than headphones. Additionally, using an external sound system instead of personal headphones allows users to have a more open experience and a participatory entertainment experience with other audiences.

[0013] According to aspects of the present invention, the audiovisual data stream comprises an audio data stream and a video data stream, i.e., an audio component and a video component. According to one embodiment, the audiovisual data stream is a movie, a television program, or a 360° video including audio, where the 360° video provides a view in all directions but does not allow interaction with a virtual environment. In a further embodiment, the audiovisual data stream is a live audiovisual production, particularly of a concert, a musical, a sporting event, an e-sports event, an exhibition, or a conference, or any kind of live interactive media. In a further embodiment, the video data stream is an XR data stream, particularly a VR data stream. In a further embodiment, the video data stream includes three-dimensional video data. In another embodiment, the audio data stream is an analog audio data stream. In yet another embodiment, the audio data stream is a digital audio data stream.

[0014] The audio data stream includes a time code signal. In the present context, the term "time code signal" refers to any timing information included in the audio data stream. In particular, the term time code signal refers to a standard time code used in cinema and / or TV and / or film and / or video production. In particular, the time code signal is a time code based on the Society of Motion Picture and Television Engineers (SMPTE) standard and / or based on other standards, such as the MIDI standard. The use of a standard time code allows for flexibility in the use of video playback systems. Using other timing information present in the audio data stream as a time code signal has the advantage that the video playback system is independent of the incoming audio data stream.

[0015] In one embodiment, the synchronization unit is a computer hardware device, particularly comprising a processing unit and / or an analog-to-digital converter and / or a digital-to-analog converter. In a further embodiment, the synchronization unit is a single-board computer. In another embodiment, the synchronization unit is configured to receive an audio data stream via a wired connection, particularly via a twisted pair cable with an RJ45 connector, and / or via a phone connector, and / or via an XLR connector. In this specification, the term "twisted pair" refers to, for example, a CAT5 or CAT6 cable. The term "phone connector," also known as a headphone jack, is a three-contact type, for example, a tip-ring-sleeve (TRS) connector. The term "XLR" refers to an electrical connector used in professional audio, as specified in the international standard IEC 61076-2-103. A wired connection provides a stable, high-throughput audio data stream as input to the synchronization unit.

[0016] According to another aspect of the present invention, the synchronization unit is configured to convert a time code signal of the audio data stream into a time stamp signal. In one embodiment, the synchronization unit is configured to extract the time code signal from the audio data stream and process the time code signal into a time stamp signal. In particular, the time stamp signal is a digital signal. In one embodiment, the time stamp signal is in a time format, while in another embodiment, the time stamp signal is in a cinema-based format, for example including a frame number. Using a digital time stamp signal in a format tailored to the needs of each head-mounted display allows for efficient and reliable synchronized playback of the audiovisual data stream.

[0017] The synchronization unit is configured to transmit the timestamp signal to the plurality of head-mounted displays, and in one embodiment, the synchronization unit is configured to transmit the timestamp signal at predetermined time intervals. In a further embodiment, the predetermined time intervals are regular or irregular. In a further embodiment, the synchronization unit is configured to transmit the timestamp signal to the plurality of head-mounted displays upon request of each of the head-mounted displays. By transmitting the timestamp signal at predetermined time intervals or upon request, an efficient synchronization process can be achieved that is tailored to the specific needs of the video playback system, in particular the number of head-mounted displays.

[0018] In an advantageous embodiment, the head mounted displays are XR headsets. In a further embodiment, the head mounted displays are configured to display a video data stream of an AR and / or MR and / or VR audiovisual data stream. In a further embodiment, each head mounted display comprises an inertial measurement unit for determining the orientation and / or position of the head mounted display in a given coordinate system. Preferably, the given coordinate system is related to a room, such as a cinema hall or a living room. In a further embodiment, each head mounted display comprises an additional tracking unit, such as an eye tracker or a head tracker. In a further embodiment, each head mounted display comprises a processing unit, such as a microprocessor, for processing the timestamp signal.

[0019] In further embodiments, each head mounted display plays the video data stream either as monoscopic 2D, i.e. one image to each eye, or as stereoscopic 3D, i.e. a different image to each eye, which provides an optimal XR experience.

[0020] In a further embodiment, each of the head mounted displays is configured to synchronize playback of the video data stream with the audio data stream using the timestamp signals, and each of the head mounted displays is configured to display a particular portion of the video data stream that corresponds to the timestamp signal received from the synchronization unit, such that each of the head mounted displays is configured to play video frames of the video data stream at the same time that corresponding audio signals of the audio data stream are played.

[0021] In yet another embodiment, each head-mounted display can be calibrated with respect to its spatial position and / or rotational orientation within the room. In a further embodiment, the video playback system includes additional sensors and / or beacons to assist in the calibration, for example, such additional sensors are located in the same room as the head-mounted displays. This allows the direction of video playback by the head-mounted displays to be aligned with the direction of playback of the audio data stream by the sound system, which is particularly important for XR media content.

[0022] In another embodiment, the video playback system further comprises an audio data source configured to provide an audio data stream to the synchronization unit, the audio data source being configured to receive the live stream as an audio data stream and / or the audio data source being configured to retrieve the audio data stream from a data store, in particular configured to decode the audio data stream from Digital Cinema Package (DCP) data. In this way, the video playback system is suitable for playback of live streams and / or locally stored audiovisual data, in particular cinema content compliant with the industry standard DCP.

[0023] In another preferred embodiment, the audio data source is configured to decode another digital cinema data format. In another preferred embodiment, the audio data source is configured to provide an audio data stream without decoding and / or decrypting data. In a further embodiment, the audio data source is configured to receive an audio data stream, particularly a live data stream, over the Internet. In another embodiment, the audio data source is configured to generate an audio data stream from an audio data memory device.

[0024] In a further embodiment, each of the head mounted displays comprises a data store having stored therein video data, and each of the head mounted displays is configured to generate a video data stream from the video data stored in the data store, which has the advantages of increasing the availability of the video data stream and reducing the energy consumption of each of the head mounted displays, thus extending battery life.

[0025] In further embodiments, the video data stored on the data storage device is cinematic and / or 360° video data and / or XR video data and / or other video data.

[0026] In a further embodiment, the time code signal is an analog signal and the time stamp signal is a digital signal, and the synchronization unit is configured to convert the analog time code signal into a digital time stamp signal, and in particular the synchronization unit is configured to detect errors in the analog time code signal, in particular errors related to consistency, and to correct the time code signal using an error correction scheme. Advantageously, the video playback system is technically compatible with existing systems. Digital signals allow for easier processing and more efficient operation. Detecting and correcting errors avoids errors in synchronization between the audio and video data streams due to poor data connections.

[0027] In a further embodiment, the audio data stream comprises at least four audio channels, particularly at least eight audio channels, particularly at least 12 audio channels, and even more particularly at least 16 audio channels, one of which carries a time code signal, particularly an SMPTE time code signal, and even more particularly an SMPTE-LTC time code signal. The LTC time code signal is a linear time code signal, also known as a longitudinal time code signal. In another embodiment, the time code signal is a MIDI time code. Multiple channels allow the user to enjoy a surround sound audio experience with higher listening quality. By placing the time code signal in a separate channel within the audio data stream, it can be easily detected within the audio data stream. This ensures that conversion to a timestamp signal is fast and reliable.

[0028] In a further embodiment, the synchronization unit is configured to send the audio data stream to the sound system unmodified, i.e., without any changes. In another embodiment, the synchronization unit is configured to filter and / or process the audio data stream before sending it to the sound system. By not making any changes, the synchronization unit is completely transparent to the audio data stream and does not interfere with the audio playback. Also, by processing the audio data stream, the synchronization unit can optimize the audio data stream, for example, to tailor it to the specific needs of the sound system, the user, or the venue.

[0029] In yet another embodiment, the synchronization unit includes a time stamp unit and a network unit, where the time stamp unit is configured to receive an audio data stream, transmit the audio data stream, and convert a time code signal of the audio data stream into a time stamp signal, and the network unit is configured to establish a wireless network connection with each of the head-mounted displays and transmit the time stamp signal from the time stamp unit to the multiple head-mounted displays via the wireless network connection. The wireless connection supports flexibility of the head-mounted displays. The wireless connection enables a better user experience, especially for XR or VR experiences.

[0030] In a further embodiment, the network unit is a computer network device, in particular a wireless router or a wireless access point. In a further embodiment, the network unit is configured to operate a wireless network connection based on Wi-Fi, in particular according to Wi-Fi 4 (IEEE 802.11n) and / or Wi-Fi 5 (IEEE 802.11ac) and / or subsequent Wi-Fi standards.

[0031] In a further embodiment, the network unit is configured to operate with at least 50, particularly at least 100, more particularly at least 150 simultaneously connected head mounted displays. In a further embodiment, the minimum range of the wireless connection from the network unit is at least 100 meters, particularly at least 200 meters, particularly at least 300 meters. These specifications allow for large audiences and large venues. In a further embodiment, each head mounted display comprises a wireless communication unit, the wireless communication unit being configured to receive the timestamp signal transmitted by the synchronization unit.

[0032] In yet another embodiment, each of the head-mounted displays is configured to request and / or read a timestamp signal from the synchronization unit, and in particular, each of the head-mounted displays is configured to request and / or read a timestamp signal from the synchronization unit at predetermined time intervals. By requesting a timestamp signal from the synchronization unit as needed, the synchronization process is efficient without requiring push transmission from the synchronization unit.

[0033] In a further embodiment, the synchronization unit comprises a timestamp server configured to provide a timestamp signal for each of the head mounted displays, in particular each of the head mounted displays configured to request and / or read a timestamp signal from the timestamp server.

[0034] In further embodiments, the predetermined time interval or requesting and / or reading the timestamp signal from the synchronization unit is every frame, or multiple times per second, or approximately once per second, or once per second, or once every multiple seconds, or less frequently. In further embodiments, the predetermined time interval is a constant time interval.

[0035] In a further embodiment, the synchronization unit and each of the head-mounted displays are configured to recalibrate the timestamp signal to compensate for individual, particularly variable, transmission delays of the timestamp signal from the synchronization unit to each of the head-mounted displays, and in particular, each of the synchronization unit and each of the head-mounted displays has an internal clock for determining a clock signal referenced to a time standard, and the synchronization unit is configured to transmit the clock signal together with the timestamp signal to the multiple head-mounted displays, and each of the head-mounted displays is configured to receive the clock signal, determine an individual transmission delay from the clock signal, and recalibrate the timestamp signal to compensate for the individual transmission delay.

[0036] Since a network connection between two separate devices, especially when connected via a wireless connection, introduces a network delay that can vary by tens of milliseconds, it is advantageous to compensate for this transmission delay. Since the transmission delay is an individual amount for each connection between the synchronization unit and each head-mounted display, the individual transmission delay is determined separately for each head-mounted display. This allows for better synchronization of the audio data stream and the video data stream of the audio-video data stream.

[0037] In a preferred embodiment, the internal clock is used to determine a clock signal that is referenced to a time standard, preferably having a time of day, for example the local time of the synchronization unit or Coordinated Universal Time (UTC).

[0038] In a further embodiment of the video playback system, the audiovisual data stream further comprises a cinematic physical effects data stream, where cinematic physical effects refers to the concept of 4D movies in which moving images are combined with cinematic physical effects, such as movement, vibration, scent, rain, mist, bubbles, fog, smoke, wind, temperature changes, etc. The physical effects are often reproduced, for example, by specially adapted seats. In a preferred embodiment, the video playback system further comprises a cinematic physical effects unit for reproducing the cinematic physical effects of the cinematic physical effects data stream, wherein the synchronization unit is configured to transmit a timestamp signal to the cinematic physical effects unit, and the cinematic physical effects unit is configured to receive the timestamp signal, reproduce the cinematic physical effects data stream, and synchronize the reproduction of the cinematic physical effects data stream with the audio data stream using the timestamp signal.

[0039] By using the 4D cinema function, viewers can have a more special experience when using a video playback system. In a further embodiment, the cinematic physical effects data stream is based on 4DX. In this specification, the term "4DX" refers to the 4D cinema format developed by CJ 4DPlex.

[0040] This object is also solved by a media playback system configured to play an audio-video data stream having an audio data stream including a time code signal and a video data stream, the playback system comprising: a video playback system according to one or more of the above-mentioned embodiments; and a sound system, wherein the synchronization unit is configured to send the audio data stream to the sound system, the sound system is configured to receive the audio data stream from the synchronization unit and play the audio data stream, and each of the head-mounted displays is configured to synchronize playback of the video data stream with audio playback of the audio data stream by the sound system using the timestamp signal.

[0041] By including a sound system in a media playback system that comprises a video playback system as described above, synchronization in the playback of audiovisual data streams is further improved. In one embodiment, the sound system is a surround sound system or a theater sound system. In a further embodiment, the synchronization unit is configured to adapt the timestamp signal for playback by the sound system to compensate for path delays in the audio data signal path from the synchronization unit. In this way, the path delays account for delays that occur in the audio signal path, for example in amplifiers or speakers.

[0042] In a further embodiment of the video playback system or media playback system, each of the head-mounted displays comprises a switch-off device configured to switch off each of the head-mounted displays and / or put each of the head-mounted displays into a standby state, in particular the switch-off device being activated when it is detected that a user of each of the head-mounted displays is inactive.

[0043] The object is also solved by a method for synchronized playback of a video data stream of an audio-video data stream having an audio data stream including a time code signal and a video data stream, the method comprising the steps of receiving the audio data stream by a synchronization unit; converting the time code signal from the audio data stream into a time stamp signal, in particular a digital signal, by the synchronization unit; transmitting the time stamp signal from the synchronization unit to a plurality of head-mounted displays, in particular via a wireless network connection; transmitting the audio data stream from the synchronization unit to a sound system; receiving the time stamp signal by each of the head-mounted displays; determining synchronized video data in the video data stream to be displayed by each of the head-mounted displays in synchronization with the audio data stream using the time stamp signal; and playing the synchronized video data by each of the head-mounted displays.

[0044] The same or similar benefits and advantageous aspects as those described with respect to the video playback system described above also apply to the methods described below, explicitly including the features of the described embodiments and technical advantages derived from the characteristics of the embodiments.

[0045] The audiovisual data stream comprises an audio data stream, i.e., an audio component, and a video data stream, i.e., a video component. In one embodiment, the audiovisual data stream is a movie or a television program, or a 360° video including audio. In a further embodiment, the audiovisual data stream is a live audiovisual production, in particular of a concert, or a musical, or a sporting event, or an e-sports event, or an exhibition, or a conference, or any kind of live interactive media. In a further embodiment, the video data stream is an XR data stream. In a further embodiment, the video data stream includes three-dimensional video data. In a preferred embodiment, the audio data stream is an analog audio data stream. In another embodiment, the audio data stream is a digital audio data stream.

[0046] In a preferred embodiment, the synchronization unit is a computer hardware device, in particular comprising a processing unit and / or an analog-to-digital converter and / or a digital-to-analog converter. In a further embodiment, the synchronization unit is a single-board computer. In a further embodiment, the method comprises receiving the audio data stream via a wired connection, in particular via a twisted pair cable with an RJ45 connector and / or via a phone connector and / or via an XLR connector.

[0047] In a further embodiment, the time code signal is timing information included in the audio data stream, the audio data stream being analog or digital, and the time code signal is processed by a synchronization unit which converts the time code signal into a time stamp signal, in particular a digital time stamp signal. In a further embodiment, converting the time code signal comprises extracting the time code signal from the audio data stream and processing the time code signal into a digital time stamp signal. The time stamp signal is preferably in the form of a time of day, or alternatively preferably in a cinema-based format, for example including a frame number.

[0048] In a further embodiment, the synchronization unit transmits the timestamp signal to the plurality of head mounted displays, in particular at predetermined time intervals. In another embodiment, the synchronization unit transmits the timestamp signal in response to a request from each of the head mounted displays. In one embodiment, the predetermined time interval is a fixed time interval.

[0049] In one embodiment for transmitting the timestamp signal from the synchronization unit to the plurality of head mounted displays, a wireless network connection is established. In a further embodiment, the wireless network is established by the synchronization unit, and in particular the synchronization unit comprises a network unit, in particular a wireless router or a wireless access point. In a further embodiment, each of the head mounted displays comprises a wireless communication unit, which establishes the wireless network connection and receives the timestamp signal transmitted by the synchronization unit. The wireless connection supports the flexibility of the head mounted display and provides a better user experience.

[0050] In a preferred embodiment, the head mounted displays are XR headsets, in particular AR headsets and / or MR headsets and / or VR headsets. In a preferred embodiment, the method comprises determining the orientation and / or position of each of the head mounted displays in a given coordinate system, in particular the given coordinate system relating to a room, for example a cinema hall or a living room. In a further embodiment, the method comprises tracking the user's gaze by an eye tracking device and / or tracking the user's head by a head tracking device.

[0051] In a further embodiment, synchronized video data is played back by each of the head mounted displays using monoscopic 2D video data or stereoscopic 3D video data, and the playback of the video data is synchronized with a timestamp signal derived from the audio data stream.

[0052] In a further embodiment, the method comprises a step of calibrating the spatial position and / or rotational orientation of each of the head mounted displays, in particular in a room, preferably in which a sound system is located, so that the video orientation for the user can be adapted to the audio orientation of the sound system.

[0053] In another embodiment, the method for synchronized playback of a video data stream further comprises providing an audio data stream by an audio data source to a synchronization unit, receiving the live stream as an audio data stream by the audio data source, and / or decoding and retrieving the audio data stream from a data store, in particular from Digital Cinema Package (DCP) data, by the audio data source.

[0054] In another embodiment, the audio data source provides an audio data stream and also decodes another digital cinema data format. In another embodiment, the audio data source does not decode and / or decode the audio data. In a further embodiment, the audio data source receives the audio data stream from the Internet. In another embodiment, the audio data source generates the audio data stream from an audio data memory device. These embodiments make the method suitable for playback of a variety of cinema content conforming to various industry standards.

[0055] In yet another embodiment, a method for synchronized playback of a video data stream comprises storing video data in a data storage device that is part of each of the head-mounted displays, and generating a video data stream from the video data stored in the data storage device. In one embodiment, movie, 360° video, or similar video data is stored in the video data storage device. Each of the head-mounted displays plays a portion of the video data stream that corresponds to a timestamp signal received from the synchronization unit, thereby playing video and audio data at the same temporal location. Storing video data locally not only increases availability, but also reduces energy consumption of the head-mounted displays, thereby extending battery life.

[0056] In yet another embodiment, a method for synchronized playback of a video data stream comprises converting a time code signal into a digital time stamp signal, and the audio data stream is an analog audio data stream, and in particular the method further comprises detecting errors in the analog time code signal, in particular errors relating to consistency, by a synchronization unit, and correcting the time code signal using an error correction scheme. Correcting errors, in particular errors relating to consistency, improves the audio experience and synchronization between the audio and video data. Correcting errors significantly improves the quality of the processed time code signal as well as the quality of the time stamp signal, especially when the audio data stream is analog.

[0057] In a further embodiment, the synchronization unit transmits the audio data stream unmodified, i.e., without any changes, to the sound system. In another embodiment, the synchronization unit filters and / or processes the audio data stream and transmits it to the sound system. This allows the synchronization unit to be transparent by simply intercepting the signal and passing it on as is, or the synchronization unit may improve the audio signal, for example, to suit the specifications of a user, venue, or speaker system.

[0058] In one embodiment, detecting the error comprises semantically verifying the acquired time code signal. In a further embodiment, correcting the time code signal comprises interpolating the time code signal.

[0059] According to further embodiments, the audio data stream comprises at least four audio channels, at least eight audio channels, at least twelve audio channels, or at least sixteen audio channels, with one channel carrying a time code signal. The time code signal is in particular an SMPTE time code signal, in particular an SMPTE-LTC time code signal. In another embodiment, the time code signal is a MIDI time code. The use of various channels not only results in a higher listening quality and a better three-dimensional sound experience, but also makes it easier for a synchronization unit to acquire the time code signal.

[0060] In yet another embodiment, the method for synchronized playback of video data streams comprises requesting, by each of the head mounted displays, a timestamp signal from the synchronization unit, in particular at predetermined time intervals, thereby reducing the amount of network traffic. In one embodiment, the predetermined time interval is a request every frame, or multiple times per second, or approximately once per second, or once every several seconds, or less frequently. In a further embodiment, the predetermined time interval is a constant time interval.

[0061] In a further embodiment, the synchronization unit provides a timestamp server that provides timestamp signals for the head-mounted devices, each of the head-mounted displays requesting and / or retrieving the timestamp signals from the timestamp server.

[0062] In yet another embodiment, a method for synchronously reproducing video data streams comprises recalibrating a timestamp signal to include an individual, particularly variable, transmission delay of the timestamp signal from a synchronization unit to each of the head-mounted displays, and in particular the method further comprises determining a clock signal referenced to a time standard by an internal clock of the synchronization unit, transmitting the clock signal together with the timestamp signal from the synchronization unit to a plurality of head-mounted displays, receiving the clock signal by each of the head-mounted displays, determining an individual transmission delay from the clock signal, and recalibrating the timestamp signal to compensate for the individual transmission delay. Network connections, in particular wireless network connections, introduce transmission delays that can vary by tens of milliseconds. Recalibrating the timestamp signal to include this individual transmission delay ensures that the audio data stream and the video data stream are reproduced in sync.

[0063] The clock signal determined by the synchronization unit is referenced to a time standard, which may be, for example, the local time of the synchronization unit or UTC. In a further embodiment, the synchronization unit transmits a clock signal together with the timestamp signal, whereby each of the head mounted displays compares multiple combinations of the timestamp signal and the clock signal, compares these values ​​with the internal clock of the head mounted display and determines an individual transmission delay on this basis.

[0064] In a further embodiment, the method for synchronously playing video data streams comprises switching off the head mounted displays or putting the head mounted displays into a standby state when each of the head mounted displays detects that the user is inactive.

[0065] In a further embodiment of the method for synchronously playing back video data streams, the audiovisual data stream further comprises a cinematic physical effects data stream. In this embodiment, the method for synchronously playing back video data streams further comprises the steps of transmitting a timestamp signal from the synchronization unit to a cinematic physical effects unit, in particular via a wireless network connection, receiving the timestamp signal by the cinematic physical effects unit, determining synchronized cinematic physical effects data in the cinematic physical effects data stream to be played by the cinematic physical effects unit using the timestamp signal in synchronization with the audio data stream, and playing back the cinematic physical effects data by the cinematic physical effects unit.

[0066] By using the 4D cinema function, viewers can have a more special experience when using a video playback system. In a further embodiment, the cinematic physical effects data stream is based on 4DX, where the term "4DX" refers to the 4D cinema format developed by CJ 4DPlex.

[0067] The object is also solved by a method for synchronized playback of an audiovisual data stream having an audio data stream including a time code signal and a video data stream, the method comprising, as described above, a method for synchronized playback of a video stream of the audiovisual data stream, and comprising transmitting the audio data stream from a synchronization unit to a sound system, playing the audio data stream using the sound system, and determining synchronized video data in the video data stream to be displayed by each of the head-mounted displays in synchronization with the playback of the audio data by the sound system.

[0068] By having a sound system and playing the audio data stream through the sound system, both playbacks of the audio and video streams are included in this message, which allows for further improved synchronization of the playback by both devices.

[0069] The object is also solved by a computer readable storage medium comprising instructions which, when executed by a computer, cause a media playback system to perform the method for synchronized playback of video data streams or the method for synchronized playback of audio-video data streams as described above, said computer readable storage medium enabling a media playback system, e.g. a video playback system or a media playback system as described above, to execute said method to synchronize the playback of video data streams and audio data streams in accordance with the invention.

[0070] Further features of the invention will become apparent from the description of the embodiments according to the invention, together with the claims and the accompanying drawings, in which: Embodiments according to the invention may feature individual features or combinations of several features.

[0071] The present invention is described below on the basis of exemplary embodiments, without limiting the general scope of the invention, and explicit reference is made to the drawings for the disclosure of all details according to the invention not described in detail in the text, the drawings being shown below: [Brief explanation of the drawings]

[0072] [Figure 1] 1 is a schematic diagram of a first embodiment of a video playback system. [Figure 2] FIG. 2 is another schematic diagram of a second embodiment of a video playback system. [Figure 3] 3 is a flowchart of the steps of a method for synchronously playing a video data stream of an audiovisual data stream, performed by a synchronization unit in the first embodiment; [Figure 4]10 is another flowchart of steps of a method for synchronously playing video data streams of audiovisual data streams performed by each of the head mounted displays in the first embodiment; [Figure 5a] FIG. 1 is a schematic diagram of a first embodiment of an audiovisual data stream. [Figure 5b] FIG. 10 is a schematic diagram of a second embodiment of an audiovisual data stream. [Figure 6] 1 is a schematic diagram of an embodiment of a head-mounted device. DETAILED DESCRIPTION OF THE INVENTION

[0073] In the drawings, elements of the same or similar type or corresponding parts are given the same reference numerals so as not to have to repeatedly describe the items.

[0074] FIG. 1 schematically illustrates a video playback system 10 according to a first embodiment. The video playback system 10 includes a synchronization unit 20, which in this embodiment is a single-board computer including a processing unit and audio input / output connectors. The synchronization unit 20 receives an audio data stream 61 including a time code signal 62 from an audio data source 30 via a wired connection 32. The audio data source decodes DCP data retrieved from a data storage unit 31 to provide the audio data stream 61. The wired connection 32 is a twisted pair cable connected to the synchronization unit 20 via an RJ45 connector. The audio data stream 61 includes an audio signal of an audio-visual data stream, which is an XR media data stream. The audio data stream 61 is an analog signal consisting of 16 channels, one of which carries a time code signal 62. The time code signal 62 is an SMPTE-LTC time code signal. Synchronization unit 20 transmits audio data stream 66, including time code signal 67, via wired connection 33 to sound system 40. In this embodiment, sound system 40 is a surround sound system configured to play audio data stream 66. In this embodiment, audio data stream 61 and time code signal 62 are identical to audio data stream 66 and time code signal 67.

[0075] The synchronization unit 20 processes an audio data stream 61 including a time code signal 62 and converts the analog time code signal 62 into a digital time stamp signal 25. The synchronization unit 20 is configured to transmit the time stamp signal 25 to a plurality of head-mounted displays 50. In this embodiment, the synchronization unit 20 is configured to establish a wireless network connection 56 with the plurality of head-mounted displays 50. To this end, the synchronization unit 20 includes a wireless access point. The wireless network connection 56 is based on Wi-Fi, specifically Wi-Fi 4 using the 2.4 GHz band and Wi-Fi 5 using the 5 GHz band. The head-mounted displays 50 receive the time stamp signal 25 via the wireless network connection 56. The head-mounted displays 50 play a video data stream that is stored locally in the data storage device of each head-mounted display 50. The head-mounted displays 50 play the video data stream in synchronization with the audio data streams 61, 66 using the time stamp signal 25. This allows the user of the head-mounted display 50 to experience a synchronized experience between the video data stream displayed by the head-mounted display and the audio data stream reproduced by the sound system 40.

[0076] FIG. 2 schematically illustrates an embodiment of the media playback system 11. The overall configuration of the embodiment illustrated in FIG. 2 is similar to the embodiment of the video playback system 10 illustrated in FIG. 1. In this embodiment, the synchronization unit 20 is divided into a timestamp unit 23 and a network unit 24. The timestamp unit 23 is a small-form-factor computer 802. The computer 802 includes a computer-readable storage medium 801 that includes instructions that, when executed by the computer 802, cause the media playback system 11 to perform the method for synchronously playing back video data streams as described above. The network unit 24 is a wireless router. The timestamp unit 23 receives an audio data stream 61, including a timecode signal 62, from an audio data source 30 via a wired connection 32. The audio data stream 61 is analog, and the timecode signal 62 is an SMPTE-LTC timecode signal. In this embodiment, the audio data source 30 is configured to receive the audio data stream 61 via the Internet and provide the audio data stream 61 without decoding the audio data. Wired connection 32 is connected via a phone connector to audio data source 30 and time stamp unit 23. Timestamp unit 23 transmits an audio data stream 66, including a time code signal 67, to audio interface 41 via wired connection 33. Via wired connection 34, audio interface 41 transmits audio data stream 66 to sound system 42, which plays audio data stream 66. In this embodiment, audio interface 41 comprises a connector for input of audio data stream 66 and an amplifier for playback of audio data stream 66. In another embodiment not shown here, sound systems 40, 42 each comprise audio interface 41.

[0077] The timestamp unit 23 converts an audio data stream 61, including an analog time code signal 62, into a digital timestamp signal 25. The timestamp unit 23 also includes an internal clock, which determines a clock signal 26. The timestamp unit 23 transmits the clock signal 26 along with the timestamp signal 25 to the network unit 24 via a wired connection 35, which is a twisted pair cable with an RJ45 connector. The network unit 24 establishes a wireless network connection 56 with multiple head-mounted displays 50, 52, and 53. The wireless network connection is a Wi-Fi 5 connection. In this embodiment, different types of head-mounted displays 50, 52, and 53 are used. In this embodiment, by way of example and without limitation to other embodiments, the head-mounted display 50 is a VR headset, the head-mounted display 52 is an MR headset, and the head-mounted display 53 is an AR headset. The network unit 24 transmits the timestamp signal 25 along with the clock signal 26 to the multiple head-mounted displays 50, 52, and 53 via the wireless network connection 56.

[0078] Each head mounted display 50, 52, 53 is configured to receive the clock signal 26 and determine an individual transmission delay from the clock signal 26. The individual transmission delay varies over time and for each head mounted display 50, 52, 53, as it depends, at least in part, on the stability of the wireless network connection 56 and the distance between the network unit 24 and the head mounted displays 50, 52, 53. Using the individual transmission delay, each head mounted display 50, 52, 53 recalibrates the timestamp signal 25 to compensate for the individual transmission delay between the timestamp unit 23 and the head mounted displays 50, 52, 53.

[0079] 3 shows a flowchart of the steps of the method for synchronized playback of a video data stream of an audio-video data stream, performed by the synchronization unit 20 in the first embodiment. The synchronization unit 20 receives 301 an audio data stream 311 provided by the audio data source 30. The audio data source 30 decodes the DCP data to provide the audio data stream 311, which is an analog audio data stream 311. In a next step, the synchronization unit 20 converts 303 the time code signal into a digital timestamp signal. In this embodiment, the synchronization unit 20 simply intercepts the audio data stream 311 without modifying it. In a subsequent step, the synchronization unit 20 transmits 305 the audio data stream 311 to a sound system that performs audio playback 321. The synchronization unit 20 also transmits 304 a timestamp signal 322, whereby the timestamp signal 322 is provided to multiple head-mounted displays 50, 52, 53.

[0080] 4 shows a flowchart of the method steps for synchronized playback of a video data stream in an audio-video data stream, as performed by each of the head-mounted displays 50, 52, and 53 in the first embodiment. A timestamp signal 422 is received by each of the head-mounted displays 50, 52, and 53, which processes the timestamp signal 401. Each of the head-mounted displays 50, 52, and 53 determines synchronized video data within the video stream 402. In this embodiment, this involves generating a video data stream from video data stored in a data storage device that is part of each of the head-mounted displays 50, 52, and 53. In a next step, each of the head-mounted displays 50, 52, and 53 plays back the synchronized video data 403, i.e., each of the head-mounted displays 50, 52, and 53 displays video that is time-aligned with the audio data. This results in synchronized video playback 431, which proceeds in sync with the audio stream data played back in the audio playback 321.

[0081] In an alternative embodiment not shown in Fig. 4, before the processing step 401, each of the head-mounted displays 50, 52, and 53 requests a timestamp signal 422 from the synchronization unit 20. This request is sent periodically at a predetermined time interval, which in this embodiment is once every several seconds. In response to this request, the timestamp signal 422 is transmitted from the synchronization unit 20 to each of the head-mounted displays 50, 52, and 53 that sent the request. Thereafter, the processing step 401 and subsequent steps are performed as shown in Fig. 4.

[0082] 5a is a schematic diagram of a first embodiment of an audiovisual data stream 501. The audiovisual data stream 501 comprises a video data stream 502 and an audio data stream 503.

[0083] Figure 5b is a schematic diagram of a second embodiment of an audiovisual data stream 501. In this embodiment, audiovisual data stream 501 comprises a cinematic physical effects data stream 504, along with a video data stream 502 and an audio data stream 503. Cinematic physical effects data stream 504 contains information on when to perform cinematic physical effects, and the playback of the cinematic physical effects is matched to the audio and video components of the audiovisual data stream.

[0084] 6 is a schematic diagram of a further embodiment of head-mounted device 50. Head-mounted device 50 comprises a data storage device 602, an inertial measurement unit 603, a screen 604, a wireless communication unit 605, and an internal clock 606. Screen 604 is used as a display for playback of video data stream 502. Internal clock 606 is used to determine a clock signal relative to a time standard.

[0085] All features listed, whether singly or in combination, are considered essential to the invention, including features that are only apparent from the drawings and individual features disclosed in combination with other features. Embodiments of the invention may be realized by individual features or combinations of multiple features. Features described in combination with the words "especially" or "especially" are to be considered preferred embodiments. [Explanation of symbols]

[0086] 10 Video playback system 11 Media Playback System 20 Synchronous Units 23 Timestamp Unit 24 Network Unit 25 Timestamp Signal 26 Clock Signal 28 Internal Clock 30 Audio Data Sources 31 Data storage unit 32, 33, 34, 35 Wired connection 40 Sound System 41 Voice Interface 42 Sound System 50 VR headsets 52 Mixed reality headset 53 AR headset 56 Wireless Network Connection 61 Audio Data Stream 62 Time Code Signal 66 audio data streams 67 Time Code Signal 301 Receive audio data stream Converting 303 timecode signals to timestamp signals 304 Transmits timestamp signals 305 Sending an audio data stream to the sound system 311 Audio Data Stream 321 Audio Playback 322 timestamp signal 401 Process Timestamp Signals 402 Determining synchronized video data within a video data stream 403 Playing synchronized video data 422 Timestamp Signal 431 Synchronized Video Playback 501 Audio-Video Data Stream 502 Video Data Stream 503 Audio Data Stream 504 Cinematic Physical Effects Data Stream 602 Data storage device 603 Inertial Measurement Unit 604 screens 605 Wireless Communication Unit 606 Internal Clock 801 Computer-readable storage medium 802 Computer

Claims

1. 1. A video playback system (10) configured to play back a video data stream (502) of an audio video data stream (501) having an audio data stream (61, 66, 503) including a time code signal (62, 67), the video data stream (502) comprising: The video playback system (10) includes a synchronization unit (20) and a plurality of head-mounted displays (50, 52, 53), The synchronization unit (20) receiving said audio data stream (61, 66) from an audio data source (30); transmitting said audio data streams (61, 66) to a sound system (40); converting the time code signals (62, 67) of the audio data stream (61, 66) into time stamp signals (25); configured to transmit the timestamp signal (25) to the plurality of head-mounted displays (50, 52, 53); Each of the head-mounted displays (50, 52, 53) receiving the timestamp signal (25); Playing the video data stream (502); A video playback system (10) configured to synchronize the playback of the video data stream (502) with the audio data stream (61, 66) using the timestamp signal (25).

2. 2. A video playback system (10) according to claim 1, an audio data source (30) configured to provide the audio data stream (61) to the synchronization unit (20); the audio data source (30) is configured to receive a live stream as the audio data stream (61); and / or A video playback system (10), wherein the audio data source (30) is configured to retrieve the audio data stream (61) from a data store (31).

3. 2. A video playback system (10) according to claim 1, each of the head-mounted displays (50, 52, 53) comprises a data storage device (602) having stored therein video data; A video playback system (10), wherein each of the head-mounted displays (50, 52, 53) is configured to generate the video data stream (502) from the video data stored in the data storage device (602).

4. A video playback system (10) according to claim 1, the time code signal (62, 67) is an analog signal and the time stamp signal (25) is a digital signal, and the synchronization unit (20) is configured to convert the analog time code signal (62, 67) into the digital time stamp signal (25); The video playback system (10), wherein the synchronization unit (20) is configured to detect consistency errors in the analog time code signals (62, 67) and to correct the time code signals (62, 67) using an error correction scheme.

5. A video playback system (10) according to claim 1, The synchronization unit (20) comprises a time stamp unit (23) and a network unit (24); the time stamp unit (23) is configured to receive the audio data stream (61), to transmit the audio data stream (61, 66), and to convert the time code signal (62) of the audio data stream (61) into a time stamp signal (25); The network unit (24) is configured to establish a wireless network connection (56) with each of the head-mounted displays (50, 52, 53) and transmit the timestamp signal (25) from the timestamp unit (23) to the plurality of head-mounted displays (50, 52, 53) via the wireless network connection (56).

6. A video playback system (10) according to claim 1, A video playback system (10), wherein each of the head-mounted displays (50, 52, 53) is configured to request and / or read the timestamp signal (25) from the synchronization unit (20) at predetermined time intervals.

7. A video playback system (10) according to any one of claims 1 to 6, the synchronization unit (20) and each of the head-mounted displays (50, 52, 53) are configured to recalibrate the timestamp signal (25) to compensate for variable, individual transmission delays of the timestamp signal (25) from the synchronization unit (20) to each of the head-mounted displays (50, 52, 53); each of the synchronization unit (20) and the head-mounted display (50, 52, 53) has an internal clock (28, 606) for determining a clock signal (26) based on a time standard; the synchronization unit (20) is configured to transmit the clock signal (26) together with the timestamp signal (25) to the plurality of head-mounted displays (50, 52, 53); a video playback system (10) configured to receive the clock signal (26), determine the individual transmission delays from the clock signal (26), and recalibrate the timestamp signal (25) to compensate for the individual transmission delays.

8. 1. A media playback system (11) configured to play an audiovisual data stream (501) having an audio data stream (61, 66, 503) including a time code signal (62, 67) and a video data stream (502), comprising: The media playback system (11) comprises the video playback system (10) according to claim 1 and a sound system (40), the synchronization unit (20) is configured to transmit the audio data streams (61, 66) to the sound system; the sound system (40) is configured to receive the audio data stream (66) from the synchronization unit (20) and to play back the audio data streams (61, 66); 10. A media playback system (11), wherein each of the head-mounted displays (50, 52, 53) is configured to synchronize playback of the video data stream (502) with audio playback of the audio data stream (61, 66) by the sound system (40) using the timestamp signal (25).

9. A method for synchronously playing back a video data stream (502) of an audio video data stream (501) having an audio data stream (311, 503) including a time code signal (62, 67) and the video data stream (502), the method comprising: receiving said audio data stream (311) by a synchronization unit (20); converting, by said synchronization unit (20), said time code signal (62, 67) from said audio data stream (311) into a digital time stamp signal (25); transmitting the timestamp signal (25) from the synchronization unit (20) to a plurality of head mounted displays (50, 52, 53) via a wireless network connection (56); transmitting said audio data stream (311) from said synchronization unit (20) to a sound system (40); receiving the timestamp signal (25) by each of the head mounted displays (50, 52, 53); determining synchronized video data within the video data stream (502) to be displayed by each of the head-mounted displays (50, 52, 53) in synchronization with the audio data stream (311) using the timestamp signal (25); and playing the synchronized video data by said each of said head mounted displays (50, 52, 53).

10. 10. A method for synchronously playing back a video data stream according to claim 9, comprising: providing said audio data stream (311) to said synchronization unit (20) by an audio data source (30); receiving a live stream as said audio data stream (61) by said audio data source (30); and / or retrieving the audio data stream (61) from a data store (31) by the audio data source (30).

11. 10. A method for synchronously playing back a video data stream according to claim 9, comprising: storing video data in a data storage device (602) that is part of each of the head mounted displays (50, 52, 53); generating the video data stream (502) from the video data stored in the data storage device (602).

12. 10. A method for synchronous playback of a video data stream (502) according to claim 9, comprising the steps of: further comprising converting the time code signal (62, 67) into a digital time stamp signal (25); the audio data stream (311, 502) is an analog audio data stream; The method comprises: detecting consistency errors in the analog time code signals (62, 67) by the synchronization unit (20); correcting the time code signal (62, 67) using an error correction scheme.

13. 10. A method for synchronous playback of a video data stream (502) according to claim 9, comprising the steps of: The method further comprises requesting, by each of the head-mounted displays (50, 52, 53), the timestamp signal (25) from the synchronization unit (20) at predetermined time intervals.

14. A method for synchronous playback of a video data stream (502) according to one of claims 9 to 13, comprising: recalibrating the timestamp signal (25) to include a variable, individual transmission delay of the timestamp signal (25) from the synchronization unit (20) to each of the head-mounted displays (50, 52, 53); The method comprises: determining a clock signal (26) referenced to a time standard by an internal clock (28) of said synchronization unit (20); transmitting the clock signal (26) together with the timestamp signal (25) from the synchronization unit (20) to the plurality of head-mounted displays (50, 52, 53); receiving the clock signal (26) by said each of said head mounted displays (50, 52, 53); determining the individual transmission delays from the clock signal (26); recalibrating the timestamp signal (25) to compensate for the individual transmission delays.

15. A computer-readable storage medium (801) comprising instructions that, when executed by a computer (802), cause a media playback system (11) to perform the method of claim 9.

Images