Video reproduction system and media reproduction system and method for synchronously reproducing video data streams of audiovisual data streams, and computer-readable storage medium

Abstract

The invention relates to a video reproduction system (10) configured to reproduce a video data stream of audiovisual data streams (61, 66), the audiovisual data streams (61, 66) comprising an audio data stream (61) comprising time code signals (62, 67) and a video data stream, characterized in that the video reproduction system (10) comprises a synchronization unit (20) and a plurality of head-mounted displays (50, 52, 53), characterized in that the synchronization unit (20) is configured to synchronize the time code signals (62, 67) and the plurality of head-mounted displays (50, 52, 53). A video reproduction system (10) comprises a synchronization unit (20) and a plurality of head-mounted displays (50, 52, 53), in particular augmented reality head-mounted displays, where the synchronization unit (20) is configured to receive an audio data stream (61, 66) from an audio data source (30), transmit the audio data stream (61, 66) to a sound system (40), convert a time code signal (62, 67) of the audio data stream (61, 66) into a time stamp signal (25), and transmit the time stamp signal (25) to the sound system (40). The timestamp signal (25) is transmitted to a plurality of head-mounted displays (50, 52, 53), each of the head-mounted displays (50, 52, 53) being configured to receive the timestamp signal (25), to reproduce the video data stream, and to synchronize the reproduction of the video data stream with the audio data stream (61, 66) using the timestamp signal (25).

Description

Technical Field

[0001] The present invention relates to a video reproduction system configured to reproduce a video data stream of an audiovisual data stream, the audiovisual data stream including an audio data stream and a video data stream containing a time code signal. Further, the present invention relates to a media reproduction system configured to reproduce an audiovisual data stream, the audiovisual data stream including an audio data stream and a video data stream containing a time code signal. The present invention also relates to a method for synchronously reproducing a video data stream of an audiovisual data stream, the audiovisual data stream including an audio data stream and a video data stream containing a time code signal. Finally, the present invention relates to a computer-readable storage medium including instructions that, when executed by a computer, cause the media reproduction system to perform the method for synchronously reproducing a video data stream. Background Art

[0002] Extended reality (XR) allows users to experience a world different from the world we usually perceive with our bare senses. Computers and wearable sensing devices drive this experience. The term XR is generally used as a collective term for all real and virtual combined environments, such as augmented reality (AR), mixed reality (MR), or virtual reality (VR).

[0003] XR hardware and software applications are generally designed for individual users. Usually, video data is displayed by a head-mounted display, and audio data is played through headphones. These individual headphones are either directly integrated into the head-mounted display or connected to it. XR applications generally use binaural sound to provide listeners with a sound feeling as if they were in a room with different audio sources.

[0004] The use of a head-mounted display and headphones may bring an isolated experience, or even a narrow impression, to users of XR devices. This usually limits the usage time and loses the fun of the user's entertainment experience.

[0005] In addition, the head-mounted display and headphones need to be precisely calibrated so that both the video displayed by the head-mounted display and the audio played by the headphones are precisely synchronized with the movement of the user's head. In some settings, XR devices must perform continuous computational work to (re)calculate a suitable audio signal corresponding to the user's head orientation.

[0006] To synchronize the audio and video reproduction of a single audiovisual data stream on different devices, such as a display or screen for video and headphones or a sound system for audio, different sound analysis methods are known.

[0007] US 10,178,487 discloses a method for binaural audio rendering. In response to an audio synchronization signal presented to a mobile device, audio information is played from the mobile device in a manner synchronized with the display of a movie on a theater screen. The audio synchronization signal may be a high-frequency signal transmitted by a speaker in the theater.

[0008] EP 2628047 B1 discloses another system and method for providing alternative audio for combined video and audio, particularly movies and television programs, wherein a position code is embedded as an audio watermark in the audio signal. SUMMARY OF THE INVENTION

[0009] The object of the present invention is to improve the sound quality and user experience of XR devices.

[0010] This object is achieved by a video reproduction system configured to reproduce a video data stream of an audiovisual data stream, the audiovisual data stream comprising an audio data stream and a video data stream containing a time code signal, characterized in that the video reproduction system comprises a synchronization unit and a plurality of head-mounted displays, particularly extended reality headsets, wherein the synchronization unit is configured to: receive the audio data stream from an audio data source, send the audio data stream to a sound system, convert the time code signal of the audio data stream into a time stamp signal, transmit the time stamp signal to the plurality of head-mounted displays, each of the head-mounted displays being configured to: receive the time stamp signal, reproduce the video data stream, and synchronize the reproduction of the video data stream with the audio data stream using the time stamp signal.

[0011] The video reproduction system is based on the following considerations. The reproduction (i.e., playback) of the audiovisual data stream is performed by two separate devices most suitable for the task, for example, one device for reproducing the audio data stream and another device for reproducing the video data stream. This creates a need to synchronize these two devices. According to aspects of the present invention, this is achieved by intercepting the audio data stream by means of a synchronization unit before the sound system reproduces the audio data stream. A digitally accessible time stamp signal is obtained from the time code signal contained in the audio data stream. The time stamp signal is then transmitted to the plurality of head-mounted displays that act as display devices for the video data stream, wherein the video data stream is a component of the audiovisual data stream.

[0012] One advantage of the video reproduction system is that it allows the combination of a personal (i.e., single-user) head-mounted display and a sound system (e.g., a surround sound system) to provide sound for multiple users simultaneously. Each user can experience a complete three-dimensional XR vision through the head-mounted display and an all-round audio experience through a powerful speaker system. Despite significant progress in headphone design, the sound of the most advanced headphones still cannot be compared with the sound experience of a complete sound system (e.g., a full-size subwoofer reproduces low frequencies better than headphones). In addition to this, using an external sound system instead of individual headphones can provide a more open experience for users and allow them to enjoy the entertainment experience together with other viewers.

[0013] According to various aspects of the present invention, the audiovisual data stream includes an audio data stream and a video data stream (i.e., an audio component and a video component). According to an embodiment, the audiovisual data stream is a movie or a television program or a 360° video including sound, where the 360° video is a video that provides a view in all directions but does not allow interaction with a virtual environment. In another embodiment, the audiovisual data stream is a live audiovisual product, particularly a concert, a musical, a sports event, an esports event, an exhibition, a conference, or any type of interactive live media. In another embodiment, the video data stream is an XR data stream, particularly a VR data stream. In another embodiment, the video data stream contains three-dimensional video data. In another embodiment, the audio data stream is an analog audio data stream. In another embodiment, the audio data stream is a digital audio data stream.

[0014] The audio data stream includes a time code signal. In the context of this specification, the term "time code signal" refers to any time information contained in the audio data stream. In particular, the term time code signal refers to the standard time code used in film and / or television and / or movie and / or video production. In particular, the time code signal is a time code according to the standards of the Society of Motion Picture and Television Engineers (SMPTE) and / or other standards (e.g., the MIDI standard). Using the time code standard allows for flexible use of the video reproduction system. Using other time information present in the audio data stream as the time code signal has the advantage that the video reproduction system is independent of the input audio data stream.

[0015] In one embodiment, the synchronization unit is a computer hardware device, particularly including a processing unit and / or an analog-to-digital converter and / or a digital-to-analog converter. In another embodiment, the synchronization unit is a single-board computer. In another embodiment, the synchronization unit is configured to receive an audio data stream via a wired connection, particularly via a twisted pair cable with an RJ45 connector and / or via a phone connector and / or via an XLR connector. The term "twisted pair" herein refers to, for example, a CAT5 or CAT6 cable; the term "phone connector" is also known as a headphone jack, such as a three-contact version called a tip-ring-sleeve-(TRS) connector; the term "XLR" refers to the electrical connector for professional audio specified in the international standard IEC 61076-2-103. The wired connection allows a stable high-throughput audio data stream as the input to the synchronization unit.

[0016] According to another aspect of the present invention, the synchronization unit is configured to convert the time code signal of the audio data stream into a time stamp signal. In one embodiment, the synchronization unit is configured to extract the time code signal from the audio data stream and process the time code signal into a time stamp signal. In particular, the time stamp signal is a digital signal. According to one embodiment, the time stamp signal adopts a time format, and in another embodiment, the time stamp signal adopts a movie-based format, for example, including a frame number. By using a digital time stamp signal in a format adjusted according to the needs of each of the head-mounted displays, efficient and reliable synchronization of the reproduction of the audiovisual data stream can be achieved.

[0017] The synchronization unit is configured to transmit the time stamp signal to a plurality of head-mounted displays, wherein, in one embodiment, the synchronization unit is set to transmit the time stamp signal at a predefined time interval. In another embodiment, these predefined time intervals are regular or irregular. In another embodiment, the synchronization unit is configured to transmit the time stamp signal to a plurality of head-mounted displays according to the request of each of the head-mounted displays. Transmitting the time stamp signal at a predefined time interval or according to the request can achieve an efficient synchronization process to adapt to the specific requirements of the video reproduction system, particularly the number of head-mounted displays.

[0018] In a preferred embodiment, the head-mounted display is an XR headset. In another embodiment, the head-mounted display is configured to display a video data stream of an AR and / or MR and / or VR audiovisual data stream. In another embodiment, each of the head-mounted displays is equipped with an inertial measurement unit for determining the orientation and / or position of the head-mounted display in a given coordinate system. Preferably, such a given coordinate system is related to a room (e.g., a cinema hall or a living room). In a further embodiment, each of the head-mounted displays is equipped with an additional tracking unit (e.g., an eye tracking device or a head tracking device). In another embodiment, each of the head-mounted displays is equipped with a processing unit (e.g., a microprocessor) for processing timestamp signals.

[0019] In another embodiment, each of the head-mounted displays reproduces the video data stream as a single field of view 2D (i.e., one image is directed to both eyes), or stereoscopic 3D (i.e., different images are directed to each eye). This allows for an optimized XR experience.

[0020] In another embodiment, each of the head-mounted displays is configured to synchronize the reproduction of the video data stream with the audio data stream using timestamp signals, wherein each of the head-mounted displays is configured to display a specific portion of the video data stream corresponding to the timestamp signal received from the synchronization unit. Thus, each of the head-mounted displays is configured to reproduce a video frame of the video data stream while reproducing the corresponding audio signal of the audio data stream.

[0021] In another embodiment, each of the head-mounted displays can be calibrated relative to its spatial position and / or rotational orientation within the room. In another embodiment, the video reproduction system includes additional sensors and / or beacons for supporting such calibration, wherein, for example, these additional sensors are located in the same room as the head-mounted display. This allows for aligning the orientation of the video reproduction of the head-mounted display with the orientation of the audio data stream reproduction of the sound system, which is particularly important for XR media content.

[0022] In another embodiment, the video reproduction system further includes an audio data source configured to provide an audio data stream to the synchronization unit, wherein the audio data source is configured to receive a live stream as the audio data stream and / or the audio data source is configured to retrieve the audio data stream from a data storage, particularly configured to decode the audio data stream from Digital Cinema Package (DCP) data. Thus, the video reproduction system is suitable for reproducing locally stored live and / or audio-visual data, particularly movie content compliant with the industry standard DCP.

[0023] In another preferred embodiment, the audio data source is configured to decode another digital movie data format. In another preferred embodiment, the audio data source is configured to provide an audio data stream without decrypting and / or decoding the data. In another embodiment, the audio data source is configured to receive an audio data stream via the Internet, particularly a real-time data stream. In another embodiment, the audio data source is configured to generate an audio data stream from an audio data storage device (memory device).

[0024] In another embodiment, each of the head-mounted displays includes a data memory device on which video data is stored, wherein each of the head-mounted displays is configured to generate a video data stream based on the video data stored on the data memory device. This has the advantages of high availability of the video data stream, low power consumption of each of the head-mounted displays, and longer battery life.

[0025] In another embodiment, the video data stored on the data memory device is video data of a movie and / or 360° video and / or XR video data and / or other video data.

[0026] In another embodiment, the timecode signal is an analog signal and the timestamp signal is a digital signal, and the synchronization unit is configured to convert the analog timecode signal into a digital timestamp signal, wherein, in particular, the synchronization unit is configured to detect errors in the analog timecode signal, particularly errors related to consistency, and correct the timecode signal through an error correction scheme. Advantageously, the video reproduction system has a high degree of technical compatibility with existing systems. Digital signals allow for simpler processing and more efficient operation. Detecting and correcting errors can avoid synchronization errors between the audio data stream and the video data stream due to data connection errors.

[0027] In another embodiment, the audio data stream includes at least four audio channels, particularly at least eight audio channels, particularly at least twelve audio channels, and further at least sixteen audio channels, wherein one of the audio channels carries a timecode signal, particularly an SMPTE timecode signal, and further particularly an SMPTE-LTC timecode signal. The LTC timecode signal is a linear timecode signal, also known as a longitudinal timecode signal. In another embodiment, the timecode signal is a MIDI timecode. The multi-channel provides the user with a surround sound audio experience with higher sound quality. Placing the timecode signal in a separate channel within the audio data stream enables simple detection within the audio dataset. This ensures a fast and reliable conversion to the timestamp signal.

[0028] In another embodiment, the synchronization unit is configured to send the audio data stream to the sound system in an unmodified form (i.e., without making changes). In another embodiment, the synchronization unit is configured to filter and / or process the audio data stream and then send it to the sound system. Without making changes, the synchronization unit is completely transparent to the audio data stream and does not interfere with audio reproduction. By processing the audio data stream, the synchronization unit can optimize the audio data stream, for example, adjust it according to the specific requirements of the sound system, user, or venue.

[0029] In another embodiment, the synchronization unit includes a timestamp unit and a network unit. The timestamp unit is configured to receive the audio data stream, send the audio data stream, and convert the time code signal of the audio data stream into a timestamp signal. The network unit is configured to establish a wireless network connection to each of the head-mounted displays and transmit the timestamp signal from the timestamp unit to the plurality of head-mounted displays through the wireless network connection. The wireless connection supports the flexibility of the head-mounted displays. It allows for a better user experience, especially for XR or VR experiences.

[0030] In another embodiment, the network unit is a computer network device, particularly a wireless router or a wireless access point. In another embodiment, the network unit is configured to operate the wireless network connection based on Wi-Fi, particularly according to Wi-Fi 4 (IEEE 802.11n) and / or Wi-Fi 5 (IEEE 802.11ac) and / or subsequent Wi-Fi standards.

[0031] In another embodiment, the network unit is configured to operate with at least 50, particularly at least 100, and further particularly at least 150 simultaneously connected head-mounted displays. In another embodiment, the minimum range of the wireless connection from the network unit is at least 100 meters, particularly at least 200 meters, and further particularly at least 300 meters. These specifications allow for a large number of viewers and large venues. In another embodiment, each of the head-mounted displays is equipped with a wireless communication unit, and the wireless communication unit is configured to receive the timestamp signal transmitted by the synchronization unit.

[0032] In another embodiment, each of the head-mounted displays is configured to request and / or read the timestamp signal from the synchronization unit. In particular, each of the head-mounted displays is configured to request and / or read the timestamp signal from the synchronization unit at predefined time intervals. By requesting the timestamp signal from the synchronization unit as needed, the synchronization process is efficient without the need for push transmission from the synchronization unit.

[0033] In another embodiment, the synchronization unit includes a timestamp server. The timestamp server is configured to provide a timestamp signal to each of the head-mounted displays, where, in particular, each of the head-mounted displays is configured to request and / or read the timestamp signal from the timestamp server.

[0034] In another embodiment, the predefined time interval or the frequency of requesting and / or reading the timestamp signal from the synchronization unit is once per frame, multiple times per second, approximately once per second, once per second, once every few seconds, or less. In another embodiment, the predefined time interval is a regular time interval.

[0035] In another embodiment, the synchronization unit and each of the head-mounted displays are configured to recalibrate the timestamp signal to compensate for the respective transmission delays, which are particularly variable, from the synchronization unit to each of the head-mounted displays. In particular, each of the synchronization unit and the head-mounted displays has an internal clock for a clock signal for determining a reference time standard. The synchronization unit is configured to transmit the clock signal together with the timestamp signal to the plurality of head-mounted displays. Each of the head-mounted displays is configured to receive the clock signal and determine its respective transmission delay based on the clock signal and recalibrate the timestamp signal in order to compensate for its respective transmission delay.

[0036] Due to the network connection between two different devices, especially when connected via a wireless connection, network latency is exhibited, which may vary in multiples of tens of milliseconds. Therefore, it is advantageous to compensate for such transmission delays. Since the transmission delay is an individual quantity for each connection between the synchronization unit and each head-mounted display, the respective transmission delays are determined independently on each of the head-mounted displays. This allows for better synchronization of the audio data stream and the video data stream of the audiovisual data stream.

[0037] In a preferred embodiment, the internal clock is used for the clock signal for determining the reference time standard. Preferably, the clock signal includes time, such as the local time of the synchronization unit or Coordinated Universal Time (UTC).

[0038] In another embodiment of the video reproduction system, the audiovisual data stream further includes a movie physical effects data stream. Herein, movie physical effects refer to the concept of 4D movies, in which moving images are combined with movie physical effects (such as motion, vibration, smell, rain, fog, bubbles, mist, smoke, wind, temperature change). The physical effects are typically reproduced through specially modified seats. In a preferred embodiment, the video reproduction system further includes a movie physical effects unit for reproducing the movie physical effects of the movie physical effects data stream, wherein the synchronization unit is configured to transmit a timestamp signal to the movie physical effects unit, and the movie physical effects unit is configured to receive the timestamp signal, reproduce the movie physical effects data stream, and synchronize the reproduction of the movie physical effects data stream with the audio data stream using the timestamp signal.

[0039] Using the functions of 4D movies creates a more special experience for the audience when using the video reproduction system. In another embodiment, the movie physical effects data stream is based on 4DX. Herein, the term "4DX" refers to the 4D movie format developed by CJ 4DPlex.

[0040] This object is also achieved by a media reproduction system configured to reproduce an audiovisual data stream including an audio data stream and a video data stream containing a timecode signal, characterized in that the reproduction system includes a video reproduction system and a sound system according to one or more of the above embodiments, wherein the synchronization unit is configured to: send the audio data stream to the sound system, and the sound system is configured to: receive the audio data stream from the synchronization unit and reproduce the audio data stream; each of the head-mounted displays is configured to: synchronize the reproduction of the video data stream with the audio reproduction of the audio data stream by the sound system using the timestamp signal.

[0041] By including the sound system in the media reproduction system, which includes the video reproduction system as described above, the synchronization of the reproduction of the audiovisual data stream is further improved. In one embodiment, the sound system is a surround sound system or a theater sound system. In another embodiment, the synchronization unit is configured to adjust the timestamp signal to compensate for the path delay in the audio data signal path from the synchronization unit to the reproduction of the sound system. In this way, the path delay takes into account the delay obtained on the audio signal path (such as the delay in the amplifier or speaker).

[0042] In another embodiment of the video reproduction system or the media reproduction system, each of the head-mounted displays includes a shutdown device configured to shut down the head-mounted display and / or put it into a standby state, wherein, in particular, when the corresponding head-mounted display is detected as inactive, the shutdown device is triggered.

[0043] This object is also achieved by a method for synchronously reproducing a video data stream of an audiovisual data stream, which audiovisual data stream includes an audio data stream and a video data stream containing a time code signal. The method includes the following steps: receiving the audio data stream by a synchronization unit, and converting, by the synchronization unit, the time code signal from the audio data stream into a time stamp signal, in particular a digital one; transmitting the time stamp signal from the synchronization unit to a plurality of head-mounted displays, in particular via a wireless network connection; sending the audio data stream from the synchronization unit to a sound system; receiving, by each of the head-mounted displays, the time stamp signal; determining synchronized video data of the video data stream for each of the head-mounted displays to use the time stamp signal to synchronize the display with the audio data stream; and reproducing, by each of the head-mounted displays, the synchronized video data.

[0044] The same or similar advantages and beneficial aspects mentioned with respect to the above video reproduction system also apply in the same or similar manner to the method described below. This explicitly includes the features of the described embodiments and the technical advantages resulting from the characteristics of the embodiments.

[0045] The audiovisual data stream includes an audio data stream (i.e., the audio component) and a video data stream (i.e., the video component). In one embodiment, the audiovisual data stream is a movie or a TV program or a 360° video including sound. In another embodiment, the audiovisual data stream is a live audio-visual production, in particular a live audio-visual production of a musical, a sports event, an esports event, an exhibition, a conference, or any type of interactive live media. In another embodiment, the video data stream is an XR data stream. In another embodiment, the video data stream contains three-dimensional video data. In a preferred embodiment, the audio data stream is an analog audio data stream. In another embodiment, the audio data stream is a digital audio data stream.

[0046] In a preferred embodiment, the synchronization unit is a computer hardware device, particularly including a processing unit and / or an analog-to-digital converter and / or a digital-to-analog converter. In another embodiment, the synchronization unit is a single-board computer. In another embodiment, the method includes receiving the audio data stream via a wired connection, in particular via a twisted pair cable with an RJ45 connector and / or via a telephone connector and / or an XLR connector.

[0047] In another embodiment, the time code signal is time information contained in the audio data stream, where the audio data stream is analog or digital, and the time code signal is processed by the synchronization unit. The synchronization unit converts the time code signal into a time stamp signal, in particular a digital time stamp signal. In another embodiment, converting the time code signal includes extracting the time code signal from the audio data stream and processing it into a digital time stamp signal. The time stamp signal preferably adopts a time format or preferably adopts a movie-based format including, for example, frame numbers.

[0048] In another embodiment, the synchronization unit transmits a timestamp signal to a plurality of head-mounted displays, particularly at predefined time intervals. In another embodiment, the synchronization unit sends the timestamp signal upon request of each of the head-mounted displays. In one embodiment, the predefined time interval is a regular time interval.

[0049] In an embodiment for transmitting a timestamp signal from a synchronization unit to a plurality of head-mounted displays, a wireless network connection is established. In another embodiment, the wireless network is established by the synchronization unit, wherein, in particular, the synchronization unit includes a network unit which is particularly a wireless router or a wireless access point. In another embodiment, each of the head-mounted displays includes a wireless communication unit, wherein the wireless communication unit establishes the wireless network connection and receives the timestamp signal transmitted by the synchronization unit. The wireless connection supports the flexibility of the head-mounted displays and creates a better user experience.

[0050] In a preferred embodiment, the head-mounted display is an XR headset, particularly an AR headset and / or an MR headset and / or a VR headset. In a preferred embodiment, the method includes the step of determining the orientation and / or position of each of the head-mounted displays in a given coordinate system, wherein the coordinate system is particularly related to a room (such as a cinema hall or a living room). In another embodiment, the method includes tracking the user's eyes by an eye-tracking device and / or tracking the user's head by a head-tracking device.

[0051] In another embodiment, the synchronized video data is reproduced by each of the head-mounted displays using monoscopic 2D video data or stereoscopic 3D video data. The reproduced video data is synchronized with the timestamp signal originating from an audio data stream.

[0052] In another embodiment, the method includes the step of calibrating the spatial position and / or the rotational orientation of each of the head-mounted displays, particularly in a room. Preferably, a sound system is arranged in the room. This enables the visual direction of the user to be aligned with the audio direction of the sound system.

[0053] In another embodiment, the method for synchronously reproducing a video data stream further includes providing an audio data stream from an audio data source to the synchronization unit, and further receiving a live stream as the audio data stream from the audio data source and / or retrieving the audio data stream from a data storage, particularly decoding the audio data stream from Digital Cinema Package (DCP) data.

[0054] In another embodiment, the audio data source provides an audio data stream and decodes another digital movie data format. In another embodiment, the audio data source does not decrypt and / or decode the audio data. In another embodiment, the audio data source receives an audio data stream from the Internet. In another embodiment, the audio data source generates an audio data stream from an audio data storage device. Through these embodiments, the method is applicable to reproducing various movie contents conforming to different industry standards.

[0055] In another embodiment, the method for synchronously reproducing a video data stream includes storing video data in a data storage device, where the data storage device is part of each of the head-mounted displays, and generating a video data stream based on the video data stored in the data storage device. In one embodiment, video data such as a movie or 360° video is stored in the video data storage device. Each of the head-mounted displays reproduces a portion of the video data stream corresponding to the timestamp signal received from the synchronization unit. This achieves the reproduction of video and audio data at the same time position. Storing the video data locally can achieve high availability and low power consumption, thereby extending the battery life of the head-mounted display.

[0056] In another embodiment, the method for synchronously reproducing a video data stream includes converting a timecode signal into a digital timestamp signal, where the audio data stream is an analog audio data stream. In particular, the method further includes the synchronization unit detecting errors in the analog timecode signal, particularly errors related to consistency, and correcting the timecode signal through an error correction scheme. Correcting errors, particularly consistency errors, can enhance the audio experience as well as the synchronization between the audio data and the video data. Particularly when the audio data stream is an analog audio data stream, error correction significantly improves the quality of the processed timecode signal and the timestamp signal.

[0057] In another embodiment, the synchronization unit sends the audio data stream to the sound system in an unmodified form (i.e., without making changes). In another embodiment, the synchronization unit filters and / or processes the audio data stream and sends it to the sound system. This allows the synchronization unit to remain transparent by only intercepting the signal and transmitting it as it is, or allows improving the audio signal, for example, adjusting it according to the specifications of the user, the venue, or the speaker system.

[0058] In one embodiment, detecting an error is semantically checking the obtained timecode signal. In another embodiment, correcting the timecode signal is interpolating the timecode signal.

[0059] According to another embodiment, the audio data stream consists of at least four audio channels, at least eight audio channels, at least twelve audio channels, at least sixteen audio channels, wherein one channel particularly carries a timecode signal. The timecode signal is particularly an SMPTE timecode signal, particularly an SMPTE-LTC timecode signal. In another embodiment, the timecode signal is a MIDI timecode. Using different channels can achieve higher sound quality and a more three-dimensional sound experience, and make it easier for the synchronization unit to obtain the timecode signal.

[0060] In another embodiment, the method for synchronously reproducing a video data stream includes each of the head-mounted displays requesting a timestamp signal from the synchronization unit, particularly within a predefined time interval. This reduces network traffic. In one embodiment, the predefined time interval is to request once per frame or multiple times per second, approximately once per second, once every few seconds, or less. In another embodiment, the predefined time interval is a regular time interval.

[0061] In another embodiment, the synchronization unit provides a timestamp server, wherein the timestamp server provides a timestamp signal for the head-mounted devices. Each of the head-mounted displays requests and / or reads the timestamp signal from the timestamp server.

[0062] In another embodiment, the method for synchronously reproducing a video data stream includes recalibrating the timestamp signal to include the particularly variable respective transmission delays of the timestamp signals from the synchronization unit to each of the head-mounted displays. Particularly, the method further includes determining a clock signal that determines a reference time standard by the internal clock of the synchronization unit, transmitting the clock signal together with the timestamp signal from the synchronization unit to a plurality of head-mounted displays, receiving the clock signal by each of the head-mounted displays, determining the respective transmission delays according to the clock signal, and recalibrating the timestamp signal to compensate for the respective transmission delays. A network connection, particularly a wireless network connection, generates transmission delays that can easily reach multiples of dozens of milliseconds. By recalibrating the timestamp signal to include the respective transmission delays, it is ensured that the audio data stream and the video data stream are synchronously reproduced.

[0063] The clock signal determines a reference time standard by the synchronization unit, wherein the time standard is, for example, the local time of the synchronization unit or UTC. In another embodiment, the synchronization unit transmits the clock signal together with the timestamp signal so that each of the head-mounted displays compares multiple pairs of timestamp signals and clock signals and compares these values with the internal clock of the head-mounted display to determine the respective transmission delays.

[0064] In another embodiment, the method for synchronously reproducing a video data stream includes turning off the head-mounted display or putting it into a standby state when the corresponding head-mounted display detects user inactivity.

[0065] In another embodiment of the method for synchronously reproducing a video data stream, the audio-visual data stream further includes a movie physical effects data stream. In this embodiment, the method for synchronously reproducing a video data stream further includes the following steps: transmitting a time stamp signal from a synchronization unit to a movie physical effects unit, particularly through a wireless network connection, receiving the time stamp signal by the movie physical effects unit, determining synchronized movie physical effects data of the movie physical effects data stream for use by the movie physical effects unit to reproduce synchronously with the audio data stream using the time stamp signal, and reproducing the movie physical effects data by the movie physical effects unit.

[0066] Using the functions of 4D movies creates a more special experience for viewers when using a video reproduction system. In another embodiment, the movie physical effects data stream is based on 4DX. In this document, the term "4DX" refers to a 4D movie format developed by CJ 4DPlex.

[0067] This object is also achieved by a method for synchronously reproducing an audio-visual data stream, the audio-visual data stream including an audio data stream and a video data stream containing a time code signal, wherein the method includes the method for synchronously reproducing the video stream of the audio-visual data stream as described above, and includes transmitting the audio data stream from the synchronization unit to a sound system, reproducing the audio data stream by the sound system, determining synchronized video data of the video data stream for each of the head-mounted displays to be displayed synchronously with the reproduction of the audio data of the sound system.

[0068] By including a sound system and reproducing the audio data stream by the sound system, both reproductions of the audio-visual stream are included in this message. In this way, the synchronization between the reproductions of the two devices can even be improved.

[0069] This object is also achieved by a computer-readable storage medium including instructions that, when executed by a computer, cause a media reproduction system to perform the method for synchronously reproducing a video data stream or the method for synchronously reproducing an audio-visual data stream as described above. Such a computer-readable storage medium allows a media reproduction system (e.g., a video reproduction system or a media reproduction system as described above) to perform such a method and synchronize the reproduction of the video data stream and the audio data stream according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] According to the embodiments of the present invention and the description of the claims and the drawings, other features of the present invention will become apparent. Separate features or combinations of several features can be implemented according to the embodiments of the present invention.

[0071] The present invention will be described based on exemplary embodiments, but does not limit the overall intention of the present invention. Among them, the drawings are explicitly referred to in order to disclose all details according to the present invention that are not explained in more detail herein. The drawings show:

[0072] Figure 1 is a schematic diagram of a first embodiment of a video reproduction system,

[0073] Figure 2 is another schematic diagram of a second embodiment of a video reproduction system,

[0074] Figure 3 is a flowchart of steps performed by a synchronization unit in a first embodiment of a method for synchronously reproducing a video data stream of an audiovisual data stream, and

[0075] Figure 4 is another flowchart of steps performed by a corresponding head-mounted display in a first embodiment of a method for synchronously reproducing a video data stream of an audiovisual data stream,

[0076] Figure 5a is a schematic diagram of a first embodiment of an audiovisual data stream,

[0077] Figure 5b is a schematic diagram of a second embodiment of an audiovisual data stream,

[0078] Figure 6 is a schematic diagram of an embodiment of a head-mounted device. Detailed Description of the Invention

[0079] In the drawings, elements or corresponding components of the same or similar type have the same reference numerals to prevent the need to reintroduce the item.

[0080] Figure 1FIG. 0 schematically shows a video reproduction system 10 according to a first embodiment. The video reproduction system 10 includes a synchronization unit 20 which is a single-board computer including a processing unit and audio input and output connectors in this embodiment. The synchronization unit 20 receives an audio data stream 61 including a time code signal 62 from an audio data source 30 via a wired connection 32. The audio data source provides the audio data stream 61 by decoding DCP data retrieved from a data memory 31. The wired connection 32 is a twisted pair cable and is connected to the synchronization unit 20 through an RJ45 connector. The audio data stream 61 includes an audio signal of an audiovisual data stream, wherein the audiovisual data stream is an XR media data stream. The audio data stream 61 is an analog signal and includes 16 channels, wherein one channel carries the time code signal 62. The time code signal 62 is an SMPTE-LTC time code signal. The synchronization unit 20 sends an audio data stream 66 including a time code signal 67 to a sound system 40 via a wired connection 33. In this embodiment, the sound system 40 is a surround sound system configured to reproduce the audio data stream 66. In this embodiment, the audio data stream 61 and the time code signal 62 are the same as the audio data stream 66 and the time code signal 67.

[0081] The synchronization unit 20 processes the audio data stream 61 including the time code signal 62 and converts the analog time code signal 62 into a digital time stamp signal 25. The synchronization unit 20 is configured to send the time stamp signal 25 to a plurality of head-mounted displays 50. In this embodiment, the synchronization unit 20 is configured to establish a wireless network connection 56 to the plurality of head-mounted displays 50. To this end, the synchronization unit 20 includes a wireless access point. The wireless network connection 56 is based on Wi-Fi, particularly Wi-Fi 4 using the 2.4 GHz band and Wi-Fi 5 using the 5 GHz band. The head-mounted display 50 receives the time stamp signal 25 via the wireless network connection 56. The head-mounted display 50 reproduces a video data stream which is locally stored on a data memory device of the corresponding head-mounted memory 50. The head-mounted display 50 reproduces the video data stream synchronized with the audio data streams 61 and 66 by using the time stamp signal 25. This brings a synchronized experience of the video data stream displayed on the head-mounted display 50 and the audio data stream reproduced by the sound system 40 to the user of the head-mounted display 50.

[0082] Figure 2 FIG. 7 schematically shows an embodiment of a media reproduction system 11. Figure 2 The overall structure of the shown embodiment is similar to Figure 1An embodiment of the video reproduction system 10 shown here. Here, the synchronization unit 20 is divided into a time stamp unit 23 and a network unit 24. The time stamp unit 23 is a small computer 802. The computer 802 is equipped with a computer-readable storage medium 801 including instructions, which, when executed by the computer 802, cause the media reproduction system 11 to execute the method of synchronously reproducing the video data stream as described above. The network unit 24 is a wireless router. The time stamp unit 23 receives an audio data stream 61 including a time code signal 62 from the audio data source 30 via a wired connection 32. The audio data stream 61 is an analog data stream, and the time code signal 62 is an SMPTE-LTC time code signal. In this embodiment, the audio data source 30 is configured to receive the audio data stream 61 via the Internet and provide the audio data stream 61 without decrypting the audio data. The wired connection 32 is connected to the audio data source 30 and the time stamp unit 23 via a telephone connector. The time stamp unit 23 sends an audio data stream 66 including a time code signal 67 to the audio interface 41 via a wired connection 33. Through the wired connection 34, the audio interface 41 sends the audio data stream 66 to the sound system 42, which reproduces the audio data stream 66. In this embodiment, the audio interface 41 includes a connector for inputting the audio data stream 66 and an amplifier for reproducing the audio data stream 66. In another embodiment not shown herein, the sound systems 40, 42 include the audio interface 41.

[0083] The time stamp unit 23 converts the audio data stream 61 including the analog time code signal 62 into a digital time stamp signal 25. In addition, the time stamp unit 23 includes an internal clock and determines a clock signal 26 through this internal clock. The time stamp unit 23 transmits the clock signal 26 together with the time stamp signal 25 to the network unit 24 via a wired connection 35, where the wired connection 35 is a twisted pair cable with an RJ45 connector. The network unit 24 establishes a wireless network connection 56 with a plurality of head-mounted displays 50, 52, 53. This wireless network connection is a Wi-Fi 5 connection. In this embodiment, different types of head-mounted displays 50, 52, 53 are used. In this embodiment, by way of example and not limitation of other embodiments, the head-mounted display 50 is a VR headset, the head-mounted display 52 is an MR headset, and the head-mounted display 53 is an AR headset. The network unit 24 transmits the time stamp signal 25 together with the clock signal 26 to the plurality of head-mounted displays 50, 52, 53 via the wireless network connection 56.

[0084] Each of the head-mounted displays 50, 52, 53 is configured to receive the clock signal 26 and determine its respective transmission delay based on the clock signal 26. The respective transmission delays depend at least in part on the stability of the wireless network connection 56 and the distance between the network unit 24 and the head-mounted displays 50, 52, 53, and thus vary over time and vary for different head-mounted displays 50, 52, 53. Using the respective transmission delays, the different head-mounted displays 50, 52, 53 recalibrate the timestamp signal 25 to compensate for the respective transmission delays between the timestamp unit 23 and the head-mounted displays 50, 52, 53.

[0085] Figure 3 A flowchart showing the steps performed by the synchronization unit 20 in a first embodiment of a method for synchronously reproducing a video data stream of an audiovisual data stream is shown. The synchronization unit 20 receives 301 an audio data stream, wherein the audio data stream 311 is provided by the audio data source 30. The audio data source 30 decodes the DCP data to provide the audio data stream 311, which is an analog audio data stream 311. In the next step, the synchronization unit 20 converts 303 the timecode signal into a digital timestamp signal. In this embodiment, the synchronization unit 20 only intercepts and does not modify the audio data stream 311. In the next step, the synchronization unit 20 sends 305 the audio data stream 311 to the sound system, wherein the sound system performs audio reproduction 321. In addition, the synchronization unit 20 transmits 304 the timestamp signal 322 such that the timestamp signal 332 is provided to the plurality of head-mounted displays 50, 52, 53.

[0086] Figure 4 A flowchart showing the steps performed by each of the head-mounted displays 50, 52, 53 in a first embodiment of a method for synchronously reproducing a video data stream of an audiovisual data stream is shown. The timestamp signal 422 is received by each of the head-mounted displays 50, 52, 53 that is processing 401 the timestamp signal. Each of the head-mounted displays 50, 52, 53 determines 402 the synchronized video data of the video stream. In this embodiment, this includes generating a video data stream based on the video data stored on the data storage device, which is part of each of the head-mounted displays 50, 52, 53. In the next step, each of the head-mounted displays 50, 52, 53 reproduces 403 the synchronized video data, i.e., it displays the video that is temporally aligned with the audio data. This achieves a synchronized video reproduction 431 that runs in synchronization with the audio stream data reproduced in the audio reproduction 321.

[0087] In Figure 4In an alternative embodiment not shown, prior to processing step 401, each of the head-mounted displays 50, 52, 53 requests a timestamp signal 422 from the synchronization unit 20. This request is sent periodically at a predefined time interval, where, in this embodiment, the predefined time interval is once every few seconds. In response to this request, the timestamp signal 422 is transmitted from the synchronization unit 20 to each of the head-mounted displays 50, 52, 53 that has sent the request. Then, as Figure 4 shown, processing step 401 and subsequent steps are performed.

[0088] Figure 5a is a schematic diagram of a first embodiment of the audiovisual data stream 501. The audiovisual data stream 501 includes a video data stream 502 and an audio data stream 503.

[0089] Figure 5b is a schematic diagram of a second embodiment of the audiovisual data stream 501. In this embodiment, the audiovisual data stream 501 includes a video data stream 502, an audio data stream 503, and a movie physical effect data stream 504. The movie physical effect data stream 504 contains information about when to perform movie physical effects, where the reproduction of the movie physical effects is adapted to the audio and video components of the audiovisual data stream.

[0090] Figure 6 shows a schematic diagram of another embodiment of the head-mounted device 50. The head-mounted device 50 includes a data storage device 602, an inertial measurement unit 603, a screen 604, a wireless communication unit 605, and an internal clock 606. The screen 604 serves as a display for reproducing the video data stream 502. The internal clock 606 is used to determine a clock signal for the reference time standard.

[0091] All features mentioned, including those extracted only from the drawings and individual features disclosed in combination with other features, are considered important for the present invention both individually and in combination. Embodiments according to the present invention can be implemented by individual features or combinations of several features. Features combined with the words "in particular" or "especially" should be regarded as preferred embodiments.

[0092] Reference List

[0093] 10 Video reproduction system

[0094] 11 Media reproduction system

[0095] 20 Synchronization unit

[0096] 23 Timestamp unit

[0097] 24 Network unit

[0098] 25 Timestamp signal

[0099] 26 Clock signal

[0100] 28 Internal clock

[0101] 30 Audio data source

[0102] 31 Data storage

[0103] 32, 33, 34, 35 Wired connection

[0104] 40 Sound system

[0105] 41 Audio interface

[0106] 42 Sound system

[0107] 50 VR headset

[0108] 52 MR headset

[0109] 53 AR headset

[0110] 56 Wireless network connection

[0111] 61 Audio data stream

[0112] 62 Time code signal

[0113] 66 Audio data stream

[0114] 67 Time code signal

[0115] 301 Receive audio data stream

[0116] 303 Convert time code signal to timestamp signal

[0117] 304 Transmit timestamp signal

[0118] 305 Send audio data stream to sound system

[0119] 311 Audio data stream

[0120] 321 Audio reproduction

[0121] 322 Timestamp signal

[0122] 401 Process timestamp signal

[0123] 402 Determine synchronized video data of video data stream

[0124] 403 Reproduce synchronized video data

[0125] 422 Timestamp signal

[0126] 431 Synchronized Video Reproduction

[0127] 501 Audio-Visual Data Stream

[0128] 502 Video Data Stream

[0129] 503 Audio Data Stream

[0130] 504 Movie Physical Effect Data Stream

[0131] 602 Data Storage Device

[0132] 603 Inertial Measurement Unit

[0133] 604 Screen

[0134] 605 Wireless Communication Unit

[0135] 606 Internal Clock

[0136] 801 Computer-Readable Storage Medium

[0137] 802 Computer

Claims

1. A video reproduction system (10), the video reproduction system (10) being configured to reproduce a video data stream (502) of an audiovisual data stream (501), the audiovisual data stream (501) including an audio data stream (61, 66, 503) and a video data stream (502) containing a time code signal (62, 67), characterized in that, The video reproduction system (10) includes a synchronization unit (20) and a plurality of head-mounted displays (50, 52, 53), in particular extended reality headsets, wherein the synchronization unit (20) is configured to: Receive audio data streams (61, 66) from an audio data source (30), Send the audio data streams (61, 66) to a sound system (40), Convert the time code signals (62, 67) of the audio data streams (61, 66) into time stamp signals (25), Transmit the time stamp signals (25) to the plurality of head-mounted displays (50, 52, 53), Each of the head-mounted displays (50, 52, 53) is configured to: Receive the time stamp signals (25), Reproduce the video data stream (502), Synchronize the reproduction of the video data stream (502) with the audio data streams (61, 66) using the time stamp signals (25).

2. The video reproduction system (10) according to claim 1 further includes an audio data source (30), the audio data source (30) being configured to provide an audio data stream (61) to the synchronization unit (20), wherein, The audio data source (30) is configured to receive a live stream as the audio data stream (61), and / or the audio data source (30) is configured to retrieve the audio data stream (61) from a data memory (31), in particular configured to decode the audio data stream (61) from digital cinema package (DCP) data.

3. The video reproduction system (10) according to claim 1 or 2, wherein, Each of the head-mounted displays (50, 52, 53) includes a data memory device (602) on which video data is stored, wherein each of the head-mounted displays (50, 52, 53) is configured to generate the video data stream (502) based on the video data stored on the data memory device (602).

4. The video reproduction system (10) according to one of the foregoing claims, wherein, The time code signals (62, 67) are analog signals, and the time stamp signals (25) are digital signals, and the synchronization unit (20) is configured to convert the analog time code signals (62, 67) into the digital time stamp signals (25), wherein, in particular, the synchronization unit (20) is configured to detect errors in the analog time code signals (62, 67), in particular errors regarding consistency, and to correct the time code signals (62, 67) through an error correction scheme.

5. The video reproduction system (10) according to one of the foregoing claims, wherein, The synchronization unit (20) includes a time stamp unit (23) and a network unit (24), wherein the time stamp unit (23) is configured to receive the audio data stream (61), send the audio data streams (61, 66) and convert the time code signal (62) of the audio data stream (61) into a time stamp signal (25), and the network unit (24) is configured to establish a wireless network connection (56) to each of the head-mounted displays (50, 52, 53) and transmit the time stamp signal (25) from the time stamp unit (23) to the plurality of head-mounted displays (50, 52, 53) via the wireless network connection (56).

6. A video reproduction system (10) according to one of the preceding claims, wherein, Each of the head-mounted displays (50, 52, 53) is configured to request and / or read a timestamp signal (25) from the synchronization unit (20), wherein, in particular, each of the head-mounted displays (50, 52, 53) is configured to request and / or read the timestamp signal (25) from the synchronization unit (20) at a predefined time interval.

7. A video reproduction system (10) according to one of the preceding claims, wherein, The synchronization unit (20) and each of the head-mounted displays (50, 52, 53) are configured to recalibrate the timestamp signal (25) to compensate for the respective, particularly variable, transmission delays of the timestamp signal (25) from the synchronization unit (20) to each of the head-mounted displays (50, 52, 53), wherein, in particular, the synchronization unit (20) and each of the head-mounted displays (50, 52, 53) each have an internal clock (28, 606) for determining a reference time standard of a clock signal (26), and the synchronization unit (20) is configured to transmit the clock signal (26) together with the timestamp signal (25) to a plurality of head-mounted displays (50, 52, 53), and each of the head-mounted displays (50, 52, 53) is configured to receive the clock signal (26) and determine its respective transmission delay based on the clock signal (26), and to recalibrate the timestamp signal (25) in order to compensate for the respective transmission delays.

8. A media reproduction system (11) configured to reproduce an audiovisual data stream (501) including an audio data stream (61, 66, 503) containing a time code signal (62, 67) and a video data stream, characterized in that, The media reproduction system (11) comprises a video reproduction system (10) and an audio system (40) according to any one of claims 1 to 8, wherein the synchronization unit (20) is configured to: Send the audio data stream (61, 66) to the audio system, The audio system (40) is configured to: Receive the audio data stream (66) from the synchronization unit (20), Reproduce the audio data stream (61, 66), Each of the head-mounted displays (50, 52, 53) is configured to: Synchronize the reproduction of the video data stream (502) with the audio reproduction of the audio data stream (61, 66) by the audio system (40) using the timestamp signal (25).

9. A method for synchronously reproducing a video data stream (502) of an audiovisual data stream (501), the audiovisual data stream (501) comprising an audio data stream (311, 503) and a video data stream (502) containing time code signals (62, 67), wherein, The method comprises the steps of: Receiving an audio data stream (311) by the synchronization unit (20), Converting the timecode signal (62, 67) from the audio data stream (311) by the synchronization unit (20) into a particularly digital timestamp signal (25), Transmitting the timestamp signal (25) from the synchronization unit (20) to a plurality of head-mounted displays (50, 52, 53), particularly via a wireless network connection (56), Sending the audio data stream (311) from the synchronization unit (20) to the audio system (40), Receiving the timestamp signal (25) by each of the head-mounted displays (50, 52, 53), Determining synchronized video data of the video data stream (502) for each of the head-mounted displays (50, 52, 53) to be displayed in synchronization with the audio data stream (311) using the timestamp signal (25), and Synchronously reproduced video data is reproduced by each of the head-mounted displays (50, 52, 53).

10. The method for synchronously reproducing a video data stream according to claim 9 further includes providing the audio data stream (311) from the audio data source (30) to the synchronization unit (20), and further receiving a live stream as the audio data stream (61) from the audio data source (30) and / or retrieving the audio data stream (61) from the data memory (31), in particular decoding the audio data stream (61) from digital cinema package (DCP) data by the audio data source (30).

11. The method for synchronously reproducing a video data stream according to claim 9 or 10 further includes storing video data on a data storage device (602), wherein, The data storage device (602) is part of each of the head-mounted displays (50, 52, 53) and generates the video data stream (502) based on the video data stored in the data memory device (602).

12. The method of synchronously reproducing a video data stream (502) according to any one of claims 9 to 11, further comprising converting the time code signal (62, 67) into a digital time stamp signal (25), wherein, The audio data streams (311, 502) are analog audio data streams. In particular, the method further includes detecting errors in the analog time code signals (62, 67) by the synchronization unit (20), in particular errors regarding consistency, and correcting the time code signals (62, 67) by an error correction scheme.

13. The method for synchronously reproducing a video data stream (502) according to any one of claims 9 to 12 further includes, by each of the head-mounted displays (50, 52, 53), in particular at a predetermined time interval, requesting the timestamp signal (25) from the synchronization unit (20).

14. The method for synchronously reproducing a video data stream (502) according to any one of claims 9 to 13 further includes recalibrating the timestamp signal (25) to include the respective, particularly variable, transmission delays of the timestamp signal (25) from the synchronization unit (20) to each of the head-mounted displays (50, 52, 53). In particular, the method further includes determining, by the internal clock (28) of the synchronization unit (20), a clock signal (26) of a reference time standard, transmitting the clock signal (26) together with the timestamp signal (25) from the synchronization unit (20) to a plurality of head-mounted displays (50, 52, 53), receiving the clock signal (26) by each of the head-mounted displays (50, 52, 53), determining the respective transmission delays based on the clock (26), and recalibrating the timestamp signal (25) to compensate for the respective transmission delays.

15. A computer-readable storage medium (801) comprising instructions that, when executed by a computer (802), cause a media reproduction system (11) to perform the method according to any one of claims 9 to 14.

Images