How to synchronize an additional signal to a primary signal

The method synchronizes additional signals to primary signals by comparing feature sequences to a database, addressing inefficiencies in existing synchronization methods and achieving precise media playback without relying on a fixed time reference.

JP7583887B2Active Publication Date: 2024-11-14レッジ·テクノロジーズ·エスピー·ゼット·オー·オー
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2023172390
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-03-02
Filing Date
2023-10-03
Publication Date
2024-11-14
Estimated Expiration
2038-12-19

AI Technical Summary

Technical Problem

Existing methods for synchronizing additional signals to primary signals are not fast, robust, or accurate, particularly when the primary signal lacks a fixed time reference, leading to inefficiencies in media playback and synchronization.

Method used

A method that extracts signal feature sequences from both primary and additional signals, compares them to a database of stored feature sequences, and assigns synchronization information based on matching sequences, allowing for precise synchronization without relying on a fixed time reference in the primary signal.

Benefits of technology

Enables fast, robust, and accurate synchronization of additional signals to primary signals, even when the primary signal lacks a fixed time reference, reducing latency and ensuring precise media playback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583887000001
    Figure 0007583887000001
  • Figure 0007583887000002
    Figure 0007583887000002
  • Figure 0007583887000003
    Figure 0007583887000003
Patent Text Reader

Abstract

To provide a quick, robust, and precise method by which an additional signal can be output synchronously to an arbitrary, even continuous, primary signal.SOLUTION: A method for synchronizing an additional signal 10 to a primary signal 4 includes: transmitting primary signals from a broadcast studio 3 to multiple reproducing devices; extracting, by a synchronization server 5, at least one signal feature sequence of the primary signal; and comparing it to DB feature sequences stored in a database 9 to generate synchronization information related to the primary signals; if the signal feature sequence matches one of the DB feature sequences to a predetermined degree, allocating synchronization information of the matched DB feature sequences to the primary signal at a position specified by the signal feature sequence; transmitting the generated synchronization information to a reproducing device 6; and outputting, by the reproducing device, an additional signal to the primary signal based on the synchronization information.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The invention relates to a method for synchronizing an additional signal with a primary signal and to a device for synchronizing an additional signal with a primary signal. The signal is a "continuous signal", a continuous signal being understood to be a signal that can be described by a characteristic sequence of time-continuous features. Typical continuous signals are audio and video signals, which can be sampled at regular intervals to generate corresponding features. However, a continuous signal can also be a signal used to transmit digitally encoded text. [Background technology]

[0002] US9,609,034B2 discloses a method for identifying media data by metadata. WO2016 / 085414A1 allows a mobile platform to recognize a television station and receive information appropriate to it during commercial breaks. EP2507790B1 describes a method and system for channel-invariant robust audio hashing with subsequent comparison of two audio hashes. The audio signal is first divided into fragments with a typical length of 2 seconds. These fragments are then further divided into frames with a typical length of 0.36 seconds. The frames are Fourier transformed and the resulting data are then normalized. A hash value is obtained by quantizing these data.

[0003] WO2012 / 049223A2 describes a method for synchronizing an alternative audio signal with a combined video and audio signal. Two possibilities in this regard are presented. Firstly, a watermark method is described, which constitutes an additional signal imperceptible to humans with respect to the video and audio signals and can be described, for example, as a modulation of the primary audio signal. Another method describes a fingerprint method. The main audio signal is characterized based on amplitude, frequency, zero crossing rate, tempo, spectral flatness, bandwidth and / or an audio fingerprint and compared with the corresponding features of the second signal. If the position of the second signal is detected, it can be adapted in time to the primary signal.

[0004] WO2012 / 049223, WO2014 / 018652A2 describe a method for synchronizing an alternative audio signal with a combined video and audio signal. Again, a fingerprinting process is used as a method. In this case, the fingerprint of the main audio signal is combined in its entirety with the second signal. The fingerprints of the second signal and the first signal are preloaded on the device to be synchronized, so that during synchronization, only the first signal is analyzed and compared with the fingerprints on the device. WO03003743A2 discloses a method and a device for providing a synchronization signal for synchronizing parallel media. A synchronization server is coupled to a communication network and connected to a broadcast media start time database. The synchronization server receives a synchronization request from a user client via the communication network. The synchronization server uses the synchronization request and the broadcast media start time database to generate synchronization data.

[0005] WO2014209179A1 describes a method and a transceiver for network diversity in long-distance communication. The method in a main node comprises the following steps for communicating with a long-distance destination node: · Synchronizing the use of at least one communication resource with a supplementary node. Generate main and auxiliary signals from the amount of information. · Transmitting a main signal to a destination node via at least one synchronized communication resource. · Transmitting an auxiliary signal to an auxiliary node via at least one synchronized communication resource. Summary of the Invention

[0006] It is an object of the present invention to provide a fast, robust and accurate method and corresponding device, which can output an additional signal synchronously to an arbitrary, continuous primary signal. Another object is to provide a streaming method with low latency. Another object is to provide a fast, robust and accurate method, which can measure and calibrate the delay between reception and output of a media playback device. Another object is to provide a method of synchronizing an additional signal to a primary signal, which measures the time delay between these two signals and adapts at least one of these signals so that the time delay is as small as possible. These objects are achieved by the features of the independent claims. Advantageous modifications and preferred embodiments are the subject of the dependent claims.

[0007] According to a first aspect of the invention, a method for synchronizing an additional signal to a primary signal is provided, comprising the following steps: - generating synchronization information for the primary signal by extracting at least one signal feature sequence of the primary signal and comparing it with DB feature sequences stored in a database, where if the signal feature sequence matches one of the DB feature sequences in a predetermined percentage, the synchronization information of the matching DB feature sequence is assigned to the primary signal at a position specified by the signal feature sequence. - transmitting synchronization information to a playback device, which outputs an additional signal in synchronization with the primary signal based on the synchronization information;

[0008] According to this method, one or more signal feature sequences of the primary signal are extracted and compared with corresponding previously stored DB features in the database, so that the primary signal does not inherently have to have specific time information, such as a starting point. For example, the primary signal can be a continuously transmitted television signal that does not have a starting point. By comparing the feature sequences, one of the signal feature sequences of the primary signal can be identified by matching it with the corresponding DB feature sequence. Thereby, synchronization information can be assigned to the primary signal at a specified position relative to the identified signal feature sequence. In other words, this means that the synchronization information relates to the location or position of the identified signal feature sequence of the primary signal.

[0009] This synchronization information is for example stored together with the DB feature sequence in a database. However, this synchronization information can also include a time, in particular a server time of a synchronization server on which the method is executed, which is for example extracted when an additional signal with this signal feature sequence is received from the synchronization server or is detected when an additional signal with this signal feature sequence is transmitted from the synchronization server to a playback device. This signal feature sequence therefore constitutes a specific location of the primary signal to which a specific time, in particular a server time, is assigned. This specific time can also be extracted from the time information contained in the signal and assigned to the respective signal feature sequence. Naturally, the method can also be used for primary signals with a specific starting point. This makes the method according to the invention significantly more flexible than a general method of the kind known from EP 1 307 833 B1, which requires a primary signal with a fixed time reference.

[0010] In the method according to the invention, for example, this time reference is generated during the passage of the primary signal (on the fly) through the synchronization server, where a comparison to the DB feature sequence is used to identify at least one signal feature sequence of the primary signal to which corresponding synchronization information can be assigned. Furthermore, it is possible to determine the time of the signal feature sequence of the primary signal that matches the DB feature sequence, and this time can be assigned to this extracted signal feature sequence as an extraction time to generate the synchronization information. By using such an extraction time, time information assigned to the primary signal is generated, which connects a location or point of the primary signal to a time. Such an assignment of time information to the primary signal is also meaningful if the primary signal already contains time information, for example in the form of a time marker. If necessary, this is used to generate a second reference, which can be connected to further information. Furthermore, the DB feature sequence can be assigned time information that defines a specific DB time associated with the DB feature sequence used to generate the synchronization information. This time information is typically stored together with the DB feature sequence in a database. For example, a particular time is indicated when a signal feature sequence matching this DB feature sequence occurs at a particular point within a larger signal segment, such as a film. Additional signals can then be synchronized on a playback device with respect to this DB time.

[0011] Synchronization information may also be assigned to the additional signal by extracting the signal feature sequence of the additional signal and comparing it to the DB feature sequences stored in the database. If the signal feature sequence matches one of the DB feature sequences in a predetermined percentage, the synchronization information is assigned to the additional signal at the location specified by the signal feature sequence. Synchronization information may also be assigned to the additional signal manually. For example, an operator may assign time information to the additional signal regarding when it is broadcast relative to the primary signal.

[0012] However, it is also possible to use an additional signal, to which synchronization information is already pre-assigned. The synchronization information can comprise one or more of the following data: - Time information identifying the time of the primary and / or additional signal, in particular this describes the time associated with the detected signal feature sequence. - one or more identification markers describing the content of the primary and / or additional signal. These identification markers may describe only the type of additional signal, for example a description of subtitles in a particular language, audio synchronization signals in a particular language, or the type of META information transmitted together with the additional signal. However, the identification markers may describe the additional signal in more detail and may reflect the structure of the additional signal. For example, if the additional signal is song lyrics, the identification markers may identify each verse of the song lyrics. However, it is also possible for the identification markers to describe other structures, such as chapters, opera repertoires, concert songs, episodes of a television series, etc. - a timestamp describing a particular time in the primary and / or additional signal, which is unrelated to one of the detected signal feature sequences, e.g., a particular point in the film, which is related to a reference point that is not related to the detected signal feature sequence, which reference point is typically the start of a particular segment of the primary signal, the additional signal, or a corresponding signal. - A time offset describing the time interval required to relay the primary and / or additional signals from a particular point in the transmission path to the actual output of the primary and / or additional signals at the regenerative device.

[0013] Therefore, this synchronization information does not constitute synchronization information describing characteristics of the primary and / or additional signals, but rather describes characteristics of the device on which the method is executed. This synchronization information can be configured very differently depending on the application. Using the above described aspects of the invention, the synchronization information of the matched DB feature sequence is assigned to the primary signal at the position specified by the signal feature sequence. Specific information in the database is assigned to the DB feature sequence. However, this information does not necessarily have to include time information. It could for example be metadata describing the meaning of the DB feature sequence (e.g. music title, opera repertoire, etc.) of the segment of the signal in which this DB feature sequence was found.

[0014] Then, the synchronization information can be generated based on the time information contained in the primary signal, for example, from which it is extracted together with the signal feature sequence, or an extraction time, which can be generated based on the extraction time combined with this meta information, thus resulting in synchronization information that the additional signal can be assigned with the same meta information. The assignment and synchronization time is derived based on the extracted time information or based on the extraction time. In other words, this means that the synchronization information is assigned to the primary signal, and there is a time relationship with the synchronization information with respect to the primary signal. The invention is also based on the discovery that media signals from different signals often have similar feature segments.

[0015] These feature segments do not have to be exactly the same. For example, if the primary signal is a high quality audio signal of a concert and the additional signal is a video signal with a low quality audio signal, based on the low quality audio signal, for example when a musician is greeted with applause, the additional signal can be synchronized very accurately, even if the quality of the two audio signals differs significantly, since the audio features here are very similar. This is equally possible with a video signal recorded by, for example, a professional camera or a video signal recorded by a mobile phone.

[0016] The inventors have found that, based on these feature segments, it is possible to perform an automatic identification of one or more specific signal feature sequences in order to synchronize different signals. The synchronization information can be generated in a synchronization server, which is implemented independently of the playback device. This synchronization information must then be transmitted to the playback device, where the additional signal is output in synchronization with the primary signal. In this case, the synchronization of the playback device and the synchronization server can also be performed, for example, by determining the time intervals required to transmit the corresponding signal from a given point, in particular from the synchronization server, to the playback device. However, the synchronization server can also be embodied in the playback device itself.

[0017] When digital transmission of data between the synchronization server and the playback device is used, this time interval varies and is therefore generally indeterminable. The playback device can synchronize the additional signal to the primary signal by outputting the primary signal and the additional signal (each including one or more time markers as synchronization information) to the playback device. Based on the time markers, the playback device can synchronously assign the additional signal to the primary signal and output it synchronously. Furthermore, based on a playback time measured by the playback device using a clock, the synchronization information can be used by the playback device to assign this playback time to the additional signal such that the additional signal is output synchronously to the primary signal.

[0018] In the former case, the primary signal and the additional signal are output by the same playback device, so the time markers of the two signals are sufficient to output the signals synchronously. However, if it is desirable not to insert time markers in the primary signal, for example if one wants to transmit the primary signal to the playback device as quickly as possible without a time lag, or if the primary signal is output by a different playback device to the additional signal, it is advantageous to use the synchronization information to assign the additional signal to a playback time measured by a clock in the playback device.

[0019] The synchronization information comprises corresponding information for assigning a playback time to the additional signal so that it is output synchronously to the primary signal. If the primary signal and the additional signal are output by different playback devices, it is advantageous to use the first time to which the output of the primary signal was assigned. This time therefore describes the time of the individual features or feature sequences in the primary signal. Basically, this can be achieved using the clock of the playback device at which the primary signal is output. However, if a synchronization server independent of the playback device for this primary signal is provided, the clock of the synchronization server can be used, in particular if the primary signal is continuously transmitted from the synchronization server to the playback device. In that case, it is only necessary to supplement the synchronization information with a corresponding time offset indicating the time interval by which the output of the primary signal on the playback device is delayed with respect to the passage on the synchronization server. For the playback device of the additional signal, synchronization information is provided which describes the relationship of the playback time at the playback device to the server time on the synchronization server. Thereby, it is possible to establish a relationship based on the playback time to the server time of the synchronization server, since the output of the primary signal is also synchronized to this server time.

[0020] If both the server time and the playback time are periodically synchronized to an independent or reference time (e.g. NTP: Network Time Protocol), the synchronization information does not need to include information about the relationship between these two times, since they should be considered identical within the framework of the existing level of measurement accuracy. A comparison of signal feature sequences with DB feature sequences shows that some signal feature sequences match to a certain degree. Therefore, it is basically possible to generate synchronization information using this plurality of signal feature sequences.

[0021] However, advantageously, the matches of all signal feature sequences that meet a predetermined degree of match within a predetermined time interval are evaluated, and the signal feature sequence with the best score is selected for assigning the synchronization information to the primary signal at the position specified in the signal feature sequence. Essentially, the goal is to achieve a unique assignment of the synchronization information to the primary signal. When using multiple signal feature sequences, this is not always guaranteed. The best synchronization can be achieved by using the signal feature sequence that best matches the DB feature sequence.

[0022] The predetermined rules for evaluating the proportion of matching of the signal feature sequence with the DB feature sequence include one or more of the following rules: -The more matching features in the feature sequence, the better the rating. The greater the number of matching features in the feature sequence relative to the number of features contained in the feature sequence, the better the rating. - A classification of the individual feature matches is performed: the shorter the distance between matching features, the better the classification assigned to the corresponding match, and this classification is taken into account in the overall assessment of the match of the feature sequences.

[0023] According to another aspect of the present invention, there is provided a method for synchronizing an additional signal with a primary signal, comprising the steps of: - Extraction of a signal feature sequence from the primary signal and storage of the signal feature sequence together with the time information in a database. Synchronization of the additional signal using this database. In particular, the feature sequence extracted from the additional signal is compared with the feature sequence stored in the database and / or the time information from the additional signal is compared with the corresponding time information in the database. This method can be used to create a database of the primary signal as it is being transmitted from the broadcasting station to the playback device. This database can be immediately used to synchronize the additional signal to this primary signal. It is thus possible to analyze and synchronize a live signal. For this reason, the database created in this way is also called a live database. The time information can be generated or extracted by means of time information of a clock provided by a corresponding server and / or based on time information contained in the primary signal.

[0024] According to another aspect of the present invention, a method for synchronizing an additional signal with a primary signal is created. A reference signal is output, which is received simultaneously by a corresponding sensor, in order to calibrate the transmission path from the server to the playback device and / or to calibrate the latency in the playback device for outputting the media signal at the playback device. The output reference signal and the reference signal received by the sensor are compared with each other to determine a time interval required for relaying the reference signal and / or actually outputting it to the playback device, which time interval is in turn used as a time offset based on time information related to the clock of the server and / or the playback device to determine the output time at which the signal is output on the media playback device.

[0025] This method can be used to automatically calibrate the transmission path or the playback device. The latency of the playback device varies greatly depending on whether, for example, the audio signal is output to a hardwired speaker, a Bluetooth connected speaker, or an underlying audio system. The time interval required to transmit the signal can vary significantly depending on the respective transmission path.

[0026] The method can be used to calibrate the transmission path and / or latency of the output device before or once or multiple times during playback of the media signal. The reference signal can include an audio signal, in which case the sensor is a microphone. The reference signal can also include a video signal, in which case a camera can be used as the sensor. The time interval can be determined by determining the transmission time and reception time of the reference signal, with the time interval being derived from the time difference between these two times.

[0027] If the transmission time and the reception time of the reference signal are measured at the same location, the determined time interval is half the time difference between these two times. If the transmission time of the reference signal is measured at the beginning of the transmission path and the reference time of the reference signal is measured directly by the sensor, the determined time interval is this time difference. One or both times can be performed by comparing the extracted reference feature sequence with one or more previously stored reference feature sequences. This method corresponds to the method described above for identifying a signal feature sequence based on a DB feature sequence. A comparison of such feature sequences can thus be used to determine the time. The accuracy of such times is limited by the length of the feature sequence used to determine the time. Typical lengths of such features are in the range of ±8 ms.

[0028] According to another aspect of the invention, a method for synchronizing an additional signal with a primary signal is provided, the additional signal being transmitted to the playback device from a synchronization server embodied to be independent of the playback device. In the device and the synchronization server, a clock provided in the synchronization server is used to generate synchronization information related to the server time of the synchronization server measured in the synchronization server. In the playback device, a playback device clock is provided for measuring the playback time, which is synchronized at least once with the server time, and a time drift of the playback time with respect to the server time is measured, which time drift is taken into account when synchronizing the additional signal with the primary signal. The additional signal is output to the playback device and can be controlled by the playback time available from the playback device. The time drift can have different causes: on the one hand, the playback device clock may not run at exactly the same speed as the server clock, and the clocks of different playback devices may not run at different speeds. On the other hand, the additional signal may be stretched or compressed in time compared to the primary signal, so that an additional signal that is exactly synchronized with the primary signal at a certain time will deviate more and more from the primary signal as the playback time increases.

[0029] Such a temporal stretching or compression is present, for example, during conversion of an analog signal to a digital signal by a corresponding analog-to-digital converter. The primary signal, the additional signal and the further signals are at least transmitted over different paths and therefore converted with different analog-to-digital converters at different locations. Each analog-to-digital converter has a clock generator (clock) that adds time information in the form of time markers to the digital signal. The times of the different clock generators can be slightly different. This leads to the fact that even if the primary signal and the additional signal are identical in the analog state, they have slightly different times in the digital state.

[0030] If they are played back at the same time on a playback device, a time offset between the two signals may occur as the playback time decreases. Time drift due to clocks or clock generators on different playback devices running at different speeds can be eliminated through regular comparison to a reference clock (e.g. atomic clock or TMP). In playback devices, the outputs often have their own clock signal and the playback devices have a control unit with its own clock. In such cases, it is advantageous if the clock signal of the playback unit is regularly synchronized with the clock of the control unit of the playback device, and the clock of the control unit of the playback device is synchronized with the reference clock regular intervals. Time drift due to compression or expansion of a signal can be measured. In this context, extraction of characteristic sequences can be used to determine compression or expansion.

[0031] Once the time drift has been determined, the additional signal can be played back in synchronization with the primary signal forever, without the need to perform a new synchronization between the additional signal and the primary signal at regular intervals. If the signal portion of the main signal with which the additional signal is synchronized is available, the synchronization of the two signals with each other can also be repeated without the need to take into account the time drift to achieve this. For example, it may be advantageous to convey the primary signal and the additional signal through a shared synchronization server before outputting them to the playback device, so that the complete primary and additional signals are available on the synchronization server and a new synchronization of the additional signal with respect to the primary signal can occur at any time. The primary signal can then be output on the playback device, even if the portion required for the synchronization of the two signals is not available, and a new synchronization between the two signals can nevertheless occur periodically.

[0032] The time drift can be induced through repeated comparisons to a reference time to calculate the respective time differences. The time drift is determined based on the deviation in the time differences. The larger the interval between the first and last comparison, the more accurately the time drift can be determined.

[0033] According to another aspect of the invention, a method for synchronizing an additional signal to a primary signal is created. In a first step, the timestamps of the available additional signal are transmitted to the playback device. It is thus possible to calculate the available buffer time. In this regard, the buffer time describes the time that is still available for the additional signal before it is played back in order to synchronize with the primary signal. At the same time, in this first transmission, the available bandwidth is determined. In a second step, the buffer time is used to encode the additional signal, to transmit it from the additional signal server to the playback device and to decode it again. In this case, the quality of the additional signal depends on the sensor available buffer time and on the available bit rate. By selecting the shortest possible encoding / decoding time with the same buffer time, a high signal quality can be achieved, but this results in a large amount of data requiring a correspondingly long transmission time, or by selecting a long encoding / decoding time, the bit rate is reduced and the transfer is accelerated. With different buffer times, hardware requirements and available bandwidth, the encoding / decoding time has to be determined again.

[0034] In practice, the method can be embodied such that on one or more servers the signals (primary signal and / or additional signal) are encoded in different ways, with different codecs, and therefore the signals are available in different qualities. At the playback device, a decision is made which signal to use and retrieve from the server. Furthermore, the additional signal is transmitted from the additional signal server to the playback device in chunks with a time length of 10 frames, corresponding to about 400 ms, in particular up to 5 frames, corresponding to a maximum of about 200 ms. At most 1 frame (corresponding to about 40 ms), and at the playback device, the additional signal is received by a local web server. By providing a direct link to the local web server via a web socket, the additional signal can essentially be received without delay. A direct connection is a connection that is maintained after the transmission event. The local web server is preferably compatible with the transmission protocol used by the playback device (in principle http), so that no changes need to be made to the playback device itself, except for the fact that a local web server needs to be added. This also allows the use of very short chunks.

[0035] In a traditional transmission process, such chunks would cause significant delays if a data connection were repeatedly established, due to the overhead caused by sequential polling operations and individual file requests. Essentially, it is possible to create a direct connection, which allows the chunks to be transmitted with almost no delay. However, this has the drawback that it can only be established between two specific partners and does not allow streaming to multiple recipients. Instead of a direct connection, a local web server can be embodied, which can request multiple chunks simultaneously, consecutively, or without waiting for the receipt of a previously requested chunk.

[0036] In a typical process, chunks are requested individually and additional chunks are requested when a previously requested chunk has already been received. This allows even very small chunks to be sent quickly. For example, common streaming techniques such as HLS or DASH can be used for this. Thus, the required transmission time can be minimized and, as a result, more time is available for encoding. This can be done at a higher data rate and can increase the signal quality. With this transmission method, the additional signal can typically arrive at the recipient in 2-3 seconds. The above-described aspects can be implemented individually or in any combination. In the context of the present invention, the additional signal can be synchronized to the primary signal. However, it is possible that multiple additional signals are synchronized to the primary signal. The present invention is explained in detail below by way of example based on the drawings. [Brief description of the drawings]

[0037] [Figure 1] A system for regenerating multiple camera signals synchronously with a primary signal is shown. [Diagram 2] 1 illustrates a system for loading additional external information related to a live broadcast. [Diagram 3] 1 illustrates a system for loading external additional information related to a television broadcast. [Figure 4] 1 illustrates a system for loading external additional information related to a television broadcast using a local server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] A first exemplary embodiment relates to a system for broadcasting a live event on a stage 1, comprising a plurality of cameras 2, which merge the camera signals of the individual cameras 2 to be converted into a primary signal by a director, and a broadcast studio 3. The broadcast studio 3 is connected to a synchronization server 5 to which a primary signal 4 is sent. The synchronization server 5 transmits the primary signal 4 as a data stream to one or more playback devices 6.

[0039] Only a single playback device is shown in Fig. 1. In reality, the primary signal 4 is transmitted to a large number of playback devices, for example in a broadcast process. From the broadcast studio, the signals of the individual cameras are transmitted as additional signals to an additional signal synchronization server 7. The additional signal synchronization server 7 is connected to a web server 8, from which the individual additional signals can be retrieved by means of Internet protocols and supplied to the respective playback devices 6 via the Internet 18. Since there is a two-way connection between the playback devices and the web server, an individual selection can be made at the playback devices for the additional signals to be retrieved. In the broadcast studio, the primary signal is pre-processed and optimized. The individual additional signals are output with or without further pre-processing. The two synchronization servers 5, 7 are each connected to a database server 9 providing a database, in which the specific feature sequences are stored according to the synchronization information assigned to them.

[0040] In this exemplary embodiment, only a single database is provided, which is accessed by both synchronization servers 5, 7. However, for quick access or to provide two databases with somewhat different data content, it may be advantageous to provide a respective copy of the database in the immediate vicinity of each of the synchronization servers 5, 7. In this arrangement, the primary signal 4 can be output on a specific playback device 6, and the user of the playback device should nevertheless also have the possibility to synchronously output at least one additional signal on the playback device 6. The primary signal 4 and the additional signal each have a video track and an audio track. The audio track of the additional signals is recorded by a microphone attached to each respective camera. The audio signal of the primary signal is recorded by a microphone system installed on the stage 1 and is of very high quality.

[0041] In the following, it is explained how the primary and additional signals are processed to allow them to be synchronized by a playback device simply, accurately and reliably. In the synchronization server 5, consecutive segments of a given length are read from the audio track and special features are extracted therefrom. To achieve this, a Fast Fourier Transform is performed to convert these segments into frequency or Fourier space. In this exemplary embodiment, the length of the segments is 16 ms. However, in any case, it should be less than 50 ms, in particular less than 32 ms, since shorter segments allow a correspondingly accurate synchronization. The shorter the segments and the time slots, the more pronounced the tendency is for low frequencies to be ignored.

[0042] However, it surprisingly turned out that with time slots of a maximum length of 8-10 ms, a sufficient number of high frequency signals are available to perform synchronization. The read and converted time slots preferably overlap each other. For example, time slot lengths of 50%, 32 ms or 16 ms, and a resolution of 16 ms or 8 ms can be achieved. In the frequency space, all frequency peaks above a certain threshold are considered as features.

[0043] In other words, features are intensity values ​​of certain frequencies that exceed a predefined threshold. These features are detected at each time slot to form feature sequences. In this exemplary embodiment, the sequence is not a time-ordered sequence but a list of frequencies ordered by frequency. The feature sequences derived from the audio tracks are referred to below as signal feature sequences. These signal feature sequences are compared with DB feature sequences stored in a database 9. The database 9 contains a large number of such DB feature sequences stored in advance. If a pop concert is performed on stage 1, the database 9 is used. In the database 9, all songs of the corresponding musical performance are converted into DB feature sequences, and songs of other artists are also frequently converted live. The feature sequences have a similarity that allows them to be assigned to each other, even if the signal for which the DB feature sequence was generated and the live signal are not identical. If the comparison of the signal feature sequence with the DB feature sequence coincides at a predefined percentage, this is evaluated as an assignment.

[0044] In the extraction of each signal feature sequence, an extraction time is measured by the synchronous server clock 11 and assigned to each feature sequence. This extraction time is used to represent the time of a particular feature sequence in the corresponding signal. The extraction time can therefore be used to uniquely represent the relative time allocation of several signal feature sequences in a signal. However, it is also possible that the process of extraction itself is subject to time fluctuations. In this case, the extraction time is hampered by errors caused by time fluctuations. For this reason, instead of the time measured by the synchronous server clock 11, it may be advantageous to use time information, contained in the primary signal, which represents the time of a particular point in time in the primary signal. Such time information is essentially contained in the primary signal and is referred to below as signal time information. For example, if the primary signal is a video signal, it has a particular frame rate at which the individual frames are recorded and played back. If the frames are counted in sequence, the time interval between two particular frames of this signal is the number of frames in the signal between these frames multiplied by the inverse of the frame rate. The number of frames of a video signal therefore constitutes this kind of signal time information. As a result, such signal time information is unambiguously encoded in the primary signal.

[0045] However, it can also be implicit, in that for example the number of frames of a video signal is counted. Thus, in the extraction, it is possible to extract a feature sequence together with signal time information indicating the time of this feature sequence in the primary signal. This results in an extraction time that is independent of the chronological order of the extraction process. The signal time information can be assigned an absolute time, for example by a synchronization server clock 11. This assignment is performed once and then maintained. This identified signal feature sequence is the assigned synchronization information that is stored together with the corresponding DB feature sequence in the database.

[0046] In the currently presented exemplary embodiment, the synchronization information includes identification markers describing each song and defining song points. The synchronization information also includes the extraction time of the corresponding signal feature sequence. In the additional signal synchronization server 7, the same process is performed with each additional signal 10. Here too, the signal feature sequence is extracted from the audio track and compared with the DB feature sequence in the database. The extraction time can be measured using the signal synchronization server clock 12 or can be extracted from the corresponding signal time information, which is transmitted to the playback device together with the synchronization information derived from the database with the allocation of each additional signal. Thus, both the primary signal and the additional signal are assigned time information describing the respective times of the feature sequences extracted in the respective signals.

[0047] This time information can be pre-synchronized by comparing the extracted feature sequence with the DB feature sequences stored in the database 9. That is, if the extracted feature sequence matches to a given degree with one of the DB feature sequences, the synchronization information or the time information of this DB feature sequence is assigned to the extracted feature sequence and a time difference is calculated. This is assigned to the primary signal and to the additional signal. The time difference is added to all extraction times of the primary signal and the additional signal. As a result, the same synchronization information or the same time information is assigned to the same feature sequence of the primary signal and the additional signal. According to a first variant, the synchronization information is combined with the respective signals. This means that for the primary signal 4, the synchronization information generated in the synchronization server 5 is combined with the primary signal and the synchronization information generated in the additional signal synchronization server 7 is combined with the corresponding additional signal.

[0048] The synchronization information is transmitted to the playback device 6 together with the corresponding signal from the respective server 5, 7. Only if a corresponding additional signal is requested by the playback device 6, the additional signal is transmitted from the web server 8 to the playback device 6. The primary signal 4 and the requested additional signal are then output to the playback device 6. These two signals are synchronized based on the synchronization information transmitted together with them. When the additional signal is output in synchronization with the primary signal, it contains a time marker that the playback device can recognize (e.g. a synchronized extraction time). In this variant, the corresponding signal is a kind of watermark, and the synchronization information is provided.

[0049] This variant is advantageous in that the synchronization at the playback device is simple, since the two signals, i.e. the primary signal and the additional signal, only need to be output in synchronism with each other based on a time marker, and no additional transmission path is needed to transmit the synchronization information between the synchronization server 5, 7 and the playback device. On the other hand, this variant is disadvantageous in that the primary signal and the additional signal need to be output to the shared playback device, and that the primary signal and the additional signal need to be modified by the addition of the synchronization information. This additional information may cause interference in the receivers of the corresponding signals that do not use this information. Integrating the synchronization information into the transmitted signal delays the transmission.

[0050] In another variant of this exemplary embodiment, the synchronization information is not combined with the primary and additional signals but is sent separately to the playback device 6. The synchronization information comprises time information combined with a specific identifier of the respective signal. If the signals are predefined segments with a defined starting point, the time signal can refer to this beginning or starting point. This can be particularly advantageous in the case of additional signals that last for example tens of seconds to minutes and each contain only omitted additional information that can be output in addition to the primary signal.

[0051] Based on the starting point and the respective time information, the playback device can then synchronize the additional signal with the primary signal. If no such starting point exists in the respective signal, the time information has to refer to another reference point. This reference point can for example be a feature sequence in the respective signal. This feature sequence can occur at different points in the signal. The playback device then needs to be provided with a module that can extract a feature sequence from the respective primary and / or additional signal and compare it with the feature sequence provided with the synchronization information. It is thus possible to obtain a unique reference of the time information for the respective primary and additional signal without a uniquely defined starting point in the primary or additional signal.

[0052] This variant has the disadvantage that the playback device must be provided with a module for extracting the feature sequence and comparing the extracted feature sequence with the feature sequence contained in the synchronization information. However, this variant has the advantage that the additional signal and / or the primary signal do not need to be modified and can be transmitted in their original form. According to a third variant of the first exemplary embodiment, the playback device clock 13 provided to the playback device 6, the synchronization server clock 11 and the additional signal synchronization server clock 12 are synchronized. In this case, the playback device clock 13 is pair-synchronized with the synchronization server clock 11 and the additional signal synchronization server clock 12, respectively. Furthermore, the transmission times of the primary signal from the synchronization server 5 to the playback device 6 and the transmission times of the additional signal from the additional signal synchronization server 7 to the playback device 6 are known.

[0053] In this case, the transmission path is embodied such that the transmission time is constant. In the case of short transmission paths such as Bluetooth links, the transmission time is generally constant. In the case of longer transmission paths, especially when data is transmitted via the Internet, the transmission time often differs significantly, so in this case this variant does not work. The time information contained in the synchronization information relates to a specific event on the synchronization server 5 or on the additional synchronization server 7. This event is typically the time of a specific signal feature sequence that could be identified based on the DB feature sequence. It is therefore known when the primary signal or the additional signal was transferred via the corresponding synchronization server 5, 7 together with the corresponding signal feature sequence. Since the transmission time from the respective synchronization server 5, 7 to the playback device 6 is also known, this can be used to determine when the signal feature sequence arrived at the playback device 6.

[0054] These signal feature sequences of the primary signal and the corresponding additional signal have been identified so that the additional signal can be time referenced to the primary signal, i.e., the additional signal can be synchronized with the primary signal. The corresponding time relationship is included in the synchronization information.

[0055] In this variant, the clock 13 of the playback device must be synchronized with the clock 11 of the synchronization server and with the clock 12 of the additional signal synchronization server, respectively, and the transmission times from the individual synchronization servers 5, 7 to the playback device must be known and stable. However, it is advantageous in this case that neither the primary nor the additional signal needs to be modified. Furthermore, a module for extracting the feature sequences does not need to be integrated into the playback device. This is a very simple solution that allows a reliable synchronization.

[0056] Another advantage of the third variant is that it can be easily implemented with two different playback devices. One playback device is provided for playing the primary signal and a second playback device is provided for playing the additional signal. The playback device clock of the primary signal playback device needs to be synchronized with the synchronization server clock 11 of the synchronization server 5, and the additional signal playback device clock needs to be synchronized with the additional signal synchronization server clock 12. Furthermore, the two playback device clocks must be synchronized with each other. For example, the primary signal playback device can be a television and the additional signal playback device can be a mobile phone.

[0057] The primary and additional signals are output in synchronism with each other. All three variants described above share the fact that a primary signal in the form of a primary signal output by the broadcast studio 3 can be output to a playback device, and an additional signal in the form of a signal of the camera 2 can also be output, not necessarily the camera from which the primary signal is output. The user is therefore free to choose the camera from which he wants to see the performance of the stage 1. As these signals all have a similar audio track, they can be easily and reliably synchronized with each other by the audio track.

[0058] Moreover, the variants described above can be combined with each other, for example by transmitting an additional signal using a playback device according to one of the three variants and transmitting an additional signal to the playback device by one of the other variants and synchronizing with the additional signal. Basically, the third variant is preferred for the transmission of the primary signal, and similarly, all three variants are of equal value for the transmission of the additional signal. A second exemplary embodiment (FIG. 2) is described below. Elements identical to those of the first exemplary embodiment are provided with the same reference signs. In the following, for identical elements, the above description applies, unless stated otherwise.

[0059] In a second exemplary embodiment, a stage 1 is again provided, scanned by a number of cameras 2. The signals of the cameras 2 are converted into a primary signal 4 in a broadcast studio 3. The broadcast studio 3 is connected to a synchronization server 5. The synchronization server 5 is coupled to a database server 9, which has a database containing DB feature sequences and associated synchronization information. A PS playback device 6 / 1 is connected to the synchronization server 5 for receiving and playing the primary signal. Again, multiple PS playback devices 6 / 1 may be provided.

[0060] The second exemplary embodiment differs from the first exemplary embodiment in that an independent source for the additional information is provided. For example, this source is an additional information database server 15. If a concert of several songs is performed on stage 1, it is advantageous to provide the corresponding lyrics to the additional information database server 15. However, the additional information database server 15 can also contain foreign language translations of the corresponding lyrics as audio tracks. There are songs known in many languages, such as the children's lullaby "Flare Jack". Preferably, the additional information stored in the additional information database server 15 is already provided with the corresponding synchronization information. In the case of a song, this can be the start time and other time markers between songs. The additional information database server 15 is connected to a web server 8. The additional information can be retrieved from the web server 8 via the Internet 18. An AS reproduction device 6 / 2 for reproducing the additional signal is connected to the Internet 14.

[0061] The synchronization server 5 also has a connection to the Internet 14, so that the synchronization information generated in the synchronization server 5 can be provided to the AS playback device 6 / 2 via the Internet 14. Then, in the synchronization server 5, a synchronization clock 11 is provided and synchronized with the playback device clock 13 / 1 of the PS playback device and with the playback device clock 13 / 2 of the AS playback device 6 / 2, respectively. In the second exemplary embodiment, the synchronization clock 11 of the synchronization server 5 is the main clock, whereas in the first exemplary embodiment, the playback device 13 is the main clock to which all other clocks are synchronized.

[0062] On the synchronization server 5, the synchronization information is generated by extracting signal feature sequences from the primary signal and comparing them with the corresponding DB feature sequences in the database server 9. The generation of the synchronization information essentially corresponds to that of the first exemplary embodiment. Furthermore, the transmission time for transmitting the primary signal from the synchronization server 5 to the PS playback device 6 / 1 is known, so that if the time at which a particular segment of the primary signal is transferred through the synchronization server 5 is known, the time at which this segment is output to the PS playback device 6 / 1 is also known. Therefore, the synchronization information transmitted from the synchronization server 5 to the AS playback device 6 / 2 includes time information describing the time of the primary signal relative to the respective detected signal feature sequence and an identification marker describing the content of the primary signal. In this exemplary embodiment, the identification marker indicates which song was played in the primary signal. The identification marker can also optionally include additional information such as a verse, a line or an excerpt of the lyrics of the song.

[0063] These lyric excerpts are preferably lyric excerpts from the time when one of the signal feature sequences was detected. The time information preferably includes an indication of the time when the corresponding signal feature sequence on the synchronization server 5 was extracted. Based on this synchronization information, the AS playback device 6 / 2 knows when each song was output on the PS playback device 6 / 1. Correspondingly, the AS playback device can output on the AS playback device 6 / 2, in synchronization with the output of the primary signal on the PS playback device 6 / 1, the additional signal received from the additional information database server 15 or received from the web server 8 and already provided with synchronization information in advance.

[0064] Optionally, an additional signal synchronization server 7 can be provided between the additional information database server 15 and the web server 8, which is embodied similarly to that of the first exemplary embodiment. If the additional information is for example in the form of lyrics coded in ASCII, it does not contain any audio signal. However, as known from speech synthesis, feature sequences like audio signals can be generated from the words contained in the lyrics. These feature sequences are then compared with DB feature sequences stored in another database server 16.

[0065] This also allows a direct comparison of the lyric segments of a song with the corresponding lyric segments stored in the database server 16. In this case, the individual characters of the lyric segments constitute the corresponding features. Each sequence of features stored on the database server 16 is an assigned synchronization information that can be added to the additional information or additional signal. Alternatively, the spoken or sung text can be converted into text form by voice recognition. The features are then text and / or character sequences also stored in the database. On the AS reproduction device 6 / 2 it is possible to display the corresponding lyrics synchronized with the video and audio reproduction of the concert taking place on the stage 1, which is output on the PS reproduction device 6 / 1. The third exemplary embodiment (FIG. 3) corresponds to the second exemplary embodiment and differs in that the synchronization server 5 is embodied independently of the connection between the broadcast station 3 and the PS reproduction device 6 / 1 for reproducing the primary signal.

[0066] Furthermore, the AS reproduction device 6 / 2 comprises a sensor 17 for detecting at least a part of the primary signal output by the PS reproduction device 6 / 1. This sensor 17 can be a microphone for detecting the audio signal of the primary signal 4 or a camera for imaging the video output of the primary signal 4. The AS reproduction device 6 / 2 is embodied with a module for extracting signal feature sequences of the primary signal 4. These signal feature sequences are extracted from the primary signal detected by the sensor 17. The extraction time can be measured by the AS reproduction device clock 13 / 2. Since the extraction process itself is subject to time variations as explained above, it can be advantageous to use signal time information to determine the extraction time. In this embodiment, instead of the signal time information inherently contained in the primary signal, it is possible to use signal time information added during recording with the sensor 17 (microphone) and describing the recording time of the signal.

[0067] Such signal time information is independent of the time variations of the extraction process and allows a unique relative time positioning of the extracted signal feature sequence. The signal feature sequence is transmitted to the synchronization server 5 as in the first and second exemplary embodiment and is analyzed and identified based on the DB feature sequence from the database server 9. On the synchronization server 5, synchronization information is then generated. The synchronization information of the third exemplary embodiment differs from that of the previous exemplary embodiment in that only the time of the clock 13 / 2 of the AS playback device is important. The synchronization information is transmitted from the synchronization server to the AS playback device 6 / 2 via the Internet 14. There, the additional signal 10 is synchronized to the primary signal 4 based on the synchronization information as in the previous exemplary embodiment.

[0068] However, in this case, the synchronization is performed only based on the playback time measured by the AS playback device clock 13 / 2. There is no need to synchronize different times between the AS playback device 6 / 2, the PS playback device 6 / 1 or the synchronization server 5. Alternatively, instead of the signal feature sequence, it is also possible to send short signal sequences, such as music snippets, which are output as additional signals to the synchronization server 5. In this case, the server identifies the signal feature sequences of the signal sequences and parses and identifies them based on the DB feature sequences from the database server 9. As a rule, the signal sequences are not longer than 60 seconds, in particular not longer than 30 seconds or not longer than 15 seconds. The third exemplary embodiment can also be modified in that a module for extracting the signal feature sequences is provided in the synchronization server instead of the AS playback device 6 / 2.

[0069] The third exemplary embodiment is a very elegant solution for outputting the additional signal to a separate AS playback device 6 / 2. In this third exemplary embodiment, the additional signal can be synchronized to the primary signal. The transmission time can be freely changed within a given range, for example between the broadcast station 3 and the PS playback device 6 / 1. The fourth exemplary embodiment (FIG. 4) corresponds essentially to the third exemplary embodiment and differs from it in that the synchronization server 5 comprises a sensor 17. The synchronization server 5 is implemented on a local computing unit, for example a computer, a minicomputer or even a game console. As in the previous exemplary embodiments, the sensor 17 can be a microphone for detecting the audio signal of the primary signal 4 or a camera for imaging the video output of the primary signal 4. The synchronization server 5 is embodied with a module for extracting a signal feature sequence of the primary signal 4.

[0070] These signal feature sequences are extracted from the primary signal detected by the sensor 17. The extraction time is measured by the synchronization clock 11. As in the first, second and third exemplary embodiment, the signal feature sequences are analyzed and identified on the synchronization server 5 based on the DB feature sequences from the database server 9. On the synchronization server 5, synchronization information is then generated. For the synchronization information, only the time of the synchronization clock 11 is important. The synchronization information is transmitted from the synchronization server 5 via the intranet 14 or another data connection, such as Bluetooth, to the AS regeneration device 6 / 2. There, as in the previous exemplary embodiment, the additional signal 10 is synchronized to the primary signal 4 based on the synchronization information. In this case, the time of the synchronization clock 11 is synchronized to the AS regeneration device 13 / 2.

[0071] The main difference between the fourth exemplary embodiment and the previous exemplary embodiments is that the synchronization server 5 is not controlled over the Internet, but instead is provided locally to the user. This has the advantage that synchronization always occurs even if the Internet is down, since there is no dependency on the Internet. However, the database server 9 can be controlled over the Internet, or is equally provided in the same computing unit as the synchronization server 5.

[0072] In the third or fourth exemplary embodiment described above, the synchronization server 5, the database server 9 and the AS playback device 6 / 2 can be embodied on a single device such as a computer (desktop, laptop, etc.) or on a mobile phone. Basically, however, the synchronization server 5 can also be provided on separate hardware from the playback device 6 / 2. The synchronization server 5 can be connected to the playback device 6 / 2 via the Internet. The amount of data exchanged between the synchronization server 5 and the playback device is small.

[0073] All the exemplary embodiments described above share the fact that the synchronization information is generated based on one or more signal feature sequences extracted from the primary signal. This allows (ad-hoc) synchronization of additional signals to a primary signal, where a specific time, such as a start time, is not known in advance. Of course, this method can also be used if a previously determined time is indicated in the respective signal and can be used for operation. Based on this synchronization information, the synchronization of the additional signals to the primary signal can be performed in various ways. Some examples are described above.

[0074] However, in the context of this invention, there are a myriad of possible modifications that can be correspondingly adapted and modified for each use. Another aspect of this invention is to adjust the quality of the additional signal streaming based not only on the available bandwidth but also on the available buffer time. In this case, the additional signal reproduction device 6 / 2 receives the synchronization information and sends a query to the additional information database server 14 about which additional signals are available.

[0075] If a corresponding additional signal 10 is found, a buffer time is also known. The associated buffer time represents the remaining time available for the additional signal before it is played back in sync with the primary signal. This query can also roughly check the available bandwidth of the network. Depending on the bandwidth and the buffer time, another encoding step is automatically selected. During the buffer time, the additional signal is encoded and transmitted from the additional signal server to the playback device and then decoded again. Depending on the encoding step, the transmitted files or parts of the files can be of different lengths and require different amounts of time to be transmitted.

[0076] Therefore, an optimal use time is formed from the buffer time, and a balance needs to be made between the encoding time and the transmission time so that the quality of the additional signal is as high as possible. The method can also be implemented such that one or more servers encode the signals with different qualities or with different encoding steps and simultaneously make them available to a playback device which retrieves and reproduces the signals, and select and re-retrieve the signals with the appropriate amount or encoding step. If the buffer time is very short, e.g. in the case of a live broadcast, it is advantageous to select the length of the chunks of additional signal to be transmitted to be as short as possible.

[0077] The signal can be sent in chunks. First the chunks have to be generated. The shorter the chunks are, the more complex their processing is, since they are sent separately. However, when searching for a chunk, it is necessary to wait at least the length of the respective chunk. Thus, the shorter the chunks are, the faster the reaction required. In this context, the length of the chunks can be reduced to a point where it corresponds to a single frame. At 25 frames per second, this corresponds to 40 ms. Thus, extremely fast transmission is possible. With certain codecs, such as the h265 codec, it is possible to set "zero latency".

[0078] This means that the time required to encode and then decode again is very short, e.g. less than one second. A certain amount of latency is unavoidable. However, in the case of a "zero latency" setting, the corresponding codec method does not introduce any additional latency. Thus, the buffer time is almost exclusively required for the actual transmission of the signal, which can also be significantly reduced with a corresponding higher bandwidth. For example, in a live concert, the organizer provides the participants with smartphones with a camera view via a web server, and a corresponding WLAN infrastructure is also provided, so that the video signal can be transmitted with almost no delay. In this case, the transmission path for the encoding of the additional signal and / or the transmission of the additional signal to the additional signal reproduction device 6 / 2 can be automatically selected as a function of the determined synchronization information.

[0079] If the synchronization information contains an indication that there is not much time left to transmit the additional signal, it is advantageous to reduce the amount of data of the additional signal through a corresponding compression encoding and to select a fast transmission path. The encoding also needs to be done very quickly. A significant reduction in the amount of data and fast compression often has a negative impact on the quality of the additional signal. However, if more time is available, a more laborious encoding and / or a lower compression rate can be used and a higher quality of the additional signal can be achieved. In all the exemplary embodiments described above, the database server 9 is provided with a previously prepared database containing the DB feature sequences and the synchronization information.

[0080] In the context of the invention, the database can also be created on the database server 9 during operation (live database). This is mainly advantageous when there is a primary signal to which an additional signal needs to be output synchronously and which primary signal is not known before. In such a case, feature sequences are extracted from the primary signal and the respective times available for extraction are measured. These extracted feature sequences are stored in the database together with the extraction times. Instead of or in addition to the extraction times, it is also possible to extract time information contained in the primary signal and store it together with the feature sequences in the database server 9. The time information in this case constitutes all or part of the synchronization information. During the system operation, the database thus created can be synchronized to other databases in which different signals have already been previously stored in feature sequences.

[0081] This database may also contain meta-information describing the content, time and meaning of these feature sequences and / or of this signal. A wide variety of media streams may be stored in this database as feature sequences. Comparison with this additional database allows the assignment of meta-information, in particular semantic information or meaning, to the feature sequences of the database generated "online" or "on the fly". The generation of such a database on the database server 9 is possible in all the exemplary embodiments described above. Users may also generate such live databases on the site locally on their user devices (computers, mobile phones, etc.). Reference List 1 Stage 2 Camera 3 Broadcast Studio 4 Primary Signals 5 Synchronization Server 6 Playback Devices 7 Additional Signal Synchronization Server 8 Web Server 9 Database Server 10 Additional Signals 11 Synchronous Clock 12 additional signal synchronous clock 13 Playback device clock 14 Additional Information Database Server 16 Database Server 17 Sensors 18 Internet

Claims

1. 1. A method for synchronizing additional signals transmitted to a plurality of playback devices with a primary signal in a broadcast process, the method comprising: generating a DB feature sequence for a database from the primary signal while the primary signal is being transmitted from a broadcast station to the playback device; generating synchronization information for the primary signal by extracting at least one signal feature sequence of the primary signal and comparing it with DB feature sequences stored in the database, the signal feature sequence and the DB feature sequence comprising features in a frequency space; if the signal feature sequence matches one of the DB feature sequences to a predetermined degree, then the synchronization information of the matching DB feature sequence is assigned to the primary signal at a position specified in the signal feature sequence; transmitting the synchronization information to one of the plurality of playback devices that outputs an additional signal in synchronization with the primary signal based on the synchronization information; In a first step, timestamps of the available additional signals are sent to the playback device, which calculates the available buffer time; in a second step, the additional signal is transmitted from the additional signal server to a playback device, the quality of the additional signal depending on the buffer time and the available bitrate; transmitting said synchronization information; The method according to claim 1,

2. The method described in claim 1, wherein the additional signal is transmitted from the additional signal server to the playback device in chunks having a time length of at most 10 frames, corresponding to approximately 400 ms, and at the playback device the additional signal is received by a local web server.

3. A reference signal is output, which is received simultaneously by a corresponding sensor, in order to calibrate a transmission path from the server to the playback device and / or to calibrate a latency in the playback device for outputting the media signal to said playback device; 2. The method of claim 1, wherein the outputted reference signal and the received reference signal are compared to each other to determine a time interval required to repeat the reference signal and / or to actually output it to the playback device, and this time interval is used as a time offset to determine the output time at which the signal is output to the playback device.

4. The method of claim 3 , wherein the reference signal comprises an audio signal and the sensor is a microphone and / or the reference signal comprises a video signal and the sensor is a camera.

5. the time interval is determined by determining a transmission time and a reception time of the reference signal; 4. The method of claim 3, wherein the time interval is derived from the time difference between these two times, and wherein measuring at least one of the two times is performed by comparing the extracted reference signal feature sequence with one or more previously stored reference signal feature sequences.

6. an additional signal is transmitted to the playback device from a synchronization server embodied independently of the playback device, and synchronization information is generated at the synchronization server; the synchronization information relates to a server time of the synchronization server, measured at the synchronization server using a distributed clock; 2. The method of claim 1, wherein in the playback device, a playback device clock, which is synchronized at least once with the server time, is provided for measuring a playback time, a time drift of the playback time relative to the server time is measured, and this time drift is taken into account when synchronizing the primary signal with the additional signal.

7. The time drift is determined by multiple transmissions of a time signal of the server time to the playback device, or by multiple transmissions of the playback time to the synchronization server and comparing the transmitted time signal with a locally present time signal and calculating the respective time difference, The method of claim 6 , wherein the time drift is determined based on a deviation of the time differences.

8. 2. The method of claim 1, wherein based on the synchronization information, a determination is made of how much time is available to transmit the additional signal from a given server to a playback device, and based on this time, any one of several encoding methods is selected or automatically adjusted and / or one or more transmission paths are selected.

Citation Information

Patent Citations

  • Fingerprint generation and detection method and apparatus for synchronizing audio and video

    JP2006528859A

  • Synchronous control system, synchronous control method and program

    JP2016146586A

  • System for providing additional information of broadcasting contents and method thereof

    KR1020120063798A

  • Wireless video transmission system

    US20060095942A1

  • Media streaming using an index file

    US20090300204A1