Method for synchronizing an auxiliary signal to a main signal

Description

[0001] The invention relates to a method for synchronizing an additional signal to a main signal and a device for synchronizing an additional signal to a main signal.

[0002] The signals are "continuous signals," where continuous signals are understood to be signals that can be described by a sequence of temporally consecutive features. Typical continuous signals are audio signals and video signals, which can be sampled at regular intervals to generate corresponding features. However, continuous signals can also be signals used to transmit digitally encoded text.

[0003] US 9,609,034 B2 discloses a method for identifying media data using metadata.

[0004] WO 2016 / 085414 A1 describes a method whereby a mobile platform recognizes the broadcaster on a television and receives corresponding information during commercial breaks. EP 2 507 790 B1 describes a method and system for channel-invariant robust audio hashing with subsequent comparison of two audio hashes. The audio signals are first divided into fragments with a typical length of 2 seconds. These fragments are then further divided into frames with a typical length of 0.36 seconds. The frames are Fourier transformed, and the resulting data is then normalized. The hash values ​​are obtained by quantizing this data.

[0005] WO 2012 / 049223 A2 describes a method for synchronizing an alternative audio signal to a combined video and audio signal. Two options are proposed for this. First, a watermarking method is described, which represents an additional signal imperceptible to humans in addition to the video and audio signal and can be represented, for example, as a modulation of the primary audio signal. The other method describes the fingerprinting method.

[0006] The main audio signal is characterized based on amplitude, frequency, zero-crossing rate, velocity, spectral flatness, bandwidth, and / or audio fingerprints and compared with the corresponding characteristics of the second signal. Once the location in the second signal has been identified, it can be time-aligned with the main signal.

[0007] WO 2014 / 018652 A2, like WO 2012 / 049223 A2, describes a method for synchronizing an alternative audio signal to a combined video and audio signal. A fingerprinting method is used. The fingerprints of the main audio signal are coupled in full length with the second signal. The second signal and the fingerprints of the first signal are transferred in advance to the device to be synchronized, so that during synchronization, only the first signal needs to be analyzed and compared with the fingerprints on the device.

[0008] WO 03003743 A2 discloses a method and apparatus for providing synchronization signals for synchronizing parallel media. A synchronization server is coupled to a communications network and connected to a transmission media start time database. The synchronization server receives a synchronization request from a user client via the communications network. The synchronization server generates synchronization data using the synchronization request and the transmission media start time database.

[0009] WO 2014209179 A1 describes methods and transceivers for network diversity in long-distance communication. The method in a master node for communicating with a target node over long distances comprises the following steps: Synchronizing the use of at least one communication resource with an auxiliary node, generating a main signal and an auxiliary signal from a set of information, transmitting the main signal to the target node through the at least one synchronized communication resource, transmitting the auxiliary signal to the auxiliary node through the at least one synchronized communication resource.

[0010] US 2014 / 0201769 A1 describes a system and method for outputting supplementary information to a television set. This supplementary information is synchronized with a main signal by creating a database of so-called fingerprints, whereby the synchronization of the supplementary signal occurs by matching the fingerprints of the supplementary signal with the fingerprints of the main signal. It is also disclosed that the latency in the broadcast process is a few seconds, whereas the latency when transmitting data over the internet is only a few milliseconds.

[0011] US 2011 / 0135283 A1 discloses a method for synchronizing media content across multiple devices. Fingerprints are created from a first media content on a first playback device. These fingerprints are compared with fingerprints from a database, from which a temporal position is then determined. Using this synchronization information, the same media content can be played simultaneously on other devices.

[0012] US 2015 / 0189347 A1 discloses methods, systems, and media for presenting additional information corresponding to on-demand media units. A web browser application first identifies the media content being played and then checks whether additional content is available. If this is the case, an indicator indicates that additional content is available. If this content is played, the media is synchronized using time information transmitted with the additional media information so that they are played simultaneously.

[0013] WO 2007 / 072326 A2 discloses a method in which a content stream and a script are synchronized to output one or more sensory effects in a multimedia system. A fingerprint is calculated from a portion of the content stream. From this, a time value corresponding to the fingerprint is determined. The time value can be stored in a fingerprint database, which is accessed using the fingerprint to retrieve the time value. A script clock is synchronized to the time value and thus to the portion of the content stream. The portion of the content stream is played back synchronously with the script using the synchronized script clock. The script is used to generate one or more sensory effects. These effects can be, for example, TV ambient lights (see page 2, last paragraph to page 3, third paragraph and summary).

[0014] WO 2014 / 178796 A1 describes a method in which, for example, a television signal is transmitted using a broadcast method. Additional signals can be synchronized to a main signal using watermarks and / or fingerprints. It also discloses that the fingerprints can be generated in real time, i.e., while the signal is being transmitted using the broadcast method. Using the watermark, the additional signals can be quickly assigned to the corresponding sections of the main signal, and the fingerprints then synchronize them precisely with the main signal. Therefore, the use of fingerprints is always used in conjunction with the use of watermarks. The fingerprints are sections of the signal in time.

[0015] D9 (WO 2017 / 181852 A1) discloses a song identification method in which a song is identified from a collection in a video interlude (e.g., an advertisement). The matching can be performed using peaks in the frequency spectrum. Once the song has been identified, the lyrics associated with the song are synchronized.

[0016] The object of the present invention is to provide a fast, robust and accurate method and a corresponding device with which an additional signal can be output synchronously to any main signal, even a continuous one.

[0017] Another task is to provide a streaming method with low latency.

[0018] Another task is to provide a fast, robust and accurate method to measure and calibrate the delay between input and output of a media player.

[0019] A further object is to provide a method for synchronizing an additional signal to a main signal, which method measures the time offset between these two signals and adjusts at least one of these signals accordingly so that the time offset is as small as possible.

[0020] The objects are achieved by the features of the independent claims. Advantageous developments and preferred embodiments form the subject matter of the dependent claims.

[0021] According to a first aspect of the invention, a method for synchronizing an additional signal to a main signal is provided, comprising the following steps, which is transmitted to many playback devices using a broadcasting method: Generating DB feature sequences for a database from the main signal while the main signal is being transmitted from a transmitting station to a playback device, generating synchronization information for the main signal in a synchronization server by extracting at least one signal feature sequence of the main signal and comparing it with DB feature sequences stored in the database, wherein the signal feature sequence as well as the DB feature sequence comprise features in the frequency domain and, if the signal feature sequence matches one of the DB feature sequences to a predetermined degree, synchronization information of the matching DB feature sequence is assigned to the main signal at a position predetermined by the signal feature sequence, wherein a sound track of the main signal is transformed into the frequency domain in sections by a Fast Fourier Transformation, wherein the sections that are transformed into the frequency domain,the audio track is no longer than 50 ms, the characteristics being frequency peaks that are above a certain threshold, transmitting the synchronization information to one of the playback devices, which, based on the synchronization information, outputs an additional signal synchronous to the main signal.

[0022] Since this method extracts one or more signal feature sequences from the main signal and compares them with corresponding pre-stored DB feature sequences in a database, it is not necessary for the main signal to be inherently provided with specific time information, such as a start point. The main signal can, for example, be a continuously transmitted television signal that has no start point. By comparing the feature sequences, one of the signal feature sequences in the main signal can be identified by a match with a corresponding DB feature sequence. This allows synchronization information to be assigned to the main signal at a predetermined position that is related to the identified signal feature sequence. In other words, this means that the synchronization information relates to the location or position of the identified signal feature sequence in the main signal.This synchronization information is stored, for example, together with the DB feature sequences in the database. However, this synchronization information can also include a time, in particular a server time of a synchronization server on which this method is executed, which is recorded when, for example, the additional signal with this signal feature sequence is received by the synchronization server, the signal feature sequence is extracted, or the additional signal with this signal feature sequence is sent from the synchronization server to the playback device. This signal feature sequence thus forms a specific location in the main signal, to which a specific time, in particular server time, is then assigned. This specific time can also be extracted from time information contained in the signal and assigned to the respective signal feature sequence.

[0023] Of course, this method can also be used with a main signal that has a specific starting point. This makes the method according to the invention significantly more flexible than conventional methods, such as those known, for example, from EP 1 307 833 B1, which require a main signal with a fixed time reference. In the method according to the invention, this time reference is created on the fly, for example, by a synchronization server. By comparing it with the DB feature sequences, at least one signal feature sequence of the main signal is identified, to which corresponding synchronization information can then be assigned.

[0024] Furthermore, the time of the signal feature sequence in the main signal can be determined, which corresponds to the DB feature sequence, and this time can be assigned to this extracted signal feature sequence as the extraction time for generating the synchronization information.

[0025] Such an extraction time point generates time information associated with the main signal, linking a location or point in the main signal to a specific time. Assigning time information to the main signal in this way can also be useful if the main signal already contains time information, for example, in the form of a time marker. This creates a second reference, which can be linked to additional information.

[0026] Furthermore, the DB feature sequence can be assigned time information that defines a specific DB time relative to the DB feature sequence, which is used to generate the synchronization information. This time information is typically stored in the database together with the DB feature sequence. For example, it specifies a specific time when a signal feature sequence that matches this DB feature sequence occurs at a specific point in a larger signal section, such as a film. The additional signals can then be synchronized with this DB time on the playback device.

[0027] Synchronization information can also be assigned to the additional signal by extracting a signal feature sequence of the additional signal and comparing it with a DB feature sequence stored in a database, wherein, if the signal feature sequence matches one of the DB feature sequences to a predetermined degree, synchronization information is assigned to the additional signal at a position provided by the signal feature sequence.

[0028] Synchronization information can also be manually assigned to the auxiliary signal. For example, an operator can assign time information to the auxiliary signal indicating when it should be broadcast relative to a main signal.

[0029] However, an additional signal can also be used to which synchronization information has already been assigned in advance.

[0030] The synchronization information may include one or more of the following data: Time information that identifies a point in time of the main and / or additional signal, in particular a point in time relative to a detected signal feature sequence being described thereby. One or more identification markers that describe the content of the main and / or additional signal. These identification markers can, for example, only describe the type of additional signal, such as subtitles in a specific language, audio synchronization signals in a specific language, or explanations of the type of META information transmitted with the additional signal. However, the identification markers can also describe the additional signal in more detail and reproduce structures of the additional signal. For example, if the additional signal is song lyrics, the identification markers can identify the respective verses of the lyrics.However, other structures such as chapters, acts of an opera, songs in a concert, episodes of a television series or the like can also be described with the identification markers. A timestamp which describes a specific point in time in the main and / or additional signal. This point in time is independent of one of the detected signal feature sequences and describes, for example, a specific point in time in a film with respect to a reference point which is not related to a detected signal feature sequence. This reference point is usually a starting point of the main signal, the additional signal or a specific section of the corresponding signal. A time offset which describes the time interval necessary to forward a main and / or additional signal from a specific point on a transmission path to the actual output of the main and / or additional signal on the playback device.This synchronization information is therefore not synchronization information that describes a property of the main or additional signal, but rather a property of the device with which the method is carried out.

[0031] The synchronization information can be composed very differently depending on the application.

[0032] In the aspect of the invention explained above, the synchronization information of the matching DB feature sequences is assigned to the main signal at a position predetermined by the signal feature sequence. Specific information in the database is assigned to the DB feature sequences. However, this information does not necessarily have to include time information. It can, for example, be metadata that describes the meaning (e.g., title of a piece of music, act of an opera, etc.) of the DB feature sequence or of a section of the signal in which this DB feature sequence is located.The synchronization information can then be generated, for example, on the basis of time information contained in the main signal, which is extracted together with the signal feature sequence, or on the basis of the extraction time, which can be combined, for example, with this meta-information and thus results in synchronization information, on the basis of which an additional signal with the same meta-information can be assigned, wherein the time of the assignment or synchronization is derived from the extracted time information or from the extraction time.

[0033] In other words, this means that synchronization information is assigned to the main signal and there is a temporal reference between the synchronization information and the main signal.

[0034] The invention is further based on the finding that media signals from different signal sources often have similar sections of features. These feature sections do not have to be exactly the same. If, for example, a main signal is a high-quality audio signal from a concert and an additional signal is a video signal with a low-quality audio signal, then the additional signal can be synchronized very precisely based on the low-quality audio signal, e.g., the welcoming applause of the musicians, because the audio characteristics here are very similar, even if the quality of the two audio signals differs considerably. This is equally possible with video signals recorded with professional cameras and those recorded, for example, with a mobile phone.

[0035] The inventors have recognized that these feature sections allow for automatic identification of one or more specific signal feature sequences for synchronizing the different signals.

[0036] The synchronization information can be generated on a synchronization server that is configured independently of a playback device. This information must then be transmitted to the playback device, on which the additional signal is output synchronously with the main signal. In this case, synchronization of the playback device and the synchronization server could also be performed, for example, by determining a time interval required to transmit the corresponding signal from a predetermined location, in particular from the synchronization server, to the playback device. However, the synchronization server can also be configured in the playback device itself. If digital data transmission between the synchronization server and the playback device is used, it is generally not possible to determine this time interval, as it varies.

[0037] The auxiliary signal can be synchronized to the main signal on the playback device by outputting the main signal and the auxiliary signal, each containing one or more time markers as synchronization information. Using the time markers, the playback device can synchronously assign the auxiliary signal to the main signal and output them synchronously.

[0038] Furthermore, the additional signal can be assigned to this playback time on the playback device using the synchronization information based on a playback time measured on the playback device by means of a clock in such a way that the additional signal is output synchronously with the main signal.

[0039] In the first case, the main signal and the auxiliary signal are output via the same playback device, so the time markers in both signals are sufficient to output the signals synchronously. However, if you do not want to insert time markers in the main signal, for example, because you want to transmit the main signal to a playback device as quickly as possible without any time delay, or if the main signal is output via a different playback device than the auxiliary signal, then it is advisable to assign the auxiliary signal to a playback time measured on the playback device using a clock using the synchronization information. The synchronization information contains the necessary information to assign the auxiliary signal to the playback time in such a way that the auxiliary signal is output synchronously with the main signal.

[0040] If the main signal and the additional signal are output via different playback devices, it is expedient to use a first time to which the output of the main signal is assigned. This time thus describes the times of the individual features or feature sequences in the main signal. In principle, the clock of the playback device with which the main signal is output could be used for this purpose. However, if a synchronization server is provided that is independent of this playback device for the main signal, the clock of the synchronization server can also be used, particularly if the main signal is transmitted continuously from the synchronization server to the playback device. In this case, a corresponding time offset must simply be added to the synchronization information. This time offset specifies the time interval by which the main signal is delayed when output at the playback device compared to its passage at the synchronization server.Synchronization information is provided for the playback device of the additional signal, which describes the relationship between the playback time on the playback device and the server time on the synchronization server. This allows a reference to the server time of the synchronization server to be established based on the playback time, since the output of the main signal is also synchronized to this server time. If both the server time and the playback time are regularly synchronized with an independent time or reference time (e.g., NTP: Network Time Protocol), the synchronization information does not need to contain any information about the relationship between these two times, since both times can be considered identical within the scope of the available measurement accuracies.

[0041] When comparing the signal feature sequences with the DB feature sequences, it may be that several signal feature sequences match the predetermined degree. In principle, the several signal feature sequences can then be used to generate synchronization information. However, it is advisable to evaluate the match of all signal feature sequences that meet the predetermined degree of match within a predetermined time interval, and select the signal feature sequence with the best rating to assign the synchronization information to the main signal at a position specified by the signal feature sequence. In principle, one wants a clear assignment of the synchronization information to the main signal. When using several signal feature sequences, this is not always guaranteed.Using the signal feature sequence that best matches a DB feature sequence also achieves the best synchronization.

[0042] The predetermined rules for evaluating the degree of agreement of the signal feature sequence with the DB feature sequence include one or more of the following rules: The greater the number of matching features in a feature sequence, the better the rating. The greater the number of matching features in a feature sequence relative to the number of features contained in the feature sequence, the better the rating. The match of the individual features is classified. The closer the distance between the matching features, the better the corresponding match is classified. This classification is then taken into account in the overall assessment of the match of a feature sequence.

[0043] According to a further aspect of the present invention, a method for synchronizing an additional signal to a main signal is provided, comprising the following steps: Extracting signal feature sequences from the main signal and storing the signal feature sequences in a database together with time information, synchronizing an additional signal using this database, wherein in particular feature sequences extracted from the additional signal are compared with the feature sequences stored in the database and / or time information from the additional signal is compared with corresponding time information of the database.

[0044] This method creates a database for a main signal while the main signal is being transmitted from a transmitting station to a playback device. This database is immediately available for synchronizing an auxiliary signal with this main signal. This allows a live signal to be analyzed and synchronized. Therefore, a database created in this way is also called a live database.

[0045] The time information can be created or extracted using a clock provided on a corresponding server and / or using time information contained in the main signal.

[0046] According to a further aspect of the present invention, a method for synchronizing an additional signal to a main signal is provided, wherein for calibrating a transmission path from a server to a playback device and / or for calibrating the latency at a playback device for outputting a media signal on the playback device, a reference signal is output which is simultaneously recorded by a corresponding sensor, wherein the output reference signal and the reference signal received by the sensor are compared with one another in order to determine the time interval necessary for forwarding the reference signal and / or the actual output to the playback device, and this time interval is used as a time offset in order to determine an output time based on time information related to the clock of the server and / or the playback device, with which output time a signal is output to the media playback device.

[0047] This method allows transmission paths or playback devices to be calibrated automatically. The latency of a playback device can vary considerably depending on whether, for example, an audio signal is output via a wired speaker, a speaker connected via Bluetooth, or a downstream audio system. The time interval required to transmit signals can also vary considerably depending on the respective transmission path. This method allows the transmission path and / or the latency of the output device to be calibrated once or several times before or during playback of the media signal, ensuring the correct offset is always present for the signals to be output.

[0048] The reference signal can comprise an audio signal, in which case the sensor is a microphone. The reference signal can also comprise a video signal, in which case a camera is used as the sensor.

[0049] The time interval can be determined by determining the time of transmission and the time of reception of the reference signal, whereby the time interval is derived from the time difference between these two times. If the time of transmission and the time of reception of the reference signal are measured at the same location, then the time interval to be determined is half the time difference between these two times. If the time of transmission of the reference signal is measured at the beginning of the transmission path and the time of reception of the reference signal is measured directly at the sensor, then the time interval to be determined is this time difference. One or both times can be determined by comparing an extracted reference feature sequence with one or more previously stored reference feature sequences.This method corresponds to the method explained above for identifying signal feature sequences based on DB feature sequences. By comparing feature sequences in this way, a time point can be determined. The accuracy of such a time point is limited by the length of the feature in the feature sequence used to determine the time point. A typical length of such a feature is in the range of ±8 ms.

[0050] According to a further aspect of the present invention, a method for synchronizing an additional signal to a main signal is provided, in which an additional signal is transmitted from a synchronization server designed independently of a playback device to the playback device, and synchronization information is generated on the synchronization server, which synchronization information is related to a server time of the synchronization server, which is measured on the synchronization server using a clock arranged there, wherein a playback device clock is provided on the playback device for measuring a playback time, which is synchronized at least once with the server time, and a time drift of the playback time relative to the server time is measured, and this time drift is taken into account when synchronizing the additional signal to the main signal. The additional signal can be output at the playback device in a manner controlled by the playback time available from the playback device.

[0051] Time drift can have various causes. On the one hand, the playback device's clock may not always run at exactly the same speed as the server's clock, or the clocks of different playback devices may run at different speeds. On the other hand, the additional signal may be temporally stretched or compressed compared to the main signal, so that an additional signal that is exactly synchronized with the main signal at a certain point in time deviates more from the main signal as the playback time increases. Such temporal stretching or compression occurs, for example, when analogue signals are converted to digital using a corresponding analogue-to-digital converter. The main signal and the additional signal(s) are usually transmitted over different distances and therefore converted at different locations using different analogue-to-digital converters.Every analog-to-digital converter has a clock, which adds time information to the digital signal in the form of time markers. The clock speeds of different clocks can vary slightly. This means that even if a main signal and an auxiliary signal are identical in the analog state, they have slightly different time information in the digital state. If they are played back simultaneously on a playback device, a time offset between the two signals can arise as the playback time increases.

[0052] Time drift due to clocks or clock generators running at different speeds on different playback devices can be eliminated by regularly synchronizing them with a reference clock (e.g., an atomic clock or TMP). In playback devices, the output unit often has its own clock signal, and the playback device has a control unit with its own clock. In such a case, it is useful to regularly synchronize the clock signal of the playback unit with the clock of the control unit of the playback device, and to synchronize the clock of the control unit of the playback device with the reference clock at regular intervals.

[0053] The time drift due to signal compression or stretching can be measured. This can be determined by extracting feature sequences. Once the time drift has been determined, the auxiliary signal can be played continuously synchronously with the main signal, without the need for resynchronization between the auxiliary signal and the main signal at regular intervals.

[0054] If the part of the main signal to which the additional signal is synchronized is available, then the two signals can be repeatedly synchronized without having to calculate any time drift. For example, it may be useful to pass the main signal and the additional signal through a common synchronization server before they are output to the playback device, so that the complete main signal and additional signal are available at the synchronization server and a new synchronization of the additional signal to the main signal can occur at any time. The main signal can then be output to the playback device without the portion required to synchronize both signals, while still allowing regular resynchronization between the two signals.

[0055] Time drift can be determined by repeatedly comparing the time with a reference time to calculate a time difference each time. The time drift is determined based on the deviations in the time difference. The longer the intervals between the first and last comparison, the more precisely the time drift can be determined.

[0056] According to a further aspect of the present invention, a method for synchronizing an additional signal with a main signal is created, wherein in a first step the time stamps of the available additional signals are transmitted to a playback device. This allows the available buffer time to be calculated. The buffer time describes the time still available for the additional signal before it must be played back in order to be synchronized with the main signal. At the same time, the available bandwidth can be determined in this first transmission. In the second step, the buffer time is used to encode the additional signal, transmit it from the additional signal server to the playback device and then decode the additional signal again. The quality of the additional signal depends on the available buffer time and the available bit rate.With the same buffer time, high signal quality can be achieved by either selecting the shortest possible encoding / decoding time, which results in large data volumes requiring correspondingly long transmission times, or by selecting a long encoding / decoding time, which reduces the bit rate and accelerates transmission. If buffer times, hardware requirements, and available bandwidth change, the encoding / decoding time must be redetermined.

[0057] In practice, this process can be implemented in such a way that the signals (main signal and / or additional signal) are encoded differently on the server(s), for example, using different codecs, so that the signals are available in different qualities. The playback device then decides which signal to use and retrieve from the server.

[0058] Furthermore, the additional signal is transmitted from the additional signal server to the playback device in chunks with time lengths of 10 frames, which corresponds to approximately 400 ms, in particular a maximum of 5 frames, which corresponds to approximately 200 ms, and preferably a maximum of 1 frame, which corresponds to approximately 40 ms. The additional signal is received at the playback device using a local web server. By providing the local web server with a direct connection via a websocket, the additional signal can be received essentially without delay. A direct connection is a connection that is maintained after a transmission process. The local web server is preferably compatible with the transmission protocol used by the playback device (usually: http), so that the playback device itself does not need to be modified, except for adding the local web server.

[0059] This also allows the use of very short chunks. With conventional transmission methods, such short chunks cause a significant delay when repeatedly establishing the data connection, due to the sequential polling behavior and the individual file queries, each of which incurs overhead. In principle, it would be possible to establish a direct connection, which would allow the transmission of chunks with almost no delay. However, this has the disadvantage that it can only be established between two specific partners and does not allow streaming to multiple recipients simultaneously.

[0060] Instead of a direct connection, the local web server can be configured to request multiple chunks simultaneously or in quick succession without waiting for the previously requested chunks to be received. With conventional methods, the chunks are queried individually, and another chunk is only requested once the previously requested chunk has been received. This allows even very small chunks to be transmitted in rapid succession. Conventional streaming techniques such as HLS or Dash can be used for this purpose.

[0061] This minimizes the required transmission time, allowing more time for encoding. This can then be performed at a high data rate, increasing signal quality.

[0062] With this transmission method, the additional signal can usually reach the receiver within two to three seconds.

[0063] The aspects explained above can be implemented individually or in any combination.

[0064] Within the scope of the invention, an additional signal can be synchronized to a main signal. However, it is also possible for several additional signals to be synchronized to a main signal.

[0065] The invention is explained in more detail below by way of example with reference to the drawings. The drawings show: Figure 1 shows a system for synchronously playing back multiple camera signals to a main signal, Figure 2 shows a system for inserting external additional information into a live broadcast, Figure 3 shows a system for inserting external additional information into a television broadcast, Figure 4 shows a system for inserting external additional information into a television broadcast with a local server.

[0066] A first embodiment relates to a system for transmitting a live event on a stage 1 with multiple cameras 2 and a broadcast studio 3 in which the camera signals from the individual cameras 2 converge to be converted into a main signal by the control room. The broadcast studio 3 is connected to a synchronization server 5, to which the main signal 4 is transmitted.

[0067] The synchronization server 5 forwards the main signal 4 as a data stream to one or more playback devices 6. In Figure 1 Only a single playback device is shown. In reality, the main signal 4 is transmitted to many playback devices, e.g., using a broadcasting process.

[0068] From the broadcast studio, the signals from the individual cameras are routed as additional signals to an additional signal synchronization server 7. The additional signal synchronization server 7 is connected to a web server 8, from which the individual additional signals can be retrieved using an internet protocol and transmitted to the respective playback devices 6 via the internet 18. A bidirectional data connection exists between the playback devices and the web server 8, allowing the playback devices to individually select the additional signals to be retrieved.

[0069] In the broadcast studio, the main signal is preprocessed and optimized. The individual auxiliary signals are output with or without further preprocessing.

[0070] The two synchronization servers 5, 7 are each connected to a database server 9, which maintains a database in which specific feature sequences are stored along with the synchronization information associated with the feature sequences. In the present exemplary embodiment, only a single database is provided, which both synchronization servers 5, 7 access. However, it may also be expedient to provide a copy of the database in the immediate vicinity of each synchronization server 5, 7, thus enabling faster access, or even to provide two databases with slightly different data contents.

[0071] With this device, the main signal 4 should be able to be output on the playback device 6 and the operator of the playback device should also have the possibility of outputting at least one of the additional signals synchronously on the playback device 6.

[0072] Both the main signal 4 and the auxiliary signals each have a video track and an audio track. The audio tracks of the auxiliary signals are recorded using a microphone mounted on the respective camera. The audio signal of the main signal is recorded using a microphone system installed on Stage 1 and is therefore of significantly higher quality.

[0073] The following explains how the main signal and the additional signals are processed so that they can be synchronized easily, precisely and reliably by the playback device.

[0074] In the synchronization server 5, successive sections of a predetermined length are read from the audio track, and specific features are extracted from them. For this purpose, these sections are transformed into the frequency domain or Fourier domain using a fast Fourier transformation. In the present exemplary embodiment, the length of the individual sections is 16 ms. According to the invention, they should in any case be no longer than 50 ms and in particular no longer than 32 ms, since short sections allow for correspondingly precise synchronization. The shorter the sections or time windows are, the more low frequencies are disregarded. However, it has surprisingly been shown that for time windows up to a maximum length of 8-10 ms, a sufficient number of high-frequency signals are present to perform the synchronization.

[0075] The read and transformed time windows are preferably overlapped. With an overlap of, for example, 50%, a resolution of 16 ms and 8 ms can be achieved with a time window length of 32 ms and 16 ms, respectively.

[0076] In the frequency domain, all frequency peaks that lie above a certain threshold are considered features. This means that the features are intensity values ​​of specific frequencies that lie above the predetermined threshold.

[0077] These features are captured in the individual time windows and form a feature sequence. In the present embodiment, the sequence is not a chronological sequence, but rather a list of features in order of frequency.

[0078] The feature sequences derived from the audio track are referred to below as signal feature sequences. These signal feature sequences are compared with DB feature sequences stored in the database.

[0079] Database 9 contains a large number of such DB feature sequences that have been created in advance.

[0080] If a pop concert is held on stage 1, then a database 9 is used in which all songs of the corresponding music band are converted into DB feature sequences and, if possible, also songs by other artists, which are often played live.

[0081] The feature sequences are so characteristic, even if the signals from which the DB feature sequences were generated and the live signals are not identical, that they still have a similarity to be able to be assigned to each other.

[0082] If a match to a predetermined degree is found when comparing the signal feature sequences with the DB feature sequences, this is evaluated as an assignment.

[0083] When extracting the respective signal feature sequence, the extraction time is recorded using the synchronization server clock 11 and assigned to the respective feature sequence.

[0084] This extraction time serves to describe the time of a specific feature sequence in the corresponding signal. Thus, the extraction times can be used to clearly describe the relative temporal assignment of several signal feature sequences within a signal. However, the extraction process itself may be subject to temporal fluctuations. In this case, the extraction times are subject to an error caused by the temporal fluctuations. Therefore, it may be expedient to use time information contained in the main signal, which describes the time of a specific point in the main signal, instead of the time measured with the synchronization server clock 11. Such time information is inherently contained in the main signal and is referred to below as signal time information. If, for example, the main signal isa video signal, then it has a specific frame rate at which individual images (= frames) are recorded and played back. If the images are counted consecutively, then the time interval between two specific images of this signal is the number of images in the signal between these images multiplied by the inverse of the frame rate. The number of an image in a video signal thus represents such signal time information. Such signal time information is usually explicitly encoded in the main signal. However, it can also be included implicitly, for example by counting the number of images in a video signal.

[0085] During extraction, the feature sequence can be extracted together with the signal timing information, which indicates the time point of this feature sequence in the main signal. This results in an extraction time point that is independent of the timing of the extraction process.

[0086] The signal time information can be assigned an absolute time, for example, using the synchronization server clock 11. This assignment is made once and is then maintained.

[0087] The identified signal feature sequences are assigned to synchronization information stored in the database along with the corresponding DB feature sequence. In the present embodiment, the synchronization information contains identification markers that describe the respective song and define the location within the song. Furthermore, the synchronization information contains the extraction time of the corresponding signal feature sequences.

[0088] The same process is performed on the additional signal synchronization server 7 with the respective additional signals 10, whereby the signal feature sequences are extracted from the audio track and compared with the DB feature sequences in the database. The extraction times can be recorded using the additional signal synchronization server clock 12 or extracted from the corresponding signal time information. The extraction times are transmitted to the playback device along with the synchronization information derived from the database, along with an assignment to the respective additional signals.

[0089] As a result, time information is assigned to both the main signal and the additional signal, which describes the respective time of the extracted feature sequences in the respective signal. This time information can be synchronized in advance by comparing the extracted feature sequences with the DB feature sequences stored in the database 9. If there is a predetermined degree of agreement between the extracted feature sequence and one of the DB feature sequences, the synchronization information or time information of this DB feature sequence is assigned to the extracted feature sequence, or a time difference is calculated and assigned to the main signal or the additional signal, wherein the time difference is added to all extraction times of the main signal or the additional signal, whereby the same synchronization information or the same time information is assigned to the same feature sequences in the main signal and in the additional signal.

[0090] According to a first variant, the synchronization information is coupled to the respective signals. This means that the synchronization information for the main signal 4 generated at the synchronization server 5 is coupled to the main signal, and the synchronization information generated at the additional signal synchronization server 7 is coupled to the corresponding additional signals. The synchronization information is transmitted together with the corresponding signals from the respective server 5, 7 to the playback device 6. Additional signals are only transmitted from the web server 8 to the playback device 6 if the corresponding additional signals have been requested by the playback device 6.

[0091] The main signal 4 and the requested additional signal are then output on the playback device 6. These two signals are synchronized using the transmitted synchronization information. The synchronization information contains time markers (e.g., the synchronized extraction times) that allow the playback device to recognize when the additional signal should be output synchronously with the main signal. In this variant, the synchronization information is imprinted on the corresponding signals as a kind of watermark.

[0092] The advantage of this variant is that the synchronization on the playback device is simple, since the two signals, the main signal and the additional signal, only need to be output synchronously to each other based on their time markers, and that no additional transmission paths are necessary to transmit the synchronization information between the synchronization servers 5, 7 and the playback device.

[0093] The disadvantage of this variant is that the main signal and the auxiliary signal must be output to a common playback device, and that the main signal and the auxiliary signal must be modified by inserting the synchronization information. This additional information can cause interference for receivers of the corresponding signals that do not use this information. Integrating the synchronization information into the signals to be transmitted delays the transmission.

[0094] In a further variant of this embodiment, the synchronization information is not coupled to the main signal and the additional signals, but is transmitted separately to the playback device 6. The synchronization information each contains time information that is linked to a specific characteristic of the respective signal. If the signal is a defined section with a defined beginning, then the time signal can refer to this beginning point or start point. This can be particularly useful for additional signals, each of which only contains a short piece of additional information, lasting, for example, a few tens of seconds up to a few minutes, and which can be output in addition to the main signal. The playback device can then synchronize the additional signal to the main signal based on the start point and the respective time information.If such a starting point is not present in the respective signal, then the time information must be related to another reference point. This reference point can, for example, be a sequence of features in the respective signal. This sequence of features can occur at any point in the signal. A module must then be provided on the playback device that can extract the sequence of features from the respective main signal and / or additional signal and compare it with the sequence of features supplied with the synchronization information. This makes it possible to obtain a clear reference of the time information to the respective main signal or additional signal without a start point clearly defined in the main signal or additional signal. The disadvantage of this variant is that a module for extracting the sequence of features and comparing the extracted sequence of features with the sequence of features contained in the synchronization information must be provided on the playback device.The advantage, however, is that with this variant the additional signal and / or the main signal do not have to be changed and can be transmitted in their original form.

[0095] According to a third variant of the first embodiment, the playback device clock 13, the synchronization server clock 11, and the additional signal synchronization server clock 12 present on the playback device 6 are synchronized. The playback device clock 13 is synchronized in pairs with the synchronization server clock 11 and the additional signal synchronization server clock 12, respectively.

[0096] Furthermore, the transmission times of the main signal from the synchronization server 5 to the playback device 6, as well as the transmission time from the additional signal synchronization server 7 to the playback device 6, are known. The transmission paths are designed in such a way that the transmission times remain constant. For short transmission paths, such as Bluetooth connections, the transmission times are generally constant. For longer transmission paths, especially when data is transmitted over the Internet, the transmission times often vary considerably, so this option does not work.

[0097] The time information contained in the synchronization information relates to a specific event at the synchronization server 5 or the additional synchronization server 7. This event is typically the time of extracting a specific signal feature sequence, which could be identified using the DB feature sequences. This makes it known when the main signal or the additional signal with the corresponding signal feature sequence was forwarded to the corresponding synchronization server 5, 7. Since the transmission time from the respective synchronization server 5, 7 to the playback device 6 is also known, it can be determined from this when the signal feature sequences arrive at the playback device 6. Since these signal feature sequences of the main signal and the corresponding additional signal have been identified, the additional signal can be temporally related to the main signal, i.e., the additional signal can be synchronized to the main signal.The corresponding time reference is contained in the synchronization information.

[0098] In this variant, the playback device clock 13 must be synchronized with the synchronization server clock 11 and the additional signal synchronization server clock 12, and the transmission times from the individual synchronization servers 5, 7 to the playback device must be known and stable. However, this approach has the advantage that neither the main signal nor the synchronization signal needs to be changed. Furthermore, the playback device does not need to integrate a module for extracting feature sequences. This is a very simple solution that allows for reliable synchronization.

[0099] A further advantage of the third variant is that it can easily be implemented with two different playback devices, one for playing the main signal and a second for playing the additional signal. A playback device clock of the main signal playback device must be synchronized with the synchronization server clock 11 of the synchronization server 5, and an additional signal playback device clock must be synchronized with the additional signal synchronization server clock 12. Furthermore, the two playback device clocks must be synchronized with each other.

[0100] For example, the main signal playback device could be a television and the auxiliary signal playback device a mobile phone. The main signal and the auxiliary signal are output synchronously.

[0101] What all three variants explained above have in common is that a main signal from broadcast studio 3 can be output to the playback device as the main signal, and a signal from camera 2 can be output as an additional signal, which does not have to be the same camera used to output the main signal. A user can thus freely choose the camera with which they want to view the performance on stage 1. Since all these signals have a similar audio track, they can be easily and reliably synchronized with each other using the audio tracks.

[0102] Furthermore, the variants explained above can be combined with each other, for example by transmitting the additional signal to the playback device according to one of the three variants and the additional signal according to one of the other variants to the playback device and synchronizing it with the additional signal.

[0103] For transmitting the main signal, the third variant is generally preferred, whereas for transmitting the additional signals, all three variants are equivalent.

[0104] Below is a second embodiment ( Figure 2 ), wherein identical elements as in the first embodiment are provided with the same reference numerals. The above explanations apply to identical elements unless otherwise stated below.

[0105] In the second embodiment, a stage 1 is again provided, which is scanned by several cameras 2. The signals from the cameras 2 are converted into a main signal 4 in a broadcast studio 3. The broadcast studio 3 is connected to a synchronization server 5. The synchronization server 5 is coupled to a database server 9, which contains a database with DB feature sequences and the associated synchronization information.

[0106] An HS playback device 6 / 1 is connected to the synchronization server 5 to receive and play the main signal. Multiple HS playback devices 6 / 1 can be provided.

[0107] The second embodiment differs from the first embodiment in that an independent source for additional information is provided. This source is, for example, an additional information database server 15. If a concert with several songs is being performed on stage 1, it may be expedient to provide the corresponding lyrics in the additional information database server 15. However, the additional information database server 15 can also contain foreign-language translations of the corresponding lyrics as audio tracks. There are songs that are known in many languages, such as the children's lullaby "Brother Jacob."

[0108] Preferably, the additional information stored on the additional information database server 15 is already provided with corresponding synchronization information. For songs, this can be the starting time and other time markers during the song.

[0109] The additional information database server 15 is connected to a web server 8. The additional information can be retrieved from the web server 8 via the internet 18. A ZS playback device 6 / 2 for playing an additional signal is connected to the internet 14. The synchronization server 5 also has a connection to the internet 14, so that synchronization information generated on the synchronization server 5 can be forwarded to the ZS playback device 6 / 2 via the internet 14.

[0110] A synchronization clock 11 is provided on the synchronization server 5, which is synchronized with a playback device clock 13 / 1 of the HS player and with a playback device clock 13 / 2 of the ZS player 6 / 2. In the second embodiment, the synchronization clock 11 of the synchronization server 5 is the master clock, whereas in the first embodiment, the playback device clock 13 was the master clock to which all other clocks are synchronized.

[0111] At the synchronization server 5, synchronization information is generated by extracting signal feature sequences from the main signal and comparing them with corresponding DB feature sequences of the database server 9. The generation of the synchronization information essentially corresponds to that of the first embodiment.

[0112] Furthermore, the transmission time for transmitting the main signal from the synchronization server 5 to the HS playback device 6 / 1 is known, so that if the time at which a certain section of the main signal is passed through to the synchronization server 5 is known, it is also known when this section is output to the HS playback device 6 / 1.

[0113] The synchronization information transmitted from the synchronization server 5 to the ZS playback device 6 / 2 thus contains time information, each of which describes a point in time of the main signal relative to a detected signal feature sequence, and identification markers that describe the content of the main signal. In the present exemplary embodiment, the identification markers indicate which song is being played back with the main signal. The identification markers can optionally contain further information, such as verses, lines, or text excerpts from the song. These text excerpts are preferably text excerpts at the point at which one of the signal feature sequences was detected. The time information preferably contains an indication of when the corresponding signal feature sequence was extracted at the synchronization server 5.

[0114] Based on this synchronization information, the ZS playback device 6 / 2 indicates when which song is output to the HS playback device 6 / 1. Accordingly, the ZS playback device can output the additional signals received from the additional information database server 15 or the web server 8, which have already been provided with synchronization information, to the ZS playback device 6 / 2 synchronously with the output of the main signal to the HS playback device 6 / 1.

[0115] Optionally, an additional signal synchronization server 7 can be provided between the additional information database server 15 and the web server 8, which is configured similarly to the first exemplary embodiment. If the additional information is song lyrics, encoded, for example, in ASCII, then the additional information does not contain any audio signals. However, audio signal-like feature sequences can be generated from the words contained in the text, as is known from speech synthesis. These feature sequences can then be compared with DB feature sequences stored in another database server 16. This also makes it possible to directly compare text sections of the songs with corresponding text sections stored in the database server 16. The individual letters of the text sections form the corresponding features.The feature sequences stored in the database server 16 are each assigned synchronization information, which can be added to the additional information or additional signals.

[0116] Alternatively, spoken or sung texts can be converted into text using speech recognition. The features are then text and / or letter sequences that are also stored in the database.

[0117] The ZS playback device 6 / 2 can thus display the corresponding lyrics synchronously with the picture and sound reproduction of the music concert taking place on stage 1, which is output on the HS playback device 6 / 1.

[0118] A third embodiment ( Figure 3) essentially corresponds to the second embodiment and differs from it in that the synchronization server 5 is configured to play the main signal independently of the connection between the transmitting station 3 and the HS playback devices 6 / 1. Furthermore, the ZS playback device 6 / 2 has a sensor 17 for detecting at least a portion of the main signal output by the HS playback device 6 / 1. This sensor 17 can be a microphone for capturing the audio signal of the main signal 4 or a camera for capturing the image output of the main signal 4.

[0119] The ZS playback device 6 / 2 is equipped with a module for extracting the signal feature sequences of the main signal 4, wherein these signal feature sequences are extracted from the main signal 4 sampled by the sensor 17. The extraction time can be detected by the ZS playback device clock 13 / 2. Since, as already explained above, the extraction process itself can be subject to temporal fluctuations, it may be expedient to use the signal time information to determine the extraction time. In this embodiment, instead of signal time information inherently contained in the main signal, signal time information added during recording with the sensor 17 (microphone) can be used, which describes the recording time of the signal.Such signal time information is independent of temporal fluctuations of the extraction process and enables a clear temporally relative arrangement of the extracted signal feature sequences.

[0120] The signal feature sequences are transmitted to the synchronization server 5, where, as in the first and second embodiments, they are analyzed and identified using the DB feature sequences from the database server 9. Synchronization information is again generated at the synchronization server 5. The synchronization information of the third embodiment differs from the synchronization information of the previous embodiments in that it is based exclusively on the time of the ZS playback device clock 13 / 2. The synchronization information is transmitted from the synchronization server via the Internet 14 to the ZS playback device 6 / 2. There, the additional signal 10 is synchronized to the main signal 4 using the synchronization information, as in the previous embodiments, although in this case the synchronization is carried out solely based on the playback time recorded by the ZS playback device clock 13 / 2.There is no need to synchronize different times between the ZS player 6 / 2, the HS player 6 / 1 or the synchronization server 5.

[0121] Alternatively, instead of the signal feature sequences, short signal sequences, such as music snippets, can be transmitted to the synchronization server 5, which are to be output as an additional signal. The server identifies the signal feature sequences of the signal sequences and analyzes and identifies them based on the DB feature sequences from the database server 9. The signal sequences are generally no longer than 60 s and, in particular, no longer than 30 s or no longer than 15 s.

[0122] The third embodiment can also be modified such that the one module for extracting the signal feature sequences is arranged on the synchronization server 5 instead of on the ZS playback device 6 / 2.

[0123] The third embodiment is a very elegant solution for outputting additional signals to a separate ZS playback device 6 / 2. In this third embodiment, the additional signal can be synchronized to a main signal, whereby the transmission duration, for example, between the transmitting station 3 and the HS playback device 6 / 1 can vary freely within a predetermined range.

[0124] A fourth embodiment ( Figure 4 ) essentially corresponds to the third embodiment and differs from it in that the synchronization server 5 has the sensor 17. The synchronization server 5 is implemented on a local processing unit, e.g., a computer, a minicomputer, or even a game console. As in the previous embodiment, the sensor 17 can be a microphone for capturing the audio signal of the main signal 4 or a camera for capturing the image output of the main signal 4.

[0125] The synchronization server 5 is designed with a module for extracting the signal feature sequences of the main signal 4, wherein these signal feature sequences are extracted from the main signal 4 sampled by means of the sensor 17.

[0126] The time of extraction is recorded using the synchronization clock 11. The signal feature sequences are analyzed and identified at the synchronization server 5, as in the first, second, and third embodiments, using the DB feature sequences from the database server 9. Synchronization information is again generated at the synchronization server 5, with the synchronization clock 11 being the sole source of the synchronization information. The synchronization information is transmitted from the synchronization server 5 via an intranet 14 or another data connection, such as Bluetooth, to the ZS playback device 6 / 2. There, the additional signal 10 is synchronized to the main signal 4 using the synchronization information, as in the previous embodiments.

[0127] The time of the synchronization clock 11 is synchronized with the ZS playback device clock 13 / 2.

[0128] The main difference between the fourth embodiment and the previous ones is that the synchronization server 5 is not accessed via the internet, but is located locally at a user's location. This has the advantage that synchronization always works, even if the internet fails, since it is not dependent on the internet.

[0129] However, the database server 9 can be accessed via the Internet or it can also be located on the same computing unit as the synchronization server 5.

[0130] In the third or fourth embodiment explained above, the synchronization server 5, the database server 9 and the ZS player 6 / 2 may be formed on a single device such as a computer (desktop, laptop, etc.) or a mobile phone.

[0131] In principle, however, the synchronization server 5 can also be provided on hardware designed separately from the playback device 6 / 2. The synchronization server 5 can be connected to the playback device 6 / 2 via the Internet. The amount of data exchanged between the synchronization server 5 and the playback device is small.

[0132] What all of the above-described embodiments have in common is that synchronization information is generated based on one or more signal feature sequences extracted from the main signal. This makes it possible to synchronize additional signals to a main signal (on the fly) for which no specific points in time, such as a start time, are known in advance. Of course, these methods can also be applied if a predetermined point in time is specified in the respective signal, which can be used as a reference.

[0133] Based on this synchronization information, the additional signals can be synchronized with the main signal in a variety of ways. Some examples are explained above. However, countless modifications are possible within the scope of the invention, which can be adapted and designed accordingly for the respective application.

[0134] A further aspect of the invention is to adjust the quality of the additional signal streaming based not only on the available bandwidth but also on the available buffer time. The additional signal playback device 6 / 2 receives the synchronization information and sends a query to the additional information database server 14 stating which additional signals are available at what time. If a corresponding additional signal 10 is found, the buffer time is also known. The buffer time describes the time still available for the additional signal before it must be played back in order to be synchronized with the main signal. This query can also be used to roughly check the available bandwidth of the network. Depending on the bandwidth and buffer time, a different encoding level is automatically selected. During the buffer time, the additional signal is encoded, transmitted from the additional signal server to the playback device, and then decoded again.Depending on the encoding level, the file to be transmitted, or rather the file portion, varies in size and requires different amounts of time to transmit. Therefore, a balance between encoding time and transmission time must be found so that the buffer time is utilized as effectively as possible and the quality of the additional signal is as high as possible.

[0135] This method can also be designed in such a way that the server(s) encode the signals in different qualities or different encoding levels and simultaneously make them available for retrieval, and the playback device that is to play the signal selects or retrieves the signal in the appropriate quality or encoding level.

[0136] If the buffer time is very short, for example, in live broadcasts, it is advantageous to select the shortest possible chunk length for the additional signal to be transmitted. The signal can be transmitted in multiple chunks, whereby the chunks must first be generated. The shorter the chunks, the more complex the handling becomes, because they are transmitted individually. However, when retrieving a chunk, you have to wait at least as long as the respective chunk is long. Therefore, the shorter the chunks, the faster the response time. The chunk length can be reduced to the point where it corresponds to a single frame. At 25 frames per second, this corresponds to 40 ms. This enables very fast transmissions. With certain codecs, such as the h265 codec, "zero latency" settings are also possible. This means that the time for encoding and subsequent decoding is very short, for example, less than 1 s.A certain amount of latency is unavoidable. However, with a "Zero Latency" setting, the corresponding codec method causes no additional latency. The buffer time is thus required almost exclusively for the actual transmission of the signal, which can be significantly reduced with sufficiently high bandwidth. For example, at a live concert where the organizer provides a camera view to attendees with smartphones via a web server, the appropriate Wi-Fi infrastructure can also be provided so that the video signal can be transmitted with virtually no delay.

[0137] With this aspect, the coding of the additional signal and / or the transmission path for transmitting the additional signal to the additional signal playback device 6 / 2 can be automatically selected depending on the determined synchronization information. If the synchronization information contains information that little time is available for transmitting the additional signal, then it is expedient to reduce the data volume of the additional signal by appropriately compressing the coding and selecting a fast transmission path. Furthermore, the coding should be very fast. A significant reduction in the data volume and rapid compression often impair the quality of the additional signal. If, on the other hand, more time is available, then a more complex coding and / or a low compression rate can be used, thereby achieving a higher quality of the additional signal.

[0138] In all embodiments explained above, a database server 9 is provided with a pre-prepared database with DB feature sequences and synchronization information.

[0139] Within the scope of the invention, the database is generated on the database server 9 during operation (live database). This is particularly useful when a main signal is present to which additional signals are to be output synchronously, whereby the main signal was previously unknown. In such a case, feature sequences are extracted from the main signal, and the time present at the time of extraction is recorded. These extracted feature sequences are stored in the database together with the extraction time. Instead of the extraction time or in addition to this, time information contained in the main signal can also be extracted and stored on the database server 9 together with the feature sequences.

[0140] The time information forms all or part of the synchronization information. The database thus generated during system operation can be compared with another database in which various signals are already stored, broken down into feature sequences. This database can also contain metadata describing the content, timing, and meaning of these feature sequences or signals. A wide variety of media streams can be stored in this database as feature sequences. By comparing the feature sequences with this additional database, metadata, particularly semantic information or meaning content, can be assigned to the feature sequences of the online or on-the-fly generated database.

[0141] Such online creation of the database on the database server 9 is possible in all of the embodiments explained above. A user can also create such a live database locally on their device (computer, mobile phone, etc.). List of reference symbols

[0142] 1Stage 2Camera 3Broadcast studio 4Main signal 5Synchronization server 6Playback device 7Auxiliary signal synchronization server 8Web server 9Database server 10Auxiliary signal 11Synchronization clock 12Auxiliary signal synchronization clock 13Playback device clock 14Auxiliary information database server 16Database server 17Sensor 18Internet

Claims

1. A method for synchronizing an additional signal with a primary signal which is transmitted to multiple playback units by a broadcast method, comprising the steps of: - generating DB feature sequences for a database from the primary signal while the primary signal is transferred from a transmitting station to a playback unit, - generating synchronization information about the primary signal in a synchronization server in that at least one signal feature sequence of the primary signal is extracted and compared with DB feature sequences stored in the database, wherein both the signal feature sequence and the DB feature sequence comprise features in the frequency space and wherein, in case of a match of the signal feature sequence with one of the DB feature sequences to a predetermined degree, synchronization information of the matching DB feature sequence is assigned to the primary signal at a position predefined by the signal feature sequence, wherein a sound track of the primary signal is transformed in sections into the frequency space by a fast Fourier transform, wherein the sections of the sound track that are transformed into the frequency space are not longer than 50 ms, wherein the features are frequency peaks, which are above a certain threshold value, - transferring the synchronization information to one of the playback units, which delivers an additional signal synchronously to the primary signal on the basis of the synchronization information.

2. The method of claim 1, characterized in that the point in time of the signal feature sequence is determined in the primary signal, which matches the DB feature sequence, and this point in time is used as extraction moment for generating the synchronization information, or a time information is assigned to the DB feature sequence, which defines a certain DB point in time relative to the DB feature sequence, which is used for generating the synchronization information.

3. The method of claim 1 or 2, characterized in that synchronization information is assigned to the additional signal in that a signal feature sequence of the additional signal is extracted and compared with DB feature sequences stored in a database, wherein, in case of a match of the signal feature sequence with one of the DB feature sequences to a predetermined degree, synchronization information is assigned to the additional signal at a position predefined by the signal feature sequence, or synchronization information is manually assigned to the additional signal, or an additional signal is used to which synchronization information is already associated beforehand.

4. The method according to any of claims 1 to 3, characterized in that the synchronization information comprises one or more of the following data: - a time information, which describes a point in time of the primary and / or additional signal relative to a detected signal feature sequence, - one or more identification markers, which describe the content of the primary and / or additional signal, - a time stamp, which describes a certain point in time in the primary and / or additional signal, - a temporal offset, which describes the required time interval for forwarding a primary and / or additional signal from a specific location on a transfer path to the actual delivery of the primary and / or additional signal at the playback unit.

5. The method according to any of claims 1 to 4, characterized in that the synchronization information is generated at a synchronization server, which is configured independently of a playback unit.

6. The method according to any of claims 1 to 5, characterized in that the additional signal is synchronized with the primary signal at the playback unit in that - the primary signal and the additional signal are delivered at the playback unit, each of which includes one or more time markers as synchronization information, on the basis of which the playback unit synchronously assigns the additional signal to the primary signal, or - on the basis of a playback time measured by means of a clock at the playback unit, to which the additional signal is assigned by means of the synchronization information in such a way that the additional signal is delivered synchronously with the primary signal.

7. The method according to any of claims 1 to 6, characterized in that if several signal feature sequences match with corresponding DB feature sequences to a predetermined degree, the several signal feature sequences are evaluated according to predetermined rules, and that the signal feature sequence having the best evaluation is selected for assigning the synchronization information to the primary signal at a position predefined by the signal feature sequence.

8. The method of claim 7, characterized in that the predetermined rules comprise one or more of the following rules: - the higher the number of matching features of a feature sequence, the better the evaluation; - the higher the number of matching features of a feature sequence in relation to the number of features comprised in the feature sequence, the better the evaluation; - the concordance of the different features is classified, and the smaller the distance of the concordant features, the better is the classification of the corresponding concordance, this classification being taken into account in the overall evaluation of the concordance of a feature sequence.

9. The method according to any of claims 1 to 8, characterized in that for the calibration of a transfer route from a server to a playback unit and / or of the latency at a playback unit for the delivery of a media signal at the playback unit, a reference signal is delivered, which is simultaneously picked up by a corresponding sensor, wherein the delivered reference signal and the received reference signal are compared with each other to determine a time interval required for forwarding the reference signal and the actual delivery at the playback unit, and this time interval is used as a temporal offset to determine a delivery time with which a signal is delivered at the media playback unit.

10. The method of claim 9, characterized in that the reference signal comprises an audio signal and the sensor is a microphone and / or the reference signal comprises a video signal and the sensor is a camera.

11. The method of claim 9 or 10, characterized in that the time interval is determined in that the point in time of sending and the point in time of receiving the reference signal are determined, wherein the time interval is derived from the time difference of these two points in time, and the measurement of at least one of the two points in time is performed by a comparison of an extracted reference signal feature sequence with one or more previously stored reference signal feature sequences.

12. The method according to any of claims 1 to 11, wherein the additional signal is transmitted to the playback unit from a synchronization server, which is configured independently of a playback unit, and the synchronization server generates synchronization information which relates to a server time of the synchronization server, which server time is measured at the synchronization server with a clock arranged there, wherein a playback unit clock, which is synchronized at least once with the server time and is used for measuring the playback time, is provided at the playback unit, and a time drift of the playback time with respect to the server time is measured and this time drift is also taken into account when synchronizing the additional signal with the primary signal.

13. The method of claim 12, characterized in that the time drift is determined by multiple transfer of a time signal of the server time to the playback unit or of the playback time to the synchronization server and comparison of the transferred time signal with the locally present time signal in order to calculate a respective time difference, wherein the time drift is determined on the basis of the deviations of the time differences.

14. The method for synchronizing an additional signal with a primary signal according to any of claims 1 to 13, wherein on the basis of the synchronization information, it is determined how much time is available for transmitting the additional signal from a predetermined server to a playback unit, and, on the basis of this time, either one of several coding methods is selected or set automatically and / or one of several transfer routes is selected.

15. The method according to any of claims 1 to 14, wherein in a first step, the time stamps of the available additional signals are transferred to a playback unit and, in this way, the available buffer time is calculated, and in a second step, the additional signal is transferred from an additional signal server to a playback unit, wherein the quality of the additional signal depends on the buffer time and the available bit rate, wherein preferably the additional signal is transferred from the additional signal server to the playback unit in fragments having time lengths of at most 10 frames, which corresponds approximately to 400 ms, in particular at most 5 frames, which corresponds approximately to 200 ms, and preferably at most 1 frame, which corresponds approximately to 40 ms, and the additional signal is received at the playback unit with a local web server.