Method and system for synchronizing a viewer effect signal of media content with a media signal of the media content

Abstract

A method for synchronizing a viewer effect signal of media content with a media signal of the media content is described. The method generally has: capturing an audio portion of the media signal from an ambient environment using a microphone; obtaining a fingerprint from the captured audio portion of the media signal; identifying a temporal position of the media content corresponding to the obtained fingerprint based on a reference fingerprint associated with at least a reference temporal position of the reference media content; obtaining a viewer effect signal associated with the identified temporal position of the media content; and outputting a viewer effect signal synchronized with the media signal using the identified temporal position of the media content for producing an effect to a viewer that is synchronized with the media signal of the media content.

Description

Technical Field

[0001] The present disclosure relates to the field of media identification and synchronization. More specifically, the present specification relates to methods and apparatus for synchronizing viewer effect signals, such as haptic signals, with video and audio streaming media. Background Art

[0002] When controlling motion feedback systems used in public, commercial or home environments, the challenge is to synchronize the haptic signals used in games, feature-length films, amusement park rides, virtual reality, etc. with the audio or video signals during playback so that the haptic signals are delivered to the motion feedback system in time to match the platform movement to the reproduced audiovisual signal. The required accuracy can be milliseconds.

[0003] In some applications, the haptic signal is not transmitted synchronously with the audio or video signal; the playback device must separately retrieve the haptic signal and play it back synchronously with the audio or video signal. In some cases, the audio signal of the media may only be available as part of the ambient sound as an output of a speaker. For example, the ambient sound can come from a mono audio setup, a stereo setup, or multi-channel surround sound audio, such as 5.1 surround sound or 7.1 surround sound. As a function of the sound capture location, as a function of the number of audio channels being used, and / or as a function of which language track of the media is being played, this can cause problems identifying the media due to factors such as reverberation, parasitic noise, unknown sound levels, distortion, echo, sound variability, etc.

[0004] Therefore, there is a need for an improved synchronization method and apparatus that addresses the above challenges. Summary of the invention

[0005] According to a first aspect of the present invention, there is provided a system for synchronizing a viewer effect signal of media content with a media signal of the media content, the system comprising: at least one microphone, the microphone capturing an audio portion of the media signal from an ambient environment; a controller, the controller being communicatively coupled to the at least one microphone, the controller having a processor and a non-transitory memory having instructions stored thereon, the instructions, when executed by the processor, performing the following steps: obtaining a fingerprint from the captured audio portion of the media signal; identifying a time position of the media content corresponding to the obtained fingerprint based on a reference fingerprint associated with a reference time position of the reference media content; obtaining the viewer effect signal associated with the identified time position of the media content; and outputting the viewer effect signal synchronized with the media signal using the identified time position of the media content, which can be used to produce an effect synchronized with the media signal to a viewer.

[0006] According further to the first aspect of the invention, for example, at least one microphone may be integrated with the viewer seat.

[0007] Furthermore, according to the first aspect of the present invention, the at least one microphone may, for example, comprise a plurality of microphones capturing different audio portions of the media signal.

[0008] Furthermore, according to the first aspect of the present invention, the plurality of microphones may be distributed in the surrounding environment, for example.

[0009] Furthermore, according to the first aspect of the present invention, the obtaining, identifying and acquiring steps may be performed, for example, for each captured audio portion of the media signal.

[0010] Furthermore, according to the first aspect of the invention, said reference fingerprint may be stored, for example, on a remote database and accessible by said controller during said identification.

[0011] Furthermore, according to the first aspect of the present invention, said obtaining a viewer effect signal may for example comprise retrieving said viewer effect signal from said remote database.

[0012] Furthermore, according to the first aspect of the present invention, the controller may, for example, have a gain control module which applies a given gain to the captured audio portion before the recognition.

[0013] Further, according to the first aspect of the present invention, the gain control module may, for example, reduce the given gain when the peak amplitude is above a saturation threshold, increase the given gain when the peak amplitude is below a detectable sound threshold, and maintain the given gain when the peak amplitude is above the detectable sound threshold and below a saturation threshold.

[0014] Still further, according to the first aspect of the present invention, the system may also, for example, include a viewer effect generating device communicatively coupled to the controller, the viewer effect generating device generating an effect to the viewer based on the output viewer effect signal synchronized with the media signal.

[0015] Furthermore, according to the first aspect of the present invention, the viewer effect generating device may be, for example, a motion feedback device, and the viewer effect signal may correspond to, for example, a tactile signal.

[0016] According to a second aspect of the present invention, a method for synchronizing a viewer effect signal of media content with a media signal of the media content is provided, the viewer effect signal being used to produce an effect to a viewer, the method comprising: capturing an audio portion of the media signal from an ambient environment using at least one microphone; obtaining a fingerprint from the captured audio portion of the media signal; identifying a time position of the media content corresponding to the obtained fingerprint based on a reference fingerprint associated with a reference time position of at least reference media content; obtaining the viewer effect signal associated with the identified time position of the media content; and outputting the viewer effect signal synchronized with the media signal using the identified time position of the media content for producing an effect synchronized with the media signal of the media content.

[0017] According further to the second aspect of the present disclosure, the system may, for example, further include generating an effect synchronized with the media signal of the media content based on the output viewer effect signal synchronized with the media signal.

[0018] Furthermore, according to the second aspect of the invention, the viewer effect signal may be, for example, a haptic signal, and the generating may, for example, comprise activating one or more actuators of a motion feedback device according to the haptic signal.

[0019] Furthermore, according to the second aspect of the present invention, the method may, for example, further comprise receiving an identity of the media content, the reference fingerprint being associated with the identity of the media content.

[0020] Furthermore, according to the second aspect of the present invention, the capturing may for example comprise capturing a plurality of audio portions of the media signal, the obtaining, identifying and acquiring steps being performed for each captured audio portion of the media signal.

[0021] Furthermore, according to the second aspect of the present invention, the reference fingerprint may be stored, for example, on a remote database, and the identifying may include accessing the reference fingerprint from the remote database.

[0022] Furthermore, according to the second aspect of the present invention, said obtaining a viewer effect signal may for example comprise retrieving said viewer effect signal from said remote database.

[0023] Furthermore, according to the second aspect of the present invention, said obtaining a fingerprint may for example comprise applying a given gain to the captured audio portion before said identifying, and the given gain may for example depend on a peak amplitude.

[0024] Further, according to the second aspect of the invention, the given gain may, for example, be reduced when the peak amplitude is above a saturation threshold, increased when the peak amplitude is below a detectable sound threshold, and maintained when the peak amplitude is above the detectable sound threshold and below a saturation threshold.

[0025] According to a third aspect of the present invention, there is provided a system for determining whether a viewer effect signal of media content is synchronized with a media signal of the media content, the system comprising: a viewer effect generating device, the viewer effect generating device generating an effect to a viewer based on the viewer effect signal; at least one microphone, the microphone capturing an audio portion of the media signal from a surrounding environment; and a controller, the controller being communicatively coupled to the viewer effect generating device and the at least one microphone, the controller having a processor and a non-transitory memory having instructions stored thereon, and when executed by the processor, the instructions performing the following steps: obtaining a fingerprint from the captured audio portion of the media signal; retrieving a reference fingerprint of the media signal at a reference time position of the media content based on a current time position of the viewer effect signal; comparing the obtained fingerprint and the reference fingerprint with each other; determining whether the viewer effect signal is synchronized with the media signal based on the comparison; and outputting the viewer effect signal synchronized with the media signal based on the determination, for generating an effect synchronized with the media content using the viewer effect generating device.

[0026] According further to the third aspect of the invention, the system may, for example, further comprise stopping outputting the viewer effect signal when it is determined that the viewer effect signal is not synchronized with the media signal.

[0027] Furthermore, according to the third aspect of the present invention, the system may, for example, further include maintaining the output for a given asynchronous delay before the stopping.

[0028] Furthermore, according to the third aspect of the present invention, the given asynchronous delay may be, for example, at least one of zero and a range between approximately 1 second and 1 minute.

[0029] Further, according to the third aspect of the present invention, the system may, for example, also include receiving acceptable asynchronous delay data associated with the media content, the acceptable asynchronous delay data including multiple acceptable asynchronous delay values ​​associated with corresponding reference time positions in the reference time positions of the media content, and determining the given asynchronous delay based on the received acceptable asynchronous delay data and the reference time position of the media content.

[0030] Further, according to the third aspect of the present invention, the system may, for example, also include receiving synchronization confidence data associated with the media content, the synchronization confidence data including multiple synchronization confidence values ​​associated with corresponding reference time positions in the reference time positions of the media content, and determining the given asynchronous delay based on the synchronization confidence data and the reference time position of the media content.

[0031] Furthermore, according to the third aspect of the present invention, the at least one microphone may be integrated with the viewer seat, for example.

[0032] Furthermore, according to the third aspect of the present invention, the controller may, for example, have a gain control module which applies a given gain to the captured audio portion before the retrieval.

[0033] Still further, according to the third aspect of the present invention, the system may, for example, further include a viewer effect generating device communicatively coupled to the controller, the viewer effect generating device generating an effect to a viewer based on the output viewer effect signal synchronized with the media signal.

[0034] Furthermore, according to the third aspect of the present invention, the viewer effect generating device may be, for example, a motion feedback device, and the viewer effect signal corresponds to a tactile signal.

[0035] According to a fourth aspect of the present invention, a method for determining whether a viewer effect signal of media content is synchronized with a media signal of the media content is provided, the method comprising: generating an effect to a viewer based on the viewer effect signal; capturing an audio portion of the media signal from the surrounding environment; obtaining a fingerprint from the captured audio portion of the media signal; retrieving a reference fingerprint of the media signal at a reference time position of the media content based on a current time position of the viewer effect signal; comparing the obtained fingerprint and the reference fingerprint with each other; determining whether the viewer effect signal is synchronized with the media signal based on the comparison; and outputting the viewer effect signal synchronized with the media signal based on the determination, for generating an effect synchronized with the media content using the viewer effect generating device.

[0036] According further to the fourth aspect of the invention, the method may, for example, further comprise, upon determining that the viewer effect signal is not synchronized with the media signal, ceasing to output the viewer effect signal.

[0037] Furthermore, according to the fourth aspect of the present invention, the method may, for example, further include, before the stopping, maintaining the output for a given asynchronous delay.

[0038] Still further, according to the fourth aspect of the present invention, the given asynchronous delay may be, for example, at least one of zero and a range between approximately 1 second and 1 minute.

[0039] Further, according to the fourth aspect of the present invention, the method may, for example, also include receiving acceptable asynchronous delay data associated with the media content, the acceptable asynchronous delay data including multiple acceptable asynchronous delay values ​​associated with corresponding reference time positions in the reference time positions of the media content, and determining the given asynchronous delay based on the received acceptable asynchronous delay data and the reference time position of the media content.

[0040] Further, according to the fourth aspect of the present invention, the method may, for example, also include receiving synchronization confidence data associated with the media content, the synchronization confidence data including multiple synchronization confidence values ​​associated with corresponding reference time positions in the reference time positions of the media content, and determining the given asynchronous delay based on the synchronization confidence data and the reference time position of the media content.

[0041] Furthermore, according to the fourth aspect of the present invention, capturing may, for example, include capturing a plurality of audio portions of the media signal, the obtaining, retrieving and comparing steps being performed for each captured audio portion of the media signal.

[0042] Furthermore, according to the fourth aspect of the present invention, the obtaining of the fingerprint may for example further comprise applying a given gain to the captured audio portion before the identifying, the given gain being dependent on the peak amplitude.

[0043] Further, according to the fourth aspect of the present invention, the given gain can be, for example, reduced when the peak amplitude is higher than the saturation threshold, increased when the peak amplitude is lower than the detectable sound threshold, and maintained when the peak amplitude is higher than the detectable sound threshold and lower than the saturation threshold.

[0044] Furthermore, according to the fourth aspect of the present invention, the viewer effect signal may be, for example, a haptic signal, and the generating comprises activating one or more actuators of a motion feedback device according to the haptic signal.

[0045] The motion feedback system is intended to include a motion platform, a motion actuator, a motion simulator, a vibrotactile transducer, and an inertial vibrator.

[0046] Motion platforms are intended to include any platform or seating device whose motion is caused by actuators and on which a person is mounted. Motion platform applications include, but are not limited to, games, feature-length films, amusement park rides, virtual reality, music with or without video, etc. Examples of motion platforms include seats or chairs for one or more people, on which one or more actuators engaged with the ground or structure are mounted. Another example is a platform for accommodating a seat, chair, or other device for accommodating a user, on which one or more actuators engaged with the ground are mounted. Another example is a shaker device that produces vibrations in a device (e.g., a chair, seat, etc.).

[0047] Based on the content to be reproduced to the viewer, the motion code, motion code signal, or motion code stream contains motion effects, or similar viewer effect signals contain other viewer effects, such as subtitles, alternative languages, audio tracks for the deaf or hard of hearing. In most cases, the haptic signal is prepared by a motion designer. In other cases, the haptic signal, or a portion thereof, is automatically generated.

[0048] Media signals should be understood to include audio signals, video signals, tactile signals, augmented reality signals, virtual reality signals, three-dimensional hologram signals, etc. For example, the media signal may cause vibrations of the device, and these vibrations may be captured for synchronization.

[0049] An audio signal or audio stream is understood to contain sound information to be reproduced during a presentation, such as an audio track of a game, a feature-length film, an amusement park ride, a virtual reality environment, indie music, music videos, etc. For example, the audio signal is in the form of encoded pulse code modulation (PCM) samples.

[0050] A video signal or video stream is understood to contain visual information to be reproduced to a user, e.g. the main screen of a game, a feature-length movie, an amusement park ride, a virtual reality environment, etc. An example of a video signal or video signal is an MPEG-4 part 10 video elementary stream.

[0051] Media content is intended to include games, feature-length films, amusement park rides, virtual reality environments, advertisements, music, etc.

[0052] A feature-length film is a film intended for projection in a cinema or home theater. It usually lasts more than 30 minutes and has separate video and audio tracks to which haptic signals, motion code streams, or motion code tracks may be associated. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Further features and advantages of the present disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0054] Figure 1Ais a schematic diagram of a surround sound setup having a system for synchronizing a viewer effects signal of media content with a media signal of the media content according to an embodiment of the present disclosure;

[0055] Figure 1B According to one embodiment Figure 1A An oblique view of an example of a microphone integrated into a viewer's seat of the device;

[0056] Figure 1C According to an embodiment Figure 1A An oblique view of an example of a standalone microphone for a device;

[0057] Figure 2 is a flow chart of a method for synchronizing a haptic signal with an audio signal according to one embodiment;

[0058] Figure 3 is a flow chart illustrating an embodiment of a method for determining whether a haptic signal corresponding to media content is synchronized with a media signal of the media content;

[0059] Figure 4 is a schematic diagram of a system for synchronizing a haptic signal with an audio signal according to one embodiment;

[0060] Figure 5A is a graph illustrating an example of an audio portion of a given media content captured using a microphone according to one embodiment;

[0061] Figure 5B is a diagram showing an application according to an embodiment Figure 5A A graph of audio gain of an audio portion of an audio device;

[0062] Figure 5C is a graph illustrating an example of a reference audio track for a given media content according to one embodiment;

[0063] Figure 5D is a graph illustrating an example of a viewer effect signal associated with a given media content according to one embodiment;

[0064] Figure 5E is a graph illustrating exemplary metadata track content in the form of a synchronization confidence delay track and an acceptable asynchronous delay track for given media content according to one embodiment;

[0065] Fig. 6A is a graph illustrating an example of an English version of an audio track of a given media content according to one embodiment;

[0066] Figure 6B is a graph illustrating an example of a French version of an audio track of a given media content according to one embodiment;

[0067] Figure 6C is a diagram showing a method according to an embodiment Fig. 6A A graph of the sample envelope of the English version of the audio track;

[0068] Fig.6D is a diagram showing a method according to an embodiment Figure 6B a graph of a sample envelope of the French version of the audio track; and

[0069] Fig. 6E is a diagram showing a method according to an embodiment Figure 6C and Fig.6D A graph of the ratio of the envelope.

[0070] It should be noted that throughout the drawings, like features are identified by like reference numerals. DETAILED DESCRIPTION

[0071] Reference is now made to the accompanying drawings, in particular Figure 1A , shows an exemplary audio-visual device through which one or more users can view and listen to media with parallel synchronized video and audio outputs. In some embodiments, the audio-visual device can also be used to play music or other types of sounds without any video output. Figure 1A In the embodiment of the present invention, the device can be domestic or small, featuring a single seat S, whether for a single occupant or multiple occupants. Figure 1A While the device of is shown as an example, the method 20 and apparatus 50 described herein can be used for larger scale devices, including theaters, performance venues, auditoriums, etc., to name a few examples. This can include more than one seat S. Therefore, reference is made herein to one or more seats, but unless otherwise stated, the synchronization concepts described herein can be applied to devices with single or multiple seats as well as other items, such as shakers, and / or devices with any other suitable tactile device type, such as tactile vests, tactile shoes, tactile gloves, tactile wristbands or watches, and tactile helmets.

[0072] In this arrangement, the seat S faces a screen television, e.g., a television, a tablet or similar portable device, a projection display, a VR helmet, etc., i.e., a video output. The seat S may be surrounded by speakers LS (also referred to as loudspeakers, including, e.g., a subwoofer). The audio signal sent to the speaker LS may come from any suitable device, e.g., a home theater receiver, an amplifier, a television, a media streamer, a player (DVD, ) etc. The loudspeaker LS may also be integrated into the screen TV (e.g., TV speakers). Thus, the occupant of the seat S is exposed to surround sound from the loudspeaker LS in sync with the video displayed on the screen TV.

[0073] The seat S is equipped or decorated with a motion feedback system 10 ( Figure 4 ), which has actuators 1 to N, shown as 10A, 10B, 10C, 10D in this example. The motion feedback system transmits vibratory powered motion to the seat S as a function of the media being viewed on the A / V device. The seat S can have fewer or more actuators, with a single actuator being an option, and have a passive joint (e.g., a ball joint) to connect the seat S to a structure or ground. The actuators 10A-10N can be linear actuators (e.g., electromechanical actuators), such as those described in U.S. Patents Nos. 9,664,267, 9,863,513, 10,012,295, which are incorporated herein by reference. As other possibilities, the actuators can also be pneumatic or hydraulic cylinders, vibrators.

[0074] The motion feedback system 10 is controlled by a device 50. The device 50 sends a signal to the actuators 10A-10N to cause the actuators 10A-10N to generate a vibration power output, vibration and / or any other motion output. The vibration power output / motion output may not be a vibration obtained from a sound channel, but may be, for example, an encoded tactile signal (motion code) that can give texture to the movement of the platform supported by the actuators 10A-10N. The concurrent effects of the actuators 10A-10N result in a vibration power movement and / or vibration of the seat S. As described above, other types of viewer effect signals may be generated, which may require the presence of other types of viewer effect generating devices, such as light tinting type lamps, hair blowing equipment, closed captioning devices, etc.

[0075] In order to synchronize the vibration dynamic movement of the seat S with the AV output (e.g., audio track, image) of the screen TV and / or the speaker LS, a microphone 12 or similar sound capture device is connected to the device 50. The microphone 12 can be any suitable type of sound capture device that captures ambient sound, including the audio output from the speaker LS and / or the screen TV. In some embodiments, the microphone 12 can be integrated with the viewer seat S, such as Figure 1B Additionally or alternatively, the microphone 12 may be part of a separate device 50, such as Figure 1CAs shown. The stand-alone device 50 can be located next to the viewer seat or anywhere within the audio-visual device environment. In these embodiments, the microphone 12 can have a microphone opening 12' facing the audio-visual device. In some embodiments, for example, when the stand-alone device 50 is not facing the audio-visual device, the audio portion captured by the microphone may or may not be ideal. Thus, for example, the captured audio portion can be processed based on the positioning of the viewer seat S and / or the stand-alone device 50 in its corresponding surrounding environment. As an alternative to the microphone, the device 50 can receive an audio track as a line-in signal. As another embodiment, the device 50 has an optical capture device that captures images and can generate a signature based on the color content of the image displayed on the screen TV.

[0076] refer to Figure 2 , a method for causing a viewer to experience an effect signal (e.g., a haptic signal corresponding to media content, such as from a feature-length film or an amusement park ride) and Figure 1A A method 20 for synchronizing media signals of media content displayed in a device shown. For simplicity, reference may be made herein to haptic signals, although this may include other types of viewer effect signals. Therefore, even though reference may be made herein to haptic signals, these concepts may also be applied to other types of viewer effect signals unless otherwise stated. The haptic signals are used to control the motion feedback system 10. The viewer effect signals may be continuous and may also be subtitles or other visual signals, additional audio tracks (e.g., audio tracks in alternative languages), audio tracks for the visually impaired (through which additional narration is added to provide context (e.g., "a car is about to arrive")), audio tracks for the deaf or hearing impaired, such as subtitles, sign language, or written narration describing sounds (e.g., "explosion"), using other devices, such as tint-type lights, steam generators, to simulate elements such as lightning, fog, or an audio track with audio commentary including actor and / or director comments. According to one embodiment, the media signal is an audio signal and / or a video signal from a speaker LS. For simplicity, the method described herein will refer to haptic signals. However, all methods are also applicable to other types of viewer effect signals, such as those listed above, and not just haptic signals.

[0077] As a starting point, for each media content for which a viewer effect or motion track or signal has been created, a signal track, also called a metadata track, has been pre-created for the duration of the audio track of the media content. The metadata track provides a quantitative indication of the audio power or any other characteristic for each time interval to assess the intensity that should be received if the audio is at its optimal volume for subsequent timely synchronization. Other types of information may be used to allow the expected intensity values ​​to be estimated at sufficient time intervals to allow timely synchronization. The metadata track may also include other types of signals or information. For example, the metadata track may describe the signal as having a pattern that repeats locally or at a specific time position in the movie, making synchronization more difficult based on the time position. The viewer effects track / motion track, metadata track for the media content is associated and stored together with a fingerprint representing the audio track and / or visual track of the media content.

[0078] In step 22, the audio portion of the media signal is captured from the surrounding environment. For example, in addition to the ambient sound, the media signal is received from the microphone 12. Therefore, depending on the device, the captured audio may include multiple different audio portions. Each audio portion can be captured using a corresponding one of the multiple microphones distributed in the surrounding environment of the audio-visual device, although the use of a single microphone 12 is considered, regardless of the ambient sound produced by multiple devices in the multi-channel device. In some embodiments, each captured audio portion can correspond to a combination of audio channels heard by any corresponding microphone. In other words, the microphone can capture audio from different physical locations in the room, thereby allowing subsequent separation of the audio channels of the media content. For example, eight microphones can be used to capture audio portions from six different speakers, and six channels can be parsed based on the eight captured audio portions. The configuration of the room where the microphone is placed can affect the audio portion captured by the microphone. For example, in one embodiment, the room can have multiple speakers and one or more microphones. In this embodiment, the audio portion including direct sound and reflected sound captured by the microphone can be mathematically processed to jointly retrieve the audio channels incoming from any given one speaker. In this way, the sound signature can be associated with different locations in the room, so in some embodiments, this can help noise removal.

[0079] In step 24, a fingerprint (also called a signature) corresponding to the received audio portion of the media signal is obtained. According to one embodiment, the step of obtaining a fingerprint corresponding to the received portion of the media signal (step 24) comprises computing a fingerprint of the received portion of the media signal in order to extract a sequence of features characterizing the audio signal over time.

[0080] In one embodiment, to address issues related to ambient sound, the audio gain from captured sound or from hardware including a microphone is adjusted to optimize the input sound level.

[0081] In step 24, in the case where the haptic signal is not yet synchronized with the media content, if the captured signal is saturated, the audio gain from the captured audio is systematically reduced. Still in the absence of synchronization, if the audio level is insufficient, the audio gain is gradually increased until a threshold value that is considered sufficient is reached. The reduction and subsequent gradual increase of the audio gain can be referred to as automatic gain control (AGC). The gain control module can perform AGC by applying the audio gain to the captured audio portion before any further steps of method 20. In some embodiments, the gain control module can reduce the audio gain when the peak amplitude is higher than the saturation threshold. The gain control module can increase the audio gain when the peak amplitude is lower than the detectable sound threshold. Otherwise, for example, when the peak amplitude of the captured audio portion is higher than the detectable sound threshold and lower than the saturation threshold, the gain control module can maintain a given audio gain. In some embodiments, a given audio gain is intended to be applied to all microphones at the same time. In some other embodiments, the audio gain module can apply independent audio gains to different microphones, thereby compensating for microphones that may need more or less audio gain, depending on their corresponding positions in the room.

[0082] In the step of calculating the fingerprint, the fingerprint of the media signal portion can be generated as a code that uniquely corresponds to the waveform of the media signal. When such a unique code (i.e., an encoded identifier) ​​is generated, as described herein, the corresponding waveform of the media signal from which the fingerprint was generated can thereafter be identified by reference to its fingerprint.

[0083] Thus, the fingerprint obtained in step 24 may be in the form of a sequence of features, for example, a sampled time domain audio source representation or a subset thereof, a frequency domain representation or a subset thereof, any transformation of the aforementioned signals, or any combination of these signals, their subsets, or their transformed subsets. For example, the envelope of the root mean square (RMS) value of a time domain signal representing the instantaneous power of the audio may be used to form a fingerprint.

[0084] In step 26, the temporal position of the media content corresponding to the obtained fingerprint is identified based on a reference fingerprint associated with a reference temporal position of at least one reference media content accessible in a database or a cloud-based database.

[0085] More specifically, in one embodiment of step 26, a database (also referred to as a storage medium) of encoded identifiers of fingerprints of the media signal at a plurality of playback time positions is queried. The fingerprints of the media signal portions are matched to reference fingerprints associated with reference time positions within the media content. According to one embodiment, the time positions of the media content are identified (step 26) on a remote network server (not shown).

[0086] According to one embodiment, method 20 may include a step of receiving an identity of at least one of the reference media contents to which the media signal corresponds before identifying the temporal position of the media content (step 26). Figure 1A In this case, the step 26 of identifying the temporal position of the media content is performed using a reference fingerprint associated with the reference temporal position of the identified reference media content. In some embodiments, the identity of the reference media content may not exist. In these embodiments, the captured fingerprint may be compared with reference fingerprints belonging to multiple different reference media content identities.

[0087] The method 20 may further include receiving at least one reference fingerprint associated with the output tactile signal, and stopping the output of the tactile signal when the obtained fingerprint does not match any of the at least one reference fingerprint. The method may also include improving and updating metadata.

[0088] In another embodiment of step 24, obtaining the fingerprint comprises retrieving an encoded identifier of the fingerprint of the media signal portion identifying the temporal location directly from the haptic signal.

[0089] In step 28, a viewer effect signal, such as a haptic signal, associated with the identified temporal position of the media content is obtained. According to one embodiment, the haptic signal associated with the identified temporal position of the media content is obtained from a remote network server (not shown).

[0090] According to an embodiment of step 28, identifying the current playback time position includes continuously performing the steps of obtaining a fingerprint, identifying a time position, and obtaining a tactile signal to keep outputting a tactile signal synchronized with the media signal, thereby ensuring continuous synchronization of the tactile signal.

[0091] According to another embodiment of step 28, identifying the current playback time position includes performing the steps of obtaining a fingerprint, identifying the time position, and obtaining a tactile signal at a given frequency to resynchronize the output tactile signal with the media signal. In the case where more than one microphone is used to capture the audio portion of the media content, these steps may also be performed for each captured audio portion.

[0092] The identified temporal position refers to the position of a media signal within at least one media content that is identified as corresponding to the media content currently being played on any one of a speaker (for audio), a screen or display (for video), and a motion feedback system (for motion) (corresponding to the obtained fingerprint).

[0093] Step 30 involves using the identified temporal position of the media content to output a haptic signal that is synchronized with the media signal. The haptic signal is ultimately sent to the motion feedback system 10 ( Figure 4 ), the motion feedback system has actuators 10A-10N for moving the seat S according to the tactile signals. The viewer effect signal is sent to an appropriate device for playback. This can include a portable device (e.g., a smart phone, a tablet, a dedicated audio device, a seat monitor) that displays additional written data (e.g., for the hearing impaired) or outputs additional audio data (e.g., for the visually impaired). This applies to any viewer effect signal described herein.

[0094] According to one embodiment, method 20 further comprises receiving a haptic signal corresponding to the media content prior to identifying the temporal position of the media content (step 26 ), wherein the reference fingerprint is retrieved directly from the received haptic signal.

[0095] According to one embodiment, the synchronized haptic signal is stored in a storage device, for example, in a buffered manner, to be eventually sent to the motion feedback system 10 .

[0096] In one example, the method 20 is performed in real time while the media signal is played, and the audio is captured using, for example, the microphone 12 .

[0097] Further, according to another embodiment, method 20 may further include receiving at least one reference fingerprint associated with the output tactile signal, and when the obtained fingerprint does not match any of the at least one reference fingerprint, stopping the output of the tactile signal despite the processing of step 24 .

[0098] Figure 3 An embodiment of a method 100 for determining whether a haptic signal corresponding to media content is synchronized with a media signal of the media content is shown. The method 100 may be performed using Figure 2 The initial synchronization of method 20 occurs after that.

[0099] In step 102, an audio portion of a media signal is captured, for example using microphone 12. In step 104, a fingerprint of the media signal at a current playback time position (ie, the obtained fingerprint) is obtained based on the captured audio portion of the media signal.

[0100] The fingerprint of the media signal may include additional information, such as a metadata track. The metadata track may be used to further process the audio capture for subsequent fingerprint generation.

[0101] A feature sequence of the audio capture and a corresponding feature sequence of a reference audio track can be compared over time to produce a distance estimate indicating how much the features differ over time. This comparison of information about feature distances over time can be used to modify the captured audio in a way that reduces the feature distances.

[0102] In one embodiment, the modification of the captured audio may be an adjustable scalar gain. In this case, an automatic gain control module may be implemented, for example, by simultaneously taking into account knowledge of the expected volume of the audio capture inferred from the volume of the reference audio track. As another form, the modification of the captured audio may include implementing a transfer function that cancels the modification applied to the original audio signal by the microphone 12 or the playback device in response to the capture. As an example, the adaptive filter may simulate the inverse acoustic response of the acoustic response of the room in which the microphone 12 captures the audio. In some embodiments, the acoustic response of the room may be determined by playing a reference audio track in the room and capturing the resulting audio signal using the microphone 12. By comparing the captured audio signal and the reference audio track to each other, the acoustic response of the room may be determined. Once determined, the acoustic response of the room may be used to modify any subsequently captured audio signal accordingly, which may simplify the identification of the corresponding fingerprint. Otherwise, in some other embodiments, the reference fingerprint may also be modified according to the acoustic response of the room. For example, based on knowledge of the playing audio track and knowledge of the audio response of the room, noise of the captured audio portion can be identified, and by comparing the captured audio portion with an expected version of the captured audio portion, the noise may be subsequently subtracted. Note that the acoustic response of the room may depend on the type of sound presented by the audiovisual device. For example, the acoustic response of the room may take into account whether the sound is produced according to stereo or surround sound. In an embodiment where the center channel, left channel, and right channel of the audiovisual device use different audio tracks, the microphone can be positioned to advantageously optimize the amount of captured sound incoming from the center channel. In some other embodiments, the microphone can be positioned to optimize the amount of captured sound incoming from the left channel and the right channel, thereby minimizing the impact of language-related fingerprints (usually carried by the center channel) that may be difficult to synchronize, especially when more than one language track is available.

[0103] As another possibility, audio processing may be implemented before the feature extraction function in the reference audio track processing path. This module will implement an acoustic model of the room, so feature extraction of both sides (reference audio and captured audio) will produce feature distances that are relatively unaffected by the room acoustics. In some embodiments, the acoustic model of the room may take into account constrained locations, such as a room containing multiple physical elements that cause sound wave reflections, such as walls, furniture, or people. In some other embodiments, the acoustic model may represent an open area.

[0104] The processing applied to the audio capture channel may also provide an estimate of the acoustic audio level relative to the expected audio level. More precisely, information derived from the feature distance estimate allows an assessment of whether the hardware audio gain (e.g., microphone 12) in the captured audio is optimal. For example, if the current hardware gain is such that the available dynamic range of the hardware chain is underutilized, knowledge of the current audio level relative to the optimal level allows the hardware gain to be adjusted so the audio will use the optimal portion of the dynamic range of the audio hardware.

[0105] In the particular case of capturing audio with microphone 12, audio processing with a priori knowledge of the information contained in the audio signal can at least partially separate the portion of the audio capture that is from the media playback from the portion that is not from the media playback. The latter portion may come from other noise sources present in the room, and most commonly may come from different locations in the microphone capture range.

[0106] In the case of one or more microphones 12 , differentiation between the intended audio (media playback) and other sources may be made by different reflection patterns in the room, since the sound sources of the media playback are typically located at different locations than the acoustic noise sources within the capture range of the microphones 12 .

[0107] In the case where more than one microphone 12 captures audio, the audio processing will have the ability to tune the microphone input gain and phase to increase sensitivity in the direction of the media audio source and reduce sensitivity in the direction of the noise source, in effect implementing beamforming. For example, in some embodiments, the microphones can be located at different locations within the room in order to find the best position for the microphones. When positioned at the best position, the microphones can pick up a smaller amount of noise, which can reduce the amount of processing that must be performed in real time to perform method 20, such as audio gain modification.

[0108] In step 106, a reference fingerprint of a media signal portion at a temporal position of the media content is retrieved based on the current position in the viewer effect signal.

[0109] In step 108, the reference fingerprint of step 106 is compared to the fingerprint obtained in step 104. In step 110, based on the comparison, if the two fingerprints are similar to each other, it is determined that the haptic signal is synchronized with the media signal. Otherwise, it is determined that the two fingerprints are different from each other and the haptic signal and the media signal are not synchronized. This may occur, for example, if the position within the haptic signal changes suddenly, such as during a fast forward or chapter skip operation. As a result, the system may automatically select to apply method 20 to restore synchronization.

[0110] In another embodiment of step 104, obtaining the fingerprint includes retrieving an encoded identifier of the fingerprint of the media signal portion identifying the temporal location directly from the haptic signal.

[0111] In step 112, based on the result of determining whether the haptic signal is synchronized with the media signal, a haptic signal synchronized with the media signal using the identified time position of the media content is output for controlling the motion feedback system (step 110).

[0112] Now, refer to Figure 4 , shows an embodiment of an apparatus 50 for synchronizing haptic signals and media signals for a feature-length film. The apparatus 50 is configured to implement at least some steps of methods 20 and 100. The apparatus 50 has an input unit 52; a processing unit 54; a memory 56; one or more databases (here, databases 58 and 60); and an output unit 62 that communicates with a motion feedback system 10 having actuators 1 to N, 10A to 10N.

[0113] The input unit 52 is used to receive a portion of the media signal. The input unit 52 may therefore receive captured audio from the microphone 12 or other capture means used (including line input, optical means, inertial sensors, etc.).

[0114] Databases 58 and 60 are used to store haptic signals, reference media signal fingerprints, and / or metadata tracks, respectively, associated with specific known media content (e.g., a known feature-length movie). Databases 58 and 60 may be within device 50, cloud-based, on a dedicated server, etc., and may be accessed in any suitable manner. Such reference fingerprints are identifiers that have been derived for known media content and are known to be uniquely associated with the known media content. Additional information may be associated with the reference fingerprint, including the metadata track of the media content.

[0115] The processing unit 54 communicates with the input unit 52, the databases 58 and 60, and the memory 56. Thus, the memory 56 is accessible by the processing unit 54 and may be a non-transitory computer-readable memory communicatively coupled to the processing unit 54 and includes computer-readable program instructions executable by the processing unit 54 to perform some steps of the methods 20 and 100. In one embodiment, the memory 56 is used to store instructions for implementing the processing unit 54 to perform a series of steps on the media signal, including accessing and retrieving data from the databases 58 and 60, and performing processing steps that allow the haptic signal to be synchronized with the media signal, as in the methods 20 and 100, including audio processing. The processing unit 54 also communicates with the output unit 62 so that the haptic signal, once retrieved and synchronized, is sent out to be stored on a database (not shown) or directly sent to the motion feedback system 10 during the playback of the media signal. Although the processing unit 54 is shown as being local in the illustrated embodiment, the processing unit may be remote in some other embodiments. For example, the processing unit may be provided in the form of one or more virtual machines accessible via one or more networks (e.g., the Internet). Note that such remote processing may advantageously reduce computing power requirements on any local processing unit, and / or also facilitate software, firmware, and / or hardware updates that may be required over time.

[0116] Upon receiving a media signal portion at the processing unit 54, via which it was transmitted from the input unit 52, the processing unit 54 obtains a fingerprint corresponding to the received portion of the media signal.

[0117] The processing unit 54 then proceeds to identify the temporal position of the media content corresponding to the obtained fingerprint from the reference fingerprints associated with the temporal position of at least one reference media content. For example, to perform such a comparison, the processing unit 54 accesses a database 60 of reference fingerprints; parsing via optionally retrieving the data to an optional internal memory (not shown) until a match is detected between the reference media signal fingerprint and the fingerprint of the media signal. When no match is detected, an error signal is sent to a user interface (not shown).

[0118] As described above in steps 24 and 104, the processing device 54 may also process the captured audio for calibration purposes. The processing device 54 continuously obtains haptic signals associated with the identified time positions of the media content. When synchronization is achieved, the processing device 54 knows the identity of the media content and the timestamp corresponding to the instant playback. When synchronization is maintained, the processing device 54 compares the received sound level with the sound level that should be achieved according to the metadata track to obtain the optimal time by looking at the power track at the timestamp. If it is determined that the received sound level exceeds a given threshold of the optimal predetermined sound level, the audio gain of the processing device 54 is modified to be close to the optimal value.

[0119] This method of optimizing sound levels helps to predict silence in the audio of the media content to improve robustness and detect the loss of an audio track that is synchronized with the motion trajectory more quickly. More specifically, synchronization must tolerate periods in which audio exists but synchronization cannot be confirmed. This is the case of silence, where the processing unit 54 may not have sufficient audio resolution to confirm synchronization. If the metadata track notifies the processing unit 54 of a waiting period during silence, the processing unit 54 can quickly detect a mismatch between the media content (if loud audio is expected) and therefore detect a loss of synchronization. When synchronization is lost, or at the end of the media content, the above-mentioned AGC algorithm is restored. In addition, in some embodiments, for example, when synchronization is temporarily lost due to a period of silence, no attempt may be made to optimize the audio gain.

[0120] Based on the identified time position of the media content, the processing unit 54 can synchronize and output the haptic signal with the media signal. For example, to do this, the processing unit 54 retrieves and sends a haptic signal from the database 58, the haptic signal starting at the identified time position corresponding to the reference time of the media signal received at the input unit 52. Thus, the haptic signal sent at the output unit 62 is synchronized with the playback time position of the media signal.

[0121] According to another embodiment, a non-transitory computer-readable memory (not shown) is provided on which statements and instructions executed by a computer are recorded to execute a method for synchronizing a haptic signal corresponding to media content with a media signal of the media content. The haptic signal is used to control a motion feedback system. The method includes Figure 2 Method 20 shown.

[0122] Finally, according to another embodiment, a non-transitory computer-readable memory (not shown) is provided, on which statements and instructions executed by a computer are recorded to execute a method for determining whether a tactile signal corresponding to media content is synchronized with a media signal of the media content. The tactile signal is used to control a motion feedback system. The method includes Figure 3 Method 100 is shown.

[0123] The audio processing and calibration described in method 100 and implemented by device 50 can use the audio track knowledge obtained by identifying the media content to perform more robust fingerprint generation to achieve more effective synchronization. Therefore, this can lead to faster detection of loss of synchronization when the audio stops or the media content changes.

[0124] Different methods can be implemented to achieve satisfactory detection of loss of synchronization. For example, Figure 5AAn example of an audio portion 500 captured using a microphone is shown. In this particular embodiment, the captured audio portion 500 is modified with a gain control module on the fly. As shown, when it is determined that the peak amplitude of the captured audio portion 500 is not satisfactory, the gain control module can apply an audio gain to the captured audio portion 500. For example, in region 502, it is determined that the peak amplitude of the captured audio portion 500 is higher than the saturation threshold Sth, which can indicate that the microphone is saturated. Therefore, the gain control module gradually reduces the audio gain of the entire region 502. In this embodiment, the audio gain is reduced a little too much because the peak amplitude of the captured audio portion 500 is lower than the detectable sound threshold Smin. In response, the gain control module gradually increases the audio gain of the entire region 504 until the peak amplitude of the captured audio portion 500 reaches a satisfactory level. For the remainder of the captured audio portion 500, when the detected peak amplitude oscillates between the detectable sound threshold Smin and the saturation threshold Sth, the audio gain applied by the gain control module remains constant. In this embodiment, the gain control module can apply an audio gain to the captured audio portion based on the detected instantaneous peak amplitude. However, in some other embodiments, the gain control module may apply an audio gain to the captured audio portion based on a peak amplitude of an envelope of the captured audio portion 500 . Figure 5B A gain evolution curve 506 corresponding to the evolution of the gain applied by the gain control module to the captured audio portion 500 is shown. In some embodiments, for example, the gain evolution curve 506 can be associated with a given media content to be saved as part of a metadata track for later use. Note that although in this embodiment the gain control module considers the peak amplitude of the captured audio signal, in some other embodiments, the gain control module can also consider other types of amplitudes, such as average amplitude, RMS amplitude, etc.

[0125] Return to reference Figure 5A Once the peak amplitude of the captured audio portion 500 is determined to be satisfactory, a fingerprint 508 of the captured audio portion 500 may be obtained. The obtained fingerprint 508 may vary from embodiment to embodiment. As described above with reference to method 20, the obtained fingerprint 508 is compared with a plurality of reference fingerprints to find a matching reference fingerprint. The reference fingerprints may be stored on a remote storage accessible via a network (e.g., the Internet). In this particular embodiment, the matching reference fingerprint 510 is stored in a remote storage. Figure 5CAs shown, the matching reference fingerprint 510 is part of an audio track 512 of the corresponding media content. As shown, a reference time position tr of the reference fingerprint 510 is identified. The reference time position may correspond to a start time position, an intermediate time position, and / or an end time position of the corresponding fingerprint. This matching allows the extraction of a viewer effect signal 514, which is associated with the corresponding media content, such as Figure 5D As shown, the media content can be used to generate a viewer effect, such as a motion track, to the viewer. More specifically, a viewer effect signal 514 associated with the identified time position tr is output to a viewer effect generating device, which generates an effect to the viewer based on the output viewer effect signal, thereby outputting a viewer effect signal synchronized with the captured audio portion 500.

[0126] Once synchronization is achieved, method 100 may be repeatedly performed to ensure that the viewer effects signal remains synchronized with the captured audio portion over time. If method 100 confirms synchronization, the viewer effects signal synchronized with the captured audio portion 500 continues to be output to the viewer effects generating device. If method 100 finds that synchronization is lost, the viewer effects signal may be stopped, thereby stopping any effects generated to the viewer via the viewer effects generating device. In this case, method 20 begins immediately with the goal of quickly returning to a synchronized state. In some embodiments, the fingerprint obtained from the captured audio portion may be matched to a reference fingerprint belonging to the same media content but in a previous or subsequent scene. In some other embodiments, the fingerprint obtained may be matched to different media content.

[0127] Note that method 100 may include different sets of steps to ensure that loss of synchronization is independent of the quality of the captured audio portion. Indeed, in some embodiments, the fingerprint of the most recently captured audio portion may not match the reference fingerprint of the corresponding media content if, for example, i) the microphone has picked up a lot of ambient sound, ii) the volume of the audiovisual device has changed, thereby creating a lag in the adjustment of the gain control module, iii) the obtained fingerprint belongs to a featureless portion of the audio track, and iv) any other suboptimal situation. In these embodiments, it is preferred to delay the switch from method 100 to method 20 and instead continue to output the viewer effects signal synchronized with the audio track for at least a given amount of time to ensure a certain continuity of the effect produced to the viewer.

[0128] Thus, in some embodiments, when it is determined that the viewer effect is no longer synchronized with the media signal, the viewer effect signal may be stopped from being output. In some embodiments, the viewer effect signal may be stopped immediately from being output. However, in some other embodiments, the viewer effect signal output may be stopped only after a given asynchronous delay. Therefore, even if it is not synchronized, the output of the viewer effect signal may be kept at least for a given duration. In some embodiments, the asynchronous delay is a predetermined value. For example, it may be zero or in a range between about 1 second and about 1 minute. As further described below, in some other embodiments, the asynchronous delay may vary with the audio track of the media content. In some embodiments, acceptable asynchronous delay data associated with the media content may be received. Acceptable asynchronous delay data may include acceptable asynchronous delay values ​​associated with the corresponding reference time positions of the media content. Therefore, in these embodiments, a given asynchronous delay may be determined based on receiving acceptable asynchronous delay data and the current reference time position of the media content. Alternatively or additionally, synchronization confidence data associated with the media content may be received. The synchronization confidence data may include synchronization confidence values ​​associated with the corresponding reference time positions of the media content. In these embodiments, a given asynchrony delay may be determined based on the synchronization confidence data and the current reference time position of the media content.The acceptable asynchrony delay data and the synchronization confidence data may be part of a metadata track associated with the media content.

[0129] In some embodiments, loss of synchronization may be avoided by anticipating silence, e.g., as shown in regions 516 and 518 of audio track 512. In practice, if the audio track 512 of the media content is known, any silence or audio level drops may be taken into account. For example, these silences or audio level drops may be taken into account in audio gain determinations, in which case the audio gain may be maintained throughout regions 516 and 518 to avoid unnecessary audio gain increases. More specifically, during these silences, the audio gain may not be increased even if the detected peak amplitude is below the detectable sound threshold Sth. Thus, it may be assumed that the synchronization between the viewer effects signal 514 and the audio track 512 is satisfactory until any expected silence ends.

[0130] In some embodiments, loss of synchronization may be avoided by using metadata tracks associated with given media content. Figure 5E shows exemplary contents of a metadata track, with as an example Figure 5CA synchronization confidence curve or track 520 associated with the media content of the metadata track is provided. In this embodiment, the synchronization confidence curve 520 distinguishes the portion of the audio track 512 that is expected to be more easily identified (i.e., the characteristic portion of the audio track 512) from the portion of the audio track 512 that is expected to be less easily identified or not characteristic. For example, the characteristic portion of the audio track 512 (which may include but is not limited to the car chase 522, the music 524, etc.) may be more easily matched with the corresponding reference fingerprint. In contrast, the non-characteristic portion of the audio track 512 may be less easy to match with the corresponding reference fingerprint, and the non-characteristic portion may include silence 516 and 518, dialogue 526 unrelated to its language track, rain 528, and other types of white noise. For example, the synchronization confidence curve 520 of the metadata track may tend to be consistent when the corresponding audio track is expected to be associated with the characteristic portion, while the synchronization confidence curve 520 may tend to zero when the corresponding audio track is expected to be associated with the non-characteristic portion, and vice versa. In some embodiments, the synchronization confidence curve 520 is useful because a loss of synchronization detected based on a fingerprint obtained within a characteristic portion of an audio track may result in a momentary cessation of the viewer effect signal. In fact, if a given fingerprint matches a corresponding reference fingerprint within a considerable confidence level according to the synchronization confidence curve 520, and the method 100 fails to do so, it can be indicated that the captured audio portion no longer corresponds to the expected media content, in which case the generation of the viewer effect should be stopped. However, if a lack of synchronization is detected with the method 100 when the fingerprint should display dialogue at the identified time position regardless of its language track, the viewer effect signal can be maintained for a given period of time until, for example, an actual loss of synchronization is detected during a more characteristic portion of the audio track. In some embodiments, the synchronization confidence curve 520 can be modified when it is determined that the captured audio portion is of poor quality. For example, if the gain control module is modifying the audio gain to obtain a satisfactory sound level, any loss of synchronization detected during this period may have a lower confidence level.

[0131] In some embodiments, the metadata track may also include an asynchronous tolerance curve or track 530. For example, if a loss of synchronization is detected within a characteristic portion of the audio track, the asynchronous tolerance delay tolerated by method 100 may be shorter than when a loss of synchronization is detected during a non-characteristic portion of the audio track. In this way, the asynchronous tolerance delay may be particularly useful in situations where the captured audio signal is poor or is during long periods of silence. Note that in this embodiment, the synchronization confidence curve 520 and the asynchronous tolerance curve 530 are shown in the form of corresponding curves. However, in some other embodiments, the information carried by these curves may be provided in any other suitable shape or form, for example, in the form of a table, etc. Such a metadata track may be useful because a smoother presentation of the viewer effects signal may be provided over time.

[0132] In some embodiments, the characteristic and non-characteristic portions of an audio track may be determined by a trained user by listening to the audio track and annotating each portion thereof in real time using appropriate software. However, in some other embodiments, the characteristic and non-characteristic portions of an audio track may be determined using a trained machine learning module that can classify each portion of the audio track. For example, such a machine learning module may be trained using previously annotated audio tracks. In some alternative embodiments, the probability of identifying a recognizable fingerprint is measured based on the signal-to-noise ratio in the audio track, which may be used as a basis for determining a corresponding synchronization confidence track.

[0133] In the above example, because the language of the audio track may be different from the default language, the conversation including speech (e.g., conversation 526) may be considered as a less identifiable part of the audio track. Fig. 6A and Figure 6C shows the instantaneous peak amplitude change and envelope change of the English audio track, while Figure 6B and Fig.6D shows the instantaneous peak amplitude change and envelope change of the same French audio track. Fig. 6E As best shown in , which shows the ratio of envelope changes for the English and French audio tracks, it can be noted that at corresponding time locations, the fingerprint or characteristic differences between the English and French audio tracks are relatively insignificant. Therefore, the envelope changes of the corresponding audio tracks can be preferably used for audio gain control purposes. In some embodiments, the instantaneous peak amplitude changes of the audio tracks are the preferred tracks for synchronization determination. For example, if method 100 is performed during a conversation, or in the event that differences are observed between the English and French audio tracks, the confidence that synchronization loss is obtained may be low, which may result in outputting a viewer effects signal that is synchronized with the audio tracks to maintain a given amount of time. For example, output can be maintained until synchronization loss is obtained during a portion of the audio track where a significant change between languages ​​is not expected.

[0134] In some embodiments, method 100 is performed during sessions, but any synchronization loss obtained during these sessions can be ignored, thereby continuing to output a viewer effect signal synchronized with the audio track. In some embodiments, performing method 20 can identify a given media content and a corresponding audio track. Note that when performing method 100, more specifically, during the comparison step between the obtained fingerprint and the reference fingerprint of the corresponding audio track, the method can include a step of confirming that the captured audio portion belongs to a default language track. In some embodiments, some differences between the obtained fingerprint and the reference fingerprint can indicate that the audio track is not the default language. In such embodiments, if these differences can be minimized or if a match can be found with a reference fingerprint belonging to a different language track, the method can include the step of finding the correct language track associated with the audio track and replacing the correct language track with the default language track. For example, in some embodiments, an association can be performed between the obtained fingerprint and the reference fingerprint of the different language tracks associated with the same media content. In these embodiments, the correct language track can be identified based on the correlation results and / or based on the synchronization confidence data.

[0135] Although embodiments have been described above and illustrated in the accompanying drawings, it will be apparent to those skilled in the art that modifications may be made thereto without departing from the scope of the present disclosure. For example, although a microphone is used to capture the audio portion of the media content, the intent is that the microphone may be removed and replaced by a camera that captures the image portion of the media content. In these embodiments, the visual fingerprint may be matched to a reference visual fingerprint including, but not limited to, watermarks, timestamps, or movie IDS in streaming media or any other type of media. Such modifications are considered possible variations that are within the scope of the present disclosure.

Claims

1. A system for synchronizing a viewer effect signal of media content with a media signal of the media content, the system comprising: at least one microphone to capture an audio portion of the media signal from an ambient environment; a controller communicatively coupled to the at least one microphone, the controller having a processor and a non-transitory memory storing instructions that, when executed by the processor, perform the following steps: applying a given gain to the captured audio portion so that the captured audio portion is within a threshold associated with the at least one microphone; obtaining a fingerprint from an audio portion of the captured media signal after applying a given gain; identifying, based on a reference fingerprint associated with a reference temporal location of reference media content, a temporal location of the media content corresponding to the obtained fingerprint; obtaining the viewer effect signal associated with the identified temporal position of the media content; as well as outputting the viewer effect signal synchronized with the media signal using the identified temporal position of the media content, capable of producing an effect synchronized with the media signal to a viewer, The controller has a gain control module, which applies a given gain to the captured audio portion before identification, wherein a metadata track is pre-created within the duration of the audio track of the media content, and the metadata track provides audio power for each time interval to evaluate the intensity, if the captured signal is saturated, the audio gain from the captured audio is reduced; if the audio level is insufficient, the audio gain is gradually increased.

2. The system according to claim 1, wherein: The at least one microphone is integrated with the viewer seating.

3. The system according to claim 1 or 2, wherein: The at least one microphone includes a plurality of microphones that capture different audio portions of the media signal.

4. The system according to claim 3, wherein: The plurality of microphones are distributed within the surrounding environment.

5. The system according to claim 3, wherein: The obtaining, identifying and acquiring steps are performed for each of the audio portions of the captured media signal.

6. The system according to claim 1, wherein: The reference fingerprint is stored on a remote database and is accessible by the controller during identification.

7. The system according to claim 1, wherein: Obtaining the viewer effects signal comprises retrieving the viewer effects signal from a remote database.

8. The system according to claim 1, wherein: The gain control module reduces the given gain when the peak amplitude is above a saturation threshold, increases the given gain when the peak amplitude is below a detectable sound threshold, and maintains the given gain when the peak amplitude is above the detectable sound threshold and below the saturation threshold.

9. The system of claim 1, further comprising a viewer effect generating device communicatively coupled to the controller, the viewer effect generating device generating an effect to a viewer based on the output viewer effect signal synchronized with the media signal.

10. The system according to claim 9, wherein: The viewer effect generating device is a motion feedback device, and the viewer effect signal corresponds to a tactile signal.

11. A method for synchronizing a viewer effect signal of media content with a media signal of the media content, the viewer effect signal being used to produce an effect to a viewer, the method comprising: capturing an audio portion of the media signal from an ambient environment using at least one microphone; Applying a given gain to the captured audio portion so that the captured audio portion is within a threshold associated with the at least one microphone; wherein a metadata track is pre-created for the duration of the audio track of the media content, the metadata track provides audio power for each time interval to evaluate the intensity, if the captured signal is saturated, the audio gain from the captured audio is reduced; if the audio level is insufficient, the audio gain is gradually increased; After applying the given gain, obtaining a fingerprint from the audio portion of the captured media signal to which the given gain is applied; identifying, based on a reference fingerprint associated with at least a reference temporal location of the reference media content, a temporal location of the media content corresponding to the obtained fingerprint; obtaining the viewer effect signal associated with the identified temporal position of the media content; and The viewer effect signal synchronized with the media signal using the identified time position of the media content is output for producing an effect synchronized with the media signal of the media content. 12 . The method of claim 11 , further comprising generating an effect synchronized with a media signal of the media content based on the output viewer effect signal synchronized with the media signal.

13. The method according to claim 12, wherein: The viewer effect signal is a haptic signal, and generating includes activating one or more actuators of a motion feedback device based on the haptic signal.

14. The method of any one of claims 11 to 13, further comprising receiving an identity of the media content, the reference fingerprint being associated with the identity of the media content.

15. The method according to claim 11, wherein: Capturing includes capturing a plurality of audio portions of the media signal, and the steps of obtaining, identifying, and obtaining are performed for each of the captured audio portions of the media signal.

16. The method according to claim 15, wherein: The reference fingerprint is stored on a remote database, and identifying includes accessing the reference fingerprint from the remote database.

17. The method according to claim 16, wherein: Obtaining the viewer effects signal comprises retrieving the viewer effects signal from a remote database.

18. The method according to claim 17, wherein: The given gain depends on the peak amplitude.

19. The method according to claim 18, wherein: When the peak amplitude is above the saturation threshold, the given gain is reduced, when the peak amplitude is below the detectable sound threshold, the given gain is increased, and when the peak amplitude is above the detectable sound threshold and below the saturation threshold, the given gain is maintained.

20. A system for determining whether a viewer effect signal of media content is synchronized with a media signal of the media content, the system comprising: a viewer effect generating device for generating an effect to a viewer based on the viewer effect signal; at least one microphone to capture an audio portion of the media signal from an ambient environment; as well as a controller communicatively coupled to the viewer effect generating device and the at least one microphone, the controller having a processor and a non-transitory memory storing instructions that, when executed by the processor, perform the following steps: Using a gain control module, applying a given gain to the captured audio portion so that the captured audio portion is within a threshold associated with the at least one microphone; obtaining a fingerprint of the audio portion of the captured media signal after applying the given gain; retrieving a reference fingerprint of the media signal at a reference time position of the media content based on a current time position of the viewer effect signal; comparing the obtained fingerprint with the reference fingerprint; based on the comparison, determining whether the viewer effects signal is synchronized with the media signal; as well as outputting the viewer effect signal synchronized with the media signal according to the determination, for generating an effect synchronized with the media content using the viewer effect generating device, The controller has a gain control module, which applies a given gain to the captured audio portion before retrieval, wherein a metadata track is pre-created within the duration of the audio track of the media content, and the metadata track provides audio power for each time interval to evaluate the intensity, if the captured signal is saturated, the audio gain from the captured audio is reduced; if the audio level is insufficient, the audio gain is gradually increased.

21. The system of claim 20, further comprising ceasing to output the viewer effects signal upon determining that the viewer effects signal is not synchronized with the media signal.

22. The system of claim 21, further comprising maintaining the output for a given asynchronous delay before stopping.

23. The system of claim 22, wherein: The given asynchronous delay is at least one of zero and a range between 1 second and 1 minute.

24. The system according to claim 22 further includes receiving acceptable asynchronous delay data associated with the media content, the acceptable asynchronous delay data including multiple acceptable asynchronous delay values ​​associated with corresponding reference time positions in the reference time positions of the media content, and determining the given asynchronous delay based on receiving the acceptable asynchronous delay data and the reference time position of the media content.

25. The system according to claim 22 further includes receiving synchronization confidence data associated with the media content, the synchronization confidence data including multiple synchronization confidence values ​​associated with corresponding reference time positions in the reference time positions of the media content, and determining the given asynchronous delay based on the synchronization confidence data and the reference time position of the media content.

26. A system according to any one of claims 20 to 25, wherein: The at least one microphone is integrated with the viewer seating.

27. The system of claim 20, further comprising a viewer effect generating device communicatively coupled to the controller, the viewer effect generating device generating an effect to a viewer based on the outputted viewer effect signal synchronized with the media signal.

28. The system of claim 27, wherein: The viewer effect generating device is a motion feedback device, and the viewer effect signal corresponds to a haptic signal.

29. A method for determining whether a viewer effect signal of media content is synchronized with a media signal of the media content, the method comprising: generating an effect to a viewer based on the viewer effect signal; capturing an audio portion of the media signal from an ambient environment; applying a given gain to the captured audio portion so that the captured audio portion is within a threshold associated with at least one microphone; obtaining a fingerprint of the audio portion of the captured media signal after applying the given gain; retrieving a reference fingerprint of the media signal at a reference time position of the media content based on a current time position of the viewer effect signal; comparing the obtained fingerprint with the reference fingerprint; based on the comparison, determining whether the viewer effects signal is synchronized with the media signal; as well as outputting the viewer effect signal synchronized with the media signal according to the determination, for generating an effect synchronized with the media content using a viewer effect generating device, Wherein obtaining the fingerprint comprises applying a given gain to the captured audio portion prior to identification, wherein a metadata track is pre-created for the duration of the audio track of the media content, the metadata track providing the audio power for each time interval to assess the intensity, and if the captured signal is saturated, the audio gain from the captured audio is reduced; if the audio level is insufficient, the audio gain is gradually increased.

30. The method of claim 29, further comprising ceasing to output the viewer effects signal upon determining that the viewer effects signal is not synchronized with the media signal.

31. The method of claim 30, further comprising, before stopping, maintaining the output for a given asynchronous delay.

32. The method according to claim 31, wherein: The given asynchronous delay is at least one of zero and a range between 1 second and 1 minute.

33. The method according to claim 31 also includes receiving acceptable asynchronous delay data associated with the media content, the acceptable asynchronous delay data including multiple acceptable asynchronous delay values ​​associated with corresponding reference time positions in the reference time positions of the media content, and determining the given asynchronous delay based on receiving the acceptable asynchronous delay data and the reference time position of the media content.

34. The method according to claim 31 further includes receiving synchronization confidence data associated with the media content, the synchronization confidence data comprising multiple synchronization confidence values ​​associated with corresponding reference time positions in the reference time positions of the media content, and determining the given asynchronous delay based on the synchronization confidence data and the reference time position of the media content.

35. The method according to any one of claims 29 to 34, wherein: Capturing includes capturing a plurality of audio portions of the media signal, and the steps of obtaining, retrieving and comparing are performed for each of the captured audio portions of the media signal.

36. The method of claim 35, wherein: The given gain depends on the peak amplitude.

37. The method of claim 36, wherein: When the peak amplitude is above the saturation threshold, the given gain is reduced, when the peak amplitude is below the detectable sound threshold, the given gain is increased, and when the peak amplitude is above the detectable sound threshold and below the saturation threshold, the given gain is maintained.

38. The method of claim 29, wherein: The viewer effect signal is a haptic signal, and generating includes activating one or more actuators of a motion feedback device based on the haptic signal.

Images