Sound object detection

Abstract

An electronic device comprises circuitry, which is configured to obtain a detected sound object based on multimodal context analysis.

Description

[0001] SOUND OBJECT DETECTION

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to the technical field of audio processing, in particular to an electronic device, a method and a program for sound object detection.

[0004] TECHNICAL BACKGROUND

[0005] During the creation of sound for a movie production, or similar, a human sound editor reviews the movie script, video sequences and sound recorded during the production to propose candidate sound objects. In the first stages of sound creation, these sound objects are placeholders for sound that might be realized in later stages of the production process. Hence, the audio for the sound objects is typically not yet created when the sound objects are chosen by the sound editor.

[0006] Although the human sound editor may still add sound objects to videos based on reviewed footage and audio, it is generally desirable to develop techniques to improve the process of generating candidate sound objects and reduce time consuming manual effort.

[0007] SUMMARY

[0008] According to a first aspect the disclosure provides an electronic device comprising circuity configured to obtain a detected sound object based on multimodal context analysis.

[0009] According to a second aspect, the present disclosure provides a method comprising obtaining a detected sound object based on multimodal context analysis.

[0010] According to a third aspect, the present disclosure provides a computer program comprising instructions which, when executed by a processor, performs the method.

[0011] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0012] BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0014] Fig. la schematically shows an embodiment of a detected sound object in relation to a candidate sound object and a realized sound object; Fig. lb schematically shows an embodiment of a detected sound object with a descriptor and a label, a candidate sound object and a realized sound object;

[0015] Fig. 2a schematically shows an example of a detected sound object;

[0016] Fig. 2b schematically shows an example of a detected sound object;

[0017] Fig. 2c schematically shows an example of a detected sound object;

[0018] Fig. 3 schematically shows an embodiment of multimodal context analysis for obtaining a detected sound object;

[0019] Fig. 4 schematically illustrates an example of a post-production workflow comprising a process of obtaining detected sound objects as described in Fig. 3;

[0020] Fig. 5 schematically shows a flow diagram of a process of obtaining a detected sound object;

[0021] Fig. 6 schematically shows timelines of multiple detected sound objects overlapping in time;

[0022] Fig. 7 shows a block diagram depicting an embodiment of an electronic device that can implement multimodal context analysis for obtaining a detected sound object.

[0023] DETAILED DESCRIPTION OF EMBODIMENTS

[0024] Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

[0025] Some embodiments pertain to an electronic device comprising circuitry configured to obtain a detected sound object based on multimodal context analysis.

[0026] The electronic device may for example be a mobile device, such as a smartphone, tablet, laptop, or it may be a stationary device.

[0027] Circuitry may include a processor, a memory (RAM, ROM or the like), a storage, input means (mouse, keyboard, camera, etc.), output means (display (e.g., liquid crystal, (organic) light emitting diode, etc.), loudspeakers, etc.) and a (wireless) interface, etc., as it is generally known for electronic devices (computers, smartphones or the like). Moreover, it may include sensors for sensing still image or video image data (image sensor, camera sensor, video sensor, etc.), for sensing sound (microphones), etc.

[0028] The detected sound object may be obtained by an automated mechanism. The detected sound object may then be presented to and reviewed by a human sound editor to possibly create a candidate sound object, for example, by further editing a list of detected sound objects, on which basis a realized sound object including an actual audio signal may be created. The audio signal of the realized sound object may then be added to an audio track, for example, an audio track of a movie. Thus, the detected sound object may be the basis of a candidate sound object and later a realized sound object whose audio signal may be added to an already existing audio signal. In this way, the detected sound object may be a placeholder for a sound object realized as the realized sound object. Furthermore, the detected sound object may be a placeholder for silence. For example, in movies oftentimes explosions lack sound to portray an intense feeling of disorientation, or to emphasize the portrayal of a person’s panic attack a lack of sound in place of dialogue may be used. In this vein, the detected sound object may be the basis, of a candidate sound object and later a realized sound object without an audio signal or with a suppressed audio signal representing silence. Therefore, the detected sound object may be a placeholder for a sound object realized as silence. In other words, the detected sound object is an abstract entity.

[0029] The detected sound object may, for example, be a placeholder for a diegetic sound object, i.e., a process-based sound object or an object-based sound object, or it may be a placeholder for a non- diegetic sound object. The detected sound object may for example be a placeholder for sound effects or dialogue. For a process-based sound object, the sound is produced by movement or dynamic interaction of objects. For an object-based sound object, the sound comes from an object that appears to be static on the outside. Non-diegetic sound objects are sound objects not existing within the narrative, but may exist in the reality of the audience or narrator of a story, for example, they may be audio-like voice overs in a movie that do not exist for the characters within a story of the movie.

[0030] The detected sound object may be a placeholder for a future on-screen sound object, for example, an object that is created later in the production process, such as a visual effect or a 3D object, where the sound is caused seemingly by an image object that is visible within one or more images, for example, of a video. The detected sound object may be a placeholder for an off-screen sound object, where the sound is (seemingly) caused by an object that is currently or not at all visible within one or more images, for example, of a video.

[0031] Moreover, the detected sound object may be an abstract entity that may leave one or more traces of itself, for example, in one or more inputs. These traces may be detectable to obtain the detected sound object. Furthermore, the detected sound object may leave one or more detectable traces in different modalities. The modalities may be an audio modality, a visual modality, a text modality or a reasoning modality, which is explained in more detail below. The obtained detected sound object is not bound to a modality, for example, the audio modality, as it is an abstract entity. The detected sound object may be a placeholder for a sound object of an audio modality, wherein, the audio modality may also include silence. That is the detected sound object may be realized in an audio modality. The audio modality may also be silence, that is, no audio signal or a suppressed audio signal.

[0032] Multimodal context analysis may relate to obtaining the detected sound object based on multiple different modalities.

[0033] Thus, the detected sound object may be obtained based on a combination of an audio input corresponding to the audio modality, an image input corresponding to the visual modality and a script input corresponding to the text modality. For example, audio input and script input, or audio input and visual input, or visual input and script input, or audio input and visual input and script input may be used. The audio input may refer to an audio wave, the image input may refer to one or more than one images, for example, a sequence of images, the script input may refer to any kind of text input.

[0034] Additionally or alternatively, the multimodal context analysis may comprise a reasoning sound object predictor. Thus, the detected sound object may be obtained based on a reasoning sound object predictor predicting a reasoning trace of the detected sound object. The reasoning sound object prediction of the reasoning trace may be based on any one or more of the audio input, the image input or the script input, but additionally the reasoning sound object predictor may include world knowledge and / or knowledge on causal relations, both of which are of the reasoning modality. Thus, multimodal context analysis may have only an audio input or only an image input or only a script input if the multimodal context analysis also includes a reasoning sound object predictor which is inherently multimodal as it inherently includes the reasoning modality and at least one other input modality.

[0035] In other words, the different modalities of the multimodal context analysis may refer to the input modalities and / or trace modalities and / or process modalities. For example, in case of the reasoning sound object predictor the reasoning modality may be an inherent modality of the predictor, i.e., the reasoning modality may be a process modality in the sense of the predictor conducting a process. For example, the reasoning modality may refer to the world knowledge and / or causal relations, any one or both of which may be an input modality of the predictor or they may be an inherent modality of the predictor, for example a process modality. Similarly, the detectors which detect the traces may have corresponding process modalities. Multimodal context analysis may also relate to the analysis of one modality, for example, the analysis of input of one modality, i.e., the one or more images (visual modality) or a script (text modality), to obtain the detected sound object, which is a placeholder for a sound object of a different modality, i.e., the audio modality, which may also include silence. That is, a placeholder for silence also falls within the audio modality. In other words, although the detected sound object is an abstract entity it is a placeholder for a sound object of an audio modality.

[0036] Thus, the multimodal context analysis may refer to a modality within the context analysis, for example, an input modality or a trace modality or a process modality, differing from the modality of the output of the multimodal context analysis, which is the obtained detected sound object. Input modality may refer to the modality of an input. For example, an audio input may correspond to the audio modality, an image input may correspond to the visual modality and a script input may correspond to the text modality. The audio input may refer to an audio wave, the image input may refer to one or more than one images, for example, a sequence of images, the script input may refer to any kind of text input.

[0037] Similarly, a trace modality may refer to the visual trace of the visual modality, the audio trace of the audio modality, the script trace of the script modality and the reasoning trace of the reasoning modality.

[0038] The process modality may refer to the processes within the multimodal context analysis, which may be, for example, the inherent modality of the predictor, but also the detectors which detect the traces may have corresponding process modalities. Thus, multimodal context analysis refers to any one of the input modalities, trace modalities and / or process modalities differing from the modality of the sound object for which the detected sound object is a placeholder, which is the audio modality, as in that case multiple modalities are involved within the multimodal context analysis.

[0039] For example, as an input modality a visual modality may be used or as an input modality a script modality may be used, both of which differ from the audio modality.

[0040] Also, the multimodal context analysis may comprise a reasoning sound object predictor. Thus, the detected sound object may be obtained based on a reasoning sound object predictor predicting a reasoning trace of the detected sound object. The reasoning sound object prediction of the reasoning trace may be based on any one or more of the audio input, the image input or the script input, but additionally the reasoning sound object predictor may include world knowledge and / or knowledge on causal relations, both of which are of the reasoning modality. Thus, the reasoning sound object predictor includes the reasoning modality, which may be an input modality, if, for example, world knowledge and / or knowledge on causal relations is an input to the predictor, or which may be a process modality, as the prediction is a process and the predictor may inherently include world knowledge and / or knowledge on causal relations. Also, the reasoning trace predicted by the reasoning sound object predictor is in the reasoning modality, which then refers to the trace modality.

[0041] Therefore, if the multimodal context analysis comprises the reasoning sound object predictor, the obtained detected sound object being a placeholder for a sound object of the audio modality, always differs from the reasoning modality, independent of any other input modalities.

[0042] However, multimodal context analysis may also be based on multiple inputs of different modalities. That is, it may be based on any combination of inputs of the audio modality, text modality and visual modality. For example, audio input and script input, or audio input and visual input, or visual input and script input, or audio input and visual input and script input may be used.

[0043] During the multimodal context analysis, depending on the modality, for example, the modality of the input, traces of the detected sound object may be detected in one or more than one modality. Multiple traces of the same detected sound object detected in different modalities, but not necessarily traces of all modalities, if for example a trace is not detected in one modality, may be merged to obtain the detected sound object. The modality may be based on the input modality, for example, an audio modality of audio input, a text modality of script input, or a visual modality of image input. However, a modality may also be based on world knowledge and / or knowledge on causal relations of detected input objects, which may be an input in the sense of an input to the reasoning sound object predictor, or which may, for example, be part of the predictor, that is, part of the process within the multi-modal context analysis, i.e., the reasoning sound object prediction. This may constitute the reasoning modality. Thus, the world knowledge or the knowledge on causal relations may constitute a reasoning modality. A trace of the detected sound object may therefore be based on any one or more of four different modalities, the audio modality, the visual modality, the script modality and the reasoning modality.

[0044] Also, different objects may be detected within the input as traces on which basis the detected sound object may be obtained.

[0045] For example, in the image input, e.g., an input of one or more images, one or more image objects may be detected and multiple image objects may be detected within one image. Based on the analysis of the image object(s) a visual trace of the detected sound object may be detected. That is, a visual trace may be an image object that should generate a sound within the context of the image. In other words, within an image used as input a visual trace may be detected. That is, the image object(s) may be detected as a visual trace of the detected sound object.

[0046] Similarly, for example, within the script input one or more script objects may be detected and based on the analysis of the script object(s), a script trace, that is, a script object that would generate sound within the context of the script, may be detected. In other words, within a script used as input a script trace may be detected. That is, the script object(s) may be detected as a script trace of the detected sound object.

[0047] Also, for example, within an audio input, for example, as an audio waveform, one or more traces of the detected sound object may be detected. That is, the audio input may comprise one or more audio objects which may be detected and based on the analysis of the audio object(s), an audio trace may be detected. Thus, the audio object(s) may be detected as an audio trace of the detected sound object.

[0048] Additionally, the multi-modal context analysis may also comprise a reasoning sound object prediction to obtain the detected sound object. Thus, the multi-modal context analysis may comprise a reasoning sound object predictor which predicts a reasoning sound object. Input to the reasoning sound object may be an image input, a script input and / or an audio input. Additionally, a detected trace, that is, the visual trace, the audio trace and / or the script trace may be an input to the reasoning sound object predictor. Also multiple traces of the same modality may be input to the reasoning sound object predictor.

[0049] Therefore, obtaining the detected sound object may for example span over multiple modalities. That is, obtaining the detected sound object may be based on a visual trace of the visual modality, a script trace of a text modality, an audio trace of an audio modality and / or a reasoning trace of a reasoning modality. Also, obtaining the detected sound object may be based on multiple traces of the same modality, a visual trace, a script trace, an audio trace and / or a reasoning trace. Obtaining the detected sound object may, however, not necessarily span over all the modalities. Therefore, an analysis of multiple inputs of different modalities may occur, but the sound object may not be detected in one of the modalities.

[0050] Obtaining the detected sound object may be based on a prediction (reasoning modality) of a sound that should occur in causal relation to the detected input object(s), i.e., the traces. In other words, obtaining the detected sound object may be based on predicting a reasoning trace occurring in a reasoning modality based on, for example, world knowledge and / or a causal relation to input object(s).

[0051] The detected input objects of different modalities may be related. Related objects may have different but overlapping timelines. For example, if an image object, such as a “car”, is detected and seems to be entering or leaving the image(s) used as input, but should continuously make a sound, such as, in the context of a car driving by, the image object “car” may be detected as a visual trace of the detected sound object. In that case, an audio trace “car” may be detected within an audio input. Thus, the audio trace “car” may be related to the image object “car” and therefore the visual trace “car” detected within the image input. However, the timeline of the audio trace “car” overlaps but is longer than the timeline of the image trace “car”.

[0052] Also, multiple detected sound objects may be detected, which may comprise completely or partially overlapping timings, for example, timelines which may comprise start timecodes and end timecodes.

[0053] In some embodiments the circuitry may be configured to obtain the detected sound object based on a trace detected in an audio wave form, a script, and / or one or more images.

[0054] Detecting a trace may include detecting an audio trace from the audio waveform, detecting a script trace from the script and / or detecting a visual trace from the one or more images as described above. The visual trace detection may be based on a detector, for example, an audio sound object detector. The script trace detection may be based on a detector, for example, a script sound object detector. The image trace detection may be based on a detector, for example, an image sound object detector.

[0055] In some embodiments the circuitry may be configured to obtain the detected sound object based on a reasoning sound object prediction. The reasoning sound object prediction may be based on one or more traces detected in an audio waveform, a script, and / or one or more images. The input to the reasoning sound object predictor may be any one or more of an audio waveform, a script and one or more images. The reasoning sound object prediction may predict a reasoning trace. However, the reasoning trace may not be directly related to the input objects of the input. That is, the reasoning trace may not be an input object, i.e., an audio object, a script object or an image object, detected in the input, i.e., the audio input, the script input, and / or the image input, respectively. Thus, there may be a causal relation to an input object actually included within the input. Also, there may be a causal relation of the reasoning trace to a trace actually appearing in the input. However, the reasoning trace may not be a trace actually appearing within the input. Thus, the reasoning sound object predictor may receive input from the detectors detecting at least one of the visual trace, the audio trace and / or the script trace. The output from the detectors received by the reasoning sound object predictor may be the visual trace, the audio trace and / or the script trace.

[0056] For example, an image input may include pupils sitting in a classroom suddenly getting up and leaving the room. The predicted reasoning trace may then be “a ringing of the school bell” which is predicted based on the context of the image input. The reasoning trace of a ringing school bell is not detectable within the image input, that is, it does not appear in the input. Other input objects may appear within the input, for example, the image objects “pupils”, which also may be image traces for other detected sound objects, for example, if the pupils talk. However, the reasoning trace is not one of the image objects, i.e., not one of the input objects appearing and detectable within the image input. Therefore, the reasoning sound object prediction may be based on an indirect causal relation to at least one input object of the input.

[0057] The electronic device may learn to suggest or predict such a non-directly related reasoning trace from training data of previous inputs, for example, previous production videos, if previous inputs that are used as training data did include the reasoning trace detectable and appearing directly within the previous input used as training.

[0058] The reasoning sound object prediction may not be grounded in the audio waveform, the script or the one or more images. This is by contrast to the detected visual trace, audio trace and script trace which are grounded in the audio waveform, the script and the one or more images respectively. Thus, obtaining the detected sound object may be grounded and / or non-grounded in input.

[0059] Furthermore, the reasoning sound object prediction may be based on world knowledge. With regard to the above “classroom” example, the world knowledge may be the association of students usually leaving a classroom only after a school bell rings.

[0060] In some embodiments the circuitry may be configured to perform multimodal merging of at least two of the group of a trace detected from an audio waveform, a trace detected from a script, a trace detected from one or more images or a reasoning trace predicted based on at least one of an audio waveform, a script or one or more images. The reasoning trace may be predicted based on a trace detected from at least one of an audio waveform, a script or one or more images. The multimodal merging may be performed on traces related to the same detected sound object.

[0061] Thus, the traces may relate to the same event within the different inputs. In other words, multimodal merging may be only performed on traces of the same event. Multimodal merging may be performed on at least two of the group of audio trace, a visual trace, a script trace and a reasoning trace. Thus, multimodal merging may span modalities. That is, obtaining the detected sound object may span modalities. Yet, not all modalities, i.e., visual modality, audio modality, text modality, reasoning modality, may be included in either multimodal merging and / or in obtaining the detected sound object. Furthermore, multimodal merging performed on at least two of the group of an audio trace, an image trace, a script trace and a reasoning trace may be performed if they refer to the same event.

[0062] The detected sound object may include a start timecode and an end timecode. The start timecode and the end timecode may be obtained based on a start timecode and an end timecode of a trace. The timecodes of the detected sound object may be related to timecodes of a trace of the detected sound object. The trace may be detected in an audio input (e.g. an audio wave form), a script input (e.g. a script), and / or an image input (e.g., one or more images). The trace may be a visual trace, an audio trace, a script trace or a reasoning trace. The start timecode and the end timecode may be obtained based on merged start timecode and end timecode of traces of different modalities. Therefore, the start time code and the end timecode of the detected sound object may be based on merged timecodes of multiple traces of the detected sound object, wherein the traces are traces of different modalities, such as, a visual trace of the visual modality, an audio trace of the audio modality, a script trace of the text modality or a reasoning trace of the reasoning modality.

[0063] The timecodes may indicate the time interval placeholder for an audio signal, which may include silence, added to a target audio. The target audio may be a multi-track audio timeline. The target audio may be related to the input or may be the same as the audio waveform input or the input of one or more images. Therefore, the target audio may be synchronized to the input. The start timecode and end timecode of the detected sound object may indicate a timeline.

[0064] A timeline of the detected sound object may be synchronized to a video timeline. For example, a start timecode and an end timecode related to a sound object may be used for synchronization with a video timeline. The video timeline may be the timeline of a video comprising images and audio. The audio may be multi-track audio including an audio timeline synchronized to the video timeline, i.e., the image timeline. The timeline of the detected sound object may therefore indicate a placeholder in a multi-track audio timeline for a single audio signal, which may include silence, added to at least one track of the multi-track audio. The detected sound object that is based on a script trace may have less precise or fuzzy timecodes and may instead include a matching of presence or absence in a particular scene. The scene in turn may include a start timecode and an end timecode. Thus, the start timecode and the end timecode of a detected sound object based on a script trace may relate to a scene in which the sound object is present or absent. Thus, the script trace detection may include an “is-present- in-scene” matching.

[0065] Obtaining the detected sound object may be based on a learned model. Therefore, the detection of traces within the input to obtain a detected sound object may be based on a learned model.

[0066] The detection of any one of the audio trace, the script trace and the visual trace may be based on a learned model. Additionally, or alternatively, multimodal merging may be based on a learned model. Additionally, or alternatively, reasoning trace prediction may be based on a learned model.

[0067] The learned model may be based on a machine learning algorithm and may, for example, be based on a support vector machine (SVM), a neural network, e.g., a convolutional neural network (CNN) or multimodal Al model, e.g., transformer model, or the like.

[0068] Thus, obtaining the detected sound object may be based on an artificial neural network. The detection of traces within the input may be based on an artificial neural network. The detection of any one of the audio trace, the script trace, the visual trace, multimodal merging and / or reasoning trace prediction may be based on an artificial neural network.

[0069] The machine learning algorithm of the learned model may be trained in advance. For example, an audio waveform, a script and / or one or more images may be used for training the learning algorithm. Furthermore, time stamps of the one or more images and / or the audio waveform may be used for training the learning algorithm. The script for training may be a production script, the images may be from a production video and the audio may be a production audio.

[0070] The time stamps may be from at least one audio track of a single track or a multi-track audio. In case of multi-track audio , the time stamps may be based on at least one audio track of the audio at the time before audio mixing, i.e., when sounds are still single and not mixed in stems. Based on the labels of an audio signal, sound object definitions may be created during preprocessing of the training data for training the learning algorithm. The labels of the audio signal may comprise time stamps, relations and / or text descriptions. The detector for detecting the audio trace may be trained on labeled production recordings, i.e., audio recordings. The detection process itself may be based on unlabeled production recordings. The detector for detecting the script trace may be trained on labeled scripts. The detection process itself may be based on unlabeled scripts. The detector for detecting the visual trace may be trained on labeled video recordings, i.e., the images of the video recordings. The detection process itself may be based on unlabeled video recordings. Also, input used as training data for the learning algorithm of the reasoning sound object predictor may include the reasoning trace detectable and appearing directly within the input used as training.

[0071] Obtaining the detected sound object may be based on sound event detection. That is, detecting a trace withing an audio input may be based on sound event detection and / or the audio sound object detector for detecting the audio trace may be a sound event detector (SED).

[0072] Obtaining the detected sound object may be based on rule-based text phrase detection. That is, detecting a trace withing a script input, i.e., detecting the script trace, may be based on rule-based text phrase detection. That is, the script sound object detector may be based on a rule-based system looking for special text phrases, for example “Glass breaking” or “Explosion”.

[0073] Obtaining the detected sound object may be based on a neural network based correlator of sentence embeddings to detected sound object using a synchronization between a script and an audio waveform. That is, detecting a trace within a script input, i.e., detecting the script trace may be based on a script sound object detector using a neural network based correlator of sentence embeddings to detected sound objects using a synchronization mechanism between a script and an audio track. The synchronization may be based on automatic speech recognition (ASR) dialogue synchronization.

[0074] Obtaining the detected sound object may be based on object detection and / or motion tracking. The detection of a trace in an image input may be based on object detection and / or motion tracking and / or the image sound object detector for detecting the image sound object may be an object detector that can detect objects and track their motion in a sequence of images, i.e., in videos.

[0075] The reasoning sound object predictor may be based on a learned model of previous inputs, for example, a previous movie, and / or world knowledge that can predict sound objects that should be there.

[0076] Obtaining the detected sound object, for example, detecting a trace within the input, that is, the detection of any one of the audio trace, the script trace and the visual trace, may be based on pattern matching and / or pattern recognition. Pattern recognition may include rule-based methods as well as learned methods, the latter may be the learned model (e.g., machine learning algorithm, neural networks) discussed above.

[0077] The detected sound object may include a detection cue indicating how the detected sound object was obtained. The detection cue may be additional information that tells a human sound editor reviewing the detected sound object which cue was the reason that the detected sound object was obtained. Thus, the review of the detected sound object by a human sound editor to possibly create a candidate sound object and in turn a realized sound object may be enabled by the detection cue. The detection cue may be one of the group of an image object visible in one or more images, an audio object present in an audio waveform, a script object located in a script.

[0078] The detection cue may be an image object including, for example, a keyframed bounding box. The bounding box may comprise as information the coordinates (x,y) and dimensions (height h, width w) of the image object indicated in pixels of the image. The bounding box may also include as bounding box information depth and / or volume information to aid identification of the image object. Keyframed may refer to the bounding box information at a certain timestep t, identified by the frame number, as the image object may be moving and may be tracked over the duration from start timecode to end timecode. Furthermore values between keyframes may be interpolated, therefore, not all timesteps must be explicitly defined. Thus, a keyframed bounding box may be a sequence of bounding box information at successive but not necessarily consecutive timesteps.

[0079] The detection cue may be found, for example, in production video recordings which may include VFX. The script object may comprise a location in a script. The detection cue may be based on existing production sound, which may, for example, be recorded at a set, and may include audio channel information. The detection cue may be a textual reasoning description.

[0080] The circuitry may be further configured to present the detected sound object as a candidate sound object for generating a realized sound object including a sound signal. As explained above, the generated realized sound object includes an actual audio signal, for example, for a movie. For that purpose, a human sound editor may review the detected sound object and create a candidate sound object. In other words, the electronic device may present the sound objects in a way that the human sound editor can choose and possibly edit it during a sound editing process to generate a candidate sound object. If multiple detected sound objects are obtained they may be presented in a list. The human sound editor may then select from the list the detected sound object(s) to keep and may add more sound object(s) or otherwise edit the list to create a list of candidate sound object(s). The detected sound object may not yet be created as an audio-signal. Therefore, the detected sound object may be a placeholder for a later created audio signal, for example, as a realized sound object. The audio signal for a realized sound object may be created based on the candidate sound object of a sound editor that in turn may be based on the detected sound object presented to and reviewed by the sound editor. It may also be possible that no candidate sound object is generated, and the realized sound object is based directly on the detected sound object.

[0081] The detected sound object may include a descriptor for the detected sound object, which may be a text specification of the needed sound signal or a proposal for the same. A human sound editor which may review the detected sound object, or a list of detected sound objects, to create at least one candidate sound object may adopt the descriptor or edit the text specification for the candidate sound object. Thus, the candidate sound object may include a descriptor too, which may be based on the detected sound object descriptor.

[0082] The descriptor may be based on a catalogue of sound object types. The descriptor may for example indicate human sounds or non-human sounds. The human sounds may be actor-specific sounds. It may be desirable to use descriptors distinguishing between human sounds, in particular, distinguishing between actor specific sounds, and non-humans sounds, because then a decision can be made, for example by a reviewing human, on whether dialogue replacement, for example, automated dialogue replacement (ADR) may be needed or not. The non-human sounds may for example include a sound of steps, a sound of car doors, a sound of house doors, a sound of room doors, a sound of computer usage, such as, typing clicking, wordy anger about certain programs, a sound of AC running, car sounds, such as, starting, running stopping or honking a car, pet sounds, office sounds, restaurant sounds, public transport sounds, or the like.

[0083] The detected sound object may include a label indicating by what generation process the audio signal the detected sound object is a placeholder for shall be created. The audio signal, possibly included in a realized sound object, may be generated by automated dialogue replacement (ADR), Foley, retrieved from a sound database, or the like. The label may, therefore, may be a specification or a proposal on how the needed sound signal shall be created. Thus, the candidate sound object may also include a label, which may be the label of the detected sound object, or a label added by the human sound editor reviewing the detected sound object, and it may be a label edited based on the label of the detected sound object.

[0084] The detected sound object may be based on sound caused by at least one image object visible in a video. However, the detected sound object may be based on sound caused by at least one image object not visible in a video. Some embodiments pertain to a method comprising obtaining a detected sound object based on multimodal context analysis. The method may also implement any one or more of all the processes described above.

[0085] Some embodiments pertain to a computer program comprising instructions which, when executed by a processor, performs the method.

[0086] Fig. la schematically shows an embodiment of a detected sound object in relation to a candidate sound object and a realized sound object. Detected sound object 1 comprises a start timecode 2 and an end timecode 3 as well as at least one detection cue 3. It may also comprise more than one detection cue 3. Detected sound object 1 is illustrated withing the boundary of candidate sound object 20 as the candidate sound object 20 is chosen based on one or more detected sound objects 1. Similarly, candidate sound object 20 is shown within the boundary of realized sound object 30, as the realized sound object 30 is based on candidate sound object 20, i.e., also on detected sound object 1. In other words, detected sound object 1 may be presented as a selection candidate, for example, in a list of detected sound objects 1. From the detected sound objects 1, for example, from the list of detected sound objects 1, a candidate sound object 20 may be chosen, for example, by a human sound editor. Afterwards, the appropriate sound may be produced or generated to generate realized sound object 30. Thus, realized sound object 30 comprises sound and may be added, for example, to a production video, not as a placeholder, such as detected sound object 1, but as sound. Therefore, contrary to the detected and candidate sound objects 1, 20 which take the form of placeholders, realized sound object 30 comprises actual sound, e.g., the final sound signal itself, which can be added to a production, for example in the audio track of a video.

[0087] Detection cue 3 of detected sound object 1 may be an audio object, for example from an existing production sound and may, therefore, comprise channel information of a multi-channel production sound. Detection cue 3 may be an image object, for example of an existing production video, and may therefore comprise information of a keyframed bounding box.

[0088] The bounding box may comprise as information the coordinates (x,y) and dimensions (height h, width w) of the image object indicated in a pixel space of the image. For example, x = 10, y=20, w=50, h=100 may be bounding box information in pixel space. The bounding box may also include as bounding box information depth and / or volume information to aid identification of the image object. Keyframed may refer to the bounding box information at a certain timestep t, identified by the frame number, as the image object may be moving and may be tracked over the duration from start timecode to end timecode. Furthermore values between keyframes may be interpolated, therefore, not all timesteps must be explicitly defined. Thus, a keyframed bounding box may be a sequence of bounding box information at successive but not necessarily consecutive timesteps.

[0089] Detection cue 3 of detected sound object 1 may be a script object, for example, from an existing production script and may, therefore, comprise script location. Detection cue 3 may comprise a textual reasoning description. Detected sound object 1 may also comprise multiple detection cues 3 at least one being an audio object, an image object, a script object or a textual reasoning description. The above-mentioned detection cue 3 may help a human sound editor understand why detected sound object 1 was detected.

[0090] Candidate sound object 20 may comprise any one or more of the features, for example, timecodes 2, 3, detection cue 3, that detected sound object 1 comprises. Similarly, realized sound object 30 may comprise any one or more of the features that candidate sound object 20, i.e., also detected sound object 1, comprise.

[0091] Fig. lb schematically shows an embodiment of a detected sound object with a descriptor and a label, a candidate sound object and a realized sound object. Detected sound object 1 comprises a start timecode 2 and an end timecode 3 as well as at least one detection cue 3. It may also comprise more than one detection cue 3. Detected sound object 1 is illustrated withing the boundary of candidate sound object 20 as candidate sound object 20 is chosen from one or more detected sound objects 1. Similarly, candidate sound object 20 is shown within the boundary of realized sound object 30, as the realized sound object 30 is based on candidate sound object 20, i.e., also on detected sound object 1.

[0092] Detected sound object 1 comprises descriptor 4 and label 5. Descriptor 4 comprises a specification, for example, a text specification, for the needed sound signal that detected sound object l is a placeholder for and which may later be realized in realized sound object 30. Label 5 comprises a specification on how the needed sound signal that detected sound object 1 is a placeholder for shall be created.

[0093] Candidate sound object 20 may comprise any one or more of the features, for example, timecodes 2, 3, detection cue 3, label 5 and / or descriptor 4, that detected sound object 1 comprises. However, candidate sound object 20 may also comprise a label and descriptor, which is not part of detected sound object 1. Similarly, realized sound object 30 may comprise any one or more of the features that detected sound object 1 and / or candidate sound object 20 comprise. Fig. 2a schematically shows an example of a detected sound object. Detected sound object 1 comprises the start timecode “fl756” and the end timecode “fl779”. In other words, detected sound object 1 starts at frame 1756 and ends at frame 1779. The frames may indicate a sequence of multiple images, i.e., frames, which may form part of a video, and / or may indicate audio frames. The frames may also indicate audio timings synchronized to frames of a sequence of images, for example, in a video. Thus, detected sound object 1 is obtained based on frames of an audio and / or image input, for example, of a video. The timecodes 2 may also indicate where in a target audio a realized sound object (e.g., 30 of Figs, la and lb) that is based on detected sound object 1, as for example explained regarding Figs, la and lb, should be added. The target audio may be an audio of a target video, the target audio may be synchronized to the multiple images of the target video. The input and the target may be the same and / or synchronized. Thus, timecodes 2 “fl756 to fl799” of detected sound object 1 are determined, for example, based on image frames and / or audio frames, and timecodes 2 may be further used as an indication of a time location for the detected sound object 1 as a placeholder for a sound signal, for example, within a multitude of images and / or within an audio signal, and / or within the frames of a video. That is, an indication where in time a sound object comprising sound, for example, realized sound object 20 of Fig. la and lb, should be added to audio, for example, the audio track of a video.

[0094] Detected sound object 1 of Fig. 2a also comprises two detection cues 3, the detection cue 3 “images(s)”, which may indicate an image object, and the detection cue 3 “script”, which may indicate a script object. The detection cue 3 “images(s)” includes area information indicating, for example, the area of an image based on which the detected sound object 1 was obtained. It may also indicate that an image object may have been detected based on that area of an image used as input. In Fig. 2a the area is x = 10, y=20, w=50, h=100 indicating the x-y coordinates as well as dimensions (width w and height h) of the image object within the pixel space of the image (see explanation of “keyframed bounding box” of Fig. 1). The detection cue 3 “script” includes the location in the script as line 841 - 842. This indicates that based on lines 841-842 of a script detected sound object 1 was obtained and it may also indicate that a script object may have been detected based on lines 841 to 842 of a script used as input. Furthermore, the two detection cues 3 may be related to each other. Thus, the image area x = 10, y=20, w=50, h=100 in pixels of an image may be related to script lines 841 to 842. That is, the detected sound object 1 was obtained based on area x = 10, y=20, w=50, h=100 of an image and lines 841 to 842 of a script. Also, detected sound object 1 may have been obtained based on an image object detected in area x = 10, y=20, w=50, h=100 of an image and based on a related script object detected in lines 841 to 842 of a script.

[0095] Detected sound object 1 of Fig. 2a may be an example of a detected sound object as placeholder for an object-based sound object, which is also an on-screen sound object, as the sound may come from an object that appears static on the outside, for example, from a body of water, that is, an image object / image sound object visible in area x = 10, y=20, w=50, h=100 of the image input.

[0096] Fig. 2b schematically shows another example of a detected sound object. Detected sound object

[0097] 1 of Fig. 2b comprises timecodes 2 comprising the start timecode “f456” and the end timecode “f480”. In other words, detected sound object 1 starts at frame 456 and ends at frame 480. The frames may indicate a sequence of multiple images, i.e., frames, which may form part of a video, and / or or may indicate audio frames. The frames may also indicate audio timings synchronized to frames of a sequence of images, for example, in a video. Thus, detected sound object 1 is obtained based on frames of an audio and / or image input, for example, of a video. The timecodes

[0098] 2 may also indicate where in a target audio a realized sound object (e.g., 30 of Figs, la and lb) that is based on detected sound object 1, as for example explained regarding Figs, la and lb, should be added. The target audio may be an audio track of a target video, the audio track may be synchronized to the multiple images of the target video. The input and the target may be the same and / or synchronized. Thus, timecodes 2 “f456 to f480” of detected sound object 1 are determined, for example, based on image frames and / or audio frames, and timecodes 2 may be further used as an indication of a time location for the detected sound object 1 as a placeholder for a sound signal, for example, within a multitude of images and / or within an audio signal, and / or within the frames of a video. That is, an indication where in time a sound object comprising sound, for example, realized sound object 20 of Fig. la and lb, should be added to audio, for example, the audio track of a video.

[0099] Detected sound object 1 of Fig. 2b also comprises three detection cues 3, the detection cue 3 “images(s)”, which may indicate an image object, the detection cue 3 “audio”, which may indicate an audio object, and the detection cue 3 “script”, which may indicate a script object. The detection cue 3 “image(s)” includes information on the content of the image(s), such as “movement (book, . . .)”. This indicates, for example, that detection cue 3, is the movement of a book or other objects. Thus, detected sound object 1 is obtained based on the movement of the book etc., visible in input images, for example, of a video. Furthermore, the movement of a book visible in the images used as input may have been detected as an image object and an image sound object for obtaining detected sound object 1.

[0100] Detection cue 3 “audio” includes frame information, f456-f480. The frames may indicate audio frames or may also indicate image frames to which an audio, i.e., an audio signal, is synchronized, for example, image frames of a video. In other words, the audio may be the audio track of a video. Also, it indicates that based on frames 456 to 480 of the audio detected sound object 1 was obtained and it may indicate that an audio object was detected based on frames 456 to 480 of the audio on which basis detected sound object 1 was obtained.

[0101] Detection cue 3 “script” includes the location in the script as line 766, which indicates that detected sound object 1 was obtained based on line 766 of a scrip. It may also indicate that a script object was detected based on line 766 of a script on which basis the detected sound object 1 was obtained.

[0102] Furthermore, the three detection cues 3 may be related to each other. Thus, the movement of a book etc. visible in the images may be related to script lines 766 and to the audio at frames f456 to f480. That is, an image object “movement of a book” may have been detected in an image and may be related to an audio object at frames f456 to f480 and to a script object detected in line 766 of a script. Thus, an image sound object “movement of a book” may have been detected and may be related to and audio sound object at f456 to f480 and to a script sound object detected in line 766 of a script. These three objects, that is traces of a single detected sound object in three modalities, may have been merged to obtain detected sound object 1.

[0103] Detected sound object 1 of Fig. 2b may be an example of a detected sound object as placeholder for a process-based sound object, which is also an on-screen sound object, as the sound comes from the movement of a book which is visible in the image input.

[0104] Fig. 2c schematically shows another example of a detected sound object. Detected sound object 1 of Fig. 2c comprises timecodes 2 comprising the start timecode “f856” and the end timecode “f869”. In other words, detected sound object 1 starts at frame 856 and ends at frame 869. The frames may indicate a sequence of multiple images, i.e., frames, which may form part of a video. Thus, the frames may also indicate audio timings synchronized to frames of a sequence of images, for example, in a video. The frames may also indicate audio frames. Therefore, detected sound object 1 is obtained based on frames of an audio and / or image input, for example, of a video. The timecodes 2 may also indicate where in a target audio a realized sound object (e.g., 30 of Figs, la and lb) that is based on detected sound object 1, as for example explained regarding Figs, la and lb, should be added. The target audio may be an audio track of a target video, the audio track may be synchronized to the multiple images of a target video. The input and the target may be the same and / or synchronized. Thus, timecodes 2 “f856 to f869” of detected sound object 1 are determined, for example, based on image frames and / or audio frames, and timecodes 2 may be further used as an indication of a time location for the detected sound object 1 as a placeholder for a sound signal, for example, within a multitude of images and / or within an audio signal, and / or within the frames of a video. That is, an indication where in time a sound object comprising sound, for example, realized sound object 20 of Fig. la and lb, should be added to audio, for example, the audio track of a video.

[0105] Detected sound object 1 of Fig. 2c also comprises a detection cue 3, the detection cue 3 “audio”, which may indicate an audio object. Detection cue 3 includes frame information f856 - f869. The frames may indicate audio frames or may also indicate image frames to which an audio, i.e., an audio signal, is synchronized, for example, image frames of a video. In other words, the audio may be the audio track of a video. Also, it indicates that based on frames 856 to 869 of the input audio detected sound object 1 was obtained and it may indicate that an audio sound object was detected based on frames 856 to 869 of the audio on which basis detected sound object 1 was obtained.

[0106] Detected sound object 1 of Fig. 2c may be an example of a detected sound object as placeholder for a process-based sound object, which is also an off-screen sound object, as the sound may come from a dialogue of a person not appearing in an image input, but detectable in the audio input at frames 856 to 869.

[0107] Fig. 3 schematically shows an embodiment of multimodal context analysis for obtaining a detected sound object. In multimodal context analysis 50 of Fig. 3 audio waveform 6, script 7 and one or more images 8 are used as input 10. Audio waveform 6 is the input 10 in the audio modality for audio sound object detector 1 la that detects one or more audio traces 12a within audio waveform 6. Script 7 is the input 10 in the text modality for script sound object detector 1 lb that detects one or more script traces 12b within script 7. One or more images 8 is the input 10 in the visual modality for image sound object detector 11c that detects one or more image traces 12c within one or more images 8 used as input 10. In other words, depending on the modality of input 10 different sound object detectors 11 detect traces of detected sound object 1 within input 10 and thus generate output 12 based on these traces. In other embodiments, traces may exist in one modality, but may not exist in other modalities. Thus, traces in a modality may exist, but may not necessarily exist. Furthermore, audio waveform 6, script 7 and one or more images 8 are used as input 10 for reasoning sound object predictor 9. Reasoning sound object predictor 9 predicts one or more reasoning traces 13. Moreover, in contrast to audio trace 12a, script trace 12b and image trace 12c which are detectable within the respective input 10 and therefore appear withing input 10, reasoning trace 13 is inferred based on input 10 as reasoning trace 13 is not directly detectable and does not appear within input 10. That is, based on the current input 10 of the aforementioned three modalities audio, visual and text, together with additional knowledge, constituting a reasoning modality, for example, based on world knowledge or based on knowledge of causal relations of detected objects, that is partially based on a learned model of a previous input 10, reasoning sound object predictor 9 may predict reasoning trace 13.

[0108] For example, if images of a video show a classroom where pupils are sitting and suddenly stand up and leave, reasoning sound object predictor 9 may predict the reasoning trace “school bell” which indicates that the current lesson is over. The “school bell” did not appear in the images used as input 10. Furthermore, the “school bell” did not appear in any other input 10, for example, also not in the audio, i.e., audio waveform 6, or the script, i.e. script 7.

[0109] On the other hand, if, for example, the “school bell” appears within audio waveform 6, it is detectable and audio sound object detector I la would detect it as audio trace 12a.

[0110] In multimodal context analysis 50 of Fig. 3 reasoning sound object predictor 9 is also informed by the output of the audio sound object detector 1 la, the script sound object detector 1 lb and the image sound object detector 11c. Therefore, reasoning sound object predictor 9 obtains, besides input 10, additional information from the sound object detectors 11 to predict reasoning trace 13.

[0111] In other embodiments reasoning sound object predictor 9 may not obtain input from the sound object detectors 11 or reasoning sound object predictor 9 may only receive information from only some of the sound object detectors 11.

[0112] Output 12 from detectors 11 and the output from the reasoning sound object predictor 9, that is, the one or more audio traces 12a, the one or more script traces 12b, the one or more visual traces 12c, as well as the one or more reasoning traces 13 are merged via multimodal merging 14 to generate detected sound object 12. Reasoning sound object 13 as output of reasoning sound object predictor 9 is in the reasoning modality. Audio trace 12a as output 12 of the audio sound object detector I la is in the audio modality. Script trace 12b as output 12 of the script sound object detector 1 lb is in the text modality and visual trace 12c as output of the image sound object detector 12c is in the visual modality. In other words, based on traces within the input 10 of the three different modalities, audio, text and visual, together with additional knowledge of the reasoning modality, which lead to traces of four different modalities audio, script, visual and reasoning, as well as based on multimodal merging 14 spanning the different modalities, detected sound object 1 is obtained. Furthermore, audio trace 12a, script trace 12b, visual trace 12c and reasoning trace 13 are related to each other. That is, audio trace 12a, script trace 12b, visual trace 12c and reasoning trace 13 are merged during the multimodal merging 14 as they are related to the same event.

[0113] In other embodiments of multimodal context analysis 50, the multimodal merging 14 may be based on traces in any combination of modalities, audio, text and visual as well as reasoning.

[0114] Furthermore, input 10 of any one of the three modalities, audio, text and visual, may be used for obtaining detected sound object 1. Therefore, input 10 for sound object detectors 11 may be of any modality. In turn, one or more of the sound object detectors 11 depending on and for each input 10 modality may be used. Furthermore, input 10 for reasoning sound object predictor 9 may be input 10 of any modality. Therefore, audio waveform 6, script 7 and / or one or more images 8 may be used as input 10 for reasoning sound object predictor 9.

[0115] In some embodiments of multimodal context analysis 50 multimodal merging 14 may be based on output 12, 13 of any combination of trace modalities, audio, text, visual, and / or the reasoning modality of reasoning trace 13. In other words, multimodal merging 14 may be based on any combination of the group of audio trace 12a, script trace 12b, visual trace 12c and reasoning trace 13.

[0116] In some embodiments of multimodal context analysis 50 no multimodal merging 14 occurs if detected sound object 1 is obtained based on only one modality. That is, detected sound object 1 may be obtained only based on reasoning sound object prediction without the sound object detectors 11. In that case, input 10 may be any one or all of audio waveform 6, script 7 and image(s) 8. Therefore, although the input 10 modality may be one or multiple depending on input 10 for reasoning sound object predictor 9, the modality of the output of reasoning sound object predictor 9 is only one, namely the reasoning modality of reasoning sound object 13.

[0117] Also, detected sound object 1 may be obtained based only on sound object detectors 11. If input 10 of different modalities and therefore multiple sound object detectors 11 of different modalities are used, then multimodal merging 14 still occurs.

[0118] However, in some embodiments input 10 of only one modality, for example script 7 or image(s) 8, is processed by one sound object detector 11 of the same modality, i.e., script sound object detector 1 lb or image sound object detector 11c respectively, then no multimodal merging 14 is used to obtain detected sound object 1. For example, script 7 may be processed by script sound object detector 1 lb to obtain detected sound object 1. Also, one or more images 8 may be processed by image sound object detector 11c to obtain detected sound object 1.

[0119] In all combination modality examples explained above multimodal context analysis 50 is used as described including multimodal merging 4. However, detected sound object 1 also comprises a modality as it is a placeholder for a sound signal. Thus, detected sound object 1 comprises the audio modality. Therefore, in the three single modality input / output examples of multimodal context analysis 50 explained above, i.e., the reasoning modality output example, the script modality input and output example and the visual modality input and output example, multimodal context analysis 50 is used as based on one modality, reasoning, script or visual, the detected sound object 1 of another modality, namely, audio modality, is obtained.

[0120] An electronic device 100 as illustrated in Fig. 7, for example, CPU 101 of Fig. 7, may implement multimodal context analysis 50 to obtain detected sound object 1 as described above.

[0121] Fig. 4 schematically illustrates an example of a post-production workflow comprising a process of obtaining detected sound objects as described in Fig. 3. Production images 8a, production audio 6a and production script 7a are used as input 10 for detecting one or more detected sound objects at S50. Obtaining detected sound object(s) at S50 is an automated process according to legend 60 and results in a list of detected sound object(s) 15. The detected sound objects (e.g., 1, Figs. 1-3) may be obtained as explained in regard to Fig. 3. Thus, production images 8a may be one or more images 8 of Fig. 3, production audio 6a, may be audio waveform 6 of Fig. 3 and production script 7a may be script 8 of Fig. 3. Based on the list of detected sound object(s) 15 sound objects are selected and completed where necessary, which is conducted by a human, for example, a sound editor, according to legend 60. Thus, the detected sound objects(s) of S50 are presented, for example, to the human sound editor, as the list of detected sound object(s) 15 in a way that the human sound editor can select from the list of detected sound object(s) and complete the list of detected sound object(s) where deemed necessary, to generate a list of candidate sound object(s) 21. A candidate sound object of list 21 may be candidate sound object 20 of Fig. la-lb. Based on the list of candidate sound object(s) 21 realized sound object candidates 30a including audio signal are produced according to various sound signal production processes 22 to 25. Although, sound signal production processes 22 to 25 are human processes according to legend 60, it does not preclude that automated processes may be used in support of the human. The audio signal production process may be one or more of automatic dialogue recording (ADR) 22, Foley creation 23, copying production audio clips 24, using sound database (DB) clips 25 based on sound database 26. Furthermore, an audio signal production process may be based on label 5 as explained in regard to Fig. lb, i.e., originally label 5 of detected sound object 1 (Fig. lb) of list 15, which is transferred as label 5 of candidate sound object 20 (Fig. lb) to candidate sound object 20 (Fig. lb) of list 21 during the selecting and completing process at S16.

[0122] Realized sound object candidate 30a may be a realized sound object 30 as explained regarding Fig. 1. It is also possible that multiple different realized sound object candidates 30a are produced, possibly via one or multiple different signal audio production processes 22 to 25, possibly based on one candidate sound object of list 21, i.e., which may correspond to one detected sound object 1 (Figs. 1-3) of list 15. A human sound editor may then choose one of the realized sound object candidates 30a as a realized sound object 30 to be added to the production video, i.e., the final audio of the video. Therefore, realized sound object candidate 30a may be a candidate, because it is merely a proposal for an audio / sound mixing process realized in later stages.

[0123] Fig. 5 schematically shows a flow diagram of a process of obtaining a detected sound object. At SI audio, script and / or images are obtained and / or world knowledge and / or knowledge on causal relations are obtained. At S2 traces of detected sound object 1 based on world knowledge and / or knowledge on causal relations and / or based on audio, script and / or images(s) are obtained. At S3 traces of different modalities are merged to obtain detected sound object 1. At S2 the reasoning sound object may be predicted based on traces of audio, script and / or images. The process of Fig. 5 may be a multimodal context analysis 50 as explained regarding Fig. 3, wherein audio, script and image(s) at SI correspond to audio waveform 6, script 7 and one or more images 8 of input 10 respectively and the traces based on world knowledge and / or causal relations at S2 may correspond to prediction of reasoning sound object 13 of Fig. 3. Thus, at S3 merging traces of different modalities may include merging of traces of the audio modality, script modality, image modality and / or reasoning modality. The reasoning modality may include world knowledge and / or knowledge of causal relations. Causal relations may include the causal relation of one or more detectable input objects, input objects being objects detectably included in input audio, script and / or image(s), to a reasoning sound object.

[0124] Fig. 6 schematically shows timelines of multiple detected sound objects overlapping in time. The timeline including start timecode 2a and end timecode 2b of detected sound object la overlaps with the timeline including start timecode 2a and end timecode 2b of detected sound object lb and partially overlaps with the timeline including start timecode 2a of detected sound object 1c. Also, the timeline, including start timecode 2a, of detected sound object Id and the timeline, including end time code 2b, of detected sound object le partially overlap.

[0125] The timelines of all detected sound objects la-le are synchronized to images, for example, of a target video.

[0126] Fig. 7 shows a block diagram depicting an embodiment of an electronic device that can implement multimodal context analysis for obtaining a detected sound object. Electronic device 100 comprises a CPU 101 as processor. The processor 101 may for example implement the multimodal context analysis 50 of Fig. 3 or the processes at SI, S2, S3 of Fig. 5 and S50 as described with regard to Fig. 4.

[0127] Electronic device 100 further comprises one or more microphones 106, one or more loudspeakers 107 and a camera 109. Loudspeaker 107 may be headphones, e.g., on-ear, in-ear, over-ear, wireless headphones and the like, or may consist of one or more loudspeakers that are distributed over a predefined space and are configured to render any kind of audio, such as 3D audio. The microphone 106 may be configured to receive any kind of audio signal. The microphone 106 may for example be used to capture the audio waveform 6 of Fig. 3. The camera 109 may, for example, be used to capture the one or more image(s) 8 of Fig. 3

[0128] Electronic device 100 further comprises a user interface 108 that is connected to the processor 101. This user interface 108 may act as a man-machine interface and may enable a dialogue between a user and the electronic device 100. For example, a user may make configurations to the system using this user interface 100. The electronic device 100 may further comprise a Bluetooth interface 104, and a WLAN interface 105. These units 104, 105 act as I / O interfaces for data communication with external devices. For example, additional loudspeakers, microphones or external databases, for example, the sound database 26 as described with regard to Fig. 4 or a database storing input 10 as described in relation to Figs. 3 and 4.

[0129] The electronic device 100 further comprises a data storage 102 and a data memory 103 (here a RAM). The data memory 103 may be arranged to temporarily store or cache data or computer instructions for processing by the processor 101. The data storage 102 may be arranged as a long-term storage, e.g., for input 10 as described in relation to Figs. 3 and 4.

[0130] The connection between the CPU 101 and the camera 109 may include a camera serial interface (CSI). The CSI is an interface between a camera 109 and a host processor 101. Thus, control signals and data from the CPU 101 to the camera 109 as well as from the camera 109 to the processor 101 may be sent.

[0131] It should be noted that the description above is only an example configuration. Alternative configurations may be implemented with additional or other sensors, storage devices, interfaces, or the like.

[0132] The electronic device 100 may be a mobile device, such as a smartphone, or any other kind of portable device, such as, a laptop, a tablet etc.

[0133] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0134] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

[0135] All embodiments described above can also be realized as a method, for example as a method for controlling an electronic device, such as electronic device 100 discussed above. Any one of the methods can also be implemented as a computer program causing a computer and / or a processor, such as processor 101 discussed above, to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the one or more methods described to be performed.

[0136] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0137] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure. Note that the present technology can also be configured as described below.

[0138] (1) An electronic device 100 comprising circuitry configured to obtain a detected sound object based on multimodal context analysis 50.

[0139] (2) The electronic device 100 of (1), wherein the circuitry is further configured to obtain the detected sound object 1 based on a trace detected in an audio waveform 6, a script 7, and / or one or more images 8.

[0140] (3) The electronic device 100 of (1) or (2), wherein the circuitry is further configured to obtain the detected sound object 1 based on a reasoning sound object prediction.

[0141] (4) The electronic device 100 of (3), wherein the reasoning sound object prediction is based on a trace detected in an audio waveform 6, a script 7, and / or one or more images 8.

[0142] (5) The electronic device 100 of (3) or (4), wherein the reasoning sound object prediction is not grounded in the audio waveform 6, the script 7 or the one or more images 8.

[0143] (6) The electronic device 100 of any one of (3) to (5), wherein the reasoning sound object prediction is based on world knowledge.

[0144] (7) The electronic device 100 of any one of (1) to (6), wherein the circuitry is configured to perform multimodal merging 14 of at least two of the group of a trace detected from an audio waveform 6, a trace detected from a script 7, a trace detected from one or more images 8 or a reasoning trace 13 predicted based on at least one of an audio waveform 6, a script 7 or one or more images 8.

[0145] (8) The electronic device 100 of (7), wherein the multimodal merging 14 is performed on traces related to the same detected sound object 1.

[0146] (9) The electronic device 100 of any one of (1) to (8), wherein the detected sound object 1 includes a start timecode 2, 2a and an end timecode 2,2a.

[0147] (10) The electronic device 100 of (8) or (9), wherein the timecodes 2, 2a, 2b are related to timecodes of a trace of the detected sound object 1, wherein the trace is detected in an audio wave form 6, a script 7, and / or one or more images 8.

[0148] (11) The electronic device 100 of any one of (8) to (10), wherein the start timecode 2, 2a and the end timecode 2, 2b are based on merged timecodes of multiple traces of the detected sound object 1, wherein the traces are traces of different modalities. (12) The electronic device 100 of any one of (1) to (11), wherein a timeline of the detected sound object 1 is synchronized to a video timeline.

[0149] (13) The electronic device 100 of any one of (1) to (12), wherein obtaining the detected sound object is based on an artificial neural network.

[0150] (14) The electronic device 100 of any one of (1) to (13), wherein obtaining the detected sound object 1 is based on sound event detection.

[0151] (15) The electronic device 100 of any one of (1) to (14), wherein obtaining the detected sound object 1 is based on rule-based text phrase detection.

[0152] (16) The electronic device 100 of any one of (1) to (15), wherein obtaining the detected sound object 1 is based on a neural network based correlator of sentence embeddings to detected sound object 1 using a synchronization between a script 7 and an audio waveform 6.

[0153] (17) The electronic device 100 of (16), wherein the synchronization is based on ASR dialogue synchronization.

[0154] (18) The electronic device 100 of any one of (1) to (17), wherein obtaining the detected sound object 1 is based on object detection and / or motion tracking.

[0155] (19) The electronic device 100 of any one of (1) to (18), wherein the detected sound object 1 includes a detection cue 3 indicating how the detected sound object 1 was obtained.

[0156] (20) The electronic device 100 of (19), wherein the detection cue 3 is one of the group of an image object visible in one or more images, an audio object present in an audio waveform, a script object located in a script.

[0157] (21) The electronic device 100 of (19) or (20), wherein the detection cue 3 is a textual reasoning description.

[0158] (22) The electronic device 100 of any one of (1) to (21), wherein the circuitry is further configured to present the detected sound object 1 as a candidate sound object 20 for generating a realized sound object 30 including a sound signal.

[0159] (23) The electronic device 100 of any one of (1) to (22), wherein the detected sound object 1 includes a descriptor 4 for the detected sound object 1.

[0160] (24) The electronic device 100 of (23), wherein the descriptor is based on a catalogue of sound object types. (25) The electronic device 100 of any one of (1) to (24), wherein the detected sound object 1 includes a label 5 indicating by what generation process the audio signal the detected sound object l is a placeholder for shall be created.

[0161] (26) The electronic device 100 of any one of (1) to (25), wherein the detected sound object 1 is based on sound caused by at least one image object visible in a video.

[0162] (27) The electronic device 100 of any one of (1) to (26), wherein the detected sound object 1 is based on sound caused by at least one image object not visible in a video.

[0163] (28) A method comprising: obtaining a detected sound object 1 based on multimodal context analysis 50.

[0164] (29) The method of (28), wherein the method further includes obtaining the detected sound object 1 based on a trace detected in an audio waveform 6, a script 7, and / or one or more images 8.

[0165] (30) The method of (28) or (29), wherein the method further includes obtaining the detected sound object 1 based on a reasoning sound object prediction.

[0166] (31) The method of (30), wherein the reasoning sound object prediction is based on a trace detected in an audio waveform 6, a script 7, and / or one or more images 8.

[0167] (32) The method of (30) or (31), wherein the reasoning sound object prediction is not grounded in the audio waveform 6, the script 7 or the one or more images 8.

[0168] (33) The method of any one of (30) to (32), wherein the reasoning sound object prediction is based on world knowledge.

[0169] (34) The method of any one of (28) to (33), wherein the method further includes performing multimodal merging 14 of at least two of the group of a trace detected from an audio waveform 6, a trace detected from a script 7, a trace detected from one or more images 8 or a reasoning trace 13 predicted based on at least one of an audio waveform 6, a script 7 or one or more images 8.

[0170] (35) The method of (34), wherein the multimodal merging 14 is performed on traces related to the same detected sound object 1.

[0171] (36) The method of any one of (28) to (35), wherein the detected sound object 1 includes a start timecode 2, 2a and an end timecode 2,2a. (37) The method of (36), wherein the timecodes 2, 2a, 2b are related to timecodes of a trace of the detected sound object 1, wherein the trace is detected in an audio wave form 6, a script 7, and / or one or more images 8.

[0172] (38) The method of any one of (36) to (37), wherein the start timecode 2, 2a and the end timecode 2, 2b are based on merged timecodes of multiple traces of the detected sound object 1, wherein the traces are traces of different modalities.

[0173] (39) The method of any one of (28) to (38), wherein a timeline of the detected sound object 1 is synchronized to a video timeline.

[0174] (40) The method of any one of (28) to (39), wherein obtaining the detected sound object is based on an artificial neural network.

[0175] (41) The method of any one of (28) to (40), wherein obtaining the detected sound object 1 is based on sound event detection.

[0176] (42) The method of any one of (28) to (41), wherein obtaining the detected sound object 1 is based on rule-based text phrase detection.

[0177] (43) The method of any one of (28) to (42), wherein obtaining the detected sound object 1 is based on a neural network based correlator of sentence embeddings to detected sound object 1 using a synchronization between a script 7 and an audio waveform 6.

[0178] (44) The method of (43), wherein the synchronization is based on ASR dialogue synchronization.

[0179] (45) The method of any one of (28) to (44), wherein obtaining the detected sound object 1 is based on object detection and / or motion tracking.

[0180] (46) The method of any one of (28) to (45), wherein the detected sound object 1 includes a detection cue 3 indicating how the detected sound object 1 was obtained.

[0181] (47) The method of (46), wherein the detection cue 3 is one of the group of an image object visible in one or more images, an audio object present in an audio waveform, a script object located in a script.

[0182] (48) The method of (46) or (47), wherein the detection cue 3 is a textual reasoning description.

[0183] (49) The method of any one of (28) to (48), wherein the method further comprises presenting the detected sound object 1 as a candidate sound object 20 for generating a realized sound object 30 including a sound signal. (50) The method of any one of (28) to (49), wherein the detected sound object 1 includes a descriptor 4 for the detected sound object 1.

[0184] (51) The method of (50), wherein the descriptor is based on a catalogue of sound object types.

[0185] (52) The method of any one of (28) to (51), wherein the detected sound object 1 includes a label 5 indicating by what generation process the audio signal the detected sound object 1 is a placeholder for shall be created.

[0186] (53) The method of any one of (28) to (52), wherein the detected sound object 1 is based on sound caused by at least one image object visible in a video.

[0187] (54) The method of any one of (28) to (53), wherein the detected sound object 1 is based on sound caused by at least one image object not visible in a video.

[0188] (55) A computer program comprising instructions which, when executed by a processor, performs the method of any one of (28) to (54).

[0189] (56) A computer program comprising program code causing a computer to perform the method according to anyone of (28) to (54), when being carried out on a computer. (22) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (28) to (54) to be performed.

Claims

CLAIMS1. An electronic device comprising circuitry configured to obtain a detected sound object based on multimodal context analysis.

2. The electronic device of claim 1, wherein the circuitry is configured to obtain the detected sound object based on a trace detected in an audio waveform, a script, and / or one or more images.

3. The electronic device of claim 1, wherein the circuitry is configured to obtain the detected sound object based on a reasoning sound object prediction.

4. The electronic device of claim 3, wherein the reasoning sound object prediction is based on a trace detected in an audio waveform, a script, and / or one or more images.

5. The electronic device of claim 3, wherein the reasoning sound object prediction is not grounded in the audio waveform, the script or the one or more images.

6. The electronic device of claim 3, wherein the reasoning sound object prediction is based on world knowledge.

7. The electronic device of claim 1, wherein the circuitry is configured to perform multimodal merging of at least two of the group of a trace detected from an audio waveform, a trace detected from a script, a trace detected from one or more images or a reasoning trace predicted based on at least one of an audio waveform, a script or one or more images.

8. The electronic device of claim 7, wherein the multimodal merging is performed on traces related to the same detected sound object.

9. The electronic device of claim 1, wherein the detected sound object includes a start timecode and an end timecode.

10. The electronic device of claim 8, wherein the timecodes are related to timecodes of a trace of the detected sound object, wherein the trace is detected in an audio wave form, a script, and / or one or more images.

11. The electronic device of claim 8, wherein the start timecode and the end timecode are based on merged timecodes of multiple traces of the detected sound object, wherein the traces are traces of different modalities.

12. The electronic device of claim 1, wherein a timeline of the detected sound object is synchronized to a video timeline.

13. The electronic device of claim 1, wherein obtaining the detected sound object is based on an artificial neural network.

14. The electronic device of claim 1, wherein obtaining the detected sound object is based on sound event detection.

15. The electronic device of claim 1, wherein obtaining the detected sound object is based on rule-based text phrase detection.

16. The electronic device of claim 1, wherein obtaining the detected sound object is based on a neural network based correlator of sentence embeddings to detected sound object using a synchronization between a script and an audio waveform.

17. The electronic device of claim 16, wherein the synchronization is based on ASR dialogue synchronization.

18. The electronic device of claim 1, wherein obtaining the detected sound object is based on object detection and / or motion tracking.

19. The electronic device of claim 1, wherein the detected sound object includes a detection cue indicating how the detected sound object was obtained.

20. The electronic device of claim 19, wherein the detection cue is one of the group of an image object visible in one or more images, an audio object present in an audio waveform, a script object located in a script.

21. The electronic device of claim 19, wherein the detection cue is a textual reasoning description.

22. The electronic device of claim 1, wherein the circuitry is further configured to present the detected sound object as a candidate sound object for generating a realized sound object including a sound signal.

23. The electronic device of claim 1, wherein the detected sound object includes a descriptor for the detected sound object.

24. The electronic device of claim 23, wherein the descriptor is based on a catalogue of sound object types.

25. The electronic device of claim 1, wherein the detected sound object includes a label indicating by what generation process the audio signal the detected sound object is a placeholder for shall be created.

26. The electronic device of claim 1, wherein the detected sound object is based on sound caused by at least one image object visible in a video.

27. The electronic device of claim 1, wherein the detected sound object is based on sound caused by at least one image object not visible in a video.

28. A method comprising obtaining a detected sound object based on multimodal context analysis.

29. A computer program comprising instructions which, when executed by a processor, performs the method of claim 28.

Images