Multimedia Content

Abstract

An apparatus includes components for: identifying an audio focus attenuation of a sound source; determining a correspondence between the sound source subject to the audio focus attenuation and a corresponding visual object; and modifying capture of an image to at least partially exclude and / or modify the visual object corresponding to the sound source subject to the audio focus attenuation.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to multimedia content, and in particular to capturing and / or processing multimedia content. Background Art

[0002] Non-professionals often record multimedia content in temporary, unplanned situations. For example, video with sound can be easily recorded on a portable camera, a camera-enabled mobile phone, or a sports action camera.

[0003] It is desirable to improve the quality of captured or recorded multimedia content. Summary of the invention

[0004] According to various but not necessarily all embodiments, there is provided an apparatus comprising means for:

[0005] Audio focus attenuation identifying the sound source;

[0006] determining a correspondence between a sound source subject to audio focus attenuation and a corresponding visual object; and

[0007] Capture of the image is modified to at least partially exclude and / or modify visual objects corresponding to sound sources subject to audio focus attenuation.

[0008] In some but not necessarily all examples, the apparatus includes components for:

[0009] determining a location of a sound source subject to audio focus attenuation;

[0010] determining a visual object associated with a location of a sound source subject to audio focus attenuation; and

[0011] The captured image is modified to exclude the determined visual object.

[0012] In some but not necessarily all examples, means for determining a location of a sound source subject to audio focus attenuation includes means for:

[0013] determining that a candidate sound source has an energy or level exceeding a threshold;

[0014] Determining that there is insufficient correlation between the target sound source and the candidate sound sources; and

[0015] If the candidate sound source has an energy or level exceeding a threshold and lacks sufficient correlation with the target sound source, the position of the candidate sound source is obtained.

[0016] Alternatively, the target sound source may be a sound source selected by the user.

[0017] For example, the localization may be performed via user selection.In some but not necessarily all examples, the target sound source is user selected and the apparatus comprises means for user selection of a visual object associated with the target sound source in the viewfinder display.

[0018] In some but not necessarily all examples, the candidate sound source can be located at any one of a plurality of locations, such as discrete locations.

[0019] In some but not necessarily all examples, candidate sound sources can be localized using spatial audio.

[0020] In some but not necessarily all examples, means for determining a visual object associated with a location of a sound source subject to audio focus attenuation includes means for object recognition.

[0021] In some but not necessarily all examples, the means for modifying the image capture is configured to reframe the captured image to exclude the visual object associated with the sound source.

[0022] In some but not necessarily all examples, the means for modifying image capture is configured to change a cropping applied to an image detected at the image sensor array, wherein the applied cropping determines the captured image.

[0023] In some but not necessarily all examples, the component for modifying image capture is configured to change the location of a crop applied to an image detected at the image sensor array.

[0024] In some but not necessarily all examples, the component for modifying image capture is configured to maintain the size of a crop applied to an image detected at the image sensor array.

[0025] In some but not necessarily all examples, the means for modifying image capture is configured to exclude all visual objects associated with a sound source subject to audio focus attenuation.

[0026] In some but not necessarily all examples, the means for modifying image capture is configured to exclude selected visual objects associated with sound sources subject to audio focus attenuation, wherein the selection of the visual objects is dependent upon one or more of:

[0027] minimizing the number of visual objects within the captured image that are associated with sound sources that are subject to audio focus attenuation;

[0028] maximizing the number of visual objects associated with sound sources subject to audio focus attenuation that are excluded from the captured image;

[0029] minimizing the total energy of sound sources that are subject to audio focus attenuation and that are associated with visual objects within the captured image;

[0030] maximizing the total energy of sound sources that are subject to audio focus attenuation and that are associated with visual objects that are excluded from the captured image;

[0031] object recognition of visual objects associated with sound sources subject to audio focus attenuation;

[0032] a proximity of a visual object associated with a sound source subject to audio focus attenuation to the device; and

[0033] The area of ​​the captured image occupied by visual objects associated with sound sources subject to audio focus attenuation is minimized.

[0034] According to various but not necessarily all embodiments, there is provided a computer program which, when run on a computer, performs:

[0035] Audio focus attenuation identifying the sound source;

[0036] determining a correspondence between a sound source subject to audio focus attenuation and a corresponding visual object; and

[0037] Capture of the image is modified to at least partially exclude and / or modify visual objects corresponding to sound sources subject to audio focus attenuation.

[0038] According to various but not necessarily all embodiments, there is provided a method comprising:

[0039] Audio focus attenuation identifying the sound source;

[0040] determining a correspondence between a sound source subject to audio focus attenuation and a corresponding visual object; and

[0041] Capture of the image is modified to at least partially exclude and / or modify visual objects corresponding to sound sources subject to audio focus attenuation.

[0042] According to various, but not necessarily all, embodiments, examples are provided as claimed in the following claims.

[0043] definition

[0044] Audio focus attenuation of a sound source applies a relative attenuation to that sound source. The level of the sound source is reduced.

[0045] Audio focus attenuation of a sound source is the application of a relative gain to that sound source. The level of the sound source is increased.

[0046] The attenuation of the audio focus of one (or more) of the multiple sound sources will inevitably lead to the gain of the audio focus of other sound sources in the multiple sound sources.

[0047] The audio focus gain of one (or more) of the multiple sound sources will inevitably lead to the audio focus attenuation of other sound sources in the multiple sound sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0049] Figure 1A and Figure 1B An example embodiment of the subject matter described herein is shown;

[0050] Figure 2A Another example embodiment of the subject matter described herein is shown;

[0051] Figure 2B Another example embodiment of the subject matter described herein is shown;

[0052] Figure 2C Another example embodiment of the subject matter described herein is shown;

[0053] Figure 3 Another example embodiment of the subject matter described herein is shown;

[0054] Figure 4A and Figure 4B Another example embodiment of the subject matter described herein is shown;

[0055] Figure 5A and Figure 5B Another example embodiment of the subject matter described herein is shown;

[0056] Fig. 6A Another example embodiment of the subject matter described herein is shown;

[0057] Figure 6B Another example embodiment of the subject matter described herein is shown;

[0058] Figure 6C Another example embodiment of the subject matter described herein is shown. DETAILED DESCRIPTION

[0059] Figure 1A and Figure 1B An example of a sound space 10 is shown which includes a plurality of sound sources 12. Sound sources 12 located at different locations are labeled with different subscripts. Figure 1A and Figure 1B Three sound sources 121 , 122 , 123 are shown at different positions relative to the origin 14 .

[0060] An audio capturing device, such as a microphone, may be placed at or near the origin 14. The audio capturing devices may, for example, be arranged in an array, such as a one-dimensional, two-dimensional or three-dimensional array.

[0061] The use of multiple audio capture devices enables the selective relative attenuation of one sound source 12 relative to another sound source 12 .

[0062] In this example, there are a plurality of sound sources 12 , including at least a target sound source 123 and one or more other non-target sound sources 121 , 122 .

[0063] The audio capture device is configured to apply relative gain to the target sound source 123 and relative attenuation to one or more other non-target sound sources 121, 122 to create an audio focus 16 for the target sound source 123. One or more non-target sound sources 122, 122 have a focus attenuation.

[0064] The target sound source 123 is located at a first position relative to the origin 14, and the one or more non-target sound sources 121, 122 are located at a second position from the origin 14, the second position being different from the first position. That is, the target sound source 123 is spatially different from the one or more non-target sound sources 121, 122.

[0065] The audio capture device may be configured to isolate the sound from the first location using beamforming. Beamforming uses a weighted linear combination of audio signals from different audio capture devices. The weighted linear combination of audio signals may be performed using hardware or software or a combination of hardware and software.

[0066] In some but not necessarily all examples, the weights of the linear combination are adaptive. For example, in some examples, they can be obtained by minimizing the energy of interfering sound sources (e.g., non-target sound sources 121, 122) under the constraint of keeping the response in the target direction towards the first position. This is a minimum variance distortion-free response.

[0067] In some but not necessarily all examples, the weights are frequency dependent.

[0068] For example, a beamformer can be used to process signals from a discrete microphone array and provide spatial filtering of the propagating sound field. Spatial filtering separates signals that have overlapping frequency content but originate from different spatial locations. Target signals from the target location (direction angle) are not attenuated, but signals from other locations (direction angles) are attenuated. The spatial filter creates a deep zero in the gain for the direction of the non-target signal. Therefore, the beamformer can operate as a spatial bandpass filter for signals from the target direction 16 and as a bandstop filter for signals from non-target directions.

[0069] The weighted linear combination of audio signals from different microphones can be expressed as y(t)=W H x(t). For a complex plane wave with direction of arrival θ, y(t) = exp(jωt). r(θ, ω) where the beamformer response r(θ, ω) = W H d(θ, ω); d(θ, ω) is the steering vector.

[0070] W H is selected such that y(t) contains a significantly larger (eg, maximum) contribution from the target sound source 123 (audio focus 16 in the target direction) and a significantly smaller (eg, minimum) contribution from the non-target sound sources 121 , 122 (non-target directions).

[0071] It can be applied across a single frequency band or separately across different frequency bands.

[0072] Weight w H This can be obtained by minimizing a cost function subject to response constraints such as a target gain in one direction and a target attenuation in another direction (e.g., enforcing zero gain), e.g., minimizing the output variance (target). This creates a minimum variance distortionless response (MVDR) beamformer that preserves the target signal (e.g., from the target sound source 123) while minimizing the contribution of non-target signals from non-target directions (e.g., from non-target sound sources 121, 122) to the output.

[0073] The weights and spatial filters can be changed in real time. For example, the steering vector d(θ, ω) can be changed in real time. For example, the weight w H Can be adjusted in real time.

[0074] Figure 1A and Figure 1B Also shown is an example of a visual space 20 including one or more visual objects 22. Visual objects 22 located at different locations are labeled with different subscripts. Figure 1A and Figure 1B Three visual objects 221 , 222 , 223 are shown at different positions relative to the origin 14 .

[0075] In this example but not necessarily all examples, each sound source 121 , 122 , 123 has a corresponding visual object 221 , 222 , 223 .

[0076] In this example, the sound space 10 corresponds to the visual space 20. The positions of the sound sources 121, 122, 123 in the sound space 10 overlap with the positions of the visual objects 221, 222, 223 in the visual space 20.

[0077] One or more image capture devices, such as cameras, may be placed at or near the origin 14. The image capture device may, for example, be arranged to capture a visual scene defined by the field of view 261. The image capture device may, for example, be a camera that captures a video including a sequence of visual scenes defined by the field of view 261.

[0078] In this example, visual objects 221, 222 correspond to attenuated sound sources 121, 122, and visual object 223 corresponds to non-attenuated sound source 123. The attenuated sound source may be, for example, a silent sound source.

[0079] from Figure 1B It can be seen that the visual object 221 corresponding to the attenuated sound source 121 is not within the field of view 261 , and therefore is not within the visual scene captured by the camera at the origin 14 .

[0080] from Figure 1B It can be seen that the visual object 223 corresponding to the non-attenuated sound source 123 is within the field of view 261 , and therefore within the visual scene captured by the camera at the origin 14 .

[0081] from Figure 1B It can be seen that the visual object 222 corresponding to the attenuated sound source 122 is within the field of view 261 , and therefore within the visual scene captured by the camera at the origin 14 .

[0082] In this example, visual object 222 corresponding to attenuated sound source 122 within field of view 261 may be a distraction to visual object 223 corresponding to non-attenuated sound source 123. This is particularly true when visual object 222 corresponding to attenuated sound source 122 within field of view 261 occupies a larger portion of the visual scene than visual object 223, or when visual object 223 is a background object and visual object 222 is a foreground object, or visual object 222 is a dynamic visual object (a visual object that moves position and / or changes).

[0083] Figure 2A An example of a visual scene is shown. Visual object 222 corresponds to attenuated sound source 122. The location of visual object 222 and its corresponding attenuated sound source 122 is within field of view 261. Visual object 222 can be seen in the captured image. Visual object 223 corresponds to non-attenuated sound source 123. The location of visual object 223 and its corresponding non-attenuated sound source 123 is within field of view 261. Visual object 223 can be seen in the captured image. The audio captured from non-attenuated sound source 123 is most prominent auditorily, while the larger-sized foreground visual object 222 is most prominent visually. The larger-sized foreground visual object 222 visually distracts attention from visual object 223 corresponding to non-attenuated sound source 123.

[0084] The following method 100( Figure 3 ) reduces distraction.

[0085] The method 100 includes:

[0086] At block 102, an audio focus attenuation of a sound source 122 is identified;

[0087] Step 104, determining a correspondence between a sound source 122 that is subject to audio focus attenuation and a corresponding visual object 222; and

[0088] At block 106 , the capture of the image is modified to at least partially exclude and / or modify the visual object 222 corresponding to the sound source 122 that is subject to the audio focus attenuation.

[0089] In some, but not necessarily all, examples, at block 102, the method may include automatically identifying an audio focus fade of a sound source 122. In some, but not necessarily all, examples, at block 104, the method may include automatically determining a correspondence between a sound source 122 that is subject to an audio focus fade and a corresponding visual object 222. In some, but not necessarily all, examples, at block 106, the method may include automatically modifying capture of an image to at least partially exclude and / or modify a visual object 222 that corresponds to a sound source 122 that is subject to an audio focus fade.

[0090] The method reduces the visual impact on the captured image of the visual object 222 corresponding to the sound source 122 that is subject to audio focus attenuation.

[0091] An example of the results of method 100 is shown in Figure 2B The method modifies the capture of an image to at least partially modify a visual object 222 corresponding to a sound source 122 that is subject to audio focus attenuation. The unmodified visual object 222 is Figure 2A The visual object 222′ shown in FIG. 1 and modified in FIG. Figure 2B . In this example, the unmodified visual object 222 (not the visual object 223) is subjected to a blur kernel of, for example, 50 pixels to produce a modified visual object 222'. In practice, the visual object 222 corresponding to the sound source 122 subjected to audio focus attenuation is visually attenuated, for example, by blurring, coloring, etc.

[0092] An example of the results of method 100 is shown in Figure 2C The method modifies the capture of the image to reframe the visual scene and at least partially exclude the visual object 222 corresponding to the sound source 122 that is subject to audio focus attenuation. The unmodified visual object 222 is shown in FIG. Figure 2A The modified visual object 222′ shown and at least partially excluded in Figure 2C. In this example, the unmodified visual object 222 (but not the visual object 223) is at least partially excluded from the captured image. This can be achieved, for example, by effectively reframing the image captured by panning or tracking and / or zooming. This can be achieved by changing the size and / or direction of the field of view 26.

[0093] Figure 4A Similar to Figure 1B . It shows that the visual object 223 corresponding to the non-attenuated sound source 123 is within the field of view 261, and therefore within the visual scene captured by the camera at the origin 14, and the visual object 222 corresponding to the attenuated sound source 122 is within the field of view 261, and therefore within the visual scene captured by the camera at the origin 14.

[0094] Figure 4B Similar to Figure 1B However, the field of view 261 has been modified to a new field of view, namely, field of view 261 ′. This reframes the captured image except for all or part of the visual object 222 corresponding to the attenuated sound source 122 .

[0095] therefore, Figure 4B It is shown that the visual object 223 corresponding to the non-attenuated sound source 123 is within the new field of view 261' and therefore within the visual scene captured by the camera at the origin 14, and the visual object 222 corresponding to the attenuated sound source 122 is at least partially excluded from the new field of view 261' and therefore no longer within the visual scene captured by the camera at the origin 14.

[0096] In some examples, reframing occurs automatically. In other examples, the user is given a user-selectable option for causing reframing.

[0097] In this example, the re-framing maintains the same resolution and is effectively a pan or track. In other examples, the re-framing additionally or alternatively changes the resolution and is effectively a zoom.

[0098] exist Figure 4A and Figure 4B In the example of , a second larger field of view 262 is shown. The large field of view 262 shows the visual scene imaged by the image sensor of the camera. The smaller fields of view 261, 261' show the visual scene captured by the image sensor. The smaller fields of view 261, 261' are sub-parts of the second larger field of view 262. Figure 4B The framing shown is achieved by cropping the large field of view image to achieve a smaller field of view image. Cropping can be done in Figure 4A and Figure 4B Change the size and / or position between . Cropping can be done in Figure 4A and Figure 4BIn this example, the capture of the modified image is configured to change a crop applied to an image detected at the image sensor array, wherein the applied crop determines the captured image. The crop defines a contiguous subset of sensors of the sensor array. The position of the crop applied to the image detected at the image sensor array is changed. The size of the crop applied to the image detected at the image sensor array may be maintained or changed. The effect of the re-cropping is Figure 5A and Figure 5B Shown in. Figure 5A An image detected at the image sensor array (field of view 262 ) and a currently applied crop (field of view 261 , which includes visual object 222 corresponding to attenuated sound source 122 ) are shown. Figure 5B The captured image is shown as defined by a new crop (field of view 261 ) newly applied to the image detected at the image sensor array (field of view 262) that excludes the visual object 222 corresponding to the attenuated sound source 122. In this example, but not necessarily all examples, the captured image retains a portion of the excluded visual object 222 corresponding to the attenuated sound source 122 (modified visual object 222').

[0099] Method 100( Figure 3 ) includes: At block 102, identifying the audio focus attenuation of the sound source 122. For example, the block may include:

[0100] Get the audio focus direction, such as the beamforming steering vector d(θ, ω);

[0101] Get the direction of the sound source;

[0102] One or more sound sources that are attenuated (eg, not in the direction of the audio focus) are determined.

[0103] The method 100 includes, at block 104 , determining a correspondence between a sound source 12 that is subject to audio focus attenuation and a corresponding visual object 22 .

[0104] This block 104 may include:

[0105] determining a location of a sound source 12 that is subject to audio focus attenuation;

[0106] determining a visual object 22 associated with a location of a sound source 12 subject to audio focus attenuation; and

[0107] The captured image is modified to exclude or modify the determined visual object 22 .

[0108] A visual object 22 is associated with a location if the visual object 22 is at or near the location of a sound source 12 that is subject to audio focus attenuation, for example, if the visual object 22 (or any portion of the visual object 22) is within a threshold distance of the location.

[0109] In some but not necessarily all examples, an image is captured prior to block 104. The captured image is then modified at block 106. In some but not necessarily all examples, the captured image is processed to identify a corresponding visual object 222.

[0110] Computer vision techniques may be used to identify the perimeter of a visual object 22 associated with a location, or otherwise recognize or characterize a visual object 22 associated with the location. Computer vision techniques may include, for example, edge or corner detection, interest point detection, and feature detection.

[0111] For example, the position can be a one-dimensional direction angle (e.g., azimuth ), two-dimensional direction angles (e.g., azimuth and polar angle θ), two-dimensional coordinates (e.g., Cartesian coordinates (x, y) or polar coordinates (r, φ)), or three-dimensional coordinates (e.g., Cartesian coordinates (x, y, z) or cylindrical polar coordinates (r, φ, z) or spherical polar coordinates (r, φ, θ)).

[0112] Modifying the captured image to modify the determined visual object 22 may, for example, include processing the captured image, for example referring to Figure 2B described.

[0113] For example, modifying the captured image to modify the determined visual object 22 may include reframing or cropping the captured image, such as with reference to Figure 2C or Figure 4A , Figure 4B , Figure 5A , Figure 5B described.

[0114] Since the candidate sound source 12 has an energy or level exceeding the threshold and there is a lack of sufficient correlation between the target sound source 123 and the candidate sound source 12 (the correlation score is below the threshold), the candidate sound source (which may or may not be the sound source 12 undergoing audio focus attenuation) can be identified as the sound source 12 undergoing audio focus attenuation.

[0115] Determining the location of a sound source 12 that is subject to audio focus attenuation may include:

[0116] Determining that the candidate sound source 12 has an energy or level exceeding a threshold;

[0117] Determining that there is insufficient correlation between the target sound source 123 and the candidate sound source 12; and

[0118] If the candidate sound source 12 has an energy or level exceeding a threshold value and lacks sufficient correlation with the target sound source 123 , the position of the candidate sound source 12 is obtained.

[0119] The threshold may be zero or may be a non-zero threshold.

[0120] In some examples, the candidate sound source 12 may be located at any one of a plurality of discrete locations, for example, defined by quantized values ​​of the beamforming steering vector d(θ, ω).

[0121] The candidate sound sources 12 can be localized using spatial audio, for example using a beamformer as described above; or from metadata associated with the candidate sound objects.

[0122] In some but not necessarily all examples, the target sound source 123 may be user selectable.The apparatus may, for example, comprise means for making a user selection in a viewfinder display.

[0123] Fig. 6A An example of an apparatus 50 is shown.

[0124] The apparatus 50 is an audio capturing apparatus and comprises an audio input circuitry 60 for inputting audio. The audio input circuitry comprises a plurality of audio capturing devices 62. In this example, the plurality of audio capturing devices 62 are microphones.

[0125] The microphones are distributed in space to form a discrete microphone array. For example, the array can be 1D, 2D or 3D.

[0126] The apparatus 50 is an image capturing apparatus and comprises an image capturing device 70. The image capturing device 70 may be a camera or an image sensor.

[0127] For example, it may be a camera 72 .

[0128] Controller 80 is configured to receive audio data from audio input circuitry 60 and visual data from image capture device 70. The data may be raw, or some pre-processing may occur at audio input circuitry 60 and / or image capture device 70.

[0129] In this example, the device 50 is a beamformer device. It is configured to spatially filter audio data from the audio input circuitry 60 to provide audio focus as previously described. The audio focus attenuates sounds from unwanted directions.

[0130] The controller 80 is also configured to:

[0131] Identify the audio focus attenuation of the sound source 12;

[0132] determining a correspondence between a sound source 12 subject to audio focus attenuation and a corresponding visual object 22; and

[0133] The capture of the image is modified to at least partially exclude and / or modify the visual object 22 corresponding to the sound source 12 that is subject to the audio focus attenuation.

[0134] Device 50 is a rendering device and includes a user output circuit system 40 for rendering images and audio simultaneously as multimedia content. In this example, user output circuit system 40 includes audio output circuit system 42, which may include one or more speakers, headphones, or interfaces to speakers or headphones. In this example, user output circuit system 40 includes image output circuit system, which may include, for example, one or more displays 44.

[0135] The apparatus 50 may be configured to display on the display 44 an image prior to the exclusion / modification of the visual object corresponding to the sound source subject to the audio focus attenuation.

[0136] The apparatus 50 may be configured to display on the display 44 an image after exclusion / modification of visual objects corresponding to sound sources subject to audio focus attenuation.

[0137] Apparatus 50 may be configured to display on display 44 a captured image that at least partially excludes and / or modifies visual objects corresponding to sound sources subject to audio focus attenuation.

[0138] In some examples, the apparatus 50 may include a user input circuit system 90 configured to enable a user to provide control commands to the apparatus 50. In some, but not necessarily all examples, the user input circuit system 90 may be integrated with the display 44 to create a touch screen display. In some, but not necessarily all examples, the display 44 or the touch screen display provides a camera viewfinder for the camera 72. The user input circuit system 90 may enable a user to identify a sound source 12 and / or a visual object 22 associated with the sound source 12.

[0139] The device 50 may be a portable or handheld device, for example, a camera or a mobile phone with a camera function.

[0140] Figure 6B An example of a controller 80 is shown. The controller 80 may be implemented as controller circuitry. The controller 80 may be implemented solely in hardware, with certain aspects of software including only firmware, or may be a combination of hardware and software (including firmware).

[0141] like Figure 6BAs shown, the controller 80 may be implemented using instructions that enable hardware functionality, for example, by using executable instructions of a computer program 86 that may be stored on a computer-readable storage medium (disk, memory, etc.) in a general or special purpose processor 82 for execution by such processor 82 .

[0142] The processor 82 is configured to read from and write to the memory 84. The processor 82 may also include an output interface via which the processor 82 outputs data and / or commands and an input interface via which data and / or commands are input to the processor 82.

[0143] The memory 84 stores a computer program 86 comprising computer program instructions (computer program code) which, when loaded into the processor 82, controls the operation of the apparatus 50. The computer program instructions of the computer program 86 provide the apparatus with the ability to perform Figure 3 The logic and routines of the method shown. By reading the memory 84, the processor 82 is able to load and execute the computer program 86.

[0144] The device 50 thus comprises:

[0145] at least one processor 82; and

[0146] at least one memory 84 including computer program code,

[0147] The at least one memory 84 and the computer program code are configured to, together with the at least one processor 82, cause the apparatus 50 to at least execute:

[0148] Audio focus attenuation identifying the sound source;

[0149] determining a correspondence between a sound source subject to audio focus attenuation and a corresponding visual object; and

[0150] Capture of the image is modified to at least partially exclude and / or modify visual objects corresponding to sound sources subject to audio focus attenuation.

[0151] like Figure 6B As shown, the computer program 86 may arrive at the apparatus 50 via any suitable delivery mechanism 88. For example, the delivery mechanism 88 may be a machine-readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a memory device, a recording medium (such as a compact disc read-only memory (CD-ROM) or a digital versatile disc (DVD) or a solid-state memory), or an article of manufacture that includes or tangibly embodies the computer program 86. The delivery mechanism may be a signal configured to reliably transmit the computer program 86. The apparatus 50 may propagate or transmit the computer program 86 as a computer data signal.

[0152] The computer program instructions are used to cause the apparatus to at least perform the following operations or to perform at least the following operations:

[0153] Audio focus attenuation identifying the sound source;

[0154] determining a correspondence between a sound source subject to audio focus attenuation and a corresponding visual object; and

[0155] Capture of the image is modified to at least partially exclude and / or modify visual objects corresponding to sound sources subject to audio focus attenuation.

[0156] The computer program instructions may be included in a computer program, a non-transitory computer readable medium, a computer program product, a machine readable medium. In some but not necessarily all examples, the computer program instructions may be distributed over more than one computer program.

[0157] Although memory 84 is shown as a single component / circuitry, it may be implemented as one or more separate components / circuitry, some or all of which may be integrated / removable and / or may provide permanent / semi-permanent / dynamic / cached storage.

[0158] Although processor 82 is shown as a single component / circuitry, it may be implemented as one or more separate components / circuitry, some or all of which may be integrated / removable.Processor 82 may be a single-core or multi-core processor.

[0159] The above example can be extended to the case where there are multiple audio focus directions.

[0160] The above example can be extended to the case where there are multiple sound sources that are subject to audio focus fade and correspond to visual objects in the candidate captured image.

[0161] Method 100 is performed for each of these sound sources.

[0162] The method may include:

[0163] At block 102, an audio focus attenuation of a plurality of sound sources 12 is identified;

[0164] At block 104, determining a correspondence between a plurality of sound sources 12 subject to audio focus attenuation and a corresponding plurality of visual objects 22 present in an image to be captured; and

[0165] At block 106 , the capture of the image is modified to at least partially exclude and / or modify at least some of the respective plurality of visual objects 22 corresponding to the plurality of sound sources 12 that are subject to audio focus attenuation.

[0166] In some but not necessarily all examples, the method completely or partially excludes and / or modifies all visual objects 22 associated with the sound source 12 that is subject to audio focus attenuation that would otherwise be within the captured image. As a result, the attenuated sound source 12 is not visible.

[0167] In some but not necessarily all examples, the capture of the modified image is configured to exclude selected visual objects 22 associated with the sound source 12 that is subject to audio focus attenuation, wherein the selection of the visual objects 22 is dependent on one or more of the following:

[0168] minimizing the number of visual objects 22 associated with sound sources 12 subject to audio focus attenuation within the captured image;

[0169] maximizing the number of visual objects 22 associated with sound sources 12 subject to audio focus attenuation that are excluded from the captured image;

[0170] minimizing the total energy of sound sources 12 that are subject to audio focus attenuation and that are associated with visual objects 22 within the captured image;

[0171] maximizing the total energy of sound sources 12 that are subject to audio focus attenuation and that are associated with visual objects 22 that are excluded from the captured image;

[0172] object identification of a visual object 22 associated with a sound source 12 subject to audio focus attenuation;

[0173] the proximity of the visual object 22 associated with the sound source 12 subject to the audio focus attenuation to the device 50;

[0174] movement or change of a visual object 22 associated with a sound source subject to audio focus attenuation;

[0175] The area of ​​the captured image occupied by the visual object 22 associated with the sound source 12 subject to the audio focus attenuation is minimized.

[0176] References to "computer-readable storage medium", "computer program product", "tangibly embodied computer program", etc. or "controller", "computer", "processor", etc. should be understood to cover not only computers with different architectures (such as single / multi-processor architectures and sequential (Von Neumann) / parallel architectures), but also special-purpose circuits such as field programmable gate arrays (FPGAs), application-specific circuits (ASICs), signal processing devices and other processing circuit systems. References to computer programs, instructions, code, etc. should be understood to cover software for programmable processors or firmware, such as the programmable content of hardware devices, whether instructions for a processor or configuration settings for a fixed-function device, gate array or programmable logic device, etc.

[0177] As used in this application, the term "circuitry" may refer to one or more or all of the following:

[0178] (a) a pure hardware circuitry implementation (such as an implementation solely in analog and / or digital circuitry), and

[0179] (b) a combination of hardware circuitry and software such as (where applicable):

[0180] (i) a combination of analog and / or digital hardware circuits and software / firmware, and

[0181] (ii) any portion of hardware processor(s) (including digital signal processor(s)), software and memory(s) with software that work together to cause a device (such as a mobile phone or server) to perform various functions, and

[0182] (c) Hardware circuit(s) and / or processor(s), such as microprocessor(s) or portion(s) of microprocessor(s), that require software (e.g., firmware) to operate, but the software may not be present when it is not required for operation.

[0183] This definition of circuitry applies to all uses of the term in this application, including in any claims. As another example, as used in this application, the term circuitry also covers an implementation of hardware-only circuits or processors and their (or their) accompanying software and / or firmware. For example, if applicable to a particular claim element, the term circuitry also covers a baseband integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0184] Figure 3 The blocks shown in the figure can represent steps in the method and / or code segments in the computer program 86. The description of the specific order of the blocks does not necessarily mean that these blocks have a required or preferred order, and the order and arrangement of the blocks can be changed. In addition, some blocks can be omitted.

[0185] Where structural features have been described, the structural features may be replaced by means of components that perform one or more of the functions of the structural features, whether or not the function or functions are explicitly or implicitly described.

[0186] In some but not necessarily all examples, device 50 is configured to transmit data from device 50 with or without local storage of the data in memory 84 at device 50, and with or without local processing of the data by circuitry or a processor at device 50.

[0187] The data can be stored remotely on one or more devices in a processed or unprocessed format. The data can be stored in the cloud.

[0188] The data may be processed remotely at one or more devices. The data may be processed partially locally and partially remotely at one or more devices.

[0189] For example, data may be transmitted wirelessly to a remote device via short range radio communications such as Wi-Fi or Bluetooth, or over a long range cellular radio link.The apparatus may comprise a communication interface, such as a radio transceiver for data communications.

[0190] The device 50 may be part of an Internet of Things forming part of a larger distributed network.

[0191] Processing of data (whether local or remote) can be used for health monitoring, data aggregation, patient monitoring, vital signs monitoring, or other purposes.

[0192] The processing of data (whether local or remote) may involve artificial intelligence or machine learning algorithms. For example, the data may be used as learning input to train a machine learning network, or may be used as a query input to a machine learning network, which provides a response. The machine learning network may, for example, use linear regression, logistic regression, vector support machines, or a non-recurrent machine learning network, such as a single or multiple hidden layer neural network.

[0193] Processing of the data (whether local or remote) may generate an output. The output may be transmitted to the device 50 where it may generate an object-sensitive output, such as an audio output, a visual output, or a tactile output.

[0194] The recording of data may include only temporary recording, or may include permanent recording, or may include both temporary and permanent recording. Temporary recording means temporarily recording data. This may occur, for example, during sensing or image capture, at a dynamic memory, at a buffer such as a circular buffer, register, cache, etc. Permanent recording means that the data is in the form of an addressable data structure that is retrievable from an addressable storage space and can therefore be stored and retrieved until deleted or overwritten, although long-term storage may or may not occur. Use of the term "capture" in relation to images refers to temporary recording of image data. Use of the term "storage" in relation to images refers to permanent recording of image data.

[0195] In the above examples, the captured image may also be a recorded image.

[0196] As used herein, a "module" refers to a unit or device that does not include certain parts / components added by the final manufacturer or user.

[0197] For example, the apparatus 50 may be physically separated into one or more of the following: audio input circuitry 60, image capture device 70, controller 80, and user output circuitry 40, which may be separated in time and / or space. The functionality of the apparatus 50 may be implemented using multiple different but functionally coupled devices.

[0198] The above example can be used as an enabling component for:

[0199] Automotive systems; telecommunications systems; electronic systems, including consumer electronics; distributed computing systems; media systems for generating or rendering media content, including audio, visual and audiovisual content and mixed, mediated, virtual and / or augmented reality; personal systems, including personal health systems or personal fitness systems; navigation systems; user interfaces, also known as human-machine interfaces; networks, including cellular, non-cellular and optical networks; ad hoc networks; the Internet; the Internet of Things; virtualized networks; and related software and services.

[0200] The term "comprising" as used in this document is inclusive rather than exclusive. That is, any reference to X including Y means that X may include only one Y or may include more than one Y. If the exclusive meaning of "comprising" is intended, it will be clearly stated in the context by referring to "including only one..." or using "consisting of..."

[0201] In this description, reference is made to various examples. The description of features or functions associated with an example indicates that these features or functions exist in the example. The use of the term "example" or "for example" or "may" or "can" in the text indicates that, whether explicitly stated or not, these features or functions are at least present in the described example, whether described as an example or not, and they may but not necessarily exist in some or all other examples. Therefore, "example", "for example", "may" or "can" refers to a specific instance in a class of examples. The attributes of an instance may be attributes of only the instance or attributes of the class or attributes of a subclass of the class including some but not all instances in the class. Therefore, it is implicitly disclosed that the features described with reference to one example rather than with reference to another example can be used as part of a working combination in the other example where possible, but not necessarily for the other example.

[0202] Although the embodiments have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the claims.

[0203] Features described in the preceding description may be used in other combinations than the combinations explicitly described above.

[0204] Although functions have been described with reference to certain features, those functions may be performable by other features whether described or not.

[0205] Although features have been described with reference to certain embodiments, those features may also be present in other embodiments whether described or not.

[0206] The terms "a" or "the" as used in this document are inclusive and not exclusive. That is, any reference to X that includes an a / the Y indicates that X may include only one Y or may include more than one Y, unless the context clearly indicates otherwise. If the exclusive meaning of "a" or "the" is intended, it will be clearly stated in the context. In some cases, "at least one" or "one or more" may be used to emphasize the inclusive meaning, but the absence of these terms should not be regarded as inferring and exclusive meaning.

[0207] The presence of a feature (or combination of features) in a claim is a reference to the feature or (combination of features) itself, as well as a reference to features that achieve substantially the same technical effect (equivalent features). Equivalent features include, for example, features that are variants and achieve substantially the same result in substantially the same way. Equivalent features include, for example, features that perform substantially the same function in substantially the same way to achieve substantially the same result.

[0208] In this specification, reference is made to various examples, using adjectives or adjective phrases to describe characteristics of the examples. Such description of characteristics associated with the examples indicates that the characteristics exist exactly as described in some examples and exist substantially as described in other examples.

[0209] While attention has been drawn in the foregoing description to such features as are regarded as important, it should be understood that the applicant may seek protection by way of claims for any patentable feature or combination of features mentioned above and / or shown in the drawings, whether emphasized or not.

Claims

1. An apparatus comprising components for: determining that a sound source has an energy or level exceeding a first threshold; determining that the correlation between the target sound source and the sound source is lower than a second threshold; In response to the energy or level of the sound source exceeding the first threshold and the correlation between the target sound source and the sound source being below the second threshold, identifying an audio focus attenuation of the sound source; determining a correspondence between the sound source subjected to audio focus attenuation and a corresponding visual object; as well as Capture of an image is modified to at least partially exclude and / or modify the visual object corresponding to the sound source subject to audio focus attenuation.

2. The apparatus according to claim 1, comprising means for: determining a location of the sound source subject to audio focus attenuation; determining a visual object associated with the location of the sound source subject to audio focus attenuation; and The captured image is modified to exclude the determined visual object.

3. The apparatus of claim 2, wherein the means for determining the location of the sound source subject to audio focus attenuation comprises means for: determining that a candidate sound source has an energy or level exceeding a threshold; Determining that there is a lack of sufficient correlation between the target sound source and the candidate sound source; as well as If the candidate sound source has energy or level exceeding the threshold and lacks sufficient correlation with the target sound source, the position of the candidate sound source is obtained. The apparatus according to claim 3 , wherein the candidate sound source can be located at any one of a plurality of locations.

5. An apparatus according to claim 3 or 4, wherein the candidate sound source can be localized using spatial audio.

6. An apparatus as claimed in claim 3 or 4, wherein the target sound source is user selected, the apparatus comprising means for user selection of a visual object associated with the target sound source in a viewfinder display.

7. The apparatus of claim 2, wherein the means for determining a visual object associated with the location of the sound source subject to audio focus attenuation comprises means for object recognition.

8. The apparatus of claim 1, wherein the means for modifying image capture is configured to reframe the captured image to exclude the visual object associated with the sound source.

9. The apparatus of claim 1, wherein the means for modifying image capture is configured to change a cropping applied to an image detected at the image sensor array, wherein the applied cropping determines the captured image.

10. The apparatus of claim 9, wherein the means for modifying image capture is configured to change a position of the crop applied to the image detected at the image sensor array.

11. The apparatus of claim 10, wherein the means for modifying image capture is configured to maintain a size of the crop applied to the image detected at the image sensor array.

12. The apparatus of claim 1, wherein the means for modifying image capture is configured to exclude all visual objects associated with sound sources subject to audio focus attenuation.

13. The apparatus of claim 1 , wherein the means for modifying image capture is configured to exclude selected visual objects associated with sound sources subject to audio focus attenuation, wherein the selection of the visual objects is dependent upon one or more of: minimizing the number of visual objects within the captured image that are associated with sound sources that are subject to audio focus attenuation; maximizing the number of visual objects associated with sound sources subject to audio focus attenuation that are excluded from the captured image; minimizing the total energy of sound sources that are subject to audio focus attenuation and that are associated with visual objects within the captured image; maximizing the total energy of sound sources that are subject to audio focus attenuation and that are associated with visual objects that are excluded from the captured image; object recognition of visual objects associated with sound sources subject to audio focus attenuation; a proximity of a visual object associated with a sound source subject to audio focus attenuation to the device; as well as An area of ​​the captured image occupied by visual objects associated with sound sources subject to audio focus attenuation is minimized.

14. A computer program product which, when run on a computer, performs: determining that a sound source has an energy or level exceeding a first threshold; determining that the correlation between the target sound source and the sound source is lower than a second threshold; In response to the energy or level of the sound source exceeding the first threshold and the correlation between the target sound source and the sound source being below the second threshold, identifying an audio focus attenuation of the sound source; determining a correspondence between the sound source subjected to audio focus attenuation and a corresponding visual object; as well as Capture of an image is modified to at least partially exclude and / or modify the visual object corresponding to the sound source subject to audio focus attenuation.

15. A method comprising: determining that a sound source has an energy or level exceeding a first threshold; determining that the correlation between the target sound source and the sound source is lower than a second threshold; In response to the energy or level of the sound source exceeding the first threshold and the correlation between the target sound source and the sound source being below the second threshold, identifying an audio focus attenuation of the sound source; determining a correspondence between the sound source subjected to audio focus attenuation and a corresponding visual object; as well as Capture of an image is modified to at least partially exclude and / or modify the visual object corresponding to the sound source subject to audio focus attenuation.

Images