Apparatus, computer program and method for processing multimedia content - Patents.com

By identifying and modifying image capture to exclude or attenuate distracting visual objects associated with audio focus attenuation, the apparatus enhances the quality of multimedia content captured in ad hoc scenarios.

JP7581359B2Active Publication Date: 2024-11-12NOKIA TECHNOLOGIES OY
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2022548695
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-02-14
Filing Date
2021-02-02
Publication Date
2024-11-12
Estimated Expiration
2041-02-02

AI Technical Summary

Technical Problem

Multimedia content captured by non-experts in ad hoc scenarios often suffers from poor quality due to audio focus attenuation, which distracts from the visual objects of interest.

Method used

An apparatus and method that identify audio focus attenuation of sound sources, determine corresponding visual objects, and modify image capture to exclude or attenuate these distracting visual objects, using techniques such as beamforming and image processing to enhance the quality of multimedia content.

Benefits of technology

The solution effectively reduces the visual impact of sound sources undergoing audio focus attenuation, improving the overall quality and focus on relevant visual objects in captured images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581359000001
    Figure 0007581359000001
  • Figure 0007581359000002
    Figure 0007581359000002
  • Figure 0007581359000003
    Figure 0007581359000003
Patent Text Reader

Abstract

Identify the audio focus attenuation of the sound source, determining a correspondence between a sound source subjected to audio focus attenuation and a corresponding visual object; Modifying the image capture to at least partially exclude and / or modify visual objects corresponding to sound sources that are subject to audio focus attenuation. An apparatus comprising means for:
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate to multimedia content, and in particular to capturing and / or processing multimedia content. [Background technology]

[0002] Multimedia content is often recorded by non-professionals in ad-hoc and unplanned scenarios. For example, video with audio may be easily recorded on a portable camera, a camera-enabled mobile phone, or a sports activity camera. Summary of the Invention [Problem to be solved by the invention]

[0003] It would be desirable to improve the quality of captured or recorded multimedia content. [Means for solving the problem]

[0004] According to various, but not necessarily all, embodiments: Identify the audio focus attenuation of the sound source, determining a correspondence between a sound source subject to audio focus attenuation and a corresponding visual object; Modifying the image capture to at least partially exclude and / or modify visual objects corresponding to sound sources that are subject to audio focus attenuation. An apparatus is provided that includes means for:

[0005] In some, but not necessarily all, embodiments, the apparatus includes: determining a location of a sound source that is to be subjected to audio focus attenuation; determining a visual object associated with a location of the sound source that is subject to audio focus attenuation; Modifying the captured image to exclude the determined visual object The measures to be taken are as follows.

[0006] In some, but not necessarily all, embodiments, the means for determining a location of a sound source that is subject to audio focus attenuation comprises: determining that the candidate sound source has an energy or level above a threshold; Determining a lack of sufficient correlation between the target sound source and the candidate sound sources; Obtaining the location of a candidate sound source if the candidate sound source has an energy or level above a threshold and lacks sufficient correlation with the target sound source The measures to be taken are as follows.

[0007] The target sound source may instead be a user-selected sound source.

[0008] Targeting may be performed, for example, via user selection: in some, but not necessarily all, embodiments, the target sound source is user selected and the device comprises means for user selection within the viewfinder display of a visual object associated with the target sound source.

[0009] In some, but not necessarily all, embodiments, a candidate source may be located in any one of multiple locations, eg, separate locations.

[0010] In some, but not necessarily all, embodiments, candidate sound sources may be located using spatial audio.

[0011] In some, but not necessarily all, embodiments, the means for determining a visual object associated with a position of a sound source subjected to audio focus attenuation comprises means for object recognition.

[0012] In some, but not necessarily all, embodiments, the means for modifying the image capture is configured to reframe the captured image to exclude visual objects associated with the sound source.

[0013] In some, but not necessarily all, embodiments, the means for modifying the image capture is configured to modify a crop applied to the image detected by the image sensor array, the crop applied determining the image that is captured.

[0014] In some, but not necessarily all, embodiments, the means for modifying the image capture is configured to change the position of a crop applied to the image detected by the image sensor array.

[0015] In some, but not necessarily all, embodiments, the means for modifying the image capture is configured to maintain a size of the crop applied to the image detected by the image sensor array.

[0016] In some, but not necessarily all, embodiments, the means for modifying the image capture is configured to exclude all visual objects associated with the sound source that is subject to audio focus attenuation.

[0017] In some, but not necessarily all, embodiments, the means for modifying the image capture is configured to exclude selected visual objects associated with the sound source that is subject to audio focus attenuation, the selection of visual objects being: Minimizing the number of visual objects in the captured image that are associated with the sound source that is subject to audio focus attenuation; maximizing the number of visual objects associated with the sound source subject to audio focus attenuation that are excluded from the captured image; Minimizing the total energy of a sound source that is subject to audio focus attenuation and associated with a visual object in a captured image; maximizing the total energy of a sound source associated with a visual object that is subject to audio focus attenuation and excluded from the captured image; Object recognition of visual objects associated with a sound source that is subject to audio focus attenuation; the proximity to the device of visual objects associated with the sound source that is subject to audio focus attenuation; and Minimizing the area of ​​a captured image occupied by visual objects associated with a sound source that is subject to audio focus attenuation It depends on one or more of:

[0018] According to various, but not necessarily all, embodiments, when executed on a computer: Identifying an audio focus attenuation of a sound source; determining a correspondence between a sound source subject to audio focus attenuation and a corresponding visual object; Modifying the image capture to at least partially exclude and / or modify visual objects corresponding to sound sources that are subject to audio focus attenuation; A computer program product is provided that implements the above.

[0019] According to various, but not necessarily all, embodiments: Identifying an audio focus attenuation of a sound source; determining a correspondence between a sound source subject to audio focus attenuation and a corresponding visual object; Modifying the image capture to at least partially exclude and / or modify visual objects corresponding to sound sources that are subject to audio focus attenuation; A method is provided that includes:

[0020] In accordance with various, but not necessarily all, embodiments, implementations are provided that are as claimed in the accompanying claims.

[0021] definition Audio focus attenuation of a sound source is the application of a relative attenuation to that sound source. The level of the sound source is reduced.

[0022] Audio focus attenuation of a sound source is the application of relative gain to that sound source: the level of the sound source is increased.

[0023] An audio focus attenuation of one (or more) of the multiple sound sources necessarily results in an audio focus gain of the other source(s) of the multiple sound sources.

[0024] An audio focus gain for one (or more) of the multiple sound sources necessarily results in an audio focus attenuation for the other sources of the multiple sound sources.

[0025] Several exemplary embodiments will now be described with reference to the accompanying drawings. [Brief description of the drawings]

[0026] [Figure 1A] FIG. 1 illustrates an exemplary embodiment of the subject matter described herein. [Figure 1B] FIG. 1 illustrates an exemplary embodiment of the subject matter described herein. [Figure 2A] FIG. 2 illustrates another example embodiment of the subject matter described herein. [Figure 2B] FIG. 2 illustrates another example embodiment of the subject matter described herein. [Figure 2C] FIG. 2 illustrates another example embodiment of the subject matter described herein. [Diagram 3] FIG. 2 illustrates another example embodiment of the subject matter described herein. [Figure 4A] FIG. 1 illustrates an exemplary embodiment of the subject matter described herein. [Figure 4B] FIG. 1 illustrates an exemplary embodiment of the subject matter described herein. [Figure 5A] FIG. 1 illustrates an exemplary embodiment of the subject matter described herein. [Figure 5B] FIG. 1 illustrates an exemplary embodiment of the subject matter described herein. [Figure 6A]FIG. 1 illustrates an exemplary embodiment of the subject matter described herein. [Figure 6B] FIG. 1 illustrates an exemplary embodiment of the subject matter described herein. [Figure 6C] FIG. 1 illustrates an exemplary embodiment of the subject matter described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] 1A and 1B show an example of an acoustic space 10 that includes multiple sound sources 12. Sound sources 12 at different positions are labeled with different subscripts. 1A and 1B show three sound sources 121, 122, 123 at different positions relative to an origin 14.

[0028] An audio capturing device, such as a microphone, may be placed at or near the origin 14. For example, the audio capturing device may be configured as an array, for example a one-dimensional, two-dimensional, or three-dimensional array.

[0029] The use of multiple sound capture devices allows for selective relative attenuation of one sound source 12 relative to another.

[0030] In this embodiment, there are three sound sources 12 including at least a target sound source 123 and one or more other non-target sound sources 121 , 122 .

[0031] The sound capture device is configured to apply a relative gain to the target sound source 123 and a relative attenuation to one or more other non-target sound sources 121, 122 to produce an audio focus 16 on the target sound source 123. There is an audio focus attenuation of the one or more non-target sound sources 122, 122.

[0032] The target sound source 123 is at a first location relative to the origin 14, and the one or more non-target sound sources 121, 122 are at a second location from the origin 14 that is different from the first location. That is, the target sound source 123 is spatially distinct from the one or more non-target sound sources 121, 122.

[0033] The sound capturing devices may be configured to separate sounds coming from a first location using beamforming, which uses a weighted linear combination of audio signals from different sound capturing devices. The weighted linear combination of audio signals may be implemented in hardware or software, or a combination of hardware and software.

[0034] In some, but not necessarily all, embodiments, the weighting for the linear combination is adaptive. The weighting may, for example, in some embodiments, be obtained by minimizing the energy of the interfering sources (e.g., non-target sources 121, 122) under the constraint of keeping the response in the target direction toward the first position. This is the minimum variance distortionless response.

[0035] In some embodiments, but not necessarily all embodiments, the weighting is frequency dependent.

[0036] For example, a beamformer may be used to process signals from an array of separate microphones and achieve spatial filtering of the propagating sound field. Spatial filtering separates signals that have overlapping frequency content but originate from different spatial locations. Target signals from the target location (azimuth) are not attenuated, while signals from other locations (azimuths) are attenuated. The spatial filter creates a deep null of gain for the directions of non-target signals. Thus, the beamformer may operate as a spatial bandpass filter for signals from the target direction 16 and as a bandstop filter for signals from non-target directions.

[0037] The weighted linear combination of the audio signals from the different microphones is y(t)=w HFor a complex plane wave with direction of arrival θ, y(t) = exp(jωt).r(θ,ω), where the beamformer response r(θ,ω) = w H d(θ,ω), where d(θ,ω) is the steering vector.

[0038] w H is selected such that y(t) contains a significantly higher (e.g., a maximum) contribution from the target sound source 123 (audio focus 16 in the target direction) and a significantly lower (e.g., a minimum) contribution from the non-target sound sources 121, 122 (non-target directions). H may be applied over a single frequency band or separately over separate frequency bands.

[0039] Weight w H can be obtained by minimizing a cost function, e.g., by minimizing the output variance (from the target) subject to response constraints such as a target gain for one direction and a target attenuation (e.g., zero force gain) for another direction. This produces a minimum variance distortionless response (MVDR) beamformer that preserves the target signal (e.g., from the target source 123) while minimizing the contribution to the output from non-target signals (e.g., from non-target sources 121, 122) from non-target directions.

[0040] The weighting and spatial filtering can be changed in real time. For example, the steering vector d(θ,ω) can be changed in real time. For example, the weights w H can be adapted in real time.

[0041] 1A and 1B also show an example of a visual space 20 that includes one or more visual objects 22. Visual objects 22 at different positions are labeled with different subscripts. Figures 1A and 1B show three visual objects 221, 222, 223 at different positions relative to the origin 14.

[0042] In this embodiment, but not necessarily in all embodiments, each sound source 121, 122, 123 has a corresponding visual object 221, 222, 223.

[0043] In this embodiment, there is a correspondence between the acoustic space 10 and the visual space 20. The positions of the sound sources 121, 122, 123 in the acoustic space 10 overlap with the positions of the visual objects 221, 222, 223 in the visual space 20.

[0044] One or more image capture devices, such as cameras, may be positioned at or near origin 14. The image capture devices may be configured, for example, to capture a visual scene defined by field of view 261. The image capture devices may be, for example, video cameras that capture a video including a sequence of the visual scene defined by field of view 261.

[0045] In this example, visual objects 221, 222 correspond to attenuated sound sources 121, 122, and visual object 223 corresponds to unattenuated sound source 123. The attenuated sound source may be, for example, a muted sound source.

[0046] From FIG. 1B it can be seen that the visual object 221 corresponding to the attenuated sound source 121 is not within the field of view 261 and therefore is not within the visual scene captured by the camera at the origin 14 .

[0047] From FIG. 1B it can be seen that a visual object 223 corresponding to the unattenuated sound source 123 is within the field of view 261 and therefore within the visual scene captured by the camera at the origin 14 .

[0048] From FIG. 1B it can be seen that a visual object 222 corresponding to the attenuated sound source 122 is within the field of view 261 and therefore within the visual scene captured by the camera at the origin 14 .

[0049] In this example, visual object 222 in field of view 261 corresponding to attenuated sound source 122 may distract from visual object 223 corresponding to unattenuated sound source 123. This is especially true when visual object 222 in field of view 261 corresponding to attenuated sound source 122 occupies a larger portion of the visual scene than visual object 223, or when visual object 222 is a foreground object while visual object 223 is a background object, or when visual object 222 is a dynamic visual object (a visual object that moves and / or changes position).

[0050] 2A shows an example of a visual scene. Visual object 222 corresponds to attenuated sound source 122. The location of visual object 222 and attenuated sound source 122 corresponding to visual object 222 are within the field of view 261. Visual object 222 can be seen in the captured image. Visual object 223 corresponds to un-attenuated sound source 123. Visual object 223 and un-attenuated sound source 123 corresponding to visual object 223 are within the field of view 261. Visual object 223 can be seen in the captured image. The captured sound from un-attenuated sound source 123 is most auditorily salient, while the larger size of foreground visual object 222 is most visually salient. The larger size of foreground visual object 222 visually distracts from visual object 223 corresponding to un-attenuated sound source 123.

[0051] The following method 100 (FIG. 3) reduces distraction.

[0052] Method 100 is In block 102, identifying an audio focus attenuation of a sound source 122; In block 104, determining a correspondence between a sound source 122 subjected to audio focus attenuation and a corresponding visual object 222; At block 106, modifying the image capture to at least partially exclude and / or modify visual objects 222 corresponding to the sound sources 122 that are subject to audio focus attenuation; Includes.

[0053] In some, but not necessarily all, embodiments, at block 102 the method may include automatically identifying audio focus attenuation for sound source 122. In some, but not necessarily all, embodiments, at block 104 the method may include automatically determining a correspondence between sound source 122 undergoing audio focus attenuation and corresponding visual object 222. In some, but not necessarily all, embodiments, at block 106 the method may include automatically modifying the capture of an image to at least partially exclude and / or modify visual object 222 corresponding to sound source 122 undergoing audio focus attenuation.

[0054] The method reduces the visual impact on the captured image of a visual object 222 that corresponds to a sound source 122 that is subject to audio focus attenuation.

[0055] An example of the result of the method 100 is shown in Fig. 2B. The method modifies the image capture to at least partially modify a visual object 222 corresponding to the sound source 122 that is subject to audio focus attenuation. An unmodified visual object 222 is shown in Fig. 2A, and a modified visual object 222' is shown in Fig. 2B. In this example, the unmodified visual object 222 (rather than visual object 223) is subjected to a blurring kernel of, for example, 50 pixels to generate a modified visual object 222'. In effect, the visual object 222 corresponding to the sound source 122 that is subject to audio focus attenuation is visually attenuated, for example by blurring, coloring, etc.

[0056] An example of the result of the method 100 is shown in FIG. 2C. The method reframes the visual scene to modify the capture of the image to at least partially exclude the visual object 222 corresponding to the sound source 122 undergoing audio focus attenuation. The unmodified visual object 222 is shown in FIG. 2A, and the at least partially excluded modified visual object 222' is shown in FIG. 2C. In this example, the unmodified visual object 222 (rather than the visual object 223) is subject to at least partial exclusion from the captured image. This may be achieved, for example, by substantially reframing the captured image by panning or tracking and / or zooming. This may be achieved by changing the size and / or direction of the field of view 26.

[0057] Figure 4A is similar to Figure 1B and shows that visual object 223, which corresponds to unattenuated sound source 123, is within the field of view 261 and therefore within the visual scene captured by a camera at origin 14, and that visual object 222, which corresponds to attenuated sound source 122, is within the field of view 261 and therefore within the visual scene captured by a camera at origin 14.

[0058] 4B is similar to FIG. 1B, except that the field of view 261 has been modified to a new field of view, field of view 261′, which reframes the captured image to exclude all or part of the visual object 222 that corresponds to the attenuated sound source 122.

[0059] Thus, FIG. 4B shows that visual object 223 corresponding to unattenuated sound source 123 is within the new field of view 261' and therefore is within the visual scene captured by the camera at origin 14, and that visual object 222 corresponding to attenuated sound source 122 is at least partially excluded from the new field of view 261' and therefore is no longer within the visual scene captured by the camera at origin 14.

[0060] In some instances, the reframing occurs automatically, while in other instances the user is given a user-selectable option to cause the reframing to occur.

[0061] In this example, the reframing maintains the same resolution and is effectively a pan or track. In other examples, the reframing additionally or alternatively changes resolution and is effectively a zoom.

[0062] In the example of Figures 4A and 4B, a second, wider field of view 262 is shown. The wider field of view 262 shows the visual scene imaged by the image sensor of the camera. The narrower fields of view 261, 261' show the visual scene captured by the image sensor. The narrower fields of view 261, 261' are sub-portions of the second, wider field of view 262. The framing shown in Figure 4B is achieved by cropping the wider field of view image to achieve the narrower field of view image. The crop may change size and / or position between Figures 4A and 4B. The crop may remain the same size between Figures 4A and 4B. In this example, modifying the image capture is configured to change the crop applied to the image detected at the image sensor array, and the applied crop determines the image captured. The crop defines a contiguous subset of sensels of the sensor array. The position of the crop applied to the image detected at the image sensor array is changed. The size of the crop applied to the image detected at the image sensor array may be maintained or changed. The effect of re-cropping is illustrated in Figures 5A and 5B. Figure 5A shows the image detected at the image sensor array (field of view 262) and the currently applied crop (field of view 261 including visual object 222 corresponding to attenuated sound source 122). Figure 5B shows the captured image (field of view 261' not including visual object 222 corresponding to attenuated sound source 122) defined by the new crop newly applied to the image detected at the image sensor array (field of view 262). In this example, but not necessarily in all examples, the captured image retains the portion of visual object 222 that corresponds to the attenuated sound source 122 that is excluded (modified visual object 222').

[0063] The method 100 (FIG. 3) includes identifying an audio focus attenuation for a sound source 122 at block 102. This block may include, for example, Obtaining an audio focus direction, e.g., a beamforming steering vector d(θ,ω); Obtaining a sound source direction; Determining one or more sound sources that are attenuated in a direction other than the audio focus direction, for example; may include.

[0064] The method 100 includes, at block 104 , determining a correspondence between the sound source 12 that is subject to audio focus attenuation and the corresponding visual object 22 .

[0065] This block 104 is determining a location of a sound source 12 that is subject to audio focus attenuation; determining a visual object 22 associated with a position of a sound source 12 that is subject to audio focus attenuation; modifying the captured image to remove or modify the determined visual object 22; may include.

[0066] If a visual object 22 is at or near a location of a sound source 12 that is subject to audio focus attenuation, for example if the visual object 22 (or any part of the visual object 22) is within a threshold distance of that location, the visual object 22 is associated with that location.

[0067] In some, but not necessarily all, embodiments, an image is captured prior to block 104. The captured image is then modified in block 106. In some, but not necessarily all, embodiments, the captured image is processed to identify corresponding visual objects 222.

[0068] Computer vision techniques may be used to identify the surroundings of a visual object 22 associated with a location or to recognize or characterize a visual object 22 associated with a location. Computer vision techniques may include, for example, edge or corner detection, interest detection, feature detection.

[0069] The position can be, for example, a one-dimensional orientation (e.g., azimuth angle φ), a two-dimensional orientation (e.g., azimuth angle φ and polar angle θ), two-dimensional coordinates (e.g., Cartesian coordinates (x,y) or polar coordinates (r,φ)), or three-dimensional coordinates (e.g., Cartesian coordinates (x,y,z) or cylindrical polar coordinates (r,φ,z) or spherical polar coordinates (r,φ,θ)).

[0070] For example, modifying the captured image to modify the determined visual object 22 may include processing the captured image, for example, as described with reference to FIG. 2B.

[0071] For example, modifying the captured image to modify the determined visual object 22 may include reframing or cropping the captured image, for example as described with reference to FIG. 2C or FIGS. 4A, 4B, 5A, 5B.

[0072] As a result of a candidate sound source (which may or may not be a sound source 12 that undergoes audio focus attenuation) 12 having an energy or level above a threshold and a lack of sufficient correlation between the target sound source 123 and the candidate sound source 12 (a correlation score below the threshold), the candidate sound source may be identified as a sound source 12 that undergoes audio focus attenuation.

[0073] Determining the location of the sound source 12 that is subject to audio focus attenuation includes: determining that a candidate sound source 12 has an energy or level above a threshold; Determining a lack of sufficient correlation between the target sound source 123 and the candidate sound sources 12; If the candidate sound source 12 has an energy or level above a threshold and lacks sufficient correlation with the target sound source 123, obtaining the position of the candidate sound source 12; may include.

[0074] The threshold value can be a zero or a non-zero threshold.

[0075] In some examples, the position of the candidate sound source 12 may be specified to any one of a number of distinct positions, for example defined by a quantized value of the beamforming steering vector d(θ,ω).

[0076] The locations of the candidate sound sources 12 may be identified using spatial audio, for example using the beamformer described above, or from metadata associated with the candidate acoustic objects.

[0077] In some, but not necessarily all, embodiments, the target sound source 123 may be user selected. The device may, for example, comprise means for user selection within a viewfinder display.

[0078] FIG. 6A illustrates one embodiment of an apparatus 50.

[0079] The apparatus 50 is an audio capture device and comprises an audio input circuit 60 for inputting audio. The audio input circuit comprises a plurality of audio capture devices 62, which in this embodiment are microphones.

[0080] The microphones are spatially distributed to form an array of separate microphones, which may be, for example, 1D, 2D, or 3D.

[0081] The apparatus 50 is an image capture apparatus and includes an image capture device 70. The image capture device 70 may be a camera or an image sensor.

[0082] The image capture device 70 may be, for example, a video camera 72 .

[0083] The controller 80 is configured to receive audio data from the audio input circuitry 60 and visual data from the image capture device 70. The data may be raw and some pre-processing may take place in the audio input circuitry 60 and / or the image capture device 70.

[0084] In this embodiment, the device 50 may be a beamformer device. The device 50 is configured to spatially filter audio data from the audio input circuitry 60 to achieve audio focus as described above. Audio focus attenuates sounds from undesired directions.

[0085] The controller 80 also Identifying an audio focus attenuation of the sound source 12; determining a correspondence between the sound source 12 subjected to audio focus attenuation and a corresponding visual object 22; Modifying image capture to at least partially exclude and / or modify visual objects 22 corresponding to sound sources 12 that are subject to audio focus attenuation It is configured as follows.

[0086] The device 50 is a rendering device and comprises a user output circuit 40 for simultaneously rendering images and sounds as multimedia content. The user output circuit 40 comprises an audio output circuit 42, which in this embodiment may comprise one of a number of speakers, headphones, or an interface to speakers or headphones. The user output circuit 40 comprises an image output circuit, which in this embodiment may comprise, for example, one of a number of displays 44.

[0087] The device 50 may be configured to display on the display 44 an image before removal / modification of visual objects corresponding to sound sources that are subject to audio focus attenuation.

[0088] The device 50 may be configured to display on the display 44 an image after removal / modification of visual objects corresponding to sound sources subjected to audio focus attenuation.

[0089] The device 50 may be configured to display on the display 44 a captured image that at least partially excludes and / or modifies visual objects corresponding to sound sources that are subject to audio focus attenuation.

[0090] Device 50 may, in some examples, include user input circuitry 90 configured to enable a user to provide control commands to device 50. In some, but not necessarily all, embodiments, user input circuitry 90 may be integrated with display 44 to create a touchscreen display. In some, but not necessarily all, embodiments, display 44 or the touchscreen display provides a camera viewfinder for video camera 72. User input circuitry 90 may enable a user to identify sound source 12 and / or visual object 22 associated with sound source 12.

[0091] The device 50 may be a portable or hand-held device. The device 50 may be, for example, a camera or a mobile phone with a camera function.

[0092] 6B shows one embodiment of the controller 80. The implementation of the controller 80 may be as a controller circuit. The controller 80 may be implemented solely in hardware, have some aspects of software including solely in firmware, or may be a combination of hardware and software (including firmware).

[0093] As shown in FIG. 6B, the controller 80 may be implemented using instructions that enable hardware functionality, for example, by using executable instructions of a computer program 86 in a general-purpose or special-purpose processor 82, which may be stored on a computer-readable storage medium (disk, memory, etc.), to be executed by such processor 82.

[0094] Processor 82 is configured to read from and write to memory 84. Processor 82 may also include an output interface, through which data and / or commands are output by processor 82, and an input interface, through which data and / or commands are input to processor 82.

[0095] The memory 84 stores a computer program 86 that includes computer program instructions (computer program code) that, when loaded into the processor 82, control the operation of the device 50. The computer program instructions of the computer program 86 provide the logic and routines that enable the device to implement the method shown in Figure 3. By reading the memory 84, the processor 82 can load and execute the computer program 86.

[0096] Thus, the device 50: At least one processor 82; at least one memory 84 containing computer program code; In conjunction with at least one processor 82, the device 50 includes: Identifying an audio focus attenuation of a sound source; determining a correspondence between a sound source subject to audio focus attenuation and a corresponding visual object; Modifying the image capture to at least partially exclude and / or modify visual objects corresponding to sound sources that are subject to audio focus attenuation; At least one memory 84 and computer program code configured to at least Equipped with.

[0097] As shown in Fig. 6B, the computer program 86 may reach the device 50 via any suitable delivery mechanism 88. The delivery mechanism 88 may be, for example, a machine-readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a memory device, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a solid-state memory, or other recording medium, an article of manufacture that includes the computer program 86 or tangibly embodies the computer program 86. The delivery mechanism may be a signal configured to reliably transfer the computer program 86. The device 50 may propagate or transmit the computer program 86 as a computer data signal.

[0098] Computer program instructions for causing an apparatus to at least: Identifying an audio focus attenuation of a sound source; determining a correspondence between a sound source subject to audio focus attenuation and a corresponding visual object; Modifying image capture to at least partially exclude and / or modify visual objects corresponding to sound sources that are subject to audio focus attenuation.

[0099] The computer program instructions may be included in a computer program, a non-transitory computer readable medium, a computer program product, a machine readable medium, etc. In some embodiments, but not necessarily all embodiments, the computer program instructions may be distributed across multiple computer programs.

[0100] Although memory 84 is shown as a single component / circuit, memory 84 may be implemented as one or more separate components / circuits, some or all of which may be integrated / removable and / or provide persistent / semi-persistent / dynamic / cached storage.

[0101] Although the processor 82 is shown as a single component / circuit, the processor 82 may be implemented as one or more separate components / circuits, some or all of which may be integrated / removable. The processor 82 may be a single-core or multi-core processor.

[0102] The above embodiment can be extended to a situation where there are multiple audio focus directions.

[0103] The above embodiment can be extended to a situation where there are multiple sound sources that are subject to audio focus attenuation and correspond to visual objects in the candidate captured image.

[0104] The method 100 is performed for each such source.

[0105] The method is: At block 102, identifying audio focus attenuation for a plurality of sound sources 12; determining, in block 104, a correspondence between a plurality of sound sources 12 subjected to audio focus attenuation and a plurality of respective visual objects 22 present in the image to be captured; At block 106, modifying the image capture to at least partially exclude and / or modify at least some of the plurality of visual objects 22 corresponding to the plurality of sound sources 12 that are subject to audio focus attenuation; may include.

[0106] In some, but not necessarily all, embodiments, the method totally or partially excludes and / or modifies all visual objects 22 associated with the audio focus attenuated sound source 12 that would otherwise be in the captured image, such that the attenuated sound source 12 is not visible.

[0107] In some, but not necessarily all, embodiments, modifying the image capture is configured to exclude selected visual objects 22 associated with the sound source 12 that is subject to audio focus attenuation, and the selection of visual objects 22 is configured to: Minimizing the number of visual objects 22 in the captured image that are associated with the sound source 12 that is subject to audio focus attenuation; maximizing the number of visual objects 22 associated with the sound source 12 subject to audio focus attenuation that are excluded from the captured image; Minimizing the total energy of the sound source 12 that is subjected to audio focus attenuation and associated with the visual object 22 in the captured image; maximizing the total energy of the sound source 12 associated with the visual object 22 that is subjected to audio focus attenuation and excluded from the captured image; Object recognition of a visual object 22 associated with a sound source 12 that is subject to audio focus attenuation; the proximity of the visual object 22 associated with the sound source 12 that is subject to audio focus attenuation to the device 50; A movement or change in the visual object 22 associated with the sound source that is subject to audio focus attenuation; Minimizing the area of ​​a captured image occupied by visual objects 22 associated with a sound source 12 that is subject to audio focus attenuation It depends on one or more of:

[0108] References to a "computer-readable storage medium," "computer program product," "tangibly embodied computer program," and the like, or to a "controller," "computer," "processor," and the like, should be understood to encompass computers having a variety of architectures, such as single / multi-processor architectures and sequential (von Neumann) / parallel architectures, as well as special purpose circuitry, such as field programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices, other processing circuitry, and the like. References to computer programs, instructions, code, and the like, should be understood to encompass software for a programmable processor or firmware, such as the programmable content of a hardware device, whether for example, instructions for a processor or configuration settings for a fixed function device, gate array, or programmable logic device, and the like.

[0109] As used herein, the term "circuitry" may refer to one or more or all of the following: (a) Hardware-only circuit implementations (e.g., analog and / or digital-only circuit implementations); and (b) A combination of hardware circuitry and software, as follows (when applicable): (i) A combination of analog and / or digital hardware circuitry and software / firmware; and (ii) any portion of hardware processors (including digital signal processors) with software, software, and memory that work together to cause a device, such as a mobile phone or a server, to perform various functions; and (c) A hardware circuit, and / or processor, such as a microprocessor or part of a microprocessor, that requires software (e.g., firmware) to operate, but the software may be absent when no software is required for operation.

[0110] This definition of circuitry applies to all uses of the term in this application, including within any claims. As another example, in this application, the term circuitry encompasses merely a hardware circuit or an implementation of a processor and its associated software and / or firmware. The term circuitry also encompasses, for example, a baseband integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device, if applicable to a particular claim element.

[0111] 3 may represent steps in a method and / or sections of code within the computer program 86. The illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order to the blocks, and the order and arrangement of the blocks may vary. Additionally, it may be possible to omit some blocks.

[0112] Where a structural feature is described, the structural feature may be replaced by a means for performing one or more of the functions of the structural feature, whether such function is explicitly described or implicitly described.

[0113] In some embodiments, but not necessarily all embodiments, device 50 is configured to communicate data from device 50 with or without local storage of the data in memory 84 at device 50, with or without local processing of the data by circuitry or a processor at device 50.

[0114] The data may be stored remotely on one or more devices in a processed or raw format. The data may be stored in the cloud.

[0115] The data may be processed remotely on one or more devices. The data may be partially processed locally and partially processed remotely on one or more devices.

[0116] Data can be transferred over Wi-Fi, for example. (Registered Trademark) or Bluetooth (Registered Trademark) or wirelessly via a longer range cellular radio link. The apparatus may comprise a communication interface, such as a wireless transceiver for communicating data.

[0117] The device 50 may be part of the Internet of Things, forming part of a larger distributed network.

[0118] The processing of the data may be local or remote and may be for health monitoring, data aggregation, patient monitoring, vital signs monitoring, or other purposes.

[0119] The processing of the data, whether local or remote, may use artificial intelligence or machine learning algorithms. The data may be used, for example, as learning input to train a machine learning network or as a query input to the machine learning network, which provides a response. The machine learning network may use, for example, an acyclic machine learning network, such as linear regression, logistic regression, vector support machine, or a single or multiple hidden layer neural network.

[0120] The processing of the data may generate an output, either locally or remotely. The output may be communicated to the device 50, which may generate an object-sensitive output, such as an audio output, a visual output, or a tactile output.

[0121] The recording of data may include only temporary recording, or the recording of data may include permanent recording, or the recording of data may include both temporary and permanent recording. Temporary recording suggests the recording of data temporarily. This may be done, for example, during sensing or image capture, in dynamic memory, in a buffer such as a circular buffer, a register, a cache, etc. Persistent recording suggests the data is in the form of an addressable data structure that is retrievable from an addressable memory space, and thus may be stored and retrieved until deleted or overwritten, although long-term storage may or may not occur. Use of the term "capture" in relation to an image relates to the temporary recording of the image's data. Use of the term "storage" in relation to an image relates to the permanent recording of the image's data.

[0122] In the above embodiment, the captured image can also be a recorded image.

[0123] Here, "module" refers to a unit or device that excludes some parts / components to be added by the manufacturer or user.

[0124] For example, apparatus 50 may be physically separated into one or more of audio input circuitry 60, image capture device 70, controller 80, and user output circuitry 40, which may be separated in time and / or space. The functionality of apparatus 50 may be implemented using multiple distinct, but functionally coupled, devices.

[0125] The above embodiment is Automotive systems, telecommunications systems, electronic systems including consumer electronic products, distributed computing systems, media systems for generating or rendering audio, visual, and audiovisual content, as well as media content including mixed reality, mediated reality, virtual reality, and / or augmented reality, personal systems including personal health systems or personal fitness systems, navigation systems, user interfaces also referred to as human-machine interfaces, networks including cellular, non-cellular, and optical networks, ad-hoc networks, the Internet, the Internet of Things, virtualized networks, and related software and services. It is applied as a usable component of the

[0126] The term "comprising" is used in this document in an inclusive rather than exclusive sense, i.e. any reference to X comprising Y means that X may comprise only one Y, or may comprise multiple Y. If the exclusive sense of "comprising" is intended, this will be made clear in the context by a reference to "comprising only one .." or by using "consisting of."

[0127] In this description, reference has been made to various embodiments. The description of a feature or function in relation to an embodiment indicates that such feature or function is present in that embodiment. Use of the terms "embodiment" or "for example" or "may" or "could" in this text, whether or not explicitly stated, indicates that such feature or function is present in at least the embodiment described, whether or not described as an example, and may, but not necessarily, be present in some or all of the other embodiments. Thus, "embodiment," "for example," "may," or "may" refers to a particular example within a class of embodiments. The properties of an example may be properties of only that example, or properties of a class, or properties of a subclass of a class that includes some but not all of the examples in the class. Thus, a feature described with reference to one embodiment but without reference to another embodiment is implicitly disclosed to be usable in that other embodiment, if possible, as part of a functional combination, but not necessarily used in that other embodiment.

[0128] Although embodiments have been described in the preceding paragraphs with reference to various examples, it should be understood that modifications can be made to the examples given without departing from the scope of the claims.

[0129] Features described in the preceding description may be used in combinations other than those explicitly described above.

[0130] Although functionality may be described with reference to certain features, such functionality may be implemented by other features, whether or not described.

[0131] Although features have been described with reference to some embodiments, such features may be present in other embodiments whether or not described.

[0132] The terms "a" or "the" are used in this document in an inclusive, not exclusive sense. That is, any reference to X comprising a / the Y indicates that X may comprise only one Y or may comprise multiple Y, unless the context clearly indicates otherwise. If "a" or "the" is intended to be used in an exclusive sense, the context will make this clear. In some circumstances, the use of "at least one" or "one or more" may be used to emphasize an inclusive sense, but the absence of such terms should not be understood to infer an exclusive sense.

[0133] The presence of a feature (combination of features) in a claim is a reference to the feature or (combination of features) per se, as well as to features that achieve substantially the same technical effect (equivalent features). Equivalent features include, for example, features that are variations and achieve substantially the same result in substantially the same way. Equivalent features include, for example, features that perform substantially the same function in substantially the same way to achieve substantially the same result.

[0134] In this description, reference has been made to various embodiments using adjectives or adjectival phrases to describe features of the embodiments. Such description of features with respect to the embodiments indicates that in some instances the feature is present exactly as described and in other instances it is present approximately as described.

[0135] Although the foregoing specification has attempted to draw attention to features which are believed to be important, it will be understood that the applicant may seek protection through the claims with respect to any patentable feature or combination of features referenced above and / or shown in the drawings, whether or not emphasis is placed thereon.

Claims

1. An apparatus comprising: At least one processor; at least one non-transitory memory containing computer program code; The at least one non-transitory memory and the computer program code, together with the at least one processor, cause the apparatus to: determining that a sound source has an energy or level that exceeds a first threshold; determining that there is a correlation between a target sound source, the target sound source being a user selected sound source, and the sound source in terms of energy or level below a second threshold; identifying an audio focus attenuation of the sound source in response to the energy or level of the sound source exceeding a first threshold and a correlation between the target sound source and the sound source being less than a second threshold; determining a correspondence between said sound sources subject to audio focus attenuation and corresponding visual objects; modifying the image capture to at least partially exclude and / or modify the visual object corresponding to the sound source that is subject to audio focus attenuation; An apparatus configured to cause

2. The at least one non-transitory memory and the computer program code, together with the at least one processor, cause the apparatus to: determining a position of the sound source that is subject to audio focus attenuation; determining a visual object based at least in part on an association between the position of the sound source and the visual object that is subject to audio focus attenuation; modifying the captured image to exclude the determined visual object; The apparatus of claim 1 configured to cause a

3. The apparatus of claim 2 , wherein the sound source is located in any one of a number of locations.

4. The apparatus of claim 2 , wherein the sound sources are positioned using spatial audio.

5. 10. The apparatus of claim 1, wherein the target sound source is user-selected, the apparatus comprising a selector in a viewfinder display of a target visual object associated with the target sound source.

6. The apparatus of claim 2 , wherein the at least one non-transitory memory and the computer program code, together with the at least one processor, cause the apparatus to perform object recognition.

7. The at least one non-transitory memory and the computer program code, together with the at least one processor, cause the apparatus to:

10. The apparatus of claim 1, configured to cause a captured image to be reframed to exclude the visual object corresponding to the sound source.

8. The at least one non-transitory memory and the computer program code, together with the at least one processor, cause the apparatus to:

2. The apparatus of claim 1, wherein modifying the image capture is configured to alter a crop applied to an image detected by an image sensor array, the crop applied determining the image captured.

9. The apparatus of claim 8, wherein modifying the image capture is configured to change a position of the crop applied to the image detected by the image sensor array.

10. The at least one non-transitory memory and the computer program code, together with the at least one processor, cause the apparatus to: The apparatus of claim 9 , configured to cause maintaining a size of the crop applied to the image detected by the image sensor array.

11. The at least one non-transitory memory and the computer program code, together with the at least one processor, cause the apparatus to:

2. The apparatus of claim 1, configured to cause the filtering out of all visual objects associated with a sound source that is subject to audio focus attenuation.

12. The at least one non-transitory memory and the computer program code, together with the at least one processor, cause the apparatus to: and causing the selection of visual objects to be removed, the visual objects being associated with a sound source that is subject to audio focus attenuation, the selection of the visual objects being configured to: Minimizing the number of visual objects in the captured image that are associated with the sound source that is subject to audio focus attenuation; maximizing the number of visual objects associated with the sound source subject to audio focus attenuation that are excluded from the captured image; Minimizing the total energy of a sound source that is subject to audio focus attenuation and associated with a visual object in a captured image; maximizing the total energy of a sound source associated with a visual object that is subject to audio focus attenuation and excluded from the captured image; Object recognition of visual objects associated with a sound source that is subject to audio focus attenuation; the proximity of a visual object associated with the sound source that is subject to audio focus attenuation to the device; or Minimizing the area of ​​a captured image occupied by the visual object associated with the sound source that is subject to audio focus attenuation. The apparatus of claim 1 , wherein the apparatus is dependent on one or more of:

13. A computer program recorded on a non-transitory computer-readable storage medium, comprising: When executed on a computer, determining that a sound source has an energy or level that exceeds a first threshold; determining that there is a correlation between a target sound source, the target sound source being a user selected sound source, and the sound source in terms of energy or level below a second threshold; identifying an audio focus attenuation of the sound source in response to the energy or level of the sound source exceeding a first threshold and a correlation between the target sound source and the sound source being less than a second threshold; determining a correspondence between said sound sources subject to audio focus attenuation and corresponding visual objects; modifying the image capture to at least partially exclude and / or modify the visual object corresponding to the sound source that is subject to audio focus attenuation; A computer program configured to perform the steps of:

14. When executed on a computer, determining a position of the sound source that is subject to audio focus attenuation; determining the visual object based at least in part on an association between the position of the sound source and the visual object that is subject to audio focus attenuation; modifying the captured image to exclude the determined visual object; 14. A computer program product as claimed in claim 13, configured to carry out the steps:

15. determining that a sound source has an energy or level that exceeds a first threshold; determining that there is a correlation between a target sound source, the target sound source being a user selected sound source, and the sound source in terms of energy or level below a second threshold; identifying an audio focus attenuation of the sound source in response to the energy or level of the sound source exceeding a first threshold and a correlation between the target sound source and the sound source being less than a second threshold; determining a correspondence between said sound sources subject to audio focus attenuation and corresponding visual objects; modifying the image capture to at least partially exclude and / or modify the visual object corresponding to the sound source that is subject to audio focus attenuation; The method includes:

Citation Information

Patent Citations

  • Video distribution terminal, program, and video distribution method

    JP2017028633A

  • Method for refocusing images captured by a preoptic camera and an audio-based refocusing image system.

    JP2018536365A

  • Shift camera focus based on speaker position

    US20150146078A1