Electronic device for controlling audio device based on image context and method of operating the electronic device

Abstract

An electronic device for controlling an audio setting of an audio device based on contextual information of an image and an operating method thereof are provided. The electronic device may detect at least one object from an image captured using a camera, identify a primary object of interest of a user from the detected at least one object, obtain contextual information of the identified primary object of interest, and transmit the contextual information to the user. And controlling an audio setting of an audio device to obtain or output an audio signal based on the obtained context information, the audio device including at least one of a microphone and a speaker.

Description

Technical Field

[0001] The present disclosure relates to an electronic device for controlling an audio device based on context information obtained from an image captured using a camera and an operating method thereof. Background Art

[0002] In general, when a user takes a photo or shoots a video by using a camera, the user selects a main object of interest (hereinafter referred to as an "object of interest") among the objects to be shot, and adjusts the camera settings (such as zooming in, zooming out, or panning) for the selected object of interest to obtain high-quality results. Even if the user adjusts the camera settings, the audio settings of the audio device will not be changed or adjusted according to the object of interest. For example, even if the user selects a singer singing in a square as the object of interest and zooms in on the singer, the microphone not only records the singer's singing, but also the noise of vehicles around the square, the cheers of the audience, or the sounds of pedestrians. In order for the user to obtain the audio signal of the object of interest (e.g., the singer's singing), the audio settings of the audio device (e.g., microphone) must be manually changed or adjusted by user input. For example, the user can change or adjust the equalizer settings in the settings of the audio device when shooting a video by using a camera. Summary of the invention

[0003] Technical Solution

[0004] Aspects of the present disclosure provide an electronic device for controlling an audio device based on an image. An electronic device according to an embodiment of the present disclosure may include: a camera; an audio device including at least one of a microphone and a speaker; a memory storing one or more instructions; and at least one processor configured to execute one or more instructions. At least one processor may detect at least one object from an image captured using a camera. At least one processor may identify a user's primary object of interest from the at least one detected object. At least one processor may obtain context information of the identified primary object of interest. At least one processor may control the audio settings of the audio device based on the obtained context information to obtain or output an audio signal by using the audio device.

[0005] Another aspect of the present disclosure provides a method for controlling an audio device based on an image performed by an electronic device. The method according to an embodiment of the present disclosure may include detecting at least one object from an image captured using a camera. The method according to an embodiment of the present disclosure may include identifying a user's primary object of interest from the at least one object detected. The method according to an embodiment of the present disclosure may include obtaining context information of the identified primary object of interest. The method according to an embodiment of the present disclosure may include controlling an audio setting of an audio device based on the obtained context information to obtain or output an audio signal, the audio device including at least one of a microphone and a speaker.

[0006] Another aspect of the present disclosure provides a computer program product including a computer-readable recording medium having a computer program recorded thereon. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present disclosure may be readily understood by referring to the following detailed description and the accompanying drawings, in which reference numerals refer to structural elements.

[0008] Figure 1 is a conceptual diagram illustrating an operation performed by an electronic device for controlling audio settings of an audio device based on context information of a primary interest object according to an embodiment of the present disclosure.

[0009] Figure 2 is a block diagram showing a structure of an electronic device according to an embodiment of the present disclosure.

[0010] Figure 3 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present disclosure.

[0011] Figure 4 is a flowchart illustrating an operation performed by an electronic device for identifying a primary object of interest from an image according to an embodiment of the present disclosure.

[0012] Figure 5 is a diagram illustrating an operation performed by an electronic device according to an embodiment of the present disclosure for obtaining an image of a main object of interest and an audio signal associated with the main object of interest.

[0013] Figure 6 is a diagram illustrating an operation performed by an electronic device according to an embodiment of the present disclosure for applying audio settings corresponding to context information of a primary focus object according to a preset mapping relationship.

[0014] Figure 7ais a diagram illustrating an operation performed by an electronic device according to an embodiment of the present disclosure for outputting equalizer presets corresponding to context information according to an image-to-audio signal mapping relationship.

[0015] Figure 7b is a diagram illustrating an operation performed by an electronic device according to an embodiment of the present disclosure for outputting equalizer presets corresponding to context information according to an image-to-audio signal mapping relationship.

[0016] Figure 8a is a diagram illustrating an operation performed by an electronic device for determining equalizer presets based on context information according to an embodiment of the present disclosure.

[0017] Figure 8b is a diagram illustrating an operation performed by an electronic device for determining equalizer presets based on context information according to an embodiment of the present disclosure.

[0018] Figure 8c is a diagram illustrating an operation performed by an electronic device for determining equalizer presets based on context information according to an embodiment of the present disclosure.

[0019] Fig. 9 is a diagram illustrating an operation for performing filtering on an audio signal based on context information of an image, performed by an electronic device according to an embodiment of the present disclosure.

[0020] Fig.10 is a diagram illustrating an operation performed by an electronic device according to an embodiment of the present disclosure for determining preset settings of an audio device based on context information of a primary object of interest.

[0021] Fig.11 is a diagram illustrating an operation performed by an electronic device according to an embodiment of the present disclosure for controlling audio settings to obtain an audio signal related to a primary object of interest.

[0022] Fig.12 is a diagram illustrating an operation performed by an electronic device according to an embodiment of the present disclosure for controlling audio settings to obtain an audio signal related to a primary object of interest.

[0023] Fig.13 is a flowchart of an operation performed by an electronic device according to an embodiment of the present disclosure for identifying a main object of interest from a cropped image and obtaining an audio signal associated with the main object of interest.

[0024] Fig.14 is a flowchart of an operation performed by an electronic device according to an embodiment of the present disclosure for identifying a main object of interest from a cropped image and obtaining an audio signal associated with the main object of interest. DETAILED DESCRIPTION

[0025] Although currently widely used general terms are selected to describe the present disclosure in consideration of the functions of the present disclosure, these general terms may vary according to the intentions of those skilled in the art, case precedents, the emergence of new technologies, etc. The terms arbitrarily selected by the applicant of the present disclosure may also be used in specific situations. In this case, their meanings need to be given in the detailed description of the embodiments of the present disclosure. Therefore, the terms must be defined based on the meaning of the terms and the content of the entire specification, rather than by simply declaring the terms.

[0026] An expression in a singular form may encompass a plural expression unless it has a clearly different meaning in the context. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0027] The terms "include" and / or "comprises" or "has" and / or "contains" used herein indicate the presence of the elements, but do not exclude the presence or addition of one or more other elements. The terms "unit", "...machine" and "module" described in this specification refer to a unit that performs at least one function or operation, and can be implemented as hardware, software, or a combination of hardware and software.

[0028] The expression "configured to (or set to)" used in this document may be used interchangeably with the following depending on the situation: for example, "suitable for", "having the ability to...", "designed for", "suitable for", "made for", or "capable of...". The expression "configured to (or set to)" may not necessarily refer to "specially designed for" in terms of hardware. On the contrary, in some cases, the expression "the system is configured to" may refer to a situation where the system, together with another device or component part, "is able to...". For example, "a processor configured to (or set to) perform A, B, and C" may refer to a dedicated processor (such as an embedded processor) for performing the corresponding operations or a general-purpose processor (such as a central processing unit (CPU) or an application processor (AP)) that can perform the corresponding operations by executing one or more software programs stored in a memory.

[0029] When an element (for example, a first element) is “coupled to” or “connected to” another element (for example, a second element), the first element may be directly coupled or connected to the second element, or, unless otherwise described, a third element may be present therebetween.

[0030] The embodiments of the present disclosure are described in detail with reference to the accompanying drawings so that the present disclosure can be easily performed by those skilled in the art. However, the present disclosure can be embodied in many different forms and should not be construed as being limited to the examples set forth herein.

[0031] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0032] Figure 1 is a conceptual diagram illustrating operations performed by the electronic device 100 for controlling the audio settings 60 of the audio device based on context information of a primary object of interest according to an embodiment of the present disclosure.

[0033] Reference Figure 1 , the electronic device 100 may obtain an image 10 of a subject by using a camera. According to an embodiment of the present disclosure, the electronic device 100 may obtain a plurality of time-sequential image frames by photographing at least one object by using a camera.

[0034] The electronic device 100 may detect an object from an image 10 obtained by a camera (operation ①). According to an embodiment of the present disclosure, the electronic device 100 may detect a plurality of objects 11 to 17 from the image 10 by using an object detection model. The object detection model may be implemented as a well-known artificial intelligence (AI) model including, for example, a convolutional neural network (CNN), but the embodiments of the present disclosure are not limited thereto. According to an embodiment of the present disclosure, the electronic device 100 may detect a plurality of objects 11 to 17 from the image 10 by performing well-known image processing (e.g., pattern matching, feature extraction, or boundary decision).

[0035] The electronic device 100 may identify objects of interest 21 to 27 from the image 10 (operation ②). The electronic device 100 may receive a user input for setting the region of interest 20 among the entire region included in the image 10. According to an embodiment of the present disclosure, the electronic device 100 may receive a zoom-in, pan, focus, or crop input about a specific region of interest from a user, and may determine the identified region as the region of interest 20 based on the received input. The electronic device 100 may identify objects 21 to 27 included in the region of interest 20 as objects of interest 21 to 27.

[0036] The electronic device 100 may obtain context information from the images of the object of interest 21 to 27 (operation ③). According to an embodiment of the present disclosure, the electronic device 100 may obtain a plurality of image frames 30 about a subject in a time series, and may obtain context information from the plurality of image frames 30 by using the context information obtaining module 146. The context information obtaining module 146 may be configured with instructions or program codes configured to obtain information about at least one of the classification of the object of interest, the action of the object of interest, and the environment, position, and time surrounding the object of interest from the images of the object of interest 21 to 27. Figure 1 In the illustrated embodiment, the electronic device 100 can classify the objects of interest 21 to 27 (e.g., person 21, guitar 22, speaker 23, tree 42, car 25, traffic light 26, and road 27) in the plurality of image frames 30 through the context information acquisition module 146, and can obtain “singer” and “street” as the context information 40 of the objects of interest 21 to 27 based on the results of the classification. For example, the electronic device 100 can also obtain the context information 40 about the action (e.g., singing while playing guitar) and time (e.g., night time) of the “singer” in the plurality of image frames 30.

[0037] The electronic device 100 may control the audio settings 60 of the audio device based on the context information (operation ④). The audio device may include but is not limited to Figure 2 Microphone 122 and Figure 2 The electronic device 100 may obtain the audio signal 50 output (or emitted) by the subject by using the microphone 122. According to an embodiment of the present disclosure, the electronic device 100 may control the audio setting 60 of the audio device to obtain only the audio signal 50 related to the context information of the objects of interest 21 to 27 from among the audio signals 50. The electronic device 100 may control at least one of the equalizer 62, the amplifier 64, and the filter among the audio setting 60 of the audio device based on the context information.

[0038] According to an embodiment of the present disclosure, the electronic device 100 may determine an equalizer preset mapped to correspond to the context information based on a preset mapping relationship between the image context and the equalizer. The electronic device 100 may enhance or attenuate the signal level of a specific frequency band in the spectrum of the audio signal 50 according to the determined equalizer preset. Figure 1 In the illustrated embodiment, the electronic device 100 can select an equalizer preset that amplifies the signal level corresponding to the frequency band of the singer's song within the frequency spectrum of the audio signal 50, and attenuates the signal level corresponding to the frequency band of the sound generated by the surrounding people 12 and 13 or the noise generated by the vehicle.

[0039] According to an embodiment of the present disclosure, the electronic device 100 may control the amplifier 64 to amplify a frequency band related to the objects of interest 21 to 27 among the frequency bands of the audio signal 50. Figure 1 In the illustrated embodiment, the electronic device 100 may amplify the signal level corresponding to the frequency band of the singer's song among the frequency bands of the audio signal 50 by controlling the amplifier 64 .

[0040] According to an embodiment of the present disclosure, the electronic device 100 may attenuate or remove frequency bands of the audio signal related to the objects 12 , 13 , 14 , 15 , 16 , and 17 other than the objects of interest 21 to 27 by using a band stop filter in the filter 66 .

[0041] Typically, when a user uses a camera to take a photo or shoot a video, the user selects an object of interest 21 to 27 among the objects to be photographed, and adjusts the camera settings, such as zooming in, zooming out, or panning, for the selected object of interest 21 to 27. In conventional technology, even if the user adjusts the camera settings, the audio settings of the audio device will not be changed or adjusted according to the objects of interest 21 to 27. For example, even if the user zooms in on a singer singing in a square, the microphone 122 records not only the singer's singing, but also the noise of vehicles around the square, the cheers of the audience, or the voices of pedestrians. In order for the user to obtain an audio signal about the object of interest (e.g., the singer's singing), it is necessary to manually change or adjust the audio settings 60 of the audio device through user input.

[0042] The present disclosure aims to provide an electronic device 100 and an operating method of the electronic device 100, wherein the electronic device 100 is used to control the audio setting of an audio device to obtain an audio signal related to an object of interest 21 to 27 set by a user when capturing image content (e.g., video) obtained using a camera.

[0043] The electronic device 100 according to an embodiment of the present disclosure obtains context information about the objects of interest 21 to 27, and controls the audio settings 60 of the audio device based on the context information to enhance only the audio signal related to the user's objects of interest, thereby providing a technical effect of improving the quality of the audio signal recorded by the microphone 122 or the audio signal output by the speaker 124. For example, in Figure 1 In the illustrated embodiment, the electronic device 100 can enhance and record only the songs of street singers or stage artists or only the music without any surrounding noise. As another example, when a worker is working in a noisy working environment, the electronic device 100 can filter or amplify only the sound of a specific machine part that the worker is interested in, thereby improving the worker's attention so that the worker can focus on the sound of the specific machine part.

[0044] Furthermore, the electronic device 100 according to an embodiment of the present disclosure may provide a technical effect of allowing a hearing-impaired user to focus only on the speaking voice of a specific speaker or to more easily hear the voice of a specific object.

[0045] Figure 2 is a block diagram illustrating components of an electronic device 100 according to an embodiment of the present disclosure.

[0046] Figure 2 The electronic device 100 shown in the figure is a device that obtains an image of a subject and controls the audio settings of an audio device based on context information of the image, and may be implemented as, for example, a smart phone, a smart TV, or a tablet personal computer (PC). However, the present disclosure is not limited thereto, and the electronic device 100 may be implemented as any of various devices (such as a laptop computer, a desktop computer, an e-book terminal, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, an MP3 player, a camera, or a wearable device).

[0047] According to an embodiment of the present disclosure, the electronic device 100 can be implemented as an augmented reality device. In the present disclosure, an "AR device" is a device capable of realizing "augmented reality", and generally includes not only augmented reality glasses worn on the user's face, but also a head-mounted display (HMD) device or an augmented reality helmet worn on the user's head.

[0048] Reference Figure 2 , the electronic device 100 may include a camera 110, an audio device 120, a processor 130, and a memory 140. The camera 110, the audio device 120, the processor 130, and the memory 140 may be electrically and / or physically connected to each other.

[0049] Figure 2 The components shown are only according to the embodiments of the present disclosure, and the components included in the electronic device 100 are not limited to Figure 2 The electronic device 100 may not include Figure 2 Some of the components shown, and may also include Figure 2Components not shown in the figure. According to an embodiment of the present disclosure, the electronic device 100 may further include a display that displays a preview image of an object captured by the camera 110. The display may be configured with a physical device including at least one of a liquid crystal display, a thin film transistor-liquid crystal display (TFT-LCD), an organic light emitting diode (OLED), a flexible display, a three-dimensional (3D) display, and an electrophoretic display. According to an embodiment of the present disclosure, the display may be configured as a touch screen including a touch interface. However, the present disclosure is not limited thereto, and in the case where the electronic device 100 is an augmented reality device in the form of glasses (i.e., augmented reality glasses), the display may include a lens, a waveguide, and an optical module (e.g., a projector).

[0050] According to an embodiment of the present disclosure, the electronic device 100 may further include an input interface for receiving user input. The input interface may include, for example, a keyboard, a mouse, a touch screen, or a voice input device (e.g., a microphone), and may include other input devices known to those skilled in the art. In the case where the display is configured as a touch screen, the display may be a component configured as a touch pad that integrates a user input interface.

[0051] According to an embodiment of the present disclosure, the electronic device 100 may be configured as a portable device, and may further include a battery for supplying driving power to the camera 110 , the audio device 120 , and the processor 130 .

[0052] The camera 110 is configured to capture a subject in a space to obtain an image. The camera 110 may obtain a video consisting of a plurality of time-sequential image frames by continuously capturing an object over time. The camera 110 may include a lens module, an image sensor, and an image processing module. The camera 110 may obtain a still image or video obtained by an image sensor (e.g., a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD)). The image processing module may process the still image or video obtained by the image sensor to extract necessary information, and send the extracted necessary information to the processor 130.

[0053] The audio device 120 is a hardware device configured to obtain an audio signal from an external source or output an audio signal. The audio device 120 may include a microphone 122 and a speaker 124.

[0054] The microphone 122 is configured to obtain speech or other sounds from an external object and convert the obtained speech or other sounds into an audio signal. According to an embodiment of the present disclosure, the microphone 122 can be configured as a microphone array including a plurality of microphone elements, a directional microphone, or a multi-mode microphone. The microphone 122 can provide the processor 130 with an audio signal from an external object.

[0055] The speaker 124 is configured to output audio signals. The speaker 124 may output audio signals according to audio settings (such as equalizer presets) under the control of the processor 130 .

[0056] The processor 130 may execute one or more instructions of a program stored in the memory 140. The processor 130 may include hardware components that perform arithmetic, logic, input / output operations, and signal processing. For example, the processor 120 may be configured as at least one of a central processing unit, a microprocessor, a graphics processing unit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), and a field programmable gate array (fPGA), but the present disclosure is not limited thereto.

[0057] The processor 130 Figure 2 1 and 2. The processor 130 is shown as a single element in FIG. 1 , but is not limited thereto. According to an embodiment, the processor 130 may be provided as one or more processors.

[0058] According to an embodiment of the present disclosure, the processor 130 may include an AI processor that performs artificial intelligence AI learning. In this case, the AI ​​processor may detect an object from an image by using an AI model and obtain context information of the detected object. The AI ​​processor may be manufactured in the form of a hardware chip dedicated to AI, or may be manufactured as part of an existing general-purpose processor (e.g., CPU or AP) or a dedicated graphics processor (e.g., GPU), and may be installed on the processor 130 within the electronic device 100.

[0059] The memory 140 may include, for example, a flash memory type storage medium, a hard disk type storage medium, a multimedia card micro storage medium, a card type memory (e.g., SD or XD memory), a random access memory (RAM), a static RAM (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), and at least one type of storage medium among optical disks. According to an embodiment of the present disclosure, the memory 140 may be accessed via a network and may be implemented as a network storage device or a cloud server that performs a storage function. In this case, the electronic device 100 may further include a communication interface, and may communicate with the network storage device or the cloud server via the communication interface, and perform data transmission and reception.

[0060] The memory 140 may store instructions or program codes that enable the electronic device 100 to obtain context information from an image and control the audio settings of an audio device based on the obtained context information. According to an embodiment of the present disclosure, the memory 140 may store at least one of instructions, algorithms, data structures, program codes, and applications that can be read by the processor 130. The instructions, algorithms, data structures, and program codes stored in the memory 140 may be implemented in a programming or scripting language such as C, C++, Java, assembly language, etc.

[0061] The memory 140 may store instructions, algorithms, data structures, or program codes related to the object detection model 142, the object of interest recognition module 144, the context information obtaining module 146, and the audio setting control module 148. The modules included in the memory 140 represent units that process functions or operations performed by the processor 130 and may be implemented as software, such as instructions, algorithms, data structures, or program codes.

[0062] According to the following embodiments, the processor 130 may be implemented by executing instructions or program codes stored in the memory 140 .

[0063] The object detection model 142 includes instructions or program codes related to operations and / or functions for detecting objects from images. According to an embodiment of the present disclosure, the object detection model 142 can be configured as an artificial neural network model. By applying a bounding box image that can be detected as an object from tens of thousands or hundreds of millions of input images as input data and applying a label value of the result of object detection as the basic fact, the object detection model 142 can be a deep neural network model trained to detect objects through a supervised learning method. The object detection model 142 can be implemented as, for example, a convolutional neural network, but the present disclosure is not limited thereto. The object detection model 142 can be implemented as, for example, a region-based convolutional neural network (R-CNN), a single-shot multi-box detector (SSD) model, YOLO v4, CenterNet, or MobileNet.

[0064] However, the object detection model 142 of the present disclosure is not limited to the aforementioned deep neural network model. The object detection model 142 can be configured as a well-known image processing model (eg, pattern matching, feature extraction, or boundary decision).

[0065] The processor 130 may obtain an image by taking a photo of a subject using the camera 110. The processor 130 may detect at least one object from the image by executing instructions or program codes associated with the object detection model 142. According to an embodiment of the present disclosure, the processor 130 may obtain a plurality of time-sequential image frames from the camera 110, and may detect at least one object from the plurality of image frames by using the object detection model 142.

[0066] The object of interest recognition module 144 is composed of instructions or program codes related to the operation and / or function of identifying the user's main object of interest from at least one detected object. According to an embodiment of the present disclosure, the processor 130 can identify the object of interest from at least one object detected in multiple image frames by executing instructions or program codes related to the object of interest recognition module 144. According to an embodiment of the present disclosure, the electronic device 100 may also include an input interface for receiving one of a user's zoom input, a user's translation input, a user's focus input, or a user's cropping input. For example, the input interface may be configured as a touch screen and may receive a touch input for zooming in, focusing, or cropping. The processor 130 may identify an area that is zoomed in, translated, focused, or cropped by a user input received through the input interface, and detect the object included in the identified area as an object of interest.

[0067] The context information acquisition module 146 is composed of instructions or program codes related to the operation and / or function of obtaining context information of the object of interest. In the present disclosure, "context information" may refer to information about a situation or context recognized from an image, and may include, for example, information about at least one of the classification, action, time, place, location, and surrounding environment of the object of interest. According to an embodiment of the present disclosure, the processor 130 may obtain context information about the object of interest by executing instructions or program codes associated with the context information acquisition module 146.

[0068] According to an embodiment of the present disclosure, the context information acquisition module 146 can be configured as an artificial neural network model. The context information acquisition module 146 can be implemented as a well-known artificial neural network model, including, for example, a convolutional neural network model. The processor 130 can obtain at least one of the classification information, action information, time information, position information, location information, and surrounding environment information of the object of interest by analyzing the object of interest via the context information acquisition module 146. In the present disclosure, "classification information" refers to information about the result of identifying the object of interest, and can be information indicating whether the object of interest is a person, an animal, or a thing, or even if the object of interest is a person, it can be information indicating whether the object of interest is a singer, an instrumentalist, an athlete, or a coach.

[0069] The audio setting control module 148 is composed of instructions or program codes related to the operation and / or function of controlling the audio setting of the audio device 120 based on the context information. According to an embodiment of the present disclosure, the processor 130 may control the audio setting for recording an audio signal through the microphone 122 or outputting an audio signal through the speaker 124 by executing the instructions or program codes associated with the audio setting control module 148. In the present disclosure, the “audio setting” may include setting information for controlling at least one of an equalizer, an amplifier, and a filter of the audio device 120.

[0070] According to an embodiment of the present disclosure, the audio device 120 may store a plurality of equalizer presets. The processor 130 may select an equalizer preset that is mapped to correspond to the context information according to a preset mapping relationship from among the plurality of equalizer presets. The preset mapping relationship may be included in the audio setting control module 148, but the present disclosure is not limited thereto.

[0071] According to an embodiment of the present disclosure, the processor 130 can detect the action of the object of interest through the context information acquisition module 146, and determine the equalizer preset mapped to correspond to the action of the object of interest according to the preset mapping relationship included in the audio setting control module 148. Figure 7a , Figure 7b as well as Figures 8a to 8c An embodiment in which the processor 130 determines equalizer presets based on context information of an object of interest is described in detail.

[0072] According to an embodiment of the present disclosure, the processor 130 may amplify a frequency band related to the object of interest among the frequency bands of the audio signal by using an amplifier of the audio device 120. For example, the processor 130 may identify the frequency band related to the object of interest by using information about the frequency band of the audio signal of each object, and may amplify only the frequency band corresponding to the identified frequency band among the frequency spectrum of the audio signal obtained by the microphone 122. As another example, the processor 130 may amplify only the audio signal related to the object of interest by using an audio-visual speech separation algorithm implemented as an artificial intelligence model or by using a pixel-by-pixel audio signal separation algorithm (e.g., PixelPlayer of The Sound of Pixel).

[0073] The processor 130 may attenuate the audio signal in the remaining frequency bands except the frequency band related to the object of interest. According to an embodiment of the present disclosure, the processor 130 may perform filtering by using a band stop filter to attenuate or cut off the frequency band of the audio signal related to the object other than the object of interest. Figures 9 to 12An embodiment in which the processor 130 amplifies or attenuates an audio signal by controlling an amplifier and / or a filter will be described in detail.

[0074] According to an embodiment of the present disclosure, the electronic device 100 may receive a cropping input from a user for cropping a specific area of ​​an image through an input interface. The processor 130 may obtain a cropped image based on the cropping input, and may amplify an audio signal related to an object of interest included in the cropped image. According to an embodiment of the present disclosure, the processor 130 may attenuate or mute an audio signal associated with at least one object not included in the cropped image. Fig.13 and Fig.14 An embodiment in which the processor 130 amplifies an audio signal related to an object of interest included in a cropped image will be described in detail.

[0075] Figure 3 is a flowchart of an operating method of the electronic device 100 according to an embodiment of the present disclosure.

[0076] In operation S310, the electronic device 100 detects at least one object from an image obtained using a camera. According to an embodiment of the present disclosure, the electronic device 100 may obtain a plurality of time-sequential image frames by continuously photographing a subject over time using a camera. The electronic device 100 may detect at least one object from a plurality of image frames by using an object detection model.

[0077] According to an embodiment of the present disclosure, the object detection model 142 may be configured as an artificial neural network model. For example, the object detection model may be implemented as a convolutional neural network. However, the present disclosure is not limited thereto, and the object detection model may be implemented as, for example, a region-based convolutional neural network (R-CNN), a single shot multi-frame detector (SSD) model, YOLO v4, CenterNet, or MobileNet.

[0078] However, the present disclosure is not limited thereto, and the electronic device 100 may detect at least one object from a plurality of image frames by using a well-known image processing model (eg, pattern matching, feature extraction, or boundary decision).

[0079] In operation S320, the electronic device 100 identifies a main object of interest from the at least one object detected. According to an embodiment of the present disclosure, the electronic device 100 may perform one of zooming in, panning, focusing, or cropping based on a user input. The processor 100 may identify an area that is zoomed in, panned, focused, or cropped by a user input, and detect an object included in the identified area from among the at least one object detected in operation S310 as an object of interest.

[0080] In operation S330, the electronic device 100 obtains context information of the identified primary object of interest. In the present disclosure, "context information" may refer to information about a situation or context identified from an image, and may include, for example, information about at least one of the classification, action, time, place, location, and surrounding environment of the object of interest. According to an embodiment of the present disclosure, the electronic device 100 may obtain at least one of the classification information, action information, time information, location information, place information, and surrounding environment information of the object of interest by analyzing the object of interest using an artificial intelligence model.

[0081] In operation S340, the electronic device 100 controls the audio settings of the audio device for recording or outputting the audio signal based on the obtained context information. Figure 2 The microphone 122 records the audio signal or Figure 2 The audio setting of the speaker 124 outputs the audio signal. In the present disclosure, the “audio setting” may include setting information for controlling at least one of an equalizer, an amplifier, and a filter of an audio device including at least one of the microphone 122 and the speaker 124.

[0082] According to an embodiment of the present disclosure, the electronic device 100 may store a plurality of equalizer presets for an audio device. The electronic device 100 may determine an equalizer preset corresponding to the context information from among a plurality of pre-stored equalizer presets by using, for example, a decision tree, fuzzy logic or other well-known algorithms.

[0083] According to an embodiment of the present disclosure, the electronic device 100 can select an equalizer preset that is mapped to correspond to the context information from among a plurality of equalizer presets according to a preset mapping relationship. The electronic device 100 can detect the action of an object of interest, and can determine an equalizer preset that is mapped to correspond to the action of the object of interest according to the preset mapping relationship.

[0084] According to an embodiment of the present disclosure, the electronic device 100 can amplify a frequency band related to an object of interest among frequency bands of an audio signal by using an amplifier of an audio device. For example, the electronic device 100 can identify a frequency band related to an object of interest by using information about the frequency band of an audio signal of each object, and can amplify only a frequency band corresponding to the identified frequency band among the frequency spectrum of an audio signal obtained through the microphone 122.

[0085] The electronic device 100 may attenuate the audio signal in the remaining frequency bands except the frequency band related to the object of interest. The electronic device 100 may perform filtering by using a band stop filter to attenuate or cut off the frequency band of the audio signal related to the object except the object of interest.

[0086] Figure 4 is a flowchart illustrating an operation performed by the electronic device 100 for recognizing an object of interest from an image according to an embodiment of the present disclosure.

[0087] Figure 4 Operations S410 and S420 are Figure 3 The detailed operation of operation S320 can be performed in Figure 3 After operation S310 is performed Figure 4 Operation S410. Figure 4 After operation S420 is performed Figure 3 Operation S330.

[0088] In operation S410, the electronic device 100 identifies an area to be enlarged, translated, or cropped from the image according to a user input. According to an embodiment of the present disclosure, the electronic device 100 may further include an input interface for receiving one of a user's enlargement input, a user's translation input, a user's focus input, or a user's cropping input. The input interface may include, for example, a keyboard, a mouse, a touch screen, a voice input device (e.g., a microphone), and may include other input devices known to those skilled in the art. For example, the input interface may be configured as a touch screen, and the electronic device 100 may receive a touch input for enlargement, focus, or cropping through the touch screen. The electronic device 100 may identify an area to be enlarged, translated, focused, or cropped by a user input received through the input interface.

[0089] In operation S420, the electronic device 100 may detect an object included in the identified area as an object of interest. According to an embodiment of the present disclosure, the electronic device 100 may detect an object included in the area enlarged by a zoom-in input as an object of interest. According to an embodiment of the present disclosure, the electronic device 100 may detect an object located at a central portion of a specific area as an object of interest by a translation input. According to an embodiment of the present disclosure, the electronic device 100 may detect an object focused by a camera as an object of interest by a focus input of a user. According to an embodiment of the present disclosure, the electronic device 100 may detect an object included in an area cropped by a crop input as an object of interest.

[0090] Figure 5 is a diagram illustrating an operation performed by the electronic device 100 according to an embodiment of the present disclosure to obtain an image of an object of interest 520 and an audio signal associated with the object of interest 520 .

[0091] Reference Figure 5 , the electronic device 100 can be used by using Figure 2The camera 1110 captures at least one object located in a real space to obtain an image 500. According to an embodiment of the present disclosure, the electronic device 100 can capture a video of at least one object, and can record voice, music, or noise output by the at least one object during the video capture by using the microphone 122. The microphone 122 can convert the recorded voice, music, noise, or other sound from the at least one object into an audio signal. Figure 5 In the illustrated embodiment, the electronic device 100 can obtain not only the audio signals of the singer's voice 511 and music 512 through the microphone 122, but also the audio signals of the noise 513 caused by cars or pedestrians around the singer.

[0092] The electronic device 100 may identify the region of interest 510 from the entire region of the image 500 based on the user input, and may detect the object of interest 520 included in the region of interest 510. For example, the electronic device 100 may receive a zoom-in input from the user, and zoom in on a specific region according to the zoom-in input. The electronic device 100 may identify the zoomed-in region as the region of interest 510. The electronic device 100 may detect the object included in the region of interest 510 by using an object detection model. According to an embodiment of the present disclosure, the object detection model may be a deep neural network model such as a convolutional neural network, but is not limited thereto. According to an embodiment of the present disclosure, the electronic device 100 may detect the object included in the region of interest 510 by using a well-known image processing model (e.g., pattern matching, feature extraction, or boundary decision). The electronic device 100 may determine the detected object in the region of interest 510 as the object of interest 520.

[0093] The electronic device 100 can obtain multiple image frames 530 by continuously photographing the object of interest 520 over time through video recording. While obtaining the multiple image frames 530, the electronic device 100 can also obtain an audio signal 540 including both the voice 511 and the music 512 output by the object of interest 520 by using the microphone 122.

[0094] like Figure 5In the illustrated embodiment, the user may wish to zoom in on the region of interest 510 including the object of interest 520 (who is a street singer) by using the camera of the electronic device 100, and to record only the voice 511 or the music 512 of the street singer by using the microphone 122. However, since the microphone 122 obtains not only the audio signals of both the voice 511 and the music 512 output by the object of interest 520, but also the audio signal of the noise 513 caused by objects (e.g., cars or pedestrians) surrounding the object of interest 520, it is necessary to attenuate or remove the audio signal of the noise 513. The present disclosure provides an electronic device 100 that is capable of enhancing or amplifying the audio signal generated by the object of interest 520 and attenuating or removing the noise 513 generated by objects other than the object of interest 520 by controlling the audio settings of an audio device including the microphone 122. Reference will be made to Figure 6 , Figure 7 and Figures 8a to 8c An embodiment of the electronic device 100 of the present disclosure controlling an equalizer among audio settings of an audio device based on context information of an object of interest 520 will be described in detail.

[0095] Figure 6 is a diagram illustrating an operation performed by the electronic device 100 according to an embodiment of the present disclosure for applying audio settings corresponding to context information of an object of interest according to a preset mapping relationship.

[0096] Reference Figure 6 , the electronic device 100 may include a camera 110, an audio device 120, and a processor 130. However, Figure 6 Only necessary components for explaining the operation of the electronic device 100 are shown, and the electronic device 100 is not limited to only including Figure 6 Configuration shown.

[0097] The electronic device 100 may obtain an original image 600 by photographing an object in a real space using the camera 110. The camera 110 may provide the processor 130 with an image frame of the obtained original image 600. According to an embodiment of the present disclosure, the camera 110 may provide the processor 130 with camera setting information set by user input. The "camera setting information" may be setting information about photographing using the camera 110 through user input. The camera setting information may include, for example, information about at least one of zooming in / out, panning, or focusing.

[0098] exist Figure 6In the illustrated embodiment, the electronic device 100 may receive a zoom-in input from the camera 110, zoom in on a specific area according to the zoom-in input, and identify the zoomed-in area as the region of interest 610. The processor 130 may provide camera setting information to the image-to-audio setting mapper 134 according to the zoom-in input (e.g., zoom ratio information or information of the region of interest 610).

[0099] The processor 130 can obtain context information from the image frame of the original image 600 obtained by the camera 110 by inputting the image frame to the image processor 132. According to an embodiment of the present disclosure, the image processor 132 can be implemented as an artificial intelligence model. For example, the image processor 132 can be implemented as, for example, a convolutional neural network, but the present disclosure is not limited thereto. The image processor 132 can also be implemented as, for example, a region-based convolutional neural network (R-CNN), a single-shot multi-box detector (SSD) model, YOLO v4, CenterNet, or MobileNet. The image processor 132 can identify an object of interest based on camera setting information, and can obtain context information from the object of interest. According to an embodiment of the present disclosure, the image processor 132 can obtain context information, including at least one of classification information, action information, time information, position information, location information, and surrounding environment information of the object of interest. In Figure 6 In the illustrated embodiment, the processor 130 may identify the object of interest within the region of interest 610 through the image processor 132, and may obtain context information of the object of interest. For example, the processor 130 may identify the object of interest within the bounding box 620 from the image frame of the original image 600, and may obtain information of "singer (classification information)", "street (location information)", or "evening time (time information)" as the context information of the object of interest.

[0100] The processor 130 may obtain a video stream 630 about the object of interest through the image processor 132 .

[0101] The processor 130 may provide the camera setting information obtained by the camera 110 and the context information obtained by the image processor 132 to the image-to-audio setting mapper 134. The image-to-audio setting mapper 134 may include information about a mapping relationship between the context information obtained from the image frame of the original image 600 and the equalizer preset settings of the audio device 120. The mapping relationship between the context information obtained from the image frame and the equalizer preset settings of the audio device 120 may be preset before the original image 600 is obtained. According to an embodiment of the present disclosure, the information about the mapping relationship between the context information and the equalizer preset settings may be stored in Figure 2 The memory 140 will be referenced Figure 7a and Figure 7bTo describe in detail the information about the mapping relationship between context information and equalizer presets.

[0102] The audio device 120 may include a microphone 122, and may obtain an audio signal output by an object by using the microphone 122. According to an embodiment of the present disclosure, the microphone 122 may obtain an original audio stream including voice, music, or noise output by an object.

[0103] The processor 130 may send an audio setting control signal to the audio device 120, and may control the audio device 120 based on the audio setting control signal. According to an embodiment of the present disclosure, the processor 130 may determine the equalizer preset mapped by the image to audio setting mapper 134 to correspond to the context information, and may control the microphone 122 to obtain an audio signal according to the determined equalizer preset. Figure 6 In the illustrated embodiment, the processor 130 may identify an equalizer preset (e.g., an equalizer preset mapped to "music") mapped by the image-to-audio setting mapper 134 to correspond to the context information of the object of interest ("singer", "street", or "night time"), and may control the microphone 122 to obtain an audio signal by using the identified equalizer preset. According to an embodiment of the present disclosure, the processor 130 may control the microphone 122 to amplify a frequency band related to the singer's voice or music among the frequency spectrum of the audio signal included in the original audio stream, and attenuate or remove a frequency band related to the noise or cheering of the surrounding environment. The microphone 122 may obtain an audio signal 640 by applying the audio setting (e.g., the equalizer preset) provided by the processor 130 to the original audio stream.

[0104] Figure 7a is a diagram illustrating an operation performed by the electronic device 100 according to an embodiment of the present disclosure for outputting equalizer presets corresponding to context information according to an image-to-audio signal mapping relationship.

[0105] Reference Figure 7a, the image context to audio setting mapping relationship storage device 700 can store information about the correspondence between multiple contexts 710-1 to 710-n and multiple equalizer presets 720-1 to 720-n. According to an embodiment of the present disclosure, the image context to audio setting mapping relationship storage device 700 can be configured as a non-volatile memory. Non-volatile memory refers to a storage medium that can store and maintain information even when power is not supplied, and the stored information can be used again when power is supplied. The non-volatile memory may include, for example, at least one of a flash memory, a hard disk, a solid-state drive (SSD), a micro multimedia card, a card-type memory (for example, an SD or XD memory), a ROM, a magnetic disk, or an optical disk. According to an embodiment of the present disclosure, the image context to audio setting mapping relationship storage device 700 may be Figure 2 The memory 140 includes or Figure 2 However, the present disclosure is not limited thereto, and the image context to audio setting mapping relationship storage device 700 may be implemented in the form of an external memory not included in the electronic device 100, or may be implemented as a network-based storage medium or cloud server connected via a wired or wireless data communication.

[0106] The image context to audio setting mapping relationship storage device 700 may store information about a plurality of equalizer presets 720-1 to 720-n mapped to correspond to the plurality of contexts 710-1 to 710-n, respectively. According to an embodiment of the present disclosure, a mapping relationship between the plurality of contexts 710-1 to 710-n and the plurality of equalizer presets 720-1 to 720-n may be preset. Figure 7a In the illustrated embodiment, a first context 710-1 representing “music” may be mapped to a first equalizer preset 720-1, a second context 710-2 representing “concert” may be mapped to a second equalizer preset 720-2, and an nth context 710-n representing “voice chat” may be mapped to an nth equalizer preset 720-n.

[0107] The processor 130 of the electronic device 100 (see Figure 2 ) can obtain context information from the image 710a by using the context information obtaining module 146. The processor 130 can determine the scene context corresponding to the equalizer preset based on the context information. The processor 130 can determine the scene context based on the context information by using, for example, a decision tree, fuzzy logic or other well-known algorithms. Figure 7aIn the illustrated embodiment, the processor 130 can obtain contextual information including "performer" (classification information), "piano performance" (action information), "concert hall" (location information) or "audience" (environmental information) from the image 710a through the contextual information acquisition module 146, and can determine that the scene context is "concert" based on the contextual information.

[0108] The processor 130 may identify an equalizer preset that is mapped with the same or similar context information as the obtained scene context from among the plurality of equalizer presets 720-1 to 720-n stored in the image context to audio setting mapping relationship storage device 700. Figure 7a In the illustrated embodiment, the processor 130 may identify, from the image context to audio setting mapping relationship storage device 700, a second equalizer preset 720-2 mapped to correspond to the second context 710-2 representing "concert". For example, the second equalizer preset 720-2 may be an equalizer preset optimized for a "concert hall" by enhancing (or amplifying) a frequency band corresponding to music at a concert. The processor 130 may control an audio device (e.g., Figure 6 The microphone 122) obtains the audio signal.

[0109] Figure 7b is a diagram illustrating an operation performed by the electronic device 100 according to an embodiment of the present disclosure for outputting equalizer presets corresponding to context information according to an image-to-audio signal mapping relationship.

[0110] In addition to the context 710-3 obtained from the image 710b and the identified equalizer preset 720-3, Figure 7b The embodiments shown in Figure 7a The embodiments shown in FIG. 1 are the same as those shown in FIG. 1 , and therefore, redundant descriptions are omitted.

[0111] Reference Figure 7b , the electronic device 100 can obtain context information from the image 710b, and can identify an equalizer preset that is mapped with context information that is the same or similar to the obtained context from among the plurality of equalizer presets 720-1 to 720-n stored in the image context to audio setting mapping relationship storage device 700. Figure 7b In the illustrated embodiment, the processor 130 (see Figure 2) can obtain contextual information including "performer" (classification information), "mountain" (classification information), "tree" (classification information), "piano playing" (action information) or "nature" (environmental information) from image 710b through the contextual information acquisition module 146, and determine that the scene context is "nature" based on the contextual information.

[0112] The processor 130 may identify, from the image context to audio setting mapping relationship storage device 700, a third equalizer preset 720-3 mapped to correspond to the third context 710-3 representing “natural”. For example, the third equalizer preset 720-3 may be an equalizer preset optimized for a “natural environment” by enhancing (or amplifying) a frequency band corresponding to a natural environment. The processor 130 may control an audio device (e.g., Figure 6 The microphone 122) obtains the audio signal.

[0113] Figure 8a is a diagram illustrating an operation performed by the electronic device 100 for determining equalizer presets based on context information according to an embodiment of the present disclosure.

[0114] Reference Figure 8a , the electronic device 100 can obtain context information from the object of interest image 800a. According to an embodiment of the present disclosure, the processor 130 (see Figure 2 ) can be done by using Figure 2 The context information obtaining module 146 analyzes the object of interest image 800a to obtain at least one of the classification information, action information, time information, position information, location information and surrounding environment information of the object of interest. Figure 8a In the illustrated embodiment, the electronic device 100 can obtain at least one piece of information of “singer” (classification information), “action of playing guitar and singing” (action information), “evening time” (time information), and “street” (location information) from the object of interest image 800a.

[0115] The electronic device 100 may display a user interface (UI) 810 for performing an operation and / or function of controlling audio settings based on the context information. For example, the electronic device 100 may display the smart audio settings UI 810 on the display. According to an embodiment of the present disclosure, the electronic device 100 may receive a touch input of a user selecting the smart audio settings UI 810. In response to receiving the touch input, the electronic device 100 determines the equalizer preset of the audio device based on the context information.

[0116] The electronic device 100 may determine the equalizer preset corresponding to the context information based on the preset mapping relationship between the context information obtained from the object of interest image 800a and the equalizer preset. According to an embodiment of the present disclosure, the mapping relationship between the context information and the equalizer preset may be preset, and the preset mapping relationship may be stored in Figure 2 or Figure 2 The mapping relationship between the context information and the equalizer preset is described in the above reference. Figure 7a The descriptions given are the same, so redundant descriptions are omitted. The electronic device 100 can determine the scene context "music" based on the context information of "singer" (classification information), "playing guitar and singing" (action information), "night time" (time information), or "street" (location information), and can determine the music equalizer preset 820 mapped to correspond to "music".

[0117] The electronic device 100 can control Figure 6 The microphone 122 is used to obtain an audio signal by using the determined music equalizer preset 820.

[0118] Figure 8b is a diagram illustrating an operation performed by the electronic device 100 for determining equalizer presets based on context information according to an embodiment of the present disclosure.

[0119] In addition to the object of interest image 800b, context information obtained from the object of interest image 800b, and the equalizer preset 822, Figure 8b The embodiments shown in Figure 8a The same, therefore, redundant description is omitted.

[0120] Reference Figure 8b , the electronic device 100 may obtain at least one of “lecturer” (classification information) and “lecture action” (action information) from the object of interest image 800b. The electronic device 100 may display an intelligent audio setting UI 810 for performing an operation and / or function of controlling audio settings based on the context information, and may receive a touch input of a user selecting the intelligent audio setting UI 810. In response to receiving the touch input, the electronic device 100 determines an equalizer preset of the audio device based on the context information.

[0121] The electronic device 100 may determine the scene context based on the action information among the context information of the object of interest, and may determine the equalizer preset 822 mapped to the scene context. Figure 8bIn the illustrated embodiment, the electronic device 100 can determine the scene context based on the action information (e.g., "lecture") in the context information, and can determine the "voice" equalizer preset 822 mapped to correspond to the scene context from among a plurality of equalizer presets according to a preset mapping relationship.

[0122] The electronic device 100 can control Figure 6 microphone 122 to obtain an audio signal by using the determined “speech” equalizer preset 822.

[0123] Figure 8c is a diagram illustrating an operation performed by the electronic device 100 for determining equalizer presets based on context information according to an embodiment of the present disclosure.

[0124] In addition to the object of interest image 800c, context information obtained from the object of interest image 800c, and the equalizer preset 824, Figure 8c The embodiments shown in Figure 8a The same, therefore, redundant description is omitted.

[0125] Reference Figure 8c , the electronic device 100 may obtain at least one piece of information of “face” (classification information) and “video call” (action information) from the object of interest image 800c. The electronic device 100 may display a smart audio setting UI 810 for performing an operation and / or function of controlling audio settings based on the context information, and may receive a touch input of a user selecting the smart audio setting UI 810. In response to receiving the touch input, the electronic device 100 determines an equalizer preset of the audio device based on the context information.

[0126] The electronic device 100 may determine the scene context based on the action information among the context information of the object of interest, and may determine the equalizer preset 824 mapped to the scene context. Figure 8c In the illustrated embodiment, the electronic device 100 can determine a scene context based on an action context (e.g., "video call"), and can determine a video call equalizer preset 824 mapped to correspond to the scene context from among a plurality of equalizer presets according to a preset mapping relationship.

[0127] The electronic device 100 can control Figure 6 microphone 122 to obtain an audio signal by using the determined video phone equalizer preset 824.

[0128] exist Figure 6 , Figure 7a , Figure 7b as well as Figures 8a to 8cIn the illustrated embodiment, the electronic device 100 may obtain context information from an image of an object of interest, and may determine an equalizer preset corresponding to the context information of the object of interest based on a preset mapping relationship between the context information and the equalizer preset. The electronic device 100 according to an embodiment of the present disclosure may obtain an audio signal through the determined equalizer preset, thereby enhancing only the frequency band of the audio signal associated with the context of the user's object of interest, and improving the quality of the audio signal recorded by the microphone 122. For example, in Figure 8a In the illustrated embodiment, the electronic device 100 may enhance and record only the songs or music of street singers without recording any surrounding noise, or may enhance and record only the songs or music of stage artists. Although not shown in the figure, the electronic device 100 may enhance (or amplify) the sound transmitted through the speaker 124 (see FIG. 1 ) included in the audio device by using the determined equalizer preset. Figure 2 ) can select a specific frequency band related to the object of interest among the frequency bands of the audio signal output by the electronic device 100, and can attenuate or remove frequency bands unrelated to the object of interest. To this end, the electronic device 100 can improve the quality of the audio signal output by the speaker 124.

[0129] Fig. 9 is a diagram illustrating an operation performed by the electronic device 100 for performing filtering on an audio signal based on context information of an image according to an embodiment of the present disclosure.

[0130] Reference Fig. 9 , the electronic device 100 may include a camera 110, an audio device 120, and a processor 130. However, Fig. 9 Only necessary components for explaining the operation of the electronic device 100 are shown, and the electronic device 100 is not limited to only including Fig. 9 Configuration shown.

[0131] The electronic device 100 may obtain an original image 900 by photographing an object in a real space using the camera 110. The camera 110 may provide an image frame of the obtained original image 900 to the processor 130. The processor 130 may detect one or more objects 910, 920, and 922 from the image frame by using an object detection model, and may identify an object of interest 910 from the detected one or more objects 910, 920, and 922. According to an embodiment of the present disclosure, the electronic device 100 may perform at least one of zooming in, panning, or focusing based on a user input, and may identify an object of interest 910 included in an area of ​​interest specified by the performed zooming in, panning, or focusing. Fig. 9 In the illustrated embodiment, the processor 130 may identify a “person” as an object of interest 910 from the image frame.

[0132] The processor 130 can obtain context information from the image frames of the original image 600 obtained by the camera 110 by inputting the image frames to the image processor 132. According to an embodiment of the present disclosure, the image processor 132 can be implemented as an artificial intelligence model. For example, the image processor 132 can be implemented as, for example, a convolutional neural network, but the present disclosure is not limited thereto. The image processor 132 can also be implemented as, for example, a region-based convolutional neural network (R-CNN), a single-shot multi-box detector (SSD) model, YOLO v4, CenterNet, or MobileNet. The image processor 132 can obtain context information from an object of interest. According to an embodiment of the present disclosure, the image processor 132 can obtain context information, including at least one of classification information, action information, time information, position information, location information, and surrounding environment information of the object of interest. In Fig. 9 In the illustrated embodiment, the processor 130 may obtain information of “singer (category information)”, “street (location information)” or “night time (time information)” as context information of the object of interest 910 through the image processor 132 .

[0133] The processor 130 may obtain a video stream 930 regarding an object of interest including the object of interest 910 through the image processor 132 .

[0134] The processor 130 may control the audio settings of the audio device 120 based on the context information obtained by the image processor 132. The audio device 120 may include, for example, a microphone, and may record, through the microphone, speech, music, or noise output by one or more objects 910, 920, and 922. According to an embodiment of the present disclosure, the audio device 120 may obtain a speech signal 912 and a music signal 914 output by an object of interest 910, and may obtain a noise signal 924 output by a car 920 and a pedestrian 922.

[0135] According to an embodiment of the present disclosure, the audio device 120 may include a band stop filter 126 and an amplifier 128. The processor 130 may control the band stop filter 126 and the amplifier 128 of the audio device 120 based on the context information. According to an embodiment of the present disclosure, the processor 130 may control the band stop filter 126 to perform filtering to attenuate or cut off a frequency band of a noise signal 924 output by the remaining objects 920 and 922 other than the object of interest 910 among the one or more objects 910, 920, and 922 detected from the image frame of the original image 900. According to an embodiment of the present disclosure, the processor 130 may amplify the frequency bands of the voice signal 912 and the music signal 914 output by the object of interest 910 by using the amplifier 128.

[0136] The audio device 120 can obtain the audio signal 940 by amplifying only the voice signal 912 and the music signal 914 related to the object of interest 910 under the control of the processor 130, and attenuating or removing the noise signal 924 related to the remaining objects 920 and 922 except the object of interest 910. Fig. 9 In the illustrated embodiment, audio signal 940 may include speech signal 912 and music signal 914 output by object of interest 910 .

[0137] exist Fig. 9 In the illustrated embodiment, the electronic device 100 can obtain context information related to the object of interest 910, and can control the band stop filter 126 and the amplifier 128 of the audio device 120 based on the context information to only enhance (or amplify) and obtain the audio signal 940 related to the object of interest to the user. Therefore, the electronic device 100 according to the embodiment of the present disclosure can obtain only the audio signal 940 desired by the user through the microphone, and effectively attenuate the noise signal 924 that the user does not want, thereby providing a technical effect of improving the quality of the recorded audio signal 940.

[0138] Fig.10 is a diagram illustrating an operation performed by the electronic device 100 for determining preset settings of an audio device based on context information of an object of interest 1010 according to an embodiment of the present disclosure.

[0139] Reference Fig.10 , the electronic device 100 may include a display, and the display may display the image 1000. For example, the electronic device 100 may be a smart TV. According to an embodiment of the present disclosure, the electronic device 100 may obtain the gaze information of the user, and detect the point of interest where the gaze of the user stays within a preset time period. The processor 100 of the electronic device 100 (see Figure 2 ) can detect one or more objects 1010 , 1020 , and 1030 from the displayed image 1000 , and can identify an object located at a point of interest among the one or more objects 1010 , 1020 , and 1030 as the object of interest 1010 .

[0140] The processor 130 may obtain the context information of the object of interest 1010. According to an embodiment of the present disclosure, the processor 140 may obtain the context information of the object of interest 1010 by using the memory 140 (see Figure 2 ) of the context information acquisition module 146 (see Figure 2) to obtain at least one of the classification, action, time, position, location and surrounding environment of the object of interest 1010. According to an embodiment of the present disclosure, the context information acquisition module 146 can be implemented as an artificial intelligence model. For example, the context information acquisition module 146 can be implemented as a convolutional neural network, but the present disclosure is not limited to this. The context information acquisition module 146 can also be implemented as, for example, a region-based convolutional neural network (R-CNN), a single-shot multi-box detector (SSD) model, YOLO v4, CenterNet or MobileNet. Fig.10 In the illustrated embodiment, the processor 130 may obtain contextual information including information about “tennis player” (classification information), “interview” (action information), “tennis court” (location information), or “audience” (environmental information) from the contextual information of the object of interest 1010 through the contextual information acquisition module 146 .

[0141] The processor 130 may determine the preset settings of the speaker 124 based on the contextual information of the object of interest 1010. Fig.10 In the illustrated embodiment, the context information of the object of interest 1010 indicates that a “tennis player” is being “interviewed” on a “tennis court,” so the processor 130 may determine that the preset setting of the speaker 124 is “voice.”

[0142] In the case where the preset setting of the speaker 124 is determined to be “voice”, the processor 130 may amplify the audio signal 1012 having a frequency band related to the object of interest 1010 among the frequency bands of the audio signals 1012, 1022, and 1032 by using the amplifier of the speaker 124. For example, the processor 130 may identify a frequency band corresponding to the voice output by the object of interest 1010 by using information about the frequency band of the audio signal of each object, and may amplify only the audio signal 1012 corresponding to the identified frequency band. As another example, the processor 130 may amplify only the audio signal 1012 related to the voice of the object of interest 1010 by using an audio-visual speech separation algorithm implemented as an artificial intelligence model or by using a pixel-by-pixel audio signal separation algorithm (e.g., PixelPlayer of The Sound of Pixel).

[0143] The processor 130 may attenuate or mute the audio signals 1022 and 1032 in the remaining frequency bands except the frequency band related to the object of interest 1010. According to an embodiment of the present disclosure, the processor 130 may perform filtering by using a band stop filter to attenuate or cut off the frequency bands of the audio signals 1022 and 1032 related to the remaining objects 1020 and 1030 except the object of interest 1010.

[0144] exist Fig.10 In the illustrated embodiment, the electronic device 100 obtains context information about the object of interest 1010 from the image 1000, and controls the audio settings of the speaker 124 based on the context information to enhance only the audio signal 1012 related to the user's object of interest (e.g., the interview voice of the tennis player), thereby providing a technical effect of improving the quality of the audio signal output through the speaker 124. In addition, the electronic device 100 according to an embodiment of the present disclosure can attenuate or mute the noise signals 1022 and 1032 (e.g., cheers from the audience) that interfere with hearing the audio signal 1012 of the object of interest 1010, thereby improving the convenience and satisfaction of use.

[0145] Fig.11 is a diagram illustrating operations performed by the electronic device 100 for controlling audio settings to obtain an audio signal 1112 related to an object of interest 1110 according to an embodiment of the present disclosure.

[0146] Reference Fig.11 , the electronic device 100 may be a glasses-type augmented reality device worn on the user's face. For example, the electronic device 100 may be an augmented reality device such as smart glasses. According to an embodiment of the present disclosure, the electronic device 100 may include an eye tracking sensor that tracks the gaze of the user's left eye and right eye to obtain information about the gaze direction of the user's left eye and right eye. The electronic device 100 may obtain information about the gaze direction of the left eye and information about the gaze direction of the right eye by using the gaze tracking sensor. The processor 130 (see Figure 2 ) The position information of the gaze point G where the gaze direction of the left eye and the gaze direction of the right eye intersect may be obtained by the gaze tracking sensor. The processor 130 may identify the object of interest 1110 based on the position of the gaze point G.

[0147] The processor 130 may control the audio settings of the audio device 120 to obtain the audio signal 1112 related to the object of interest 1110. According to an embodiment of the present disclosure, the processor 130 may amplify the audio signal 1112 having a frequency band related to the object of interest 1110 among the frequency bands of the audio signals 1112 and 1122 by using the amplifier of the microphone 122.

[0148] According to an embodiment of the present disclosure, the processor 130 may attenuate or mute the audio signal 1122 in the remaining frequency bands except the frequency band related to the object of interest 1110. According to an embodiment of the present disclosure, the processor 130 may perform filtering by using a band stop filter to attenuate or cut off the frequency band of the audio signal 1122 related to the remaining objects 1120 except the object of interest 1110.

[0149] According to an embodiment of the present disclosure, the microphone 122 may be configured as a directional microphone or a multi-mode microphone. The processor 130 may control the microphone 122 to obtain the audio signal 1112 only within a predetermined angle range corresponding to the position of the object of interest 1110 .

[0150] According to an embodiment of the present disclosure, the processor 130 may output the audio signal 1112 obtained through the microphone 122 through the speaker 124 .

[0151] exist Fig.11 In the illustrated embodiment, since the electronic device 100 only amplifies and records the voice (audio signal 1112) of the person who is talking to the user (object of interest 1110) among the people around the user, and outputs the recorded voice through the speaker 124, a technical effect is provided that allows the user to listen more attentively to the voice of the person who is talking to the user. In addition, in the case where the user has a hearing impairment, the electronic device 100 can provide a hearing aid-like function to the hearing-impaired person by amplifying the voice of the person who is talking to the user (object of interest 1110) and attenuating or cutting off the sound of the object 1120 that is not related to the person talking to the user.

[0152] Fig.12 is a diagram illustrating operations performed by the electronic device 100 for controlling audio settings to obtain an audio signal 1212 related to an object of interest 1210 according to an embodiment of the present disclosure.

[0153] Reference Fig.12 , the electronic device 100 may be a glasses-type augmented reality device worn on the user's face. For example, the electronic device 100 may be an augmented reality device such as smart glasses. According to an embodiment of the present disclosure, the processor 130 (see Figure 2 ) The position information of the gaze point G where the gaze direction of the user's left eye and the gaze direction of the user's right eye intersect can be obtained by the gaze tracking sensor. The embodiment of obtaining the position information of the gaze point G performed by the processor 130 is the same as that described above with reference to Fig.11 The described embodiments are the same, and thus redundant descriptions thereof will be omitted. The processor 130 may identify the object of interest 1210 based on the position of the gaze point G.

[0154] The processor 130 may control the audio setting of the audio device 120 to obtain the audio signal 1212 related to the object of interest 1210. According to an embodiment of the present disclosure, the processor 130 may amplify the audio signal 1212 having a frequency band related to the object of interest 1210 among the frequency bands of the audio signals 1212 and 1222 by using the amplifier of the microphone 122.

[0155] According to an embodiment of the present disclosure, the processor 130 may attenuate or mute the audio signal 1222 in the remaining frequency bands except the frequency band related to the object of interest 1210. According to an embodiment of the present disclosure, the processor 130 may perform filtering by using a band stop filter to attenuate or cut off the frequency band of the audio signal 1222 related to the remaining objects 1220 except the object of interest 1210.

[0156] According to an embodiment of the present disclosure, the microphone 122 may be configured as a directional microphone or a multi-mode microphone. The processor 130 may control the microphone 122 to obtain the audio signal 1212 only within a predetermined angle range corresponding to the position of the object of interest 1210 .

[0157] According to an embodiment of the present disclosure, the processor 130 may output the audio signal 1212 obtained through the microphone 122 through the speaker 124 .

[0158] exist Fig.12 In the illustrated embodiment, since the electronic device 100 only amplifies and records the machine sound (audio signal 1212) output by the machine (object of interest 1210) used by the worker, and outputs the recorded machine sound through the speaker 124, a technical effect of improving the worker's concentration and work efficiency is provided.

[0159] Fig.13 is a flowchart of an operation performed by an electronic device according to an embodiment of the present disclosure for identifying an object of interest from a cropped image and obtaining an audio signal associated with the object of interest.

[0160] Fig.13 Operations S1310 and S1320 are Figure 3 The detailed operation of operation S320 can be performed in Figure 3 After operation S310 is performed Fig.13 Operation S1310. Fig.13 Operation S1330 is Figure 3 A detailed version of operation S340 is shown.

[0161] Fig.14 is a diagram illustrating an operation performed by the electronic device 100 according to an embodiment of the present disclosure for identifying an object of interest 1440 from a cropped image 1430 and obtaining an audio signal 1412 associated with the object of interest 1440 .

[0162] Now refer to Fig.13 and Fig.14 The operation of the electronic device 100 will be described in detail.

[0163] Reference Fig.13In operation S1310, the electronic device 100 obtains a cropped image based on a user's cropping input. Fig.13 and Fig.14 , the electronic device 100 may include a display and an input interface. The electronic device 100 may display the image 1400 on the display. The processor 130 (see Figure 2 ) can detect one or more objects 1410 and 1420 from the displayed image 1400. The electronic device 100 can receive a user's cropping input for cropping a specific area of ​​the image 1400 through an input interface. The processor 130 can obtain a cropped image 1430 based on the cropping input. Fig.14 In the illustrated embodiment, processor 130 may detect a first object 1410 (which is a person speaking) and a second object 1420 (which is a cat) from image 1400 , and may obtain a cropped image 1430 including only first object 1410 according to a cropping input received from a user.

[0164] Reference Fig.13 In operation S1320, the electronic device 100 identifies an object of interest included in the cropped image. Fig.13 and Fig.14 , the processor 130 of the electronic device 100 may identify the first object 1410 included in the cropped image 1430 as an object of interest.

[0165] Reference Fig.13 In operation S1330, the electronic device 100 amplifies the audio signal related to the object of interest, and attenuates or mutes the audio signal related to the object not included in the cropped image among the at least one object. Fig.13 and Fig.14 , the processor 130 of the electronic device 100 may control the amplifier of the audio device 120 to amplify the audio signal 1412 related to the object of interest 1410 included in the cropped image 1430. For example, the processor 130 may identify the frequency band corresponding to the voice output by the object of interest 1410 by using information about the frequency band of the audio signal of each object, and may amplify only the audio signal 1412 corresponding to the identified frequency band. As another example, the processor 130 may amplify only the audio signal 1412 related to the voice of the object of interest 1410 by using an audio-visual speech separation algorithm implemented as an artificial intelligence model or by using a pixel-by-pixel audio signal separation algorithm (e.g., PixelPlayer of The Sound of Pixel). The processor 130 may obtain the amplified audio signal 1412 through the microphone 122.

[0166] According to an embodiment of the present disclosure, the processor 130 may attenuate or mute the audio signal 1422 in the remaining frequency bands except the frequency band related to the object of interest 1410. For example, the processor 130 may perform filtering by using a band stop filter to attenuate or cut off the frequency band of the audio signal 1422 related to the remaining objects 1420 except the object of interest 1410.

[0167] exist Fig.13 and Fig.14 In the illustrated embodiment, the electronic device 100 can only amplify and obtain the audio signal 1412 related to the object of interest 1410 included in the cropped image 1430 cropped by the user. Therefore, the electronic device 100 according to an embodiment of the present disclosure can obtain an audio signal that meets the user's preference by attenuating or cutting off the cat's sound (audio signal 1422) (which is noise that the user does not expect), and can improve the convenience of the user.

[0168] The present disclosure provides an electronic device 100 for controlling an audio device based on an image. The electronic device 100 according to an embodiment of the present disclosure may include: Figure 2 The audio device 120 includes Figure 2 Camera 110, Figure 2 Microphone 122 and Figure 2 at least one of the speakers 124; Figure 2 A memory 140 storing one or more instructions; and Figure 2 At least one processor 130 of the audio device 120 executes one or more instructions. The at least one processor 130 can detect at least one object from an image captured using the camera 110. The at least one processor 130 can identify a primary object of interest of the user from among the at least one detected object. The at least one processor 130 can obtain context information of the identified primary object of interest. The at least one processor 130 can control the audio settings of the audio device 120 based on the obtained context information to obtain or output an audio signal by using the audio device 120.

[0169] According to an embodiment of the present disclosure, the electronic device 100 may further include an input interface for receiving a user input of one of zooming in, panning, focusing or cropping an image. The at least one processor 130 may identify an area to be zoomed in, panned, focused or cropped based on the user input received through the input interface. The at least one processor 130 may detect an object included in the identified area as a primary object of interest.

[0170] According to an embodiment of the present disclosure, the image processor 130 may obtain at least one of classification information, action information, time information, position information, location information, and surrounding environment information of the main interest object from the image of the main interest object.

[0171] According to an embodiment of the present disclosure, the at least one processor 130 may control at least one of a filter, an amplifier, and an equalizer of the audio device 120 based on the context information.

[0172] According to an embodiment of the present disclosure, the at least one processor 130 may determine the equalizer preset mapped to the context information from among a plurality of pre-stored equalizer presets based on a decision tree or fuzzy logic.

[0173] According to an embodiment of the present disclosure, at least one processor 130 may determine the equalizer preset mapped to correspond to the context information according to a preset mapping relationship.

[0174] According to an embodiment of the present disclosure, at least one processor 130 may detect the motion of the primary object of interest from the image of the primary object of interest. At least one processor 130 may determine the equalizer preset mapped to the detected motion of the primary object of interest according to a preset mapping relationship.

[0175] According to an embodiment of the present disclosure, the at least one processor 130 may amplify a frequency band related to a main object of interest among frequency bands of an audio signal by using an amplifier.

[0176] According to an embodiment of the present disclosure, the at least one processor 130 may perform filtering by using a band stop filter to attenuate or cut off a frequency band of an audio signal related to the remaining objects except the main object of interest among the at least one object.

[0177] According to an embodiment of the present disclosure, the electronic device 100 may further include an input interface configured to receive a user's cropping input for cropping an image. The at least one processor 130 may obtain a cropped image based on the cropping input received through the input interface. The at least one processor 130 may amplify an audio signal related to a primary object of interest included in the cropped image, and may attenuate or mute an audio signal related to an object among at least one object that is not included in the cropped image.

[0178] The present disclosure provides a method for controlling an audio device 120 based on an image, performed by an electronic device 100. The method according to an embodiment of the present disclosure may include operation S310: detecting at least one object from an image captured using a camera 110. The method according to an embodiment of the present disclosure may include operation S320: identifying a user's main object of interest from the at least one detected object. The method according to an embodiment of the present disclosure may include operation S330: obtaining context information of the identified main object of interest. According to an embodiment of the present disclosure, the method may include operation S340: controlling an audio setting of an audio device 120 based on the obtained context information to obtain or output an audio signal, the audio device 120 including at least one of a microphone and a speaker.

[0179] According to an embodiment of the present disclosure, in operation S330 of obtaining context information, at least one of classification information, action information, time information, position information, location information, and surrounding environment information of the main interest object may be obtained from the image of the main interest object.

[0180] According to an embodiment of the present disclosure, in operation S340 of controlling the audio setting of the audio device 120, at least one of a filter, an amplifier, and an equalizer of the audio device 120 may be controlled based on the context information.

[0181] According to an embodiment of the present disclosure, operation S340 of controlling the audio setting of the audio device 120 may include determining an equalizer preset setting mapped to the context information from among a plurality of pre-stored equalizer preset settings based on a decision tree or fuzzy logic.

[0182] According to an embodiment of the present disclosure, the operation S340 of controlling the audio setting of the audio device 120 may include determining, according to a preset mapping relationship, an equalizer preset mapped to correspond to the context information.

[0183] According to an embodiment of the present disclosure, in operation S330 of obtaining context information, the action of the primary object of interest may be detected from the image of the primary object of interest. In the operation of selecting an equalizer preset, the equalizer preset mapped to the detected action of the primary object of interest may be determined according to a preset mapping relationship.

[0184] According to an embodiment of the present disclosure, operation S340 of controlling the audio setting of the audio device 120 may include amplifying a frequency band related to the main object of interest among frequency bands of the audio signal by using an amplifier.

[0185] According to an embodiment of the present disclosure, operation S340 of controlling audio settings of the audio device 120 may include performing filtering by using a band stop filter to attenuate or cut off a frequency band of an audio signal related to remaining objects other than the main object of interest among the at least one object.

[0186] According to an embodiment of the present disclosure, operation S320 of identifying a main object of interest may include operation S1310: obtaining a cropped image based on a user's cropping input for cropping the image. According to an embodiment of the present disclosure, operation S340 of controlling an audio setting of the audio device 120 may include operation S1330: amplifying an audio signal related to the main object of interest included in the cropped image, and attenuating or muting an audio signal related to an object among at least one object that is not included in the cropped image.

[0187] The present disclosure provides a computer program product including a computer-readable storage medium. According to an embodiment of the present disclosure, the storage medium may include instructions that can be read by the electronic device 100 to enable the electronic device 100 to perform: operation S310, detecting at least one object from an image captured using the camera 110; operation S320, identifying a user's main object of interest from the detected at least one object; operation S330, obtaining context information of the identified main object of interest; and operation S340, controlling the audio settings of the audio device 120 based on the obtained context information to obtain or output an audio signal, the audio device 120 including at least one of a microphone and a speaker.

[0188] The program executed by the electronic device 100 in the present disclosure may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. The program may be executed by any system capable of executing computer-readable instructions.

[0189] Software may include computer programs, codes, instructions, or a combination of one or more of the foregoing, and may constitute a processing device so that the processing device may operate as desired, or may independently or collectively issue instructions to the processing device.

[0190] Software may be implemented as a computer program comprising instructions stored in a computer-readable storage medium. Examples of computer-readable recording media include magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical recording media (e.g., CD-ROM or digital versatile disk (DVD)). Computer-readable recording media may be distributed on network-coupled computer systems so that computer-readable code is stored and executed in a distributed manner. These media may be read by a computer, stored in a memory, and executed by a processor.

[0191] The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not include a signal and is tangible, but does not distinguish whether the data is stored in the storage medium semi-permanently or temporarily. For example, the non-transitory recording medium may include a buffer that temporarily stores data.

[0192] The program according to various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product (which is a commodity) may be traded between a seller and a buyer.

[0193] A computer program product may include a software program and a computer-readable storage medium that stores a software program stored thereon. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by a manufacturer of the electronic device 100 or an electronic market (e.g., Samsung Galaxy Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium, or may be temporarily created. In this case, the storage medium may be a storage medium of a server of a manufacturer of the electronic device 100, a server of an electronic market, or a relay server for temporarily storing software programs.

[0194] In a system consisting of the electronic device 100 and / or the server, the computer program product may include a storage medium of the server or a storage medium of the electronic device 100. Alternatively, if there is a third device (e.g., a mobile device) that communicates with the electronic device 100, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the software program itself that is sent from the electronic device 100 to the third device or from the third device to the electronic device 100.

[0195] In this case, one of the electronic device 100 and the third device may execute the computer program product to perform the method according to the disclosed embodiment. Alternatively, at least two of the electronic device 100 and the third device may execute the computer program product to distribute and perform the method according to the disclosed embodiment.

[0196] For example, the electronic device 100 may execute Figure 2 The computer program product stored in the memory 140 is used to control another electronic device (eg, a mobile device) communicating with the electronic device 100 to execute the method according to the disclosed embodiment.

[0197] As another example, the third device may execute the computer program product to control an electronic device communicating with the third device to perform the method according to the disclosed embodiments.

[0198] When the third device executes the computer program product, the third device may download the computer program product from the electronic device 100 and execute the downloaded computer program product. Alternatively, the third device may execute the computer program product provided in a preloaded state to perform the method according to the disclosed embodiment.

[0199] Although the present disclosure has been specifically shown and described with reference to the examples of the present disclosure, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims. For example, even if the above techniques are performed in a different order from the above methods, and / or components such as the above computer systems or modules are coupled or combined in a different form from the above methods, or the above components are replaced or substituted with other components or their equivalents, appropriate results can be obtained.

Claims

1. A method for controlling an audio device based on an image, the method comprising: Camera (110); an audio device (120) comprising at least one of a microphone (122) and a speaker (124); A memory (140) storing one or more instructions; as well as At least one processor (130) configured to execute one or more instructions, The at least one processor (130) is configured to execute the one or more instructions to perform the following operations: detecting at least one object from an image captured using the camera (110); identifying a primary object of interest to a user from the at least one detected object; Obtaining contextual information of the identified primary object of interest; as well as Based on the obtained context information, audio settings of the audio device (120) are controlled to obtain or output an audio signal by using the audio device (120).

2. The electronic device (100) according to claim 1, wherein: The at least one processor (130) is further configured to control at least one of a filter, an amplifier, and an equalizer of the audio device (120) based on the context information.

3. The electronic device (100) according to claim 1 or 2, wherein: The at least one processor (130) is further configured to determine an equalizer preset mapped to the context information from a plurality of pre-stored equalizer presets based on a decision tree or fuzzy logic.

4. The electronic device (100) according to one of claims 1 to 3, wherein: The at least one processor (130) is further configured to: determine, according to a preset mapping relationship, an equalizer preset mapped to correspond to the context information.

5. The electronic device (100) according to one of claims 1 to 4, wherein: The at least one processor (130) is further configured to: amplify a frequency band related to the primary object of interest among the frequency bands of the audio signal by using the amplifier.

6. The electronic device (100) according to one of claims 1 to 5, wherein: The at least one processor (130) is further configured to perform filtering to attenuate or cut off a frequency band of the audio signal related to the remaining objects among the at least one object except the main object of interest by using a band stop filter.

7. The electronic device (100) according to one of claims 1 to 6, further comprising: an input interface configured to receive a cropping input from the user for cropping the image, Wherein, the at least one processor (130) is further configured to: obtaining a cropped image based on a cropping input received through the input interface, An audio signal associated with a primary object of interest included in the cropped image is amplified, and an audio signal associated with an object among the at least one object that is not included in the cropped image is attenuated or muted.

8. A method for controlling an audio device (120) based on an image, performed by an electronic device (100), the method comprising: detecting at least one object from an image captured using the camera (110) (operation S310); identifying a main object of interest to a user from among the at least one detected object (operation S320); Obtaining context information of the identified primary object of interest (operation S330 ); and Based on the obtained context information, audio settings of the audio device (120) are controlled to obtain or output an audio signal, the audio device (120) including at least one of a microphone and a speaker (operation S340).

9. The method according to claim 8, wherein: Controlling the audio settings of the audio device (120) (operation S340) includes: Based on the context information, at least one of a filter, an amplifier and an equalizer of the audio device (120) is controlled.

10. The method according to claim 8 or 9, wherein: Controlling the audio settings of the audio device (120) (operation S340) includes: Based on a decision tree or fuzzy logic, an equalizer preset mapped to the context information is determined from a plurality of pre-stored equalizer presets.

11. The method according to one of claims 8 to 10, wherein: Controlling the audio settings of the audio device (120) (operation S340) includes: According to a preset mapping relationship, an equalizer preset mapped to correspond to the context information is determined.

12. The method according to claim 8, wherein: Controlling the audio settings of the audio device (120) (operation S340) includes: By using the amplifier, a frequency band related to the main object of interest among frequency bands of the audio signal is amplified.

13. The method according to one of claims 8 to 12, wherein: Controlling the audio settings of the audio device (120) (operation S340) includes: By using a band stop filter, filtering is performed to attenuate or cut off a frequency band of the audio signal related to the remaining objects except the main object of interest among the at least one object.

14. The method according to one of claims 8 to 13, wherein: Identifying the primary object of interest (operation S320) includes: obtaining a cropped image based on a cropping input of the user for cropping the image (operation S1310), Controlling the audio settings of the audio device (120) (operation S340) includes: An audio signal associated with a primary object of interest included in the cropped image is amplified, and an audio signal associated with an object among the at least one object that is not included in the cropped image is attenuated or muted.

15. A computer program product comprising a computer-readable storage medium, The storage medium includes instructions related to a method for controlling an audio device (120) based on an image and executed by an electronic device (100), the method comprising: detecting at least one object from an image captured using a camera (110); identifying a primary object of interest to a user from the at least one detected object; Obtaining contextual information of the identified primary object of interest; as well as Based on the obtained context information, audio settings of the audio device (120) are controlled to obtain or output an audio signal, the audio device (120) comprising at least one of a microphone and a speaker.

Images