Electronic device for providing sound corresponding to characteristic information about object and control method therefor

Abstract

This electronic device may comprise a camera, a memory for storing instructions, and at least one processor including processing circuitry, wherein the instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: acquire an image including an object by using the camera; identify characteristic information about the object; and generate a first sound corresponding to the object on the basis of the characteristic information.

Description

Electronic device for providing sound corresponding to characteristic information of an object and method for controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more specifically, to an electronic device and a method for controlling the same that provide a sound corresponding to characteristic information of an object.

[0002] Thanks to advancements in electronic technology, various types of electronic devices are being developed. In particular, devices such as wine cellars for storing wine have recently become widespread, improving user convenience.

[0003] Users can use a smartphone to photograph the wine, recognize the label, and then obtain wine information or identify the location within the wine cellar.

[0004] However, conventionally, smartphones or wine cellars did not recommend wines based on their taste or suggest wines suitable for the current situation.

[0005] In other words, while users wish to select the wine most appropriate for their current situation, it can be difficult to grasp the characteristics of all the numerous wines stored in the cellar or to look them up every time.

[0006] According to one embodiment of the present disclosure for achieving the above objectives, an electronic device comprises at least one processor including a camera, a memory for storing instructions, and a processing circuitry, and when the instructions are executed individually or collectively by the at least one processor, the electronic device can acquire an image including an object using the camera, identify characteristic information about the object, and generate a first sound corresponding to the object based on the characteristic information.

[0007] Additionally, the memory further stores a first neural network model trained to output data including vector data representing sound when input data is input, and when the instructions are executed individually or collectively by the at least one processor, the electronic device inputs at least one of the characteristic information or the image into the first neural network model to obtain first data including a first vector and can generate the first sound based on the first vector.

[0008] And, the memory further stores a plurality of second vectors obtained by encoding a plurality of sound sources, and when the instructions are executed individually or collectively by the at least one processor, the electronic device can identify the second vector among the plurality of second vectors that has the greatest similarity to the first vector and generate the first sound based on the second vector.

[0009] In addition, the first sound can be generated by decoding the first data.

[0010] And, the memory further stores a second neural network model trained to output data including vector data representing a voice type when input data is input, and a third neural network model trained to output sound including a voice sound when data is input, and when the instructions are executed individually or collectively by the at least one processor, the electronic device can input at least one of the characteristic information or the image into the second neural network model to obtain second data, and input the second data into the third neural network model to generate the first sound in the form of a voice sound.

[0011] Additionally, the device further includes at least one shelf and a speaker, and when the instructions are executed individually or collectively by the at least one processor, the electronic device can acquire the image by photographing the shelf through the camera and output the first sound through the speaker when the position of one of the at least one shelves is changed.

[0012] And, when the above instructions are executed individually or collectively by the at least one processor, the electronic device can identify one object when a user designates one of the plurality of objects included in the image.

[0013] Additionally, the electronic device may further include a communication interface, and when the instructions are executed individually or collectively by the at least one processor, the electronic device may receive the characteristic information through the communication interface.

[0014] And, the object is wine, and when the instructions are executed individually or collectively by the at least one processor, the electronic device can identify the characteristic information including at least one of the winery information, type, grape variety, grape region, style, alcohol content, sweetness, acidity, tannin, or body of the wine based on the label of the image.

[0015] Additionally, the electronic device further includes a microphone and a speaker, and when the instructions are executed individually or collectively by the at least one processor, the electronic device can identify one of a plurality of objects within the electronic device when a second sound is received through the microphone, and output the first sound corresponding to the one object through the speaker.

[0016] And, further comprising at least one shelf, and when the instructions are executed individually or collectively by the at least one processor, the electronic device may change the position of the shelf on which one of the at least one shelves is placed.

[0017] Additionally, the electronic device further includes a communication interface, and when the instructions are executed individually or collectively by the at least one processor, the electronic device can identify one of a plurality of objects within the electronic device based on the second sound when a second sound is received from a user terminal device through the communication interface.

[0018] Meanwhile, according to one embodiment of the present disclosure, a control method for an electronic device may include the steps of acquiring an image including an object using a camera, identifying characteristic information about the object, and generating a first sound corresponding to the object based on the characteristic information.

[0019] Additionally, the step of acquiring the first sound may involve inputting at least one of the characteristic information or the image into a first neural network model to acquire first data including a first vector, generating the first sound based on the first data, and the first neural network model may be trained to output data including vector data representing the sound when input data is input.

[0020] The above control method may further include the step of identifying the second vector with the greatest similarity to the first vector among a plurality of second vectors obtained by encoding a plurality of sound sources, and the step of generating the first sound based on the identified second vector.

[0021] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings. In the drawings:

[0022] FIG. 1 is a block diagram showing the configuration of an electronic system according to one or more embodiments of the present disclosure.

[0023] FIG. 2 is a block diagram showing the configuration of an electronic device according to one or more embodiments of the present disclosure.

[0024] FIG. 3 is a block diagram showing the detailed configuration of an electronic device according to one or more embodiments of the present disclosure.

[0025] FIG. 4 is a drawing for explaining an object according to one or more embodiments of the present disclosure.

[0026] FIG. 5 is a drawing for explaining the structure of an electronic device according to one or more embodiments of the present disclosure.

[0027] FIG. 6 is a drawing for illustrating a screen for an inventory state according to one or more embodiments of the present disclosure.

[0028] FIG. 7 is a drawing for explaining the operation when a user selects an object according to one or more embodiments of the present disclosure.

[0029] FIG. 8 is a drawing for explaining the operation when an object is selected through a user terminal device according to one or more embodiments of the present disclosure.

[0030] FIG. 9 is a drawing for explaining characteristic information of an object according to one or more embodiments of the present disclosure.

[0031] FIG. 10 is a drawing for explaining a method for generating a first sound according to one or more embodiments of the present disclosure.

[0032] FIG. 11 is a drawing for explaining a method of generating a first sound as a voice according to one or more embodiments of the present disclosure.

[0033] FIG. 12 is a drawing for explaining the operation when a second sound is received according to one or more embodiments of the present disclosure.

[0034] FIG. 13 is a drawing for explaining the operation when a second sound is received according to one or more embodiments of the present disclosure.

[0035] FIG. 14 is a drawing for explaining a method of learning and using a generative neural network model according to one or more embodiments of the present disclosure.

[0036] FIG. 15 is a drawing for explaining a method of learning and using a generative neural network model according to one or more embodiments of the present disclosure.

[0037] FIG. 16 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.

[0038] Embodiments of the present disclosure may be modified in various ways. Accordingly, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, the present disclosure is not limited to specific embodiments and should be understood to include all modifications, equivalents, and substitutions that do not depart from the spirit and scope of the present disclosure. Furthermore, detailed descriptions of known functions or configurations that may unnecessarily obscure the essence of the present disclosure are omitted.

[0039] The purpose of the present disclosure is to provide an electronic device that provides a sound corresponding to characteristic information of an object and a method for controlling the same.

[0040] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0041] The terms used in the embodiments of this disclosure have been selected to be as widely used as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0042] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, operations, or components such as parts) and do not exclude the presence of additional features.

[0043] The expression "at least one of A or B" should be understood to mean "A" or "B" or "A and B".

[0044] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0045] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "consisting of" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0046] In this specification, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0047] Various embodiments of the present disclosure will be described in more detail below with reference to the attached drawings.

[0048] FIG. 1 is a block diagram showing the configuration of an electronic system (1000) according to one or more embodiments of the present disclosure. As shown in FIG. 1, the electronic system (1000) includes an electronic device (100) and a user terminal device (200).

[0049] The electronic device (100) is a device that outputs sound and can be implemented as a storage device such as a wine cellar or a refrigerator. However, it is not limited thereto, and the electronic device (100) may be a device that generates sound and transmits the generated sound to a speaker, soundbar, TV, projector, desktop PC, laptop, smartphone, tablet PC, smart glasses, smart watch, etc. However, it is not limited thereto, and the electronic device (100) may be any device that can output sound or transmit the generated sound to another electronic device.

[0050] The electronic device (100) can identify an object from an image and acquire a sound corresponding to the object based on characteristic information about the object. For example, the electronic device (100) can identify a wine included in an image and acquire a sound corresponding to the wine based on at least one of the winery information, type, grape variety, grape region, style, alcohol concentration, sweetness, acidity, tannin, or body. However, it is not limited thereto, and the electronic device (100) can acquire a sound based on any other object.

[0051] The electronic device (100) receives information about an object from a user terminal device (200) and may acquire a sound corresponding to the object based on characteristic information corresponding to the object. In this case, the electronic device (100) may transmit the acquired sound to the user terminal device (200).

[0052] Alternatively, the electronic device (100) may receive sound from a user terminal device (200) and identify a recommended object among a plurality of objects placed inside the electronic device (100) based on the sound. In this case, the electronic device (100) may transmit information about the recommended object to the user terminal device (200).

[0053] The user terminal device (200) may be a device that transmits information or sound about an object to the electronic device (100) and receives information about a corresponding object from the electronic device (100). For example, the user terminal device (200) may be implemented as a smartphone, tablet PC, smart glasses, smart watch, speaker, soundbar, TV, projector, desktop PC, laptop, etc. However, it is not limited thereto, and the user terminal device (200) may be any device capable of communicating with the electronic device (100).

[0054] FIG. 2 is a block diagram showing the configuration of an electronic device (100) according to one or more embodiments of the present disclosure.

[0055] According to FIG. 2, the electronic device (100) includes memory (110) and a processor (120).

[0056] Memory (110) may refer to hardware that stores information, such as data, in an electrical or magnetic form so that a processor (120), etc., can access it. To this end, memory (110) may be implemented as at least one piece of hardware among non-volatile memory, volatile memory, flash memory, hard disk drive (HDD) or solid-state drive (SSD), RAM, ROM, etc.

[0057] At least one instruction required for the operation of an electronic device (100) or a processor (120) may be stored in the memory (110). Here, the instruction is a unit of code that directs the operation of the electronic device (100) or the processor (120), and may be written in machine language, which is a language that a computer can understand. Alternatively, a plurality of instructions that perform a specific task of the electronic device (100) or the processor (120) may be stored in the memory (110) as an instruction set.

[0058] Data that is information in bit or byte units capable of representing characters, numbers, images, etc. can be stored in the memory (110). For example, a neural network model, sound source information, etc. can be stored in the memory (110).

[0059] Here, the neural network model may include at least one of a first neural network model trained to output a sound corresponding to the first input data as vector-type data (vector data), a second neural network model trained to output a voice type corresponding to the second input data as vector-type data and output setting information describing the second input data, or a third neural network model trained to output a sound of voice and content corresponding to the third input data. Additionally, the sound source information may include a plurality of second vectors obtained by encoding a plurality of sound sources.

[0060] The memory (110) is accessed by the processor (120), and the processor (120) may perform read / write / modify / delete / update, etc. on instructions, instruction sets, or data.

[0061] The processor (120) controls the overall operation of the electronic device (100). Specifically, the processor (120) can control the overall operation of the electronic device (100) by being connected to each component of the electronic device (100). For example, the processor (120) can control the operation of the electronic device (100) by being connected to a component such as a memory (110).

[0062] One or more processors (120) may include one or more of a CPU, a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), a NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator. One or more processors (120) may control one or any combination of other components of the electronic device (100) and may perform operations or data processing related to communication. One or more processors (120) may execute one or more programs or instructions stored in memory (110). For example, one or more processors (120) may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in memory (110).

[0063] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0064] One or more processors (120) may be implemented as a single-core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When one or more processors (120) are implemented as multicore processors, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by multiple cores may be included in the multicore processor. Additionally, each of the multiple cores included in the multicore processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure.

[0065] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0066] In the embodiments of the present disclosure, one or more processors (120) may refer to a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. However, for convenience of explanation, the operation of the electronic device (100) is described below using the expression "processor (120)."

[0067] The processor (120) can acquire an image containing an object. For example, the processor (120) can acquire an image through a camera included in the electronic device (100). Alternatively, the processor (120) may receive an image from another electronic device, such as a user terminal device (200).

[0068] Here, the image may contain an object. For example, the image may be an image that contains at least one wine as an object. However, it is not limited thereto, and the image may be an image that contains food ingredients, etc., as an object, and any image that contains an object is acceptable.

[0069] The processor (120) can obtain characteristic information about an object. For example, the object is wine, and the processor (120) can obtain at least one of the winery information, type, grape variety, grape region, style, alcohol concentration, sweetness, acidity, tannin, or body of the wine as characteristic information based on a label included in an image.

[0070] Object-specific characteristic information may be stored in the memory (110). In this case, the processor (120) may identify the object and read out characteristic information corresponding to the object based on the information stored in the memory (110). Alternatively, the processor (120) may identify the object, transmit information about the identified object to a server, and receive characteristic information about the object from the server. In this case, the object-specific characteristic information may be stored in the server.

[0071] The processor (120) can obtain a first sound corresponding to an object based on feature information. For example, the processor (120) can obtain a first vector by inputting at least one of feature information or an image into a first neural network model, and obtain a first sound based on the first vector. Here, the first neural network model may be a neural network model trained to output a sound corresponding to the first input data as vector-shaped data.

[0072] The processor (120) can generate a first sound by decoding a first vector. That is, the processor (120) can directly generate a first sound corresponding to an object using a first neural network model.

[0073] Alternatively, the memory (110) may further store a plurality of second vectors obtained by encoding a plurality of sound sources, and the processor (120) may identify the second vector with the greatest similarity to the first vector among the plurality of second vectors, and obtain a first sound by decoding the identified second vector. In this case, the processor (120) may use the sound source that corresponds most to the object among the existing sound sources as the first sound. For example, similarity may be identified in the following manner.

[0074] similarity(A, B) = cos(θ) = (A·B) / (∥A∥∥B∥)

[0075] Alternatively, the processor (120) may input at least one of feature information or an image into a second neural network model to obtain a vector representing a voice type corresponding to an object and descriptive information describing the object, and input the vector and descriptive information into a third neural network model to generate a first sound. Here, the second neural network model may be a neural network model trained to output a voice type corresponding to the second input data as vector-shaped data and to output descriptive information describing the second input data, and the third neural network model may be a neural network model trained to output a sound of voice and content corresponding to the third input data. In this case, the processor (120) may obtain a first sound describing the object.

[0076] The processor (120) can determine at least one of the timbre, pitch, or size of the first sound based on characteristic information. That is, as in the above embodiment, the first sound generated directly, the first sound generated from a sound source, and the first sound describing an object may all have at least one of the timbre, pitch, or size similar.

[0077] The electronic device (100) further includes at least one shelf (rack), a camera, and a speaker, and the processor (120) can acquire an image by photographing one shelf through the camera when the position of one of the at least one shelf changes, and can output the first sound through the speaker when the first sound is acquired. For example, the processor (120) can acquire an image by photographing the shelf through the camera when the shelf protrudes outward from the internal space of the electronic device (100), acquire the first sound based on characteristic information of an object included in the image, and output the first sound through the speaker.

[0078] Alternatively, the processor (120) may acquire an image by photographing one shelf through a camera when the position of at least one shelf is changed and a user's hand is identified on the top of one shelf, and when multiple objects are identified in the image, identify an object among the multiple objects based on the user's hand, acquire a first sound based on the characteristic information of the object, and output the first sound through a speaker. That is, the processor (120) may acquire an image through a camera when the shelf protrudes or when a user's hand is identified.

[0079] However, it is not limited to this, and the processor (120) may acquire an image when the shelf protrudes and identify the object pointed to by the user's hand in the image. In this case, if the user's hand is not identified in the image, the processor (120) may perform re-shooting until the user's hand is identified.

[0080] The camera may be in a turned-off state. Subsequently, the processor (120) may turn on the camera when the door included in the electronic device (100) is opened, and may perform shooting when the shelf protrudes or the user's hand is identified. Subsequently, the processor (120) may turn off the camera when the door is closed.

[0081] Alternatively, the processor (120) may acquire multiple images by continuously taking photos through a camera after the door is opened, and identify an object among the multiple images based on the user's hand. In this case, if the object pointed to by the user's hand changes, the first sound output through the speaker may also change. For example, the processor (120) may continuously acquire multiple images through a camera, and if the user's hand points to the first object, output a first sound corresponding to the characteristic information of the first object through the speaker, and subsequently, if the user's hand changes from the first object to the second object, output a first sound corresponding to the characteristic information of the second object through the speaker.

[0082] The electronic device (100) further includes a communication interface, and the processor (120) can control the communication interface to obtain characteristic information about the object when information about the object is received from the user terminal device (200) through the communication interface, obtain a first sound based on the characteristic information, and transmit the first sound to the user terminal device (200).

[0083] However, it is not limited thereto, and when the processor (120) receives information about an object from the user terminal device (200), it may acquire characteristic information about the object, acquire a first sound based on the characteristic information, and output the first sound through a speaker included in the electronic device (100). Additionally, the processor (120) may control a communication interface to transmit characteristic information about the object to the user terminal device (200).

[0084] The electronic device (100) further includes a microphone and a speaker, and when a second sound is received through the microphone, the processor (120) identifies a recommended object among a plurality of objects placed inside the electronic device (100) based on the second sound and can output a third sound corresponding to the recommended object through the speaker.

[0085] For example, when the processor (120) receives a second sound through a microphone, it may identify a recommended object among a plurality of objects placed inside the electronic device (100) based on the second sound, and output a third sound through a speaker that guides and recommends the location of the recommended object. Additionally, the electronic device (100) may further include at least one shelf, and the processor (120) may change the position of the shelf where the recommended object is placed among the at least one shelf. For example, the processor (120) may protrude the shelf where the recommended object identified based on the second sound is placed from the internal space of the electronic device (100) to the outside.

[0086] The electronic device (100) further includes a communication interface, and when a second sound is received from a user terminal device (200) through the communication interface, the processor (120) may control the communication interface to identify a recommended object among a plurality of objects placed inside the electronic device (100) based on the second sound and transmit information about the recommended object to the user terminal device (200).

[0087] Meanwhile, the artificial intelligence-related functions according to the present disclosure can be operated through the processor (120) and memory (110).

[0088] The processor (120) may be composed of one or more processors. In this case, the one or more processors may be a general-purpose processor such as a CPU, AP, DSP, etc., a graphics-dedicated processor such as a GPU, VPU (Vision Processing Unit), or an artificial intelligence-dedicated processor such as an NPU.

[0089] One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory (110). Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0090] Here, "created through learning" means that a basic artificial intelligence model is trained using multiple learning data by a learning algorithm, thereby creating a predefined rule of operation or an artificial intelligence model configured to perform a desired characteristic (or objective). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0091] An artificial intelligence model can be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through calculations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights can be updated during the learning process so that the loss or cost values ​​obtained by the artificial intelligence model are reduced or minimized.

[0092] Artificial neural networks may include deep neural networks (DNNs), such as, but are not limited to, Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), Generative Adversarial Networks (GANs), or Deep Q-Networks.

[0093] FIG. 3 is a block diagram showing the detailed configuration of an electronic device (100) according to one or more embodiments of the present disclosure. The electronic device (100) may include a memory (110) and a processor (120). Additionally, according to FIG. 3, the electronic device (100) may further include a shelf (130), a camera (140), a speaker (150), a display (155), a communication interface (160), a user interface (170), and a microphone (180). Detailed descriptions of parts of the components shown in FIG. 3 that overlap with the components shown in FIG. 2 are omitted.

[0094] The shelf (130) is positioned in the internal space of the electronic device (100) and can be protruded outward from the internal space of the electronic device (100) by a user. Alternatively, the shelf (130) may include a driving unit, and the processor (120) may control the driving unit to protrude the shelf (130) outward from the internal space of the electronic device (100).

[0095] The shelf (130) can be implemented in a form for storing objects. For example, the shelf (130) may be implemented in a ladder form for storing wine. However, it is not limited to this, and the shelf (130) may be in any form as long as it can store objects.

[0096] The camera (140) is configured to capture still images or video. The camera (140) can capture a still image at a specific point in time, but can also capture a series of still images.

[0097] The camera (140) can capture the front of the electronic device (100) to capture the actual environment in front of the electronic device (100). The processor (120) may also identify a region of interest from the image captured through the camera (140).

[0098] The camera (140) includes a lens, a shutter, an aperture, a solid-state image sensor, an AFE (Analog Front End), and a TG (Timing Generator). The shutter controls the time when light reflected from a subject enters the camera (140), and the aperture controls the amount of light incident on the lens by mechanically increasing or decreasing the size of the opening through which light enters. When light reflected from a subject accumulates as photocharge, the solid-state image sensor outputs an image based on the photocharge as an electrical signal. The TG outputs a timing signal for reading out pixel data from the solid-state image sensor, and the AFE samples and digitizes the electrical signal output from the solid-state image sensor.

[0099] The speaker (150) is a component that outputs various audio data processed by the processor (120), as well as various notification sounds or voice messages.

[0100] The display (155) is configured to display an image and can be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, and a PDP (Plasma Display Panel). The display (155) may also include a driving circuit, a backlight unit, etc., which can be implemented in the form of an a-si TFT, an LTPS (low temperature poly silicon) TFT, or an OTFT (organic TFT). Meanwhile, the display (155) can be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0101] The communication interface (160) is a configuration that performs communication with various types of external devices according to various types of communication methods. For example, an electronic device (100) can perform communication with a user terminal device or a server through the communication interface (160).

[0102] The communication interface (160) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, and a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0103] The Wi-Fi module and Bluetooth module perform communication using the Wi-Fi and Bluetooth methods, respectively. When using the Wi-Fi or Bluetooth module, various connection information, such as the SSID and session key, is transmitted and received first; after establishing a communication connection using this information, various data can be transmitted and received. The infrared communication module performs communication according to infrared communication (IrDA, Infrared Data Association) technology, which wirelessly transmits data over short distances using infrared rays that lie between visible light and millimeter waves.

[0104] In addition to the communication method described above, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0105] Alternatively, the communication interface (160) may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, etc.

[0106] In addition, the communication interface (160) may include at least one of a LAN (Local Area Network) module, an Ethernet module, or a wired communication module that performs communication using a pair cable, a coaxial cable, or a fiber optic cable.

[0107] The user interface (170) may be implemented as a button, touchpad, mouse, and keyboard, or as a touch screen capable of performing display functions and operation input functions. Here, the button may be a various type of button, such as a mechanical button, touchpad, or wheel, formed in any area of ​​the exterior of the main body of the electronic device (100), such as the front, side, or back.

[0108] The microphone (180) is configured to receive sound input and convert it into an audio signal. The microphone (180) is electrically connected to the processor (120) and can receive sound under the control of the processor (120).

[0109] For example, the microphone (180) may be formed as an integrated unit on the upper side, front side, or side side of the electronic device (100). Alternatively, the microphone (180) may be provided in a remote control or the like, separate from the electronic device (100). In this case, the remote control may receive sound through the microphone (180) and provide the received sound to the electronic device (100).

[0110] The microphone (180) may include various configurations such as a microphone that collects analog sound, an amplifier circuit that amplifies the collected sound, an A / D conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0111] Meanwhile, the microphone (180) may be implemented in the form of a sound sensor, and any configuration capable of collecting sound is acceptable.

[0112] As described above, the electronic device (100) can identify an object included in an image and provide sound based on the characteristic information of the object, thereby increasing user convenience.

[0113] The operation of the electronic device (100) will be described in more detail below through FIGS. 4 to 15. FIGS. 4 to 15 describe individual embodiments for convenience of explanation. However, the individual embodiments of FIGS. 4 to 15 may be implemented in any combination.

[0114] FIG. 4 is a drawing for explaining an object according to one or more embodiments of the present disclosure.

[0115] The processor (120) can identify objects included in the image. For example, the processor (120) can identify wine in the image as shown in FIG. 4 and obtain information about the taste of the wine (light, smooth, dry, soft, etc.), comments from users, etc. Alternatively, the processor (120) may obtain at least one of the winery information, type, grape variety, grape production region, style, alcohol content, sweetness, acidity, tannin, or body of the wine based on the wine label.

[0116] However, it is not limited to this, and the processor (120) may obtain various information about the object.

[0117] FIG. 5 is a drawing for explaining the structure of an electronic device (100) according to one or more embodiments of the present disclosure.

[0118] The electronic device (100) may include a plurality of shelves (130) for storing objects inside the electronic device (100). The shelves (130) may protrude outward from the internal space of the electronic device (100). For example, a user may pull a desired shelf (130) to protrude the shelf (130) outward from the internal space of the electronic device (100). Alternatively, the user may select one of the plurality of shelves (130) via a user terminal device (200), and the user terminal device (200) may transmit information regarding the shelf (130) selected by the user to the electronic device (100). A processor (120) may protrude the shelf (130) corresponding to the information received from the user terminal device (200). In this case, each shelf (130) may further include a driving unit.

[0119] The electronic device (100) includes a camera (140), and the processor (120) can obtain an image by photographing the protruding shelf (130) through the camera (140).

[0120] FIG. 6 is a drawing for illustrating a screen for an inventory state according to one or more embodiments of the present disclosure.

[0121] The user can check the inventory status of objects stored in the electronic device (100) through the user terminal device (200). For example, the user terminal device (200) may provide a first screen (610) containing information such as the date of receipt and storage location of the wine, as shown on the left side of FIG. 6. Additionally, the user terminal device (200) may provide a second screen (620) showing the inventory status of wine stored on a plurality of shelves (130) of the electronic device (100), as shown on the right side of FIG. 6.

[0122] According to one or more embodiments, the user terminal device (200) may provide a first screen (610) that displays information about a selected wine in accordance with a user command to select an icon of a specific wine on a second screen (620).

[0123] The processor (120) may perform different operations when an object is being put into or taken out and when a stored object is selected. For example, the processor (120) may acquire multiple images through the camera (140), analyze the multiple images to cause the shelf (130) to protrude, and when a specific object is selected by the user's hand on the shelf (130), output a first sound corresponding to the selected object based on characteristic information of the selected object. Alternatively, the processor (120) may acquire multiple images through the camera (140), analyze the multiple images to identify that an object is being stored in an empty space on the shelf (130), acquire information about the stored object, and provide the acquired information and storage location, etc., to the user terminal device (200). The user terminal device (200) may update the first screen (610) and the second screen (620) based on the received information. For example, the user terminal device (200) can update the second screen (620) based on an image of an object among the received information, and update the first screen (610) based on information describing the object among the received information. The processor (120) can also obtain similar information when an object is released and provide it to the user terminal device (200).

[0124] FIG. 7 is a drawing for explaining the operation when a user selects an object according to one or more embodiments of the present disclosure.

[0125] When the user protrudes the shelf (130), the processor (120) can obtain an image by photographing the shelf (130) through the camera (140) and output a first sound corresponding to the object based on the characteristic information of the object included in the image.

[0126] For example, as illustrated in FIG. 7, when a user protrudes the shelf (130), the processor (120) can obtain an image by photographing the shelf (130) through a camera (140), identify an object pointed to by the user's hand in the image, obtain characteristic information of the object, input at least one of the characteristic information or the image into a first neural network model to obtain a first vector, and output a first sound obtained by decoding the first vector. For instance, if the object is wine, the processor (120) can obtain at least one of the winery information, type, grape variety, grape production region, style, alcohol concentration, sweetness, acidity, tannin, or body as characteristic information based on the wine label.

[0127] If the processor (120) does not identify an object from the image, it may input the image into a first neural network model to obtain a first vector, and decode the first vector to output the obtained first sound.

[0128] However, it is not limited to this, and the processor (120) may continuously acquire multiple images through the camera (140) when the door included in the electronic device (100) is opened.

[0129] FIG. 8 is a drawing for explaining the operation when an object is selected through a user terminal device (200) according to one or more embodiments of the present disclosure.

[0130] The user terminal device (200) may output a first sound associated with an object under the control of the user. For example, the user may take a picture of wine (810) using the user terminal device (200), as shown at the top of FIG. 8. In this case, the user terminal device (200) may transmit the captured image containing the wine (810) to the electronic device (100).

[0131] When the processor (120) receives a captured image from the user terminal device (200), it can acquire a first sound corresponding to the wine (810) based on characteristic information of the wine (810) containing the captured image, and transmit the first sound to the user terminal device (200). The user terminal device (200) can output the first sound received from the electronic device (100).

[0132] The processor (120) identifies the distance to the user terminal device (200), and if the distance to the user terminal device (200) is greater than or equal to a preset distance, it transmits the first sound to the user terminal device (200), and if the distance to the user terminal device (200) is less than a preset distance, it may output the first sound directly.

[0133] Alternatively, the user may select an icon (830) of a specific wine on a second screen (820) displayed by the user terminal device (200), as shown at the bottom of FIG. 8. The user terminal device (200) may transmit information about the selected wine to the electronic device (100) in accordance with the user command to select an icon of a specific wine on the second screen (820).

[0134] When the processor (120) receives information about a selected wine from the user terminal device (200), it may acquire a first sound corresponding to the wine based on the characteristic information of the wine and transmit the first sound to the user terminal device (200). The user terminal device (200) may output the first sound received from the electronic device (100). Additionally, the user terminal device (200) may output the first sound and display a first screen showing information about the selected wine.

[0135] The processor (120) identifies the distance to the user terminal device (200), and if the distance to the user terminal device (200) is greater than or equal to a preset distance, it transmits the first sound to the user terminal device (200), and if the distance to the user terminal device (200) is less than a preset distance, it may output the first sound directly.

[0136] FIG. 9 is a drawing for explaining characteristic information of an object according to one or more embodiments of the present disclosure.

[0137] The processor (120) can identify objects from an image. When multiple objects are identified from an image, the processor (120) can identify one of the multiple objects based on the user's hand.

[0138] When an object is identified, the processor (120) can obtain characteristic information of the object. For example, when a wine is identified, the processor (120) can obtain at least one of the winery information, type, grape variety, grape production region, style, alcohol concentration, sugar content, acidity, tannin, or body of the wine as characteristic information based on the wine label. For instance, the characteristic information of the wine may include type and specific information by category, as shown in FIG. 9.

[0139] However, it is not limited to this, and the processor (120) may acquire characteristic information of various objects, and the categories included in each characteristic information may also be diverse.

[0140] FIG. 10 is a drawing for explaining a method for generating a first sound according to one or more embodiments of the present disclosure. In FIG. 10, for convenience of explanation, the electronic device (100) is described as a wine cellar (1010) and the object is described as wine.

[0141] The wine cellar (1010) can identify wine from an image and obtain information (1020) about the wine by identifying the wine.

[0142] A prompt generator (1030) can obtain characteristic information of the wine based on the identified wine and generate an input prompt (1040) based on the characteristic information of the wine. For example, the prompt generator (1030) can generate an input prompt (1040) as follows.

[0143]

[0144] prompt = f"""Create a music piece that pairs perfectly with the following wine:

[0145] Winery: {winery}

[0146] Type: {wine_type}

[0147] Grapes: {grapes}

[0148] Region: {region}

[0149] Wine Style: {wine_style}

[0150] Alcohol Content: {alcohol_content}%

[0151] Sweetness: {sweetness} / 5

[0152] Acidity: {Acidity} / 5

[0153] Tannin: {tannin} / 5

[0154] Body: {Body} / 5

[0155] The music should evoke the characteristics and ambiance of this wine, capturing its essence and the experience of enjoying it."""

[0156]

[0157] The first neural network model (generative AI model, 1060) can receive an input prompt (1040) and a wine image (1050) and output a first sound corresponding to the wine.

[0158] Although the above describes the case where the wine is identified, the wine may not be identified. In this case, the prompt generator (1030) may generate an input prompt (1040) as follows.

[0159]

[0160] prompt = f"""Create a music piece that pairs perfectly with the given image.

[0161] The music should evoke the characteristics and ambiance of this wine, capturing its essence and the experience of enjoying it."""

[0162]

[0163] The first neural network model (1060) can receive an input prompt (1040) and a wine image (1050) and output a first sound corresponding to the wine, and in this case, the first neural network model (1060) can output a first sound based on the wine image (1050).

[0164] FIG. 11 is a drawing for explaining a method of generating a first sound as a voice according to one or more embodiments of the present disclosure. In FIG. 11, for convenience of explanation, the electronic device (100) is described as a wine cellar (1110) and the object is described as wine.

[0165] The wine cellar (1110) can identify wine from an image and obtain information (1120) about the wine by identifying the wine.

[0166] The second neural network model (AI model, 1140) receives information about the wine (1120) and a wine image (1130) as input, and can output a vector (1150) representing a voice type corresponding to the wine and a prompt (description information, 1160) describing the wine.

[0167] However, it is not limited to this, and the second neural network model (1140) may receive wine characteristic information and a wine image (1130) generated through a prompt generator, and output a vector (1150) representing a voice type corresponding to the wine and a prompt (1160) describing the wine.

[0168] A third neural network model (generative AI model, 1170) can receive a vector (1150) and a prompt (1160) as inputs and output a first sound corresponding to wine. Here, the first sound may be information that converts the content of the prompt (1160) into sound using a voice corresponding to the vector (1150).

[0169] The second neural network model (1140) may be a model that has been trained with sample information about wine and sample wine images as input data, and sample vectors representing voice types corresponding to wine and sample prompts describing wine as output data.

[0170] The first neural network model (1060) of FIG. 10 and the third neural network model (1170) of FIG. 11 are generative neural network models, and FIG. 14 explains an example of the learning method.

[0171] FIGS. 12 and FIGS. 13 are drawings for explaining the operation when a second sound is received according to one or more embodiments of the present disclosure.

[0172] When the processor (120) receives a second sound through the microphone (180), it can identify a recommended object among a plurality of objects placed inside the electronic device (100) based on the second sound and change the position of the shelf (130) on which the recommended object is placed.

[0173] For example, the user can output a second sound through a user terminal device (200), and the processor (120) can receive the second sound through a microphone (180), as shown in the upper left of FIG. 12. The processor (120) identifies the wine that best matches the second sound as a recommended wine, and can protrude a shelf (130) on which the recommended wine is placed from the internal space of the electronic device (100) to the outside, as shown in the upper right of FIG. 12.

[0174] Alternatively, the electronic device (100) further includes a plurality of light-emitting elements corresponding to each of a plurality of shelves (130), and the processor (120) may turn on a light-emitting element (1210-1, 1210-2) adjacent to a shelf (130) on which a recommended wine is placed, as shown at the bottom of FIG. 12.

[0175] When the processor (120) receives a second sound from a user terminal device (200) through the communication interface (160), it may identify a recommended object among a plurality of objects placed inside the electronic device (100) based on the second sound and control the communication interface (160) to transmit information about the recommended object to the user terminal device (200).

[0176] For example, the user terminal device (200) may provide a screen for recording sound, as shown on the left side of FIG. 13, and when a second sound is recorded according to the user's control, the second sound may be transmitted to the electronic device (100). The processor (120) may identify a recommended object among a plurality of objects placed inside the electronic device (100) based on the second sound, and may control the communication interface (160) to transmit information about the recommended object to the user terminal device (200). The user terminal device (100) may provide a second screen including a focus (1330) that displays the recommended object based on information about the recommended object, as shown on the right side of FIG. 13.

[0177] FIGS. 14 and 15 are drawings for explaining a method of learning and using a generative neural network model according to one or more embodiments of the present disclosure.

[0178] First, the processor (120) can learn the model (1470) as shown in FIG. 14.

[0179] For example, the processor (120) can obtain a first vector (1420) that vectorizes the input audio (1410) in the forward process, and obtain a first vector (1440) with added noise by adding the first vector (1420) and a random noise vector (1430).

[0180] The processor (120) can obtain predicted noise (1480) by inputting the first vector (1440) with added noise, the wine image (1450), and the prompt (1460) into the model (1470) in a backward process.

[0181] The processor (120) can train a model (1470) so that the difference between the random noise vector (1430) in the forward process and the prediction noise (1480) in the backward process is reduced.

[0182] In FIG. 14, for convenience of explanation, the model (1470) is described as being one, but it is not limited thereto. For example, the processor (120) may perform learning by using multiple models (1470) in succession.

[0183] When the processor (120) is completed, it can acquire a first sound by successively using a plurality of models (1540, 1540') that have been completed, as shown in FIG. 15.

[0184] For example, the processor (120) may input noise (1510), a wine image (1520), and a prompt (1530) to the first model (1540) and input the output to the second model (1540'). The processor (120) may repeat this process to obtain predicted noise (1550), and by subtracting the predicted noise (1550) from the noise (1510), a predicted vector (1560) may be obtained. The processor (120) may decode the predicted vector (1560) to obtain a first sound (predicted audio).

[0185] FIG. 16 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.

[0186] First, an image containing an object is obtained (S1610). Then, characteristic information about the object is obtained (S1620). Then, based on the characteristic information, a first sound corresponding to the object is obtained (S1630).

[0187] Additionally, the step of acquiring the first sound (S1630) involves inputting at least one of feature information or an image into a first neural network model to acquire a first vector, and acquiring the first sound based on the first vector, and the first neural network model may be a neural network model trained to output a sound corresponding to the first input data as vector-shaped data.

[0188] And, the step of acquiring the first sound (S1630) can identify the second vector with the greatest similarity to the first vector among the multiple second vectors acquired by encoding multiple sound sources, and acquire the first sound by decoding the identified second vector.

[0189] Additionally, the step of acquiring the first sound (S1630) can generate the first sound by decoding the first vector.

[0190] And, the step of acquiring the first sound (S1630) inputs at least one of feature information or an image into a second neural network model to acquire a vector representing a voice type corresponding to an object and descriptive information describing the object, and inputs the vector and descriptive information into a third neural network model to generate the first sound, wherein the second neural network model is a neural network model trained to output a voice type corresponding to the second input data as vector-shaped data and output descriptive information describing the second input data, and the third neural network model may be a neural network model trained to output a sound of voice and content corresponding to the third input data.

[0191] Additionally, the step of acquiring an image (S1610) may further include the step of acquiring an image by photographing one shelf through a camera included in the electronic device when the position of one of at least one shelf included in the electronic device is changed, and the control method may further include the step of outputting the first sound through a speaker included in the electronic device when the first sound is acquired.

[0192] And, the step of acquiring an image (S1610) involves changing the position of at least one shelf and, when a user's hand is identified on the top of one shelf, capturing one shelf through a camera to acquire an image, and, when multiple objects are identified in the image, identifying an object among the multiple objects based on the user's hand.

[0193] Additionally, the step of acquiring characteristic information (S1620) may further include the step of acquiring characteristic information for an object when information about an object is received from a user terminal device, and the control method may further include the step of transmitting a first sound to the user terminal device.

[0194] And, the object is wine, and the step of acquiring characteristic information (S1620) can acquire at least one of the winery information, type, grape variety, grape production region, style, alcohol concentration, sweetness, acidity, tannin, or body of the wine as characteristic information based on the label included in the image.

[0195] Additionally, when a second sound is received through a microphone included in the electronic device, the method may further include the steps of identifying a recommended object among a plurality of objects placed inside the electronic device based on the second sound, and outputting a third sound corresponding to the recommended object through a speaker included in the electronic device.

[0196] And, the method may further include a step of changing the position of the shelf on which the recommended object is placed among at least one shelf included in the electronic device.

[0197] Additionally, when a second sound is received from a user terminal device, the method may further include the steps of identifying a recommended object among a plurality of objects placed inside an electronic device based on the second sound and transmitting information about the recommended object to the user terminal device.

[0198] And, the step of acquiring the first sound (S1630) can determine at least one of the timbre, pitch, or magnitude of the first sound based on characteristic information.

[0199] According to various embodiments of the present disclosure as described above, an electronic device can identify an object included in an image and provide sound based on characteristic information of the object, thereby enhancing user convenience.

[0200] Meanwhile, according to one or more embodiments of the present disclosure, the various embodiments described above may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily on the storage medium.

[0201] Additionally, according to one or more embodiments of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed online in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0202] Additionally, according to one or more embodiments of the present disclosure, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented as the processor itself. According to software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software. Each of the software may perform one or more functions and operations described herein.

[0203] Meanwhile, computer instructions for performing processing operations of the device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When computer instructions stored in such a non-transitory computer-readable medium are executed by the processor of a specific device, they cause the specific device to perform processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of a non-transitory computer-readable medium may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, ROMs, etc.

[0204] Additionally, each component (e.g., module or program) according to the various embodiments described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the respective components prior to integration. The operations performed by the module, program, or other components according to the various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

[0205] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device, camera; Memory for storing instructions; and at least one processor including processing circuitry; and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An image containing an object is obtained using the above camera, and Identify characteristic information regarding the above object, and An electronic device that generates a first sound corresponding to the object based on the above characteristic information.

2. In Paragraph 1, The above memory is, A first neural network model is further stored that is trained to output data including vector data representing sound when input data is input, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, At least one of the above characteristic information or the above image is input into the first neural network model to obtain first data including a first vector, and An electronic device that generates the first sound based on the first vector.

3. In Paragraph 2, The above memory is, A plurality of second vectors obtained by encoding a plurality of sound sources are further stored, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Identify the second vector among the plurality of second vectors that has the greatest similarity to the first vector, and An electronic device that generates the first sound based on the second vector.

4. In Paragraph 2, An electronic device that generates the first sound by decoding the first data to generate the first sound.

5. In Paragraph 1, The above memory is, Further storing a second neural network model trained to output data including vector data representing a voice type when input data is input, and a third neural network model trained to output sound including a voice sound when data is input, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, At least one of the above characteristic information or the above image is input into the second neural network model to obtain second data, and An electronic device that inputs the second data into the third neural network model to generate the first sound in the form of a voice sound.

6. In Paragraph 1, At least one shelf; and Includes additional speakers, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When the position of one of the above at least one shelf is changed, the one shelf is photographed through the camera to obtain the image, and An electronic device that outputs the first sound through the speaker.

7. In Paragraph 6, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that identifies one object when a user designates one of a plurality of objects included in the above image.

8. In Paragraph 1, It further includes a communication interface; and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that receives the characteristic information through the communication interface.

9. In Paragraph 1, The above object is, It is wine, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that identifies specific information including at least one of the winery information, type, grape variety, grape region, style, alcohol content, sweetness, acidity, tannin, or body of the wine based on the label of the image above.

10. In Paragraph 1, Microphone; and Includes additional speakers, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When a second sound is received through the microphone, one of the plurality of objects inside the electronic device is identified, and An electronic device that outputs the first sound corresponding to the above-mentioned object through the speaker.

11. In Paragraph 10, It further includes at least one shelf, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that changes the position of the shelf on which the object is placed among the at least one shelf.

12. In Paragraph 1, It further includes a communication interface; and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that identifies one of a plurality of objects within the electronic device based on the second sound when a second sound is received from a user terminal device through the communication interface.

13. In a method for controlling an electronic device, A step of acquiring an image containing an object using a camera; A step of identifying characteristic information for the above object; and A control method comprising the step of generating a first sound corresponding to the object based on the above characteristic information.

14. In Paragraph 13, The step of acquiring the first sound above is, At least one of the above characteristic information or the above image is input into a first neural network model to obtain first data including a first vector, and Generating the first sound based on the first data above, The above first neural network model is, A control method that is trained to output data containing vector data representing sound when input data is input.

15. In Paragraph 14, A step of identifying the second vector with the greatest similarity to the first vector among a plurality of second vectors obtained by encoding a plurality of sound sources; and A control method further comprising the step of generating the first sound based on the second vector identified above.

Images