Method for providing information and electronic device for supporting the same

By using input modules and processors in electronic devices to identify and control speaker output, the problem of indistinguishable virtual keyboard audio and user voice is solved, and clear distinction and processing of multiple input information is achieved.

CN118984984APending Publication Date: 2024-11-19SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380028577.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-07-29
Filing Date
2023-02-22
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

When existing electronic devices output audio generated by virtual keyboards, it is difficult to distinguish between user voice and audio output from the device, resulting in difficult to distinguish between output and process when processing multiple inputs such as voice, posture and facial expressions.

Method used

Using an electronic device including an input module, speaker and processor, by identifying the input type and controlling the speaker output, ensures that the output is distinguishable from the input corresponding to the input.

Benefits of technology

It realizes that when processing multiple inputs, the information output by the speaker can be distinguished, improving the clarity of user experience and information transmission.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984984A_ABST
    Figure CN118984984A_ABST
Patent Text Reader

Abstract

An electronic device according to various embodiments of the present disclosure comprises: one or more input modules; one or more speakers; a display; and at least one processor operably connected to the one or more input modules, the one or more speakers, and the display, where the at least one processor can be configured to: acquire an input through the one or more input modules; confirming the type of the input; determining whether information corresponding to the input is to be output through the one or more speakers; and based on a determination that the information corresponding to the input is to be output through the one or more speakers, controlling the one or more speakers such that the information corresponding to the input is output through the one or more speakers according to a type of the input. Other various embodiments are possible.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Various embodiments of the present disclosure relate to a method for providing information and an electronic device supporting the method. Background Art

[0002] With the development of communication technology, electronic devices are becoming smaller and lighter so as to be used without significant inconvenience even when worn on the user's body. For example, wearable electronic devices such as head mounted devices (HMDs), smart watches (or bands), contact lens type devices, ring type devices, glove type devices, shoe type devices, or clothing type devices have been commercialized. Wearable electronic devices are worn directly on the body, so that portability and user usability can be improved.

[0003] An HMD is a device worn on the user's head or face, and can provide augmented reality (AR) to the user. For example, an HMD providing AR can be configured in the form of glasses, and can provide information about an object to the user in the form of an image or text in at least a portion of the space within the user's field of view. An HMD can provide virtual reality (VR) to the user. For example, an HMD can output independent images for both eyes of the user, respectively, and output content provided from external input to the user in the form of video or sound, thereby providing an excellent sense of immersion. In addition, an HMD can provide mixed reality (MR) and / or extended reality (XR).

[0004] The HMD can receive or send information through the input / output device. For example, the HMD can obtain external sound (e.g., the voice of the other party) through a microphone. The HMD can convert the obtained voice into an image or audio form and output the voice, thereby providing the user wearing the HMD (e.g., a hearing-impaired person) with information indicated by the obtained voice. In addition, the HMD can generate information to be delivered to the other party via a virtual keyboard displayed on the HMD, can convert the generated information into audio, and can output the audio through an output device (e.g., a speaker). Summary of the invention

[0005] Technical issues

[0006] When an electronic device (e.g., HMD) outputs information generated using a virtual keyboard as audio, the audio output by the electronic device may be indistinguishable from the user's voice (sound). For example, the electronic device may obtain input via a virtual keyboard. The electronic device may convert the obtained input into text, and may output audio corresponding to the converted text through a speaker. When a user speaks while audio is output from the electronic device, the other party may not be able to distinguish between the audio output by the electronic device and the voice uttered by the user.

[0007] The electronic device may receive various inputs at the same time, such as the user's voice or facial expression and the user's gesture. In this case, the electronic device may only process and output the input related to the user's gesture among the various inputs entered at the same time, and may have difficulty processing other inputs.

[0008] When outputs corresponding to various inputs entered into an electronic device (eg, a user's voice, a user's gesture, and a user's facial expression) are output through one output device, the outputs corresponding to the various inputs may be indistinguishable from each other.

[0009] Various embodiments of the present disclosure relate to a method of providing information and an electronic device supporting the method, which are capable of distinguishably outputting outputs corresponding to various inputs entered into the electronic device.

[0010] Technical aspects to be implemented in the present disclosure are not limited to the above-mentioned technical aspects, and other technical aspects not mentioned will be clearly understood by those skilled in the art from the following description.

[0011] Technical Solution

[0012] According to various embodiments of the present disclosure, an electronic device may include: one or more input modules; one or more speakers; a display; and at least one processor, wherein the at least one processor is operably connected to the one or more input modules, the one or more speakers and the display, wherein the at least one processor may be configured to: obtain input through the one or more input modules; identify the type of the input; determine whether to output information corresponding to the input through the one or more speakers; and based on determining that the information corresponding to the input is to be output through the one or more speakers, control the one or more speakers so that the information corresponding to the input is output through the one or more speakers based on the type of the input.

[0013] According to various embodiments of the present disclosure, a method for an electronic device to provide information may include: obtaining input through one or more input modules of the electronic device; identifying the type of the input; determining whether to output information corresponding to the input through the one or more speakers; and based on determining that the information corresponding to the input is to be output through the one or more speakers, controlling the one or more speakers so that the information corresponding to the input is output through the one or more speakers based on the type of the input.

[0014] Beneficial Effects

[0015] A method of providing information and an electronic device supporting the same according to various embodiments of the present disclosure enable outputs corresponding to various inputs entered into the electronic device to be distinguishably output.

[0016] Effects obtainable from the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram of an electronic device in a network environment according to various embodiments;

[0018] Figure 2 shows the structure of an electronic device according to various embodiments;

[0019] Figure 3 shows the structure of an eye tracking camera of an electronic device according to various embodiments;

[0020] Figure 4 An electronic device according to various embodiments is shown;

[0021] Figure 5 is a block diagram of an electronic device according to various embodiments;

[0022] Figure 6 is a block diagram of a processor according to various embodiments;

[0023] Figure 7 is a flowchart illustrating a method of providing information according to various embodiments;

[0024] Figure 8 is a flowchart illustrating a method of providing information corresponding to a gesture according to various embodiments;

[0025] Fig. 9 is a flowchart illustrating a method of providing information corresponding to a gesture according to various embodiments;

[0026] Fig.10 is a flowchart illustrating a method of providing information corresponding to a gesture and information corresponding to a voice according to various embodiments;

[0027] Fig.11 is a flowchart illustrating a method of providing information corresponding to a gesture while conducting a call according to various embodiments;

[0028] Fig.12 is a flowchart illustrating a method of providing information corresponding to a user's voice while making a call according to various embodiments;

[0029] Fig.13is a flowchart illustrating a method of providing information corresponding to a gesture and information corresponding to a voice while a call is being made according to various embodiments;

[0030] Fig.14 is a flowchart illustrating a method of providing information corresponding to a gesture and information corresponding to a voice while a call is being made according to various embodiments;

[0031] Fig.15 is a flowchart illustrating a method of providing information corresponding to a gesture and information corresponding to a voice according to various embodiments;

[0032] Fig.16 is a flowchart illustrating a method for providing information based on surrounding environment according to various embodiments;

[0033] Fig.17 is a flow chart illustrating a method of providing information corresponding to a plurality of inputs according to various embodiments. DETAILED DESCRIPTION

[0034] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments.

[0035] Reference Figure 1 , the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the above components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the above-described components (eg, sensor module 176, camera module 180, or antenna module 197) may be implemented as a single component (eg, display module 160).

[0036] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 coupled to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operationally independent or combined with the main processor 121. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or as part of the main processor 121.

[0037] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated by machine learning. For example, such learning can be performed by the electronic device 101 where the artificial intelligence model is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0038] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0039] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0040] The input module 150 may receive a command or data to be used by another component (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0041] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as playing multimedia or playing records. The receiver may be used to receive incoming calls. Depending on the embodiment, the receiver may be implemented as a separate part from the speaker, or as part of the speaker.

[0042] The display module 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display module 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0043] The audio module 170 can convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) coupled to the electronic device 101 or wirelessly coupled.

[0044] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a posture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0045] The interface 177 may support one or more specific protocols for coupling the electronic device 101 with an external electronic device (e.g., electronic device 102) (e.g., wired) or wirelessly. According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0046] The connection end 178 may include a connector, and the electronic device 101 may be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0047] The haptic module 179 may convert the electric signal into mechanical stimulation (eg, vibration or movement) or electric stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0048] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0049] The power management module 188 may manage power supply to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0050] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0051] The communication module 190 can support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)), and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 198 (e.g., a short-range communication network such as Bluetooth TM , Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (for example, a long-distance communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (for example, a LAN or a wide area network (WAN)))) to communicate with an external electronic device. These various types of communication modules can be implemented as a single component (for example, a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (for example, multiple chips). The wireless communication module 192 can use the user information (for example, the International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196 to identify and authenticate the electronic device 101 in the communication network (such as the first network 198 or the second network 199).

[0052] The wireless communication module 192 can support 5G networks after 4G networks and next-generation communication technologies (e.g., new radio (NR) access technologies). NR access technologies can support enhanced mobile broadband (eMBB), massive machine type communications (mMTC), or ultra-reliable low-latency communications (URLLC). The wireless communication module 192 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for ensuring performance on high-frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 can support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 can support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane latency for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip).

[0053] The antenna module 197 may transmit a signal or power to the outside of the electronic device 101 (e.g., an external electronic device) or receive a signal or power from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiating element, the radiating element being formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component other than the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

[0054] According to various embodiments, the antenna module 197 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, an RFIC, and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a specified high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the specified high frequency band.

[0055] At least some of the above components can be coupled to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0056] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 coupled to the second network 199. Each of the electronic device 102 or the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be run at the electronic device 101 may be run at one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically perform a function or service or should perform a function or service in response to a request from a user or another device, the electronic device 101 may request one or more external electronic devices to perform at least part of the function or service instead of running the function or service, or the electronic device 101 may request one or more external electronic devices to perform at least part of the function or service in addition to running the function or service. The one or more external electronic devices that receive the request may perform at least the requested part of the function or service, or perform another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 can provide the result as at least a partial reply to the request when the result is further processed or when the result is not further processed. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology or client-server computing technology, for example, can be used. The electronic device 101 can use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 can be applied to intelligent services (e.g., smart homes, smart cities, smart cars or health care) based on 5G communication technology or IoT-related technologies.

[0057] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those described above.

[0058] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or replacement forms for corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the noun in the singular form corresponding to the item may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being "coupled", "coupled to", "connected to" or "connected to" another element (e.g., the second element) with or without the terms "operably" or "communicatively", it means that the element may be directly (e.g., wired) coupled to the other element, wirelessly connected to the other element, or coupled to the other element via a third element.

[0059] As used in connection with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, depending on the embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0060] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor (e.g., processor 120) of a machine (e.g., electronic device 101), the processor may call at least one of the one or more instructions stored in the storage medium and execute at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0061] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., PlayStore). TM ) The computer program product may be distributed (e.g., downloaded or uploaded) online, or may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If distributed online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a forwarding server).

[0062] According to various embodiments, each component (e.g., module or program) in the above-mentioned components may include a single entity or multiple entities, and some of the multiple entities may be separately arranged in different components. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform one or more functions of each component in the multiple components in the same or similar manner as a corresponding one of the multiple components performing one or more functions before integration. According to various embodiments, the operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operation may be run in a different order or omitted, or one or more other operations may be added.

[0063] Figure 2 The structure of an electronic device 201 according to various embodiments is shown.

[0064] Reference Figure 2 In an embodiment, the electronic device 201 (e.g., the electronic device 101) may include one or more first cameras 211-1 and 211-2, one or more second cameras 212-1 and 212-2, and one or more third cameras 213. In an embodiment, images obtained by the one or more first cameras 211-1 and 211-2 may be used to detect a user's hand gesture, track the user's head, and / or recognize a space. In an embodiment, the one or more first cameras 211-1 and 211-2 may be global shutter (GS) cameras.

[0065] In an embodiment, the one or more first cameras 211-1 and 211-2 may perform a simultaneous localization and mapping (SLAM) operation through depth photography. In an embodiment, the one or more first cameras 211-1 and 211-2 may perform space recognition for six degrees of freedom (6DoF).

[0066] In an embodiment, the image obtained by the one or more second cameras 212-1 and 212-2 may be used to detect and track the pupil of the user. In an embodiment, one or more of the second cameras 212-1 and 212-2 may be a GS camera. In an embodiment, the one or more second cameras 212-1 and 212-2 may correspond to the left eye and the right eye, respectively, and the performance of the one or more second cameras 212-1 and 212-2 is the same.

[0067] In an embodiment, the one or more third cameras 213 may be high-resolution cameras. In an embodiment, the one or more third cameras 213 may perform an auto focus (AF) function and an image stabilization function. In an embodiment, the one or more third cameras 213 may be GS cameras or rolling shutter (RS) cameras.

[0068] In an embodiment, the electronic device 201 may include one or more light emitting elements 214-1 and 214-2. In an embodiment, the light emitting elements 214-1 and 214-2 may be different from a light source configured to emit light to a screen display area of ​​a display to be described later. In an embodiment, when the user's pupil is detected and tracked by one or more second cameras 212-1 and 212-2, the light emitting elements 214-1 and 214-2 may emit light to facilitate pupil detection.

[0069] In an embodiment, the light emitting elements 214-1 and 214-2 may each include a light emitting diode (LED). In an embodiment, the light emitting elements 214-1 and 214-2 may emit light in the infrared region. In an embodiment, the light emitting elements 214-1 and 214-2 may be attached adjacent to the frame of the electronic device 201. In an embodiment, the light emitting elements 214-1 and 214-2 may be disposed adjacent to one or more first cameras 211-1 and 211-2, and when the electronic device 201 is used in a dark environment, one or more first cameras 211-1 and 211-2 may be assisted in gesture detection, head tracking, and spatial recognition. In an embodiment, the light emitting elements 214-1 and 214-2 may be disposed adjacent to one or more third cameras 213, and when the electronic device 201 is used in a dark environment, one or more third cameras 213 may be assisted in obtaining images.

[0070] In an embodiment, the electronic device 201 may include batteries 235 - 1 and 235 - 2 . The batteries 235 - 1 and 235 - 2 may store power to operate the remaining components of the electronic device 201 .

[0071] In an embodiment, the electronic device 201 may include a first display 251, a second display 252, one or more input optical members 253-1 and 253-2, and one or more transparent members 290-1 and 290-2, and one or more screen display portions 254-1 and 254-2.

[0072] In an embodiment, the first display 251 and the second display 252 may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a silicon liquid crystal display (liquid crystal on silicon (LCoS)), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0073] In an embodiment, when the first display 251 and the second display 252 may include one of LCD, DMD or LCoS, the electronic device 201 may include a light source configured to emit light to a screen display area of ​​the display. In an embodiment, when the first display 251 and the second display 252 are capable of autonomously generating light (e.g., including one of OLED or micro-LED), the electronic device 201 may provide a user with a virtual image with relatively good quality even without including a separate light source.

[0074] In an embodiment, when the user wears the electronic device 201, one or more transparent members 290-1 and 290-2 may be arranged to face the user's eyes. In an embodiment, one or more transparent members 290-1 and 290-2 may include at least one of a glass plate, a plastic plate, or a polymer. In an embodiment, when the electronic device 201 is worn, the user can see the outside world through one or more transparent members 290-1 and 290-2. In an embodiment, one or more input optical members 253-1 and 253-2 may guide the light generated by the first display 251 and the second display 252 to the user's eyes. In an embodiment, an image based on the light generated by the first display 251 and the second display 252 may be formed on one or more screen display portions 254-1 and 254-2 on one or more transparent members 290-1 and 290-2, and the user can see the image formed on the one or more screen display portions 254-1 and 254-2.

[0075] In an embodiment, the electronic device 201 may include one or more optical waveguides (not shown). The optical waveguide may transmit light generated by the first display 251 and the second display 252 to the user's eyes. The electronic device 201 may include one optical waveguide corresponding to each of the left eye and the right eye. In an embodiment, the optical waveguide may include at least one of glass, plastic, or polymer. In an embodiment, the optical waveguide may include a nano pattern, such as a polygonal or curved grating structure, disposed on one of the inner surface or the outer surface. In an embodiment, the optical waveguide may include a free-form prism, in which case the optical waveguide may provide incident light to the user through a reflector. In an embodiment, the optical waveguide may include at least one of at least one diffractive element (e.g., a diffractive optical element (DOE) and a holographic optical element (HOE)) or a reflective element (e.g., a reflector), and the display light emitted from the light source may be guided to the user's eyes via at least one diffractive element or reflective element included in the optical waveguide. In an embodiment, the diffractive element may include an input / output optical member. In an embodiment, the reflective element may include a member that causes total reflection.

[0076] In an embodiment, the electronic device 201 may include one or more voice input devices 262-1, 262-2, and 262-3 and one or more voice output devices 263-1 and 263-2.

[0077] In an embodiment, the electronic device 201 may include a first PCB 270-1 and a second PCB 270-2. The first PCB 270-1 and the second PCB 270-2 may be configured to send electrical signals to components included in the electronic device 201, such as one or more first cameras 211-1 and 211-2, one or more second cameras 212-1 and 212-2, one or more third cameras 213, a display, an audio module, and a sensor. In an embodiment, the first PCB 270-1 and the second PCB 270-2 may include a flexible printed circuit board (FPCB). In an embodiment, the first PCB 270-1 and the second PCB 270-2 may each include a first substrate, a second substrate, and a transfer board disposed between the first substrate and the second substrate.

[0078] Figure 3 The structure of an eye tracking camera of the electronic device 201 according to various embodiments is shown.

[0079] Reference Figure 3 , the electronic device 201 may include an eye tracking (ET) camera 311 (e.g., one or more second cameras 212-1 and 212-2), a display 301, an input optical component 305, a first waveguide tube 307, an output optical component 309, a first splitter 313, a second waveguide tube 315, or at least one of a second splitter 317.

[0080] In an embodiment, the user's pupil 319 may be photographed by the ET camera 311 through the first splitter 313 (e.g., an eye tracking splitter), the second waveguide 315, and the second splitter 317. In an embodiment, the ET camera 311 may detect the pupil 319 in the captured image, and may identify movement of the detected pupil 319, thereby tracking the user's gaze.

[0081] In an embodiment, an image output through the display 301 may be reflected by the input optical member 305 and the first waveguide 307 and displayed through the output optical member 309. In an embodiment, the electronic device 201 may output an image through the display 301 and may simultaneously recognize the movement of the user's pupil 319, thereby tracking (e.g., recognizing) the user's gaze (e.g., the direction of the user's gaze).

[0082] Figure 4 An electronic device 401 according to various embodiments is shown.

[0083] Reference Figure 4 In an embodiment, the electronic device 401 (e.g., the electronic device 101 or the electronic device 201) may include a first microphone 411, a second microphone 412, a first camera 421, a second camera 422, a communication module 431, a first speaker 441, a second speaker 442 and / or a display 451.

[0084] In an embodiment, the first microphone 411 may be configured to obtain a voice of a user (hereinafter, referred to as "user") wearing the electronic device 401 from the user. The first microphone 411 may be disposed on the electronic device 401 adjacent to the user's mouth to obtain the user's voice.

[0085] In an embodiment, the second microphone 412 may obtain audio introduced from the surroundings of the electronic device 401 (e.g., the voice of the other party and sounds from the surroundings of the electronic device 401). In an embodiment, the second microphone 412 may include multiple microphones. When the second microphone 412 includes multiple microphones, the electronic device 401 may determine the position of the other party based on the audio introduced through the multiple microphones. In an embodiment, when the second microphone 412 includes multiple microphones, the electronic device 401 may strengthen the audio (e.g., an audio signal corresponding to the voice of the other party) introduced in a specific direction (e.g., a direction from the position of the other party to the position of the electronic device 401) through beamforming through the multiple microphones. In an embodiment, the second microphone 412 may perform beamforming with Figure 2 The functions of one or more voice input devices 262-1, 262-2 and 262-3 included in the electronic device 201 are the same as or similar to those of the one or more voice input devices 262-1, 262-2 and 262-3 included in the electronic device 201.

[0086] In an embodiment, the first camera 421 may sense the user's gesture or the other party's gesture. For example, the first camera 421 may continuously obtain a plurality of frames including the user's gesture (eg, hand gesture). The first camera 421 may send a signal to a processor (eg, Figure 5 Processor 570) sends multiple obtained frames.

[0087] In an embodiment, the first camera 421 may sense the surroundings of the electronic device 401. For example, the first camera 421 may send an image of the surroundings of the electronic device 401 to the processor 570. When the other party is located in the surroundings of the electronic device 401, the processor 570 may detect the position of the other party based on the image of the surroundings of the electronic device 401. The detected position of the other party may be used for beamforming of multiple microphones. In an embodiment, the first camera 421 may perform the same as Figure 2 One or more first cameras (eg, Figure 2The functions of the first cameras 211-1 and 211-2) are the same or similar functions.

[0088] In an embodiment, the second camera 422 may be configured to obtain an image of the user's face. The second camera 422 may send the image of the user's face to the processor 570. In an embodiment, the processor 570 may recognize the user's face based on the image of the user's face. In an embodiment, the processor 570 may detect the user's facial expression and / or facial muscle movement based on the image of the user's face.

[0089] In an embodiment, the communication module 431 (eg, the communication module 190 ) may establish communication between the electronic device 401 and an external electronic device (eg, the electronic device 102 , the electronic device 104 , and / or a server (eg, a cloud server or server 108 )).

[0090] In an embodiment, the communication module 431 may enable the electronic device 401 to perform a call function. In an embodiment, the communication module 431 may send an audio signal related to the user's gesture and / or the user's voice to the other party's electronic device while the user is talking to the other party. The communication module 431 may receive an audio signal related to the other party's gesture and / or the other party's voice from the other party's electronic device while the user is talking to the other party.

[0091] In an embodiment, the communication module 431 can establish communication (e.g., establish wireless communication) between the electronic device 401 and an external input device in order to obtain the user's posture (e.g., the user's finger movement) and / or information related to the user's health. The external input device may include an input device using a visual method, a glove-type input device, and / or an armband and a wristband.

[0092] In an embodiment, the input device using the visual method may include an LED, and the electronic device 401 may obtain the user's hand movement by sensing light emitted from the LED.

[0093] In an embodiment, the glove type input device may include an inertial measurement unit (IMU) sensor and / or a metal member whose resistance changes according to a change in tension to sense hand movement of a user wearing the glove type input device. The glove type input device may transmit the sensed hand movement of the user to the electronic device 401.

[0094] In an embodiment, the armband and wristband may be worn on the arm and wrist of the user, respectively. The armband and / or wristband may sense finger movement by measuring the electrical signals of neurons. For example, the armband and / or wristband may sense finger movement by measuring the electrical signals of neurons via a plurality of electrodes included in the armband and / or wristband. The armband and / or wristband may send the sensed finger movement to the electronic device 401.

[0095] In an embodiment, the armband and / or wristband may obtain health information about the user (eg, heart rate and body fat percentage) through a biometric sensor. The armband and / or wristband may send the obtained health information to the electronic device 401.

[0096] However, the external input device for obtaining the user's posture and / or information related to the user's health is not limited to the above-mentioned input device using the visual method, the glove-type input device and / or the armband and wristband. In an embodiment, the external input device may include any input device capable of providing input to the electronic device 401. For example, the external electronic device may include a keyboard and / or a mouse that can establish wired or wireless communication with the electronic device 401 and provide the electronic device 401 with input typed by the user.

[0097] In an embodiment, the first speaker 441 may output audio corresponding to the user's gesture and / or audio corresponding to the user's voice (voice emitted by the user). For example, when the user's gesture is input to the electronic device 401 (for example, when an image of the user's gesture is obtained by the first camera 421), the first speaker 441 may output information (content) indicated by the user's gesture in audio form. In another example, when the user's voice is input to the electronic device 401 (for example, when a voice emitted by the user is obtained by the first microphone 411), the first speaker 441 may output information indicated by the user's voice in audio form. In an embodiment, the first speaker 441 may output audio corresponding to the user's gesture and / or audio corresponding to the user's voice, thereby delivering the user's intention expressed by the output audio to the other party located near the user. In an embodiment, the first speaker 441 may be set at a position adjacent to the user's mouth on the electronic device 401. In an embodiment, the first speaker 441 may be a mono speaker.

[0098] In an embodiment, the second speaker 442 may output audio corresponding to the user's gesture and / or audio corresponding to the other party's voice (voice emitted by the other party) (as well as ambient sound). For example, when the other party's gesture is input to the electronic device 401 (for example, when an image of the other party's gesture is obtained through the second camera 422), the second speaker 442 may output information (content) indicated by the other party's gesture in audio form. In another example, when the other party's voice (or ambient sound) is input to the electronic device 401 (for example, when a voice emitted by the other party is obtained through the second microphone 412), the second speaker 442 may output information indicated by the other party's voice in audio form. In an embodiment, the second speaker 442 may output audio corresponding to the other party's gesture and / or audio corresponding to the other party's voice, thereby delivering the other party's intention expressed by the output audio to the user. In an embodiment, the second speaker 442 may be set at a position adjacent to the user's ear on the electronic device 401. In an embodiment, the second speaker 442 may be a plurality of stereo speakers. In an embodiment, the second speaker 442 may perform the same Figure 2 The functions of one or more voice output devices 263-1 and 263-2 included in the electronic device 201 are the same as or similar to those of the one or more voice output devices 263-1 and 263-2 included in the electronic device 201.

[0099] In embodiments, the display 451 (eg, display module 160 ) (and optical module) may be configured to provide a virtual image to a user and / or an opponent.

[0100] In an embodiment, the display 451 may display a screen (hereinafter referred to as an "internal screen") including content (e.g., text and / or an image) that is usually visible to the user through a transparent member and / or a screen (hereinafter referred to as an "external screen") including voice content that is usually visible to the other party through a transparent member. For example, when the content to be shown to the user is the text "abcd", the display 451 may display an internal screen including the text "abcd" on a transparent member so that the text "abcd" is shown to the user. When the internal screen including the text "abcd" is displayed on a transparent member, the text symmetrical to the text "abcd" may be shown to the other party. When the content to be shown to the other party is the text "abcd", the display 451 may display an external screen including the text "abcd" on a transparent member so that the text "abcd" is shown to the other party. When the external screen including the text "abcd" is displayed on a transparent member, the text symmetrical to the text "abcd" may be shown to the user. In an embodiment, the display 451 may be provided at a position adjacent to the user's eyes in the electronic device 401. In an embodiment, when the electronic device 401 includes an additional display in addition to the display 451, the processor may display a screen including content (e.g., text and / or images) that is normally visible to the user through a transparent member on the display 451, and may display a screen including voice content that is normally visible to the other party on the additional display.

[0101] although Figure 4 The electronic device 401 is shown to include a first microphone 411, a second microphone 412, a first camera 421, a second camera 422, a communication module 431, a first speaker 441, a second speaker 442, and a display 451, but the electronic device 401 is not limited thereto. For example, the electronic device 401 may also include Figure 1 The electronic device 101 or Figure 2 At least one of the components included in the electronic device 201. In another example, according to an embodiment, the electronic device 401 may not include one or more of the first microphone 411, the second microphone 412, the first camera 421, the second camera 422, the communication module 431, the first speaker 441, the second speaker 442, and the display.

[0102] Figure 5 is a block diagram of an electronic device 401 according to various embodiments.

[0103] Reference Figure 5 In an embodiment, the electronic device 401 may include a microphone 510 , a camera 520 , a communication module 530 , a speaker 540 , a display 550 , a memory 560 , and / or a processor 570 .

[0104] In an embodiment, microphone 510 may include Figure 4 The first microphone 411 and / or the second microphone 412.

[0105] In an embodiment, the camera 520 may include Figure 4 The first camera 421 and / or the second camera 422.

[0106] Microphone 510 and camera 520 configured to obtain audio and gestures as input may be referred to as an “input module”.

[0107] In an embodiment, the communication module 530 may be included in Figure 1 The communication module 190 may include Figure 4 The communication module 431.

[0108] In an embodiment, the speaker 540 may include Figure 4 The first speaker 441 and / or the second speaker 442.

[0109] In an embodiment, the display 550 may include Figure 4 Display 451.

[0110] In an embodiment, the memory 560 may be included in Figure 1 in the memory 130.

[0111] In an embodiment, the memory 560 may store information for performing at least a portion of the operation of providing information. The information for performing at least a portion of the operation of providing information stored in the memory 560 will be described in detail later.

[0112] In an embodiment, the memory 560 may include a text-to-speech (TTS) tone database (DB) 561 (also referred to as a “tone TTS model”).

[0113] In an embodiment, the TTS tone DB 561 may include unique characteristics of audio (hereinafter, referred to as “tone”) to be applied to audio to be output from the electronic device 401 .

[0114] In an embodiment, the electronic device 401 (eg, the processor 570 ) may generate various tones to be applied to audio to be output from the electronic device 401 , and may store the tones in the memory 560 .

[0115] In an embodiment, the electronic device 401 may generate a tone corresponding to a person.

[0116] For example, the electronic device 401 may generate a tone corresponding to the user. The electronic device 401 may detect that the user is speaking based on the facial image of the user obtained by the second camera 422. When it is detected that the user is speaking, the electronic device 401 may obtain the voice emitted by the user through the first microphone 411. The electronic device 401 may obtain (e.g., extract) the tone corresponding to the user based on the obtained voice of the user. The electronic device 401 may store the obtained tone corresponding to the user in the memory 560. In an embodiment, when the tone corresponding to the user is generated, the electronic device 401 may map the user information (e.g., information representing the user) and the tone corresponding to the user and store them in the memory 560.

[0117] In another example, the electronic device 401 may generate a tone corresponding to the other party. The electronic device 401 may identify the other party based on an image of the other party (e.g., the other party located adjacent to the user wearing the electronic device 401) obtained by the first camera 421. The electronic device 401 may obtain a voice emitted by the other party through the second microphone 412. The electronic device 401 may obtain (e.g., extract) the other party's tone based on the obtained voice of the other party. The electronic device 401 may store the obtained tone of the other party in the memory 560. In an embodiment, the electronic device 401 may identify the position of the other party based on the image obtained by the first camera 421 and / or the voice obtained by the second microphone 412. When the position of the other party is identified, the electronic device 401 may strengthen the voice of the other party (e.g., an audio signal corresponding to the voice of the other party) introduced from the other party to the electronic device 401 via beamforming of multiple microphones included in the second microphone 412. In an embodiment, when a tone corresponding to the counterpart is generated, the electronic device 401 may map counterpart information (eg, information indicating the counterpart) and the tone corresponding to the counterpart and store them in the memory 560 .

[0118] In an embodiment, the electronic device 401 may generate a tone corresponding to an object capable of making a sound. For example, the electronic device 401 may identify an object (e.g., a pet or an object) (and the position of the object) that is making a sound based on an image obtained by the first camera 421. The electronic device 401 may obtain a sound made by an object through the first microphone 411. The electronic device 401 may obtain a tone corresponding to the object based on the obtained sound. The electronic device 401 may store the obtained tone corresponding to the object in the memory 560. In an embodiment, when a tone corresponding to an object is generated, the electronic device 401 may map the object (e.g., information representing the object) and the tone corresponding to the object and store them in the memory 560.

[0119] In an embodiment, at least a portion of the operation of the electronic device 401 to generate a tone may be performed by an external electronic device (e.g., server 108). In an embodiment, the electronic device 401 (or server) may generate a tone corresponding to a user, an opponent, and / or an object via a specified algorithm and / or artificial intelligence model.

[0120] In an embodiment, the electronic device 401 may receive a tone from an external electronic device through the communication module 530. The electronic device 401 may store the received tone in the memory 560.

[0121] In an embodiment, the processor 570 may be included in Figure 1 In the processor 120.

[0122] In an embodiment, the processor 570 may perform overall control of the operation of providing information. In an embodiment, the processor 570 may include one or more processors to perform the operation of providing information. Figure 6 Components included in the processor 570 to perform the operation of providing information are described in detail.

[0123] In the embodiment, although Figure 5 The electronic device 401 is shown to include a microphone 510, a camera 520, a communication module 530, a speaker 540, a display 550, a memory 560 and / or a processor 570, but the electronic device 401 is not limited thereto. For example, the electronic device 401 may also include Figure 1 The electronic device 101 or Figure 2 At least one of the components included in the electronic device 201. In another example, according to an embodiment, the electronic device 401 may not include one or more of the microphone 510, the camera 520, the communication module 530, the speaker 540, the display 550, the memory 560 and / or the processor 570.

[0124] Figure 6 is a block diagram of a processor 570 according to various embodiments.

[0125] Reference Figure 6 In an embodiment, the processor 570 may include an input acquisition unit 610 , an input analyzer 620 , an output determination unit 630 and / or a signal processor 640 .

[0126] In an embodiment, the input acquisition unit 610 may obtain various inputs via an input module (eg, the camera 520 and the microphone 510 ), the communication module 530 , and / or an external input device.

[0127] In an embodiment, the input acquisition unit 610 may acquire the user's gesture and / or the other party's gesture as input through the first camera 421. For example, the input acquisition unit 610 may acquire an image of the user's gesture (e.g., the user's hand movement) and / or the other party's gesture (e.g., the other party's hand movement) through the first camera 421. In an embodiment, the user's gesture and / or the other party's gesture may include the user's hand movement and / or the other party expressing sign language (also referred to as "pointing language"). In an embodiment, the user's gesture and / or the other party's gesture may include movement corresponding to a specified content (or meaning).

[0128] In an embodiment, the input acquisition unit 610 may acquire the user's facial expression as input through the second camera 422. For example, the input acquisition unit 610 may acquire an image including the user's face through the second camera 422.

[0129] In an embodiment, the input acquisition unit 610 may obtain as input the user's voice input through the first microphone 411 and / or the other party's voice input through the second microphone 412. In an embodiment, the input acquisition unit 610 may obtain as input the ambient sound of the electronic device 401 input through the second microphone 412.

[0130] In an embodiment, the input acquisition unit 610 may obtain, as input, an audio signal of the other party received through the communication module 530 while the user is talking to the other party.

[0131] In an embodiment, the input acquisition unit 610 may acquire, as input, a user's gesture (eg, a hand movement of the user) obtained by sensing light emitted from an LED of an input device using a visual method.

[0132] In an embodiment, the input acquisition unit 610 may obtain a user's gesture (eg, a user's hand movement) received from a glove-type input device as input.

[0133] In an embodiment, the input acquisition unit 610 may obtain the user's hand movement and / or health information received from an armband and / or a wristband as input.

[0134] However, the input that can be obtained by the input acquisition unit 610 is not limited to the aforementioned input.

[0135] In an embodiment, input analyzer 620 may identify the type of input.

[0136] In an embodiment, the input analyzer 620 may recognize that the type of input obtained by the input acquisition unit 610 corresponds to at least one of a gesture, a voice (and / or an environmental sound), a user's facial expression, and health information.

[0137] In an embodiment, the input analyzer 620 may identify whether the input gesture corresponds to the user's gesture or the other party's gesture by analyzing the image obtained by the first camera 421 .

[0138] In an embodiment, the input analyzer 620 can identify whether the input voice is a voice issued by the user or a voice issued by the other party. For example, the input analyzer 620 can identify whether the input voice is a voice issued by the user or a voice issued by the other party by identifying the component (e.g., device) that has obtained the voice. When the voice is obtained through the first microphone 411 (e.g., when the intensity of the audio signal received by the first microphone 411 is greater than the intensity of the audio signal received by the second microphone 412), the input analyzer 620 can identify that the input voice is a voice issued by the user. When the voice is obtained through the second microphone 412 (e.g., when the intensity of the audio signal received by the second microphone 412 is greater than the intensity of the audio signal received by the first microphone 411), the input analyzer 620 can identify that the input voice is a voice issued by the other party. In another example, the input analyzer 620 can identify whether the input voice is a voice issued by the user or a voice issued by the other party based on the tone of the input voice. When the tone of the input voice corresponds to the user, the input analyzer 620 can identify that the input voice is a voice issued by the user. When the tone of the input voice corresponds to the other party, the input analyzer 620 may recognize that the input voice is a voice uttered by the other party.

[0139] In an embodiment, the input analyzer 620 may recognize that the input is a facial expression of the user by analyzing an image obtained through the second camera 422 .

[0140] In an embodiment, when health information is received from an external input device (eg, an armband and / or a wristband), the input analyzer 620 may recognize that the input is health information.

[0141] In an embodiment, the input analyzer 620 may convert the input obtained by the input acquisition unit 610 into text.

[0142] In an embodiment, the input analyzer 620 may identify whether the input obtained by the input acquisition unit 610 is convertible into text. When the obtained input is convertible into text, the input analyzer 620 may convert information corresponding to the obtained input (e.g., information indicated by the obtained input) into text. For example, the input analyzer 620 may identify whether an input gesture (e.g., a user's gesture and / or a gesture of the other party) corresponds to a sign language. The input analyzer 620 may identify whether the input gesture corresponds to a sign language by analyzing an image of a gesture obtained by the first camera 421. When the input gesture corresponds to a sign language, the input analyzer 620 may identify that the input gesture is convertible into text. The input analyzer 620 may convert the content of the sign language indicated by the input gesture into text. In another example, the input analyzer 620 may identify that the input gesture (e.g., a user's gesture and / or a gesture of the other party) is a specified gesture (e.g., a gesture mapped to a specified content and stored in the memory 560). The input analyzer 620 may identify whether the input gesture corresponds to a specified gesture by analyzing an image of a gesture obtained by the first camera 421. When the input gesture corresponds to a specified gesture, the input analyzer 620 may recognize that the input gesture is convertible into text. The input analyzer 620 may convert the input gesture into text including information corresponding to the specified gesture. In yet another example, the input analyzer 620 may recognize whether the user's input facial expression corresponds to a specified facial expression (e.g., a facial expression stored in the memory 560). The input analyzer 620 may recognize the user's facial expression by analyzing an image of the user's face obtained by the second camera 422. When the user's facial expression corresponds to a specified facial expression, the input analyzer 620 may recognize that the user's input facial expression is convertible into text. The input analyzer 620 may convert the user's input facial expression into text including content representing the user's input facial expression (or an emotion corresponding to the user's input facial expression). In yet another example, the input analyzer 620 may recognize whether the user's input voice and / or the other party's voice is convertible into text. When the user's input voice and / or the other party's voice is convertible into text, the input analyzer 620 may convert the user's input voice and / or the other party's voice into text. For example, the input analyzer 620 can convert the user's input voice and / or the other party's voice into text via a speech-to-text (STT) program. In another example, the input analyzer 620 can identify whether the other party's audio signal received by the communication module 530 while the user is talking to the other party can be converted into text. When the other party's audio signal is convertible into text, the input analyzer 620 can convert the audio signal of the other party's signal into text. In another example, the input analyzer 620 can identify whether the health information received from an external input device (e.g., an armband and / or wristband) can be converted into text.When the health information is convertible into text, the input analyzer 620 may convert the health information into text.

[0143] In an embodiment, the output determination unit 630 may determine whether to output information corresponding to the input through one or more speakers. For example, the output determination unit 630 may determine whether to output information corresponding to the input through one or more speakers (e.g., the first speaker 441 and / or the second speaker 442) and / or whether to output information corresponding to the input through the display 550.

[0144] In an embodiment, the output determination unit 630 may determine, based on the user input, whether to output information corresponding to the input through the speaker 540. For example, the output determination unit 630 may determine, based on the user input through the virtual keyboard displayed on the display 550, to output information corresponding to the input through the speaker 540. In another example, when the user's input voice and / or the user's gesture corresponds to the user's designated voice and / or the user's gesture, the output determination unit 630 may determine to output the information corresponding to the input through the speaker 540.

[0145] In an embodiment, when it is determined not to output the information corresponding to the input through the speaker 540 , the output determination unit 630 may determine whether to display the information corresponding to the input on the display 550 .

[0146] In an embodiment, the output determination unit 630 may determine that information corresponding to the input is to be output via both the speaker 540 and the display 550 .

[0147] In an embodiment, the output determination unit 630 may determine a component to output information corresponding to the input based on the type of the input.

[0148] In an embodiment, when the type of input is the user's voice, the output determination unit 630 may output information corresponding to the user's voice through the first speaker 441. When the type of input is the user's voice, the output determination unit 630 may determine to display an internal screen including information corresponding to the user's voice (e.g., text corresponding to the user's voice) on the display 550.

[0149] In an embodiment, when the type of input is the user's voice, the output determination unit 630 may determine to output information corresponding to the user's voice through the first speaker 441 and simultaneously display an internal screen including information corresponding to the user's voice (e.g., text corresponding to the user's voice) on the display 550. When the type of input is the other party's voice, the output determination unit 630 may determine to output information corresponding to the other party's voice through the second speaker 442 and simultaneously display an external screen including information corresponding to the other party's voice on the display 550. However, the present disclosure is not limited thereto. For example, when the type of input is the user's voice, the output determination unit 630 may determine to output information corresponding to the user's voice through the first speaker 441 and simultaneously display an external screen including information corresponding to the user's voice on the display 550. In another example, when the type of input is the other party's voice, the output determination unit 630 may determine to output the other party's voice through the second speaker 442 and simultaneously display an internal screen including information corresponding to the other party's voice on the display 550.

[0150] In an embodiment, when the type of input is the user's voice or the other party's voice, the output determination unit 630 may translate the information corresponding to the user's voice or the other party's voice into a language different from the language of the user's voice or the other party's voice. The output determination unit 630 may determine to display the information corresponding to the user's voice or the other party's voice translated into a different language on the display 550.

[0151] In an embodiment, when the type of input is a user's gesture, the output determination unit 630 may output information corresponding to the user's gesture through the first speaker 441. When the type of input is a user's gesture, the output determination unit 630 may determine to display an external screen including information corresponding to the user's gesture (e.g., text indicating information corresponding to the user's gesture) on the display 550. When the type of input is a gesture of the other party, the output determination unit 630 may output information corresponding to the other party's gesture through the second speaker 442. When the type of input is a gesture of the other party, the output determination unit 630 may determine to display an internal screen including information corresponding to the other party's gesture (e.g., text indicating information corresponding to the other party's gesture) on the display 550.

[0152] In an embodiment, when the type of input is a user's gesture, the output determination unit 630 may determine to output information corresponding to the user's gesture through the first speaker 441 and simultaneously display an external screen including information corresponding to the user's gesture (e.g., text indicating information corresponding to the user's gesture) on the display 550. When the type of input is a counterpart's gesture, the output determination unit 630 may determine to output information corresponding to the counterpart's gesture through the second speaker 442 and simultaneously display an internal screen including information corresponding to the counterpart's gesture (e.g., text indicating information corresponding to the counterpart's gesture) on the display 550.

[0153] In an embodiment, the output determination unit 630 may determine whether to output information corresponding to the input through the speaker 540 based on the surrounding environment of the electronic device 401 (eg, ambient sound, brightness outside the electronic device 401 , and the distance between the electronic device 401 and the other party).

[0154] In an embodiment, the output determination unit 630 may determine whether to output the user's input facial expression (and / or the user's emotion corresponding to the user's input facial expression) and / or health information through the speaker 540 .

[0155] In an embodiment, the output determination unit 630 may obtain the user's emotion or the other party's emotion based on the user's input voice or the other party's voice. The output determination unit 630 may determine whether to output information indicating the user's emotion or the other party's emotion through the speaker 540.

[0156] In an embodiment, when multiple inputs are obtained, the output determination unit 630 may determine whether to output information corresponding to the multiple inputs through the speaker 540. For example, when multiple inputs are obtained, the output determination unit 630 may determine whether to output information corresponding to the multiple inputs through the speaker 540 and / or the display 550 based on the user input or the type of the multiple inputs.

[0157] In an embodiment, when multiple inputs are obtained, the output determination unit 630 may determine to output audio indicating information corresponding to the multiple inputs (e.g., audio into which text indicating information corresponding to each of the multiple inputs is converted) through the speaker 540. For example, when a first input and a second input are obtained, the output determination unit 630 may determine to output a first audio indicating information corresponding to the first input and a second audio indicating information corresponding to the second input simultaneously through the speaker 540. In an embodiment, when it is determined that audio indicating information corresponding to multiple inputs is to be output through the speaker 540, the output determination unit 630 may enable the audio indicating the information corresponding to the multiple inputs to have different tones. For example, when a user's gesture and a gesture of the other party are input simultaneously, the output determination unit 630 may determine to apply the user's tone to the audio indicating the information corresponding to the user's gesture and to apply the other party's tone to the audio indicating the information corresponding to the other party's gesture. However, the present disclosure is not limited thereto. In an embodiment, the output determination unit 630 may enable the audio indicating the information corresponding to the multiple inputs to have different levels (or volumes). For example, when the user's gesture and the other party's gesture are input at the same time, the output determination unit 630 may enable the audio indicating the information corresponding to the user's gesture to have a first level and enable the audio indicating the information corresponding to the other party's gesture to have a second level higher than the first level. In an embodiment, the output determination unit 630 may determine to output the audio indicating the information corresponding to the multiple inputs sequentially. For example, when the user's gesture and the other party's gesture are input at the same time, the output determination unit 630 may determine to output the audio indicating the information corresponding to the user's gesture, and then successively output the audio indicating the information corresponding to the other party's gesture.

[0158] In an embodiment, when multiple inputs are obtained, the output determination unit 630 may assign (or give) priorities to the multiple inputs. For example, the output determination unit 630 may assign a higher priority to the user's gesture than to the other party's gesture.

[0159] In an embodiment, the output determination unit 630 may determine the tone of the audio indicating the information corresponding to the multiple inputs, the level of the audio, and / or the output order of the audio based on the priority of the multiple inputs. For example, when the first input is assigned a higher priority than the second input, the output determination unit 630 may determine to output the audio having the first tone for the first input and the audio having the second tone for the second input. In another example, when the first input is assigned a higher priority than the second input, the output determination unit 630 may determine to output the audio having the first level for the first input and the audio having the second level lower than the first level for the second input. In another example, when the first input is assigned a higher priority than the second input, the output determination unit 630 may determine to output the audio indicating the information corresponding to the first input before the audio indicating the information corresponding to the second input.

[0160] In an embodiment, the output determination unit 630 may set priorities assigned to the plurality of inputs based on a user input. However, the method by which the output determination unit 630 sets priorities for the plurality of inputs is not limited to the user input.

[0161] In an embodiment, when multiple inputs are obtained, the output determination unit 630 may determine that information corresponding to the multiple inputs is to be output through the display 550. For example, when at least two inputs are obtained among the user's posture, the user's voice, and health information about the user, the output determination unit 630 may determine that the at least two inputs obtained are to be output through the display 550. The output determination unit 630 may determine that the text (and / or image) indicating the information corresponding to the at least two inputs is to be output in different sizes, different colors, different text display areas on the transparent member, and / or different languages, so that the texts are distinguished from each other. However, the method by which the output determination unit 630 outputs text indicating information corresponding to the at least two inputs obtained to be distinguished from each other through the display 550 is not limited to the aforementioned example.

[0162] In an embodiment, when multiple inputs are obtained, the output determination unit 630 may determine to output information corresponding to the multiple inputs through the display 550 and the speaker 540 via an avatar. For example, when a user's gesture, a user's voice, and a user's facial expression (as well as health information) are input, the output determination unit 630 may control the display 550 so that the avatar corresponding to the user performs a movement corresponding to the user's gesture and makes an expression corresponding to the user's facial expression. The output determination unit 630 may determine to apply a tone corresponding to the avatar corresponding to the user to the audio indicating information corresponding to the user's voice.

[0163] In an embodiment, the output determination unit 630 may determine whether to send an audio signal related to the user's gesture and / or the user's voice to the other party's electronic device while the user is talking on the other party. In an embodiment, the output determination unit 630 may determine whether to determine an audio signal related to the other party's gesture and / or the other party's voice through a speaker while the user is talking on the other party.

[0164] In an embodiment, the signal processor 640 can process the information corresponding to the input based on the determination of the output determination unit 630. For example, when the output determination unit 630 determines that the audio indicating the information corresponding to the input is to be output through a speaker, the signal processor 640 can convert the text indicating the information corresponding to the input into audio via a text-to-speech (TTS) program. When the text indicating the information corresponding to the input is converted into audio, the audio can be output through a speaker. In another example, when the output determination unit 630 determines that the first tone is to be applied to the audio indicating the information corresponding to the input, the signal processor 640 can obtain the first tone from the TTS tone DB 561. The signal processor 640 can apply the first tone to the audio indicating the information corresponding to the input. In another example, when the output determination unit 630 determines that the information corresponding to multiple inputs is to be output simultaneously through a speaker, the signal processor 640 can mix the audio indicating the information corresponding to the multiple inputs, thereby simultaneously outputting the audio indicating the information corresponding to the multiple inputs. However, the present disclosure is not limited to this example, and the signal processor 640 can perform a signal processing operation according to the determination of the output determination unit 630.

[0165] The electronic device 401 according to various embodiments of the present disclosure may include: one or more input modules (e.g., microphone 510 and camera 520); one or more speakers (e.g., speaker 540), a display (e.g., display 550); and at least one processor (e.g., processor 570) operably connected to the one or more input modules (e.g., microphone 510 and camera 520), one or more speakers (e.g., speaker 540), and a display (e.g., display 550), wherein the at least one processor (e.g., processor 570) may be configured to: obtain input through one or more input modules (e.g., microphone 510 and camera 520), identify the type of input, determine whether to output information corresponding to the input through one or more speakers (e.g., speaker 540), and based on determining that the information corresponding to the input is to be output through one or more speakers (e.g., speaker 540), control the one or more speakers so that the information corresponding to the input is output through the one or more speakers (e.g., speaker 540) based on the type of input.

[0166] In various embodiments, at least one processor (eg, processor 570) may be further configured to convert the obtained input into text.

[0167] In various embodiments, at least one processor (e.g., processor 570) may be configured to determine whether to output information corresponding to an input through one or more speakers (e.g., speaker 540) based on at least one of the user input or the type of input.

[0168] In various embodiments, at least one processor (eg, processor 570) may be configured to determine a tone to be applied to audio indicating information corresponding to an input.

[0169] In various embodiments, at least one processor (e.g., processor 570) may be configured to obtain multiple inputs through one or more input modules (e.g., microphone 510 and camera 520) and, based on the types of the multiple inputs, determine different tones to be applied to audio that indicates information corresponding to the multiple inputs and that is to be output simultaneously.

[0170] In various embodiments, at least one processor (e.g., processor 570) may be configured to: when the input type is a user's gesture and / or voice, determine to output information corresponding to the gesture and / or voice of the user of the electronic device 401 through a first speaker 441 among one or more speakers; and when the input type is a gesture and / or voice of the other party, determine to output information corresponding to the gesture and / or voice of the other party through a second speaker 442 among one or more speakers.

[0171] In various embodiments, at least one processor (e.g., processor 570) may be configured to obtain gesture input and voice input of a user of the electronic device 401 through one or more input modules (e.g., microphone 510 and camera 520), and determine a first tone to be applied to first audio indicating information corresponding to the user's gesture input and a second tone to be applied to second audio indicating information corresponding to the user's voice input and to be output simultaneously with the first audio.

[0172] In various embodiments, at least one processor (e.g., processor 570) may be configured to: when the type of input is a user's gesture and / or voice, determine to display information corresponding to the user's gesture and / or voice of the electronic device 401 on an external screen of a display (e.g., display 550); and when the type of input is the other party's gesture and / or voice, determine to output information corresponding to the other party's gesture and / or voice on an internal screen of the display (e.g., display 550).

[0173] In various embodiments, the electronic device 401 may also include a communication module (e.g., communication module 530), and at least one processor (e.g., processor 570) may be further configured to: when a user's gesture is received while the electronic device 401 is talking on a call with an external electronic device, send information corresponding to the gesture of the user of the electronic device 401 to the external electronic device through the communication module (e.g., communication module 530).

[0174] In various embodiments, at least one processor (e.g., processor 570) may be further configured to obtain information about the surrounding environment of the electronic device 401, and based on the information about the surrounding environment of the electronic device 401, determine whether to output information corresponding to the input through one or more speakers (e.g., speaker 540).

[0175] Figure 7 is a flow chart 700 illustrating a method of providing information according to various embodiments.

[0176] Reference Figure 7 , in operation 701, in an embodiment, the processor 570 may obtain input through one or more input modules.

[0177] In an embodiment, the processor 570 may obtain the user's gesture and / or the other party's gesture as input through the first camera 421. For example, the processor 570 may obtain an image of the user's gesture (e.g., the user's hand movement) and / or the other party's gesture (e.g., the other party's hand movement) through the first camera 421. In an embodiment, the user's gesture and / or the other party's gesture may include hand movements of the user and / or the other party expressing sign language (also referred to as "pointing language"). In an embodiment, the user's gesture and / or the other party's gesture may include movements corresponding to specified content (or meaning).

[0178] In an embodiment, the processor 570 may obtain the user's facial expression as input through the second camera 422. For example, the processor 570 may obtain an image including the user's face through the second camera 422.

[0179] In an embodiment, the processor 570 may obtain as input the user's voice input through the first microphone 411 and / or the other party's voice input through the second microphone 412. In an embodiment, the processor 570 may obtain as input the ambient sound of the electronic device 401 input through the second microphone 412.

[0180] In an embodiment, the processor 570 may obtain input through the communication module 530. For example, the processor 570 may obtain as input an audio signal of the other party received through the communication module 530 while the user is talking to the other party.

[0181] In an embodiment, the processor 570 may obtain input via an external input device. For example, the processor 570 may obtain a user's gesture (e.g., a user's hand movement) obtained by sensing light emitted from an LED of an input device using a visual method as input. In another example, the processor 570 may obtain a user's gesture (e.g., a user's hand movement) received from a glove-type input device as input. In yet another example, the processor 570 may obtain a user's hand movement and / or health information received from an armband and / or wristband as input.

[0182] However, the inputs obtainable by the processor 570 are not limited to the aforementioned inputs.

[0183] In operation 703 , in an embodiment, the processor 570 may determine the type of input.

[0184] In an embodiment, the processor 570 may recognize that the type of input obtained in operation 701 corresponds to at least one of a gesture, a voice (and / or an environmental sound), a facial expression of the user, and health information.

[0185] In an embodiment, the processor 570 may recognize whether the input gesture corresponds to the user's gesture or the other party's gesture by analyzing the image obtained by the first camera 421 .

[0186] In an embodiment, the processor 570 can identify whether the input voice is a voice issued by the user or a voice issued by the other party. For example, the processor 570 can identify whether the input voice is a voice issued by the user or a voice issued by the other party by identifying a component (e.g., a device) that has obtained the voice. When the voice is obtained by the first microphone 411, the processor 570 can identify that the input voice is a voice issued by the user. When the voice is obtained by the second microphone 412, the processor 570 can identify that the input voice is a voice issued by the other party. In another example, the processor 570 can identify whether the input voice is a voice issued by the user or a voice issued by the other party based on the tone of the input voice. When the tone of the input voice corresponds to the user, the processor 570 can identify that the input voice is a voice issued by the user. When the tone of the input voice corresponds to the other party, the processor 570 can identify that the input voice is a voice issued by the other party.

[0187] In an embodiment, the processor 570 may recognize that the input is a facial expression of the user by analyzing an image obtained through the second camera 422 .

[0188] In an embodiment, when health information is received from an external input device (eg, an armband and / or a wristband), the processor 570 may recognize that the input is health information.

[0189] In an embodiment, processor 570 may convert the obtained input into text.

[0190] In an embodiment, the processor 570 may identify whether the input obtained is convertible into text. When the input obtained is convertible into text, the processor 570 may convert the information corresponding to the input obtained (e.g., information indicated by the input obtained) into text. For example, the processor 570 may identify whether the input gesture (e.g., the user's gesture and / or the other party's gesture) corresponds to sign language. The processor 570 may identify whether the input gesture corresponds to sign language by analyzing the image of the gesture obtained by the first camera 421. When the input gesture corresponds to sign language, the processor 570 may identify that the input gesture is convertible into text. The processor 570 may convert the content of the sign language indicated by the input gesture into text. In another example, the processor 570 may identify that the input gesture (e.g., the user's gesture and / or the other party's gesture) is a specified gesture (e.g., a gesture mapped to specified content and stored in the memory 560). The processor 570 may identify whether the input gesture corresponds to the specified gesture by analyzing the image of the gesture obtained by the first camera 421. When the input gesture corresponds to the specified gesture, the processor 570 may identify that the input gesture is convertible into text. The processor 570 may convert the input gesture into text including information corresponding to the specified gesture. In yet another example, the processor 570 may identify whether the user's input facial expression corresponds to a specified facial expression (e.g., a facial expression stored in the memory 560). The processor 570 may identify the user's facial expression by analyzing an image of the user's face obtained by the second camera 422. When the user's facial expression corresponds to a specified facial expression, the processor 570 may identify that the user's input facial expression is convertible into text. The processor 570 may convert the user's input facial expression into text including content representing the user's input facial expression (or an emotion corresponding to the user's input facial expression). In yet another example, the processor 570 may identify whether the user's input voice and / or the other party's voice is convertible into text. When the user's input voice and / or the other party's voice is convertible into text, the processor 570 may convert the user's input voice and / or the other party's voice into text. For example, the processor 570 may convert the user's input voice and / or the other party's voice into text via an STT program. In yet another example, the processor 570 may identify whether the other party's audio signal received by the communication module 530 while the user is talking to the other party is convertible into text. When the audio signal of the other party can be converted into text, the processor 570 can convert the audio signal of the other party's signal into text. In another example, the processor 570 can identify whether the health information received from an external input device (e.g., an armband and / or a wristband) can be converted into text. When the health information can be converted into text, the processor 570 can convert the health information into text.

[0191] In operation 705, in an embodiment, the processor 570 may determine whether to output information corresponding to the input through the speaker 540. For example, the processor 570 may determine whether to output information corresponding to the input through one or more speakers (e.g., the first speaker 441 and / or the second speaker 442) and / or whether to output information corresponding to the input through the display 550.

[0192] In an embodiment, the processor 570 may determine, based on the user input, whether to output information corresponding to the input through the speaker 540. For example, the processor 570 may determine, based on the user input through the virtual keyboard displayed on the display 550, to output information corresponding to the input through the speaker 540. In another example, when the user's input voice and / or the user's gesture corresponds to the user's designated voice and / or the user's gesture, the processor 570 may determine to output information corresponding to the input through the speaker 540.

[0193] In an embodiment, when it is determined not to output the information corresponding to the input through the speaker 540 , the processor 570 may determine whether to display the information corresponding to the input on the display 550 .

[0194] In an embodiment, the processor 570 may determine to output information corresponding to the input via both the speaker 540 and the display 550 .

[0195] In an embodiment, the processor 570 may determine whether to output information corresponding to the input through the speaker 540 based on the type of the input.

[0196] In an embodiment, the processor 570 may determine whether to output information corresponding to the input through the speaker 540 based on the user input and / or the type of the input.

[0197] In an embodiment, the processor 570 may determine a component for outputting information corresponding to the input based on the type of input. In an embodiment, when the type of input is the user's voice, the processor 570 may output information corresponding to the user's voice through the first speaker 441. When the type of input is the user's voice, the processor 570 may determine to display an internal screen including information corresponding to the user's voice (e.g., text corresponding to the user's voice) on the display 550.

[0198] In an embodiment, when the input type is the user's voice, the processor 570 may determine to output information corresponding to the user's voice through the first speaker 441 and simultaneously display an internal screen including information corresponding to the user's voice (e.g., text corresponding to the user's voice) on the display 550. When the input type is the other party's voice, the processor 570 may determine to output information corresponding to the other party's voice through the second speaker 442 and simultaneously display an external screen including information corresponding to the other party's voice on the display 550. However, the present disclosure is not limited thereto. For example, when the input type is the user's voice, the processor 570 may determine to output information corresponding to the user's voice through the first speaker 441 and simultaneously display an external screen including information corresponding to the user's voice on the display 550. In another example, when the input type is the other party's voice, the processor 570 may determine to output the other party's voice through the second speaker 442 and simultaneously display an internal screen including information corresponding to the other party's voice on the display 550.

[0199] In an embodiment, when the type of input is the user's voice or the other party's voice, the processor 570 may translate the information corresponding to the user's voice or the other party's voice into a language different from the language of the user's voice or the other party's voice. The processor 570 may determine to display the information corresponding to the user's voice or the other party's voice translated into a different language on the display 550.

[0200] In an embodiment, when the type of input is a user's gesture, the processor 570 may output information corresponding to the user's gesture through the first speaker 441. When the type of input is a user's gesture, the processor 570 may determine to display an external screen including information corresponding to the user's gesture (e.g., text indicating information corresponding to the user's gesture) on the display 550. When the type of input is a gesture of the other party, the processor 570 may output information corresponding to the other party's gesture through the second speaker 442. When the type of input is a gesture of the other party, the processor 570 may determine to display an internal screen including information corresponding to the other party's gesture (e.g., text indicating information corresponding to the other party's gesture) on the display 550.

[0201] In an embodiment, when the type of input is a user's gesture, the processor 570 may determine to output information corresponding to the user's gesture through the first speaker 441 and simultaneously display an external screen including information corresponding to the user's gesture (e.g., text indicating information corresponding to the user's gesture) on the display 550. When the type of input is a counterpart's gesture, the processor 570 may determine to output information corresponding to the counterpart's gesture through the second speaker 442 and simultaneously display an internal screen including information corresponding to the counterpart's gesture (e.g., text indicating information corresponding to the counterpart's gesture) on the display 550.

[0202] In an embodiment, the processor 570 can determine whether to output information corresponding to the input through the speaker based on the surrounding environment of the electronic device 401 (e.g., ambient sound, brightness outside the electronic device 401 (e.g., the intensity of light entering the electronic device 401), and the distance between the electronic device 401 and the other party).

[0203] In an embodiment, the processor 570 may determine whether to output the user's input facial expression (and / or the user's emotion corresponding to the user's input facial expression) and / or health information through the speaker 540.

[0204] In an embodiment, the processor 570 may obtain the user's emotion or the other party's emotion based on the user's input voice or the other party's voice. The processor 570 may determine whether to output information indicating the user's emotion or the other party's emotion through the speaker 540.

[0205] In an embodiment, when multiple inputs are obtained, the processor 570 may determine whether to output information corresponding to the multiple inputs through the speaker 540. For example, when multiple inputs are obtained, the processor 570 may determine whether to output information corresponding to the multiple inputs through the speaker 540 and / or the display 550 based on the user input or the type of the multiple inputs.

[0206] In an embodiment, when multiple inputs are obtained, the processor 570 may determine that information corresponding to the multiple inputs (e.g., audio into which text indicating information corresponding to each of the multiple inputs is converted) is to be output through the speaker 540. For example, when a first input and a second input are obtained, the processor 570 may determine that a first audio indicating information corresponding to the first input and a second audio indicating information corresponding to the second input are to be simultaneously output through the speaker 540. In an embodiment, when it is determined that information corresponding to multiple inputs is to be output through the speaker 540, the processor 570 may enable the audio indicating the information corresponding to the multiple inputs to have different tones. For example, when a user's gesture and a gesture of the other party are input simultaneously, the processor 570 may determine that the user's tone is to be applied to the audio indicating the information corresponding to the user's gesture and the other party's tone is to be applied to the audio indicating the information corresponding to the other party's gesture. However, the present disclosure is not limited thereto. In an embodiment, the processor 570 may enable the audio indicating the information corresponding to the multiple inputs to have different levels (or volumes). For example, when the user's gesture and the other party's gesture are input at the same time, the processor 570 may enable the audio indicating the information corresponding to the user's gesture to have a first level and enable the audio indicating the information corresponding to the other party's gesture to have a second level higher than the first level. In an embodiment, the processor 570 may determine to output the audio indicating the information corresponding to the multiple inputs sequentially. For example, when the user's gesture and the other party's gesture are input at the same time, the processor 570 may determine to output the audio indicating the information corresponding to the user's gesture, and then successively output the audio indicating the information corresponding to the other party's gesture.

[0207] In an embodiment, when multiple inputs are obtained, the processor 570 may assign priorities to the multiple inputs. For example, the processor 570 may assign a higher priority to the user's gesture than the other party's gesture.

[0208] In an embodiment, the processor 570 may determine the tone of the audio indicating the information corresponding to the multiple inputs, the level of the audio, and / or the output order of the audio based on the priority of the multiple inputs. For example, when the first input is assigned a higher priority than the second input, the processor 570 may determine to output the audio having a first tone for the first input and to output the audio having a second tone for the second input. In another example, when the first input is assigned a higher priority than the second input, the processor 570 may determine to output the audio having a first level for the first input and to output the audio having a second level lower than the first level for the second input. In another example, when the first input is assigned a higher priority than the second input, the processor 570 may determine to output the audio indicating the information corresponding to the first input before the audio indicating the information corresponding to the second input.

[0209] In an embodiment, the processor 570 may set priorities assigned to the plurality of inputs based on user input (eg, user input through a virtual keyboard). However, the method by which the processor 570 sets priorities for the plurality of inputs is not limited to user input.

[0210] In an embodiment, when multiple inputs are obtained, the processor 570 may determine to output information corresponding to the multiple inputs through the display 550. For example, when at least two inputs are obtained among the user's gesture, the user's voice, the user input through the virtual keyboard, and the health information about the user, the processor 570 may determine to output the at least two inputs obtained through the display 550. The processor 570 may determine to output the text (and / or image) indicating the information corresponding to the at least two inputs in different sizes, different colors, different areas on the transparent member on which the text is displayed, and / or different languages, so that the texts are distinguished from each other. However, the method by which the processor 570 outputs the text indicating the information corresponding to the at least two inputs obtained through the display 550 to distinguish from each other is not limited to the aforementioned example.

[0211] In an embodiment, when multiple inputs are obtained, the processor 570 may determine to output information corresponding to the multiple inputs via the avatar through the display 550 and the speaker 540. For example, when the user's gesture, the user's voice, and the user's facial expression (as well as health information) are input, the processor 570 may control the display 550 so that the avatar corresponding to the user performs a movement corresponding to the user's gesture and makes an expression corresponding to the user's facial expression. The processor 570 may determine to apply the tone of the avatar corresponding to the user to the audio indicating the information corresponding to the user's voice.

[0212] In an embodiment, the processor 570 may determine whether to send an audio signal related to the user's gesture and / or the user's voice to the other party's electronic device while the user is talking on the phone with the other party. In an embodiment, the processor 570 may determine whether to determine an audio signal related to the other party's gesture and / or the other party's voice through the speaker 540 while the user is talking on the phone with the other party.

[0213] In operation 707, in an embodiment, the processor 570 may output information corresponding to the input through the speaker 540 based on the type of the input. For example, the processor 570 may output information corresponding to the input through the speaker 540 based on determining that the information corresponding to the input is to be output through the speaker 540.

[0214] In an embodiment, the processor 570 can process the information corresponding to the input. For example, when it is determined that the audio indicating the information corresponding to the input is to be output through the speaker, the processor 570 can convert the text indicating the information corresponding to the input into the audio via the TTS program. After the text indicating the information corresponding to the input is converted into the audio, the processor 570 can output the converted audio through the speaker 540. In another example, when it is determined that the first tone is to be applied to the audio indicating the information corresponding to the input, the processor 570 can obtain the first tone from the TTS tone DB 561. The processor 570 can apply the first tone to the audio indicating the information corresponding to the input. In another example, when the output determination unit determines that the information corresponding to multiple inputs is to be output simultaneously through the speaker 540, the processor 570 can mix the audio indicating the information corresponding to the multiple inputs, thereby simultaneously outputting the audio indicating the information corresponding to the multiple inputs. However, the present disclosure is not limited to this example.

[0215] although Figure 7 It is shown that after performing operations 701 and 703, operation 705 of determining whether to output information corresponding to the input through the speaker 540 is performed, but the present disclosure is not limited thereto. In an embodiment, the processor 570 may determine whether to output the input to be obtained through one or more input modules through the speaker 540 (and / or through the display 550) before performing operations 701 and / or 703. For example, the processor 570 may determine whether to output the input to be obtained through one or more input modules through the speaker 540 (and / or through the display 550) based on the user input and / or settings of the electronic device 401 before performing operations 701 and / or 703.

[0216] Figure 8 is a flow chart 800 illustrating a method of providing information corresponding to a gesture according to various embodiments.

[0217] Reference Figure 8 In operation 801, in an embodiment, the processor 570 may obtain input through one or more input modules. For example, the processor 570 may obtain gesture input related to the user's gesture and / or the other party's gesture through one or more input modules (e.g., the first camera 421). In another example, the processor 570 may obtain gesture input related to the user's gesture via an external input device (e.g., an input device using a visual method, a glove-type input device, or an armband and / or a wristband).

[0218] In operation 803, in an embodiment, the processor 570 may recognize that the type of input is a gesture input. For example, the processor 570 may recognize that the type of input is a gesture input related to the user's gesture and / or the other party's gesture.

[0219] In an embodiment, the processor 570 may determine whether the input gesture corresponds to one of the user's gesture and the other party's gesture by analyzing the image obtained by the first camera 421 .

[0220] In an embodiment, the processor 570 can identify that the type of input is a user's gesture input by analyzing information received from an external input device (e.g., an input device using a visual method, a glove-type input device, or an armband and / or wristband) through the communication module 530.

[0221] In an embodiment, the processor 570 may convert the gesture input into text. In an embodiment, the processor 570 may identify whether the gesture input is convertible into text. For example, when the gesture input (e.g., the user's gesture input and / or the other party's gesture input) corresponds to sign language, the processor 570 may identify that the gesture input is convertible into text. When it is identified that the gesture input is convertible into text, the processor 570 may convert the gesture input (e.g., the content of the sign language) into text. In another example, when the gesture input (e.g., the user's gesture input and / or the other party's gesture input) corresponds to a specified gesture input, the processor 570 may identify that the gesture input is convertible into text. When it is identified that the gesture input is convertible into text, the processor 570 may convert the gesture input (e.g., information corresponding to the specified gesture) into text.

[0222] In operation 805, in an embodiment, the processor 570 may determine whether to output information corresponding to the gesture input through the speaker 540. For example, the processor 570 may determine whether to output information corresponding to the gesture input through one or more speakers (e.g., the first speaker 441 and the second speaker 442) and / or the display 550.

[0223] In an embodiment, the processor 570 may determine, based on the setting, that audio indicating information corresponding to the user's gesture is to be output through the second speaker 442. For example, the user may configure a configuration (or "mode") (hereinafter referred to as a "first setting") that enables audio indicating information corresponding to the user's gesture to be output through the second speaker 442 in order to listen to the audio indicating information corresponding to the user's gesture.

[0224] In an embodiment, when the first setting is set and the type of input is recognized as a user's gesture input, the processor 570 may determine to apply the user's tone (or a tone specified by the user) to the audio indicating information corresponding to the user's gesture.

[0225] In an embodiment, when the first setting is set and a user's facial expression is input through the second camera 422 in addition to the user's gesture input, the processor 570 may determine that the user's tone and / or the tone corresponding to the user's facial expression (or the emotion corresponding to the user's facial expression) is to be applied to the audio indicating information corresponding to the user's gesture.

[0226] In an embodiment, when the first setting is set and health information about the user (e.g., heart rate, body fat percentage, and electrical signals of neurons) is received from an external input device (e.g., an armband and / or wristband) in addition to the user's gesture input, the processor 570 may determine to apply the user's tone and / or the tone corresponding to the health information about the user to the audio indicating the information corresponding to the user's gesture.

[0227] In an embodiment, when the first setting is set, a user's facial expression is input through the second camera 422, and health information about the user is received from an external input device (e.g., an armband and / or wristband) in addition to the user's gesture input, the processor 570 may determine to apply the user's tone, the tone corresponding to the user's facial expression, and / or the tone corresponding to the health information about the user to the audio indicating information corresponding to the user's gesture.

[0228] In an embodiment, the processor 570 may determine, based on the setting, that the audio indicating the information corresponding to the user's gesture is to be output through the first speaker 441. For example, the user may set a setting (hereinafter, referred to as a "second setting") that enables the audio indicating the information corresponding to the user's gesture to be output through the first speaker 441, so that the other party located adjacent to the electronic device 401 (or the user of the electronic device 401) listens to the audio indicating the information corresponding to the user's gesture.

[0229] In an embodiment, when the second setting is set and the type of input is recognized as a user's gesture input, the processor 570 may determine to apply the user's tone (or a tone specified by the user) to the audio indicating information corresponding to the user's gesture.

[0230] In an embodiment, when the second setting is set and a user's facial expression is input through the second camera 422 in addition to the user's gesture input, the processor 570 may determine that the user's tone and / or the tone corresponding to the user's facial expression (or the emotion corresponding to the user's facial expression) is to be applied to the audio indicating information corresponding to the user's gesture.

[0231] In an embodiment, when the second setting is set and health information about the user (e.g., heart rate, body fat percentage, and electrical signals of neurons) is received from an external input device (e.g., an armband and / or wristband) in addition to the user's gesture input, the processor 570 may determine to apply the user's tone and / or the tone corresponding to the health information about the user to the audio indicating the information corresponding to the user's gesture.

[0232] In an embodiment, when the second setting is set, a user's facial expression is input through the second camera 422, and health information about the user is received from an external input device (e.g., an armband and / or wristband) in addition to the user's gesture input, the processor 570 may determine to apply the user's tone, the tone corresponding to the user's facial expression, and / or the tone corresponding to the health information about the user to the audio indicating information corresponding to the user's gesture.

[0233] In an embodiment, when the second setting is set, the processor 570 may determine the tone of the audio to be applied to the information corresponding to the user's gesture input based on the other party information (e.g., the other party's gender, age, and / or occupation). For example, when the second setting is set and the type of input is the user's gesture input, the processor 570 may determine that the tone of the cartoon character is to be applied to the audio indicating the information corresponding to the user's gesture input based on the other party being a baby or a child.

[0234] In an embodiment, when the second setting is set and the type of input is a user's gesture input, the processor 570 may determine the level of the audio indicating the information corresponding to the user's gesture input based on the distance between the electronic device 401 and the other party. For example, when the second setting is set and the type of input is a user's gesture input, the processor 570 may increase the level of the audio indicating the information corresponding to the user's gesture input as the distance between the electronic device 401 and the other party increases.

[0235] In an embodiment, the processor 570 may determine, based on the setting, that the audio indicating the information corresponding to the other party's gesture is to be output through the second speaker 442. For example, the user may set a setting (hereinafter, referred to as a "third setting") that enables the audio indicating the information corresponding to the other party's gesture to be output through the second speaker 442 in order to listen to the audio indicating the information corresponding to the other party's gesture.

[0236] In an embodiment, when the third setting is set and the type of input is recognized as the other party's gesture input, the processor 570 may determine to apply the other party's tone (or the tone specified by the user) to the audio indicating the information corresponding to the other party's gesture. When the information about the other party's tone is not stored in the memory 560, the processor 570 may determine to apply the user's tone (or the tone specified by the user) to the audio indicating the information corresponding to the other party's gesture.

[0237] In an embodiment, the processor 570 may set the first setting, the second setting, and / or the third setting based on the user input, the type of input, and / or the presence of the other party (e.g., whether the other party is located adjacent to the electronic device 401). For example, when the other party is within a specified distance (e.g., approximately 2m) from the electronic device 401, the processor 570 may set the second setting and the third setting. In another example, when the other party does not exist within a specified distance from the electronic device 401 (e.g., when the other party does not exist adjacent to the electronic device 401), the processor 570 may set the first setting. In another example, when the type of input is a gesture input of the user and the other party exists within a specified distance from the electronic device 401, the processor 570 may set the second setting. In another example, when the type of input is a gesture input of the other party and the other party exists within a specified distance from the electronic device 401, the processor 570 may set the third setting. In another example, when the type of input is a gesture input of the user and the other party does not exist within a specified distance from the electronic device 401, the processor 570 may set the first setting.

[0238] In operation 807 , in an embodiment, when it is determined that the information corresponding to the gesture is to be output through the speaker 540 , the processor 570 may output the information corresponding to the gesture through the speaker 540 .

[0239] In an embodiment, when the first setting is set and it is determined that the user's gesture input is to be output through the second speaker 442, the processor 570 may output audio indicating information corresponding to the user's gesture through the second speaker 442. When the second setting is set and it is determined that the user's gesture input is to be output through the first speaker 441, the processor 570 may output audio indicating information corresponding to the user's gesture through the first speaker 441. When the third setting is set and it is determined that the other party's gesture input is to be output through the second speaker 442, the processor 570 may output audio indicating information corresponding to the other party's gesture through the second speaker 442.

[0240] In an embodiment, the processor 570 may output audio corresponding to the gesture input and having the tone determined in operation 805 through the speaker. In an embodiment, the processor 570 may output audio corresponding to the gesture input and having the audio level determined in operation 805 through the speaker.

[0241] Fig. 9 is a flow chart 900 illustrating a method of providing information corresponding to a gesture according to various embodiments.

[0242] In an embodiment, Fig. 9 It may be a diagram showing an operation performed when a user's voice is input while information corresponding to the user's gesture is output through a speaker (eg, the first speaker 441 ).

[0243] Reference Fig. 9 , in operation 901, in an embodiment, the processor 570 may obtain a user's voice input while outputting information corresponding to a gesture input (e.g., audio indicating information corresponding to the gesture input) through the speaker 540. For example, the processor 570 may obtain a user's voice input through the first microphone 411 while outputting information corresponding to the user's gesture input through the speaker 540. However, the present disclosure is not limited thereto, and the processor 570 may recognize that the user is uttering a voice based on an image obtained by the second camera 422 (e.g., an image of the user's facial muscle movement) while outputting information corresponding to the user's gesture input through the speaker 540.

[0244] In operation 903, in an embodiment, when the user's voice input is obtained while the information corresponding to the gesture input is output through the speaker (or the user is recognized to be uttering a voice based on the image of the user's facial muscle movement), the processor 570 may stop outputting the information corresponding to the gesture input through the speaker 540. For example, when the user's voice input is obtained while the information corresponding to the user's gesture input is output through the first speaker 441, the processor 570 may stop outputting the information corresponding to the user's gesture input through the first speaker 441, so that the voice emitted by the user is accurately sent to the other party. In another example, when the user's voice input is obtained while the information corresponding to the user's gesture input is output through the first speaker 441 and the other party is located within a specified distance from the electronic device 401, the processor 570 may stop outputting the information corresponding to the user's gesture input through the first speaker 441, so that the voice emitted by the user is accurately sent to the other party. In another example, when the user's voice input is obtained while the information corresponding to the user's gesture input is output through the first speaker 441 and the other party does not exist within a specified distance from the electronic device 401, the processor 570 may not stop outputting the information corresponding to the user's gesture input through the first speaker 441.

[0245] Fig.10 1000 is a flowchart illustrating a method of providing information corresponding to a gesture and information corresponding to a voice according to various embodiments.

[0246] Reference Fig.10 In operation 1001, in an embodiment, the processor 570 may obtain a user's voice input while outputting information corresponding to a gesture input (e.g., audio indicating information corresponding to the gesture input) through the speaker 540. For example, the processor 570 may obtain a user's voice input through the first microphone 411 while outputting information corresponding to the user's gesture input through the speaker 540.

[0247] In operation 1003, in an embodiment, the processor 570 may obtain information corresponding to the user's voice input. For example, the processor 570 may obtain information corresponding to the user's voice input by converting the user's voice into text.

[0248] In operation 1005, in an embodiment, the processor 570 may convert information corresponding to the gesture input into a first audio having a first tone, and may convert information corresponding to the user's voice input into a second audio having a second tone.

[0249] In an embodiment, the processor 570 may determine that different tones are to be applied to the first audio indicating information corresponding to the gesture input and the second audio indicating information corresponding to the user's voice input. For example, the processor 570 may determine that a male tone is to be applied to the first audio indicating information corresponding to the gesture input and a female tone is to be applied to the second audio indicating information corresponding to the user's voice input.

[0250] In operation 1007, in an embodiment, the processor 570 may output the first audio and the second audio through the speaker 540. For example, the processor 570 may mix the first audio having the first tone and the second audio having the second tone, and may output the mixed first audio and second audio through the first speaker 441. However, the present disclosure is not limited to this example, and the processor 570 may output the first audio and the second audio through the first speaker 441 without mixing the first audio and the second audio.

[0251] In an embodiment, when obtaining a user's voice input while outputting information corresponding to a gesture input through the speaker 540, the processor 570 may Fig. 9 As described above, the information corresponding to the gesture input is stopped from being output through the speaker 540 or based on the user input as described above. Fig.10 As described, information corresponding to the gesture input (eg, first audio) and information corresponding to the user's voice input (eg, second audio) to which different tones are applied are simultaneously output through the speaker.

[0252] Fig.11 11 is a flow chart 1100 illustrating a method of providing information corresponding to a gesture while conducting a call according to various embodiments.

[0253] Reference Fig.11 In operation 1101, in an embodiment, the processor 570 may obtain a gesture input while the user is talking to the other party. For example, the processor 570 may obtain a gesture input of the user based on an image obtained by the first camera 421 while the user is talking to the other party via the electronic device 401.

[0254] In operation 1103, in an embodiment, the processor 570 may identify information corresponding to the gesture input. For example, the processor 570 may convert the user's gesture input into text, thereby identifying information corresponding to the gesture input (e.g., the content of the sign language or information corresponding to the specified gesture).

[0255] In operation 1105, in an embodiment, the processor 570 may transmit information corresponding to the gesture input to an external electronic device (e.g., the electronic device 102 or the electronic device 104) through the communication module 530. For example, the processor 570 may apply the user's tone (or the tone specified by the user) to the audio indicating the information corresponding to the gesture input. The processor 570 may transmit a signal of the audio to which the user's tone is applied to the other party's electronic device through the communication module 530, so that the signal of the audio is output from the other party's electronic device with which the user is performing a call.

[0256] In an embodiment, the processor 570 may send information corresponding to the user's gesture to an external electronic device through the communication module 530 while the user is talking to the other party, thereby enabling the user to talk to the other party by using gestures in a quiet place such as a library.

[0257] although Fig.11 The processor 570 is shown to send a signal indicating an audio of information corresponding to the gesture input to the external electronic device through the communication module 530, but the present disclosure is not limited thereto. For example, the processor 570 may send a text indicating information corresponding to the gesture input to the external electronic device through the communication module 530 (e.g., without converting the text into audio via a TTS program).

[0258] In an embodiment, the processor 570 may receive an audio signal indicating information corresponding to the other party's gesture input obtained by the other party's electronic device from the external electronic device through the communication module 530. The processor 570 may output the received audio signal through the second speaker 442.

[0259] In an embodiment, the processor 570 may receive text indicating information corresponding to the other party's gesture input obtained by the other party's electronic device from the external electronic device through the communication module 530. The processor 570 may convert the received text into audio via a TTS program and may output the converted audio through the second speaker 442.

[0260] In an embodiment, when the processor 570 obtains a user's gesture input based on an image obtained by the first camera 421, while the user is talking on the other party by using the electronic device 401, when an audio signal indicating information corresponding to the other party's voice obtained by the other party's electronic device is received from an external electronic device through the communication module 530, the processor 570 can convert the received audio signal into text and can output the text through the display 550.

[0261] Fig.1212 is a flowchart 1200 illustrating a method of providing information corresponding to a user's voice while a call is ongoing according to various embodiments.

[0262] Reference Fig.12 In operation 1201, in an embodiment, the processor 570 may obtain a gesture input while the user is talking to the other party. For example, the processor 570 may obtain a gesture input of the user based on an image obtained by the first camera 421 while the user is talking to the other party by using the electronic device 401.

[0263] In operation 1203 , in an embodiment, the processor 570 may transmit information corresponding to the gesture input to an external electronic device (eg, the electronic device 102 or the electronic device 104 ) through the communication module 530 .

[0264] In operation 1205, in an embodiment, the processor 570 may obtain the user's voice input through the microphone 510 while sending information corresponding to the gesture input to the external electronic device through the communication module 530. For example, the processor 570 may obtain the user's voice input through the first microphone 411 while sending information corresponding to the gesture input to the external electronic device through the communication module 530.

[0265] In operation 1207 , in an embodiment, the processor 570 may stop sending information corresponding to the gesture input to the external electronic device through the communication module 530 , and may send information corresponding to the user's voice input to the external electronic device through the communication module 530 .

[0266] In an embodiment, when a user moves to a place where a call can be made using voice while talking to the other party by using gestures in a quiet place such as a library, the processor 570 can stop sending information corresponding to the gesture input to the external electronic device through the communication module 530, and can send information corresponding to the user's voice input (an audio signal indicating information corresponding to the user's voice input) to the external electronic device through the communication module 530.

[0267] Fig.13 13 is a flowchart 1300 illustrating a method of providing information corresponding to a gesture and information corresponding to a voice while a call is ongoing according to various embodiments.

[0268] In an embodiment, Fig.13 Operations of the electronic device 401 performed when the user displays information corresponding to the user's gesture input to the other party on the display 550 and transmits information corresponding to the user's voice input to the other party in the call are shown.

[0269] Reference Fig.13 In operation 1301, in an embodiment, the processor 570 may obtain a user's gesture input and a user's voice input while making a call. For example, while making a call with the other party, the processor 570 may obtain a user's gesture input based on an image obtained by the first camera 421, and obtain a user's voice input through the first microphone 411.

[0270] In operation 1303, in an embodiment, the processor 570 may identify information corresponding to the user's gesture input. For example, the processor 570 may convert the user's gesture input into text, thereby identifying information corresponding to the gesture input (e.g., the content of the sign language or information corresponding to the specified gesture).

[0271] In operation 1305, in an embodiment, the processor 570 may recognize information corresponding to the user's voice input. For example, the processor 570 may convert the user's voice input into text, thereby obtaining information corresponding to the user's voice input.

[0272] In operation 1307, in an embodiment, the processor 570 may output information corresponding to the gesture input through the display 550. For example, the processor 570 may display an external screen including text indicating information corresponding to the gesture input on the display 550 so that a party adjacent to the electronic device 401 may view the external screen.

[0273] In operation 1309, in an embodiment, the processor 570 may send information corresponding to the user's voice input to an external electronic device through the communication module 530. For example, the processor 570 may send an audio signal indicating information corresponding to the user's voice input to the other party's electronic device in the call through the communication module 530.

[0274] Fig.14 1400 is a flowchart illustrating a method of providing information corresponding to a gesture and information corresponding to a voice while a call is ongoing according to various embodiments.

[0275] In an embodiment, Fig.14 Operations of the electronic device 401 performed when the user transmits information corresponding to the user's gesture input to a counterpart in a call and transmits information corresponding to the user's voice input to a counterpart adjacent to the electronic device 401 are shown.

[0276] Reference Fig.14In operation 1401, in an embodiment, the processor 570 may obtain a user's gesture input and a user's voice input while making a call. For example, the processor 570 may obtain a user's gesture input based on an image obtained by the first camera 421, and obtain a user's voice input through the first microphone 411 while making a call with the other party.

[0277] In operation 1403, in an embodiment, the processor 570 may identify information corresponding to the user's gesture input. For example, the processor 570 may convert the user's gesture input into text, thereby identifying information corresponding to the gesture input (e.g., the content of the sign language or information corresponding to the specified gesture).

[0278] In operation 1405, in an embodiment, the processor 570 may recognize information corresponding to the user's voice input. For example, the processor 570 may convert the user's voice input into text, thereby obtaining information corresponding to the user's voice input.

[0279] In operation 1407, in an embodiment, the processor 570 may send information corresponding to the user's gesture input to an external electronic device through the communication module 530. For example, the processor 570 may apply the user's tone (or the tone specified by the user) to the audio indicating the information corresponding to the user's gesture input. The processor 570 may send a signal of the audio to which the user's tone is applied to the electronic device of the other party in the call through the communication module 530. In an embodiment, the processor 570 may control the information corresponding to the user's voice input not to be sent to the electronic device of the other party in the call while sending the information corresponding to the user's gesture input to the external electronic device through the communication module 530.

[0280] In operation 1409, in an embodiment, the processor 570 may output information corresponding to the user's voice input through the speaker 540. For example, the processor 570 may output information corresponding to the user's voice input through the first speaker 441. However, the present disclosure is not limited to this example. The processor 570 may not output information corresponding to the user's voice input through the first speaker 441, or may deactivate the first speaker 441 so that the voice issued by the user is directly sent to the other party adjacent to the electronic device 401.

[0281] In an embodiment, the processor 570 may, based on the user input, output information corresponding to the user's gesture through the first speaker 441 and send information corresponding to the user's voice input to an external electronic device through the communication module 530, or may send information corresponding to the user's gesture to an external electronic device through the communication module 530 and output information corresponding to the user's voice input through the first speaker 441.

[0282] Fig.15 15 is a flowchart 1500 illustrating a method of providing information corresponding to a gesture and information corresponding to a voice according to various embodiments.

[0283] Reference Fig.15 , in operation 1501, in an embodiment, the processor 570 may obtain a user's gesture input and a user's voice input.

[0284] In an embodiment, in addition to the user's gesture input and the user's voice input, the processor 570 may also obtain the user's facial expression and / or health information about the user.

[0285] In operation 1503 , in an embodiment, the processor 570 may recognize information corresponding to a gesture input of a user.

[0286] In operation 1505, in an embodiment, the processor 570 may recognize information corresponding to a user's voice input.

[0287] In an embodiment, the processor 570 may recognize content indicating a user's facial expression (or an emotion corresponding to the user's input facial expression) and health information.

[0288] In operation 1507 , in an embodiment, the processor 570 may output information corresponding to the user's gesture input and information corresponding to the user's voice input (and / or content indicating the user's facial expression and health information) through the display 550 .

[0289] In an embodiment, the processor 570 can control the display 550 to output text of information corresponding to the user's gesture input and information corresponding to the user's voice input (and / or content indicating the user's facial expressions and health information) in different sizes, different colors, different text display areas on the transparent component, and / or different languages, so that the information corresponding to the user's gesture input and the information corresponding to the user's voice input (and / or content indicating the user's facial expressions and health information) can be distinguished from each other.

[0290] Despite Fig.15Although not shown in the figure, in an embodiment, the processor 570 may output information corresponding to the user's gesture input and information corresponding to the user's voice input (and / or content indicating the user's facial expression and health information) via the avatar corresponding to the user. For example, the processor 570 may control the display 550 so that the avatar corresponding to the user performs a movement corresponding to the user's gesture input and makes an expression corresponding to the user's facial expression (or displays a graphic object or text indicating that the avatar is thinking). The processor 570 may display an external screen including an avatar on the display 550 so that an image (and text) using the avatar is usually seen by the other party. In an embodiment, the processor 570 may display an external screen including an image (and text) using an avatar in a first area of ​​the display 550, and may display an internal screen including an image (and text) using an avatar in a second area of ​​the display 550. The processor 570 may apply the tone of the avatar corresponding to the user to the audio indicating the information corresponding to the user's voice input, and may output the audio to which the tone of the avatar is applied through the first speaker 441.

[0291] Fig.16 16 is a flowchart 1600 illustrating a method of providing information based on surrounding environment according to various embodiments.

[0292] Reference Fig.16 In operation 1601, in an embodiment, the processor 570 may obtain a gesture input. For example, the processor 570 may obtain a gesture input of a user and / or a gesture input of a counterpart based on an image obtained by the first camera 421.

[0293] In operation 1603, in an embodiment, the processor 570 may obtain information about the surrounding environment of the electronic device 401. For example, the processor 570 may obtain ambient sound, brightness outside the electronic device 401, and / or a distance between the electronic device 401 and the other party.

[0294] In operation 1605, in an embodiment, the processor 570 may determine, based on the information about the surrounding environment, whether to output information corresponding to the gesture input through the speaker 540. For example, the processor 570 may determine, based on the information about the surrounding environment, whether to output information corresponding to the gesture input through one or more speakers (e.g., the first speaker 441 and the second speaker 442) and / or through the display 550.

[0295] In an embodiment, the processor 570 may determine whether to output information corresponding to the gesture input through the speaker 540 based on the ambient sound. For example, when the level of the ambient sound is a specified level or higher, the processor 570 may determine to output text indicating information corresponding to the other party's gesture input through the display 550. When the level of the ambient sound is lower than the specified level, the processor 570 may determine to output audio indicating information corresponding to the other party's gesture input through the second speaker 442.

[0296] In an embodiment, the processor 570 may determine whether to output information corresponding to the gesture input through the speaker 540 based on the brightness outside the electronic device 401. For example, when the brightness outside the electronic device 401 is a specified brightness or higher, the processor 570 may determine to output audio indicating information corresponding to the other party's gesture input through the second speaker 442. When the brightness outside the electronic device 401 is lower than the specified brightness, the processor 570 may determine to output an internal screen including text indicating information corresponding to the other party's gesture input through the display 550.

[0297] In an embodiment, the processor 570 may determine whether to output information corresponding to the gesture input through a speaker based on the distance between the electronic device 401 and the other party. For example, when the distance between the electronic device 401 and the other party is a specified distance or greater, the processor 570 may determine to output an audio indicating information corresponding to the gesture input of the other party through the second speaker 442. When the distance between the electronic device 401 and the other party is less than the specified distance, the processor 570 may determine to output an internal screen including text indicating information corresponding to the gesture input of the other party through the display 550. In another example, when the distance between the electronic device 401 and the other party increases, the processor 570 may determine to output an audio indicating information corresponding to the gesture input of the other party through the second speaker 442. When the distance between the electronic device 401 and the other party decreases, the processor 570 may determine to output an internal screen including text indicating information corresponding to the gesture input of the other party through the display 550.

[0298] Fig.17 is a flow chart 1700 illustrating a method of providing information corresponding to a plurality of inputs according to various embodiments.

[0299] Reference Fig.17 In operation 1701, in an embodiment, the processor 570 may obtain a user's gesture input, a user's facial expression, and / or health information about the user (as well as a user input via a virtual keyboard).

[0300] In operation 1703, in an embodiment, the processor 570 may recognize information corresponding to a gesture input of a user, information corresponding to a facial expression of a user, and / or information corresponding to health information about the user.

[0301] In operation 1705 , in an embodiment, the processor 570 may output information corresponding to a user's gesture input, information corresponding to a user's facial expression, and / or information corresponding to a user's health information through the speaker 540 .

[0302] In an embodiment, the processor 570 may apply different tones to information corresponding to the user's gesture input, information corresponding to the user's facial expression, and / or information corresponding to the user's health information, respectively. For example, the processor 570 may apply a female tone to a first audio indicating information corresponding to the user's gesture input, a male tone to a second audio indicating information corresponding to the user's facial expression, and a machine tone to information corresponding to health information about the user.

[0303] In an embodiment, the processor 570 may apply different audio levels (e.g., volumes) to information corresponding to a user's gesture input, information corresponding to a user's facial expression, and / or information corresponding to health information about the user, respectively.

[0304] In an embodiment, the processor 570 may determine the order in which information corresponding to the user's gesture input, information corresponding to the user's facial expression, and / or information corresponding to the user's health information is sequentially output.

[0305] In an embodiment, the processor 570 may output information corresponding to the user's gesture input, information corresponding to the user's facial expression, and / or information corresponding to the user's health information through the first speaker 441 based on the determined tone, audio level, and / or output order. For example, in an embodiment, the processor 570 may mix a first audio indicating information corresponding to the user's gesture input, a second audio indicating information corresponding to the user's facial expression, and / or a third audio indicating information corresponding to the user's health information based on the determined tone, audio level, and / or output order, and may output the audio through the first speaker 441.

[0306] In an embodiment, the processor may determine whether to output information corresponding to the input through a speaker based on information about the user and / or information about the other party. For example, when the user is hearing-impaired, the processor may display the other party's voice in text on the display 451. For example, when a gesture indicating sign language is input from the other party and the user does not understand sign language, the processor may output a gesture indicating the sign language input by the other party in voice and / or text. For example, in a case where the other party is hearing-impaired (e.g., a hearing-impaired person who understands sign language) and the user does not understand sign language, when the user's voice is input, the processor may display the user's input voice in text on the display 451.

[0307] According to various embodiments of the present disclosure, the method for the electronic device 401 to provide information may include: obtaining input through one or more input modules (e.g., microphone 510 and camera 520) of the electronic device 401; identifying the type of input; determining whether to output information corresponding to the input through one or more speakers (e.g., speaker 540); and based on determining that the information corresponding to the input is to be output through one or more speakers (e.g., speaker 540), controlling one or more speakers (e.g., speaker 540) so that the information corresponding to the input is output through the one or more speakers (e.g., speaker 540) based on the type of input.

[0308] In various embodiments, the method may further include converting the obtained input into text.

[0309] In various embodiments, determining whether information corresponding to the input is to be output through one or more speakers (e.g., speaker 540) may include determining whether information corresponding to the input is to be output through one or more speakers (e.g., speaker 540) based on at least one of the user input or the type of input.

[0310] In various embodiments, determining whether to output information corresponding to an input through one or more speakers may include: when the type of input is a user's gesture and / or voice, determining that information corresponding to the gesture and / or voice of a user of the electronic device 401 is to be output through a first speaker 441 among the one or more speakers; and when the type of input is a gesture and / or voice of the other party, determining that information corresponding to the gesture and / or voice of the other party is to be output through a second speaker 442 among the one or more speakers.

[0311] In various embodiments, determining whether to output information corresponding to the input through one or more speakers (eg, speakers 540 ) may include determining a tone to be applied to audio indicating the information corresponding to the input.

[0312] In various embodiments, obtaining input may include obtaining multiple inputs through one or more input modules (e.g., microphone 510 and camera 520), and determining the tone to be applied to the audio indicating information corresponding to the input may include determining different tones to be applied to the audio indicating information corresponding to the multiple inputs and to be output simultaneously based on the types of the multiple inputs.

[0313] In various embodiments, obtaining multiple inputs through one or more input modules (e.g., microphone 510 and camera 520) may include obtaining a user's gesture input and a user's voice input of the electronic device through one or more input modules (e.g., microphone 510 and camera 520), and determining different tones to be applied to audio indicating information corresponding to the multiple inputs and to be output simultaneously based on the types of the multiple inputs may include determining a first tone to be applied to first audio indicating information corresponding to the user's gesture input and a second tone to be applied to second audio indicating information corresponding to the user's voice input and to be output simultaneously with the first audio.

[0314] In various embodiments, determining whether to output information corresponding to an input through one or more speakers may include: when the type of input is a user's gesture and / or voice, determining that information corresponding to the user's gesture and / or voice of the electronic device 401 is to be displayed on an external screen of a display (e.g., display 550); and when the type of input is the other party's gesture and / or voice, determining that information corresponding to the other party's gesture and / or voice is to be output on an internal screen of the display (e.g., display 550).

[0315] In various embodiments, the method may also include: when a user's gesture is received while the electronic device 401 is talking with an external electronic device, sending information corresponding to the user's gesture of the electronic device 401 to the external electronic device through the communication module (e.g., communication module 530) of the electronic device 401.

[0316] In various embodiments, the method may further include obtaining information about the surrounding environment of the electronic device 401, and determining whether to output information corresponding to the input through one or more speakers (e.g., speaker 540) based on the information about the surrounding environment of the electronic device 401.

[0317] In various embodiments, the electronic device 401 may include AR glasses, and the input may include at least one of the following: a gesture of a user of the electronic device, a gesture of a counterpart positioned adjacent to the electronic device 401, the user's voice, the counterpart's voice, the user's facial expression, or health information about the user.

[0318] The data structures used in the foregoing embodiments of the present disclosure can be recorded in a computer-readable recording medium by various methods. The computer-readable recording medium may include a storage medium, such as a magnetic storage medium (e.g., ROM, floppy disk, or hard disk) or an optical reading medium (e.g., CD-ROM or DVD).

Claims

1. An electronic device, the electronic device include: one or more input modules; one or more loudspeakers; monitor; as well as at least one processor operably connected to the one or more input modules, the one or more speakers and the display, Wherein, the at least one processor is configured to: obtaining input via the one or more input modules; identifying a type of the input; determining whether information corresponding to the input is to be output through the one or more speakers; and Based on determining that the information corresponding to the input is to be output through the one or more speakers, the one or more speakers are controlled so that the information corresponding to the input is output through the one or more speakers based on the type of the input.

2. The electronic device according to claim 1, in, The at least one processor is further configured to convert the obtained input into text.

3. The electronic device according to claim 1, in, The at least one processor is configured to determine whether to output the information corresponding to the input through the one or more speakers based on at least one of a user input or a type of the input.

4. The electronic device according to claim 1, in, The at least one processor is configured to: When the type of the input is a gesture and / or a voice of a user of the electronic device, determining to output information corresponding to the gesture and / or the voice of the user through a first speaker among the one or more speakers; as well as When the type of the input is a gesture and / or a voice of the other party, it is determined that information corresponding to the gesture and / or the voice of the other party is to be output through a second speaker among the one or more speakers.

5. The electronic device according to claim 1, in, The at least one processor is configured to determine a tone to be applied to audio indicative of the information corresponding to the input.

6. The electronic device according to claim 5, in, The at least one processor is configured to: obtaining a plurality of inputs through the one or more input modules; as well as Based on the types of the plurality of inputs, different tones to be applied to audio indicating information corresponding to the plurality of inputs and to be simultaneously output are determined.

7. The electronic device according to claim 6, in, The at least one processor is configured to: obtaining a gesture input of a user of the electronic device and a voice input of the user through the one or more input modules; as well as A first tone to be applied to first audio indicating information corresponding to the gesture input of the user and a second tone to be applied to second audio indicating information corresponding to the voice input of the user and to be output simultaneously with the first audio are determined.

8. The electronic device according to claim 1, in, The at least one processor is configured to: When the type of the input is a gesture and / or a voice of a user of the electronic device, determining to display information corresponding to the gesture and / or the voice of the user through an external screen of the display; as well as When the type of the input is a gesture and / or voice of the other party, it is determined that information corresponding to the gesture and / or voice of the other party is to be output on an internal screen of the display.

9. The electronic device according to claim 1, further comprising a communication module, in, The at least one processor is further configured to: When a gesture of a user of the electronic device is received while the electronic device is talking with an external electronic device, information corresponding to the gesture of the user is transmitted to the external electronic device through the communication module.

10. The electronic device according to claim 1, in, The at least one processor is further configured to: obtaining information about the surrounding environment of the electronic device; and Based on the information about the surrounding environment of the electronic device, it is determined whether the information corresponding to the input is to be output through the one or more speakers.

11. A method for providing information by an electronic device, the method include: obtaining input via one or more input modules of the electronic device; identifying a type of the input; determining whether to output information corresponding to the input through one or more speakers; as well as Based on determining that the information corresponding to the input is to be output through the one or more speakers, the one or more speakers are controlled so that the information corresponding to the input is output through the one or more speakers based on the type of the input.

12. The method according to claim 11, further comprising: include: Convert the input obtained into text.

13. The method according to claim 11, in, Determining whether to output the information corresponding to the input through the one or more speakers includes: Based on at least one of a user input or a type of the input, it is determined whether the information corresponding to the input is to be output through the one or more speakers.

14. The method according to claim 11, in, Determining whether to output the information corresponding to the input through the one or more speakers includes: When the type of the input is a gesture and / or a voice of a user of the electronic device, determining to output information corresponding to the gesture and / or the voice of the user through a first speaker among the one or more speakers; and When the type of the input is a gesture and / or a voice of the other party, it is determined that information corresponding to the gesture and / or the voice of the other party is to be output through a second speaker among the one or more speakers.

15. The method according to claim 11, in, Determining whether to output the information corresponding to the input through the one or more speakers includes: A pitch to be applied to audio indicative of the information corresponding to the input is determined.