Wearable device, method, and non-transitory computer-readable storage medium for providing notification

Abstract

At least one processor is configured to: identify at least one keyword, acquired from an electronic device connected to the wearable device, from an audio signal, acquired by using the microphone, on the basis of the at least one keyword and the audio signal; use a language model to determine whether the at least one keyword in the audio signal is used to call a user; on the basis of determining that the at least one keyword is used to call the user, use the language model to determine whether a gesture responding to the call for the user is detected; provide a notification for the user on the basis of determining that the gesture is not detected; and refrain from providing the notification for the user on the basis of determining that the gesture is detected.

Description

Wearable device, method, and non-transitory computer-readable storage medium for providing notifications

[0001] The following descriptions relate to a wearable device, a method, and a non-transitory computer-readable storage medium for providing notifications.

[0002] A wearable device can receive audio signals from an electronic device connected to the wearable device. The wearable device can then use a speaker to provide a sound corresponding to the audio signal to the user. While the sound is being provided, the user wearing the wearable device may not be able to hear surrounding sounds.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] A wearable device is provided. The wearable device may include at least one sensor. The wearable device may include a microphone. The wearable device may include a speaker. The wearable device may include a memory storing instructions and including one or more storage media. The wearable device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify at least one keyword from an audio signal based on at least one keyword obtained from an electronic device connected to the wearable device and an audio signal obtained using the microphone. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine, using a language model, whether the at least one keyword in the audio signal is used to call a user. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine, using the language model, whether a gesture responsive to a call to the user is detected based on a determination that the at least one keyword is used to call the user. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to provide a notification for the user based on a determination that the gesture is not detected. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to refrain from providing a notification for the user based on a determination that the gesture is detected.

[0005] A method performed by a wearable device is provided. The method may include an operation of identifying at least one keyword from an audio signal obtained from an electronic device connected to the wearable device and an audio signal obtained using the microphone. The method may include an operation of determining, using a language model, whether the at least one keyword in the audio signal is used to call a user. The method may include an operation of determining, using the language model, whether a gesture responding to a call to the user is detected, based on a determination that the at least one keyword is used to call the user. The method may include an operation of providing a notification for the user, based on a determination that the gesture is not detected. The method may include an operation of refraining from providing a notification for the user, based on a determination that the gesture is detected.

[0006] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device including at least one processor, cause the wearable device to identify at least one keyword from an audio signal based on at least one keyword obtained from an electronic device connected to the wearable device and an audio signal obtained using the microphone. The one or more programs may include instructions that, when executed by the wearable device including at least one processor, cause the wearable device to determine, using a language model, whether the at least one keyword in the audio signal is used to call a user. The one or more programs may include instructions that, when executed by the wearable device including at least one processor, cause the wearable device to determine, using the language model, whether a gesture responsive to a call to the user is detected based on a determination that the at least one keyword is used to call the user. The one or more programs may include instructions that, when executed by a wearable device including at least one processor, cause a notification to be provided to the user based on a determination that the gesture is not detected. The one or more programs may include instructions that, when executed by a wearable device including at least one processor, cause a notification to be refrained from being provided to the user based on a determination that the gesture is detected.

[0007] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0008] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0009] Figure 2a illustrates components of a wearable device and an electronic device.

[0010] FIG. 2b illustrates components of a wearable device and an electronic device for providing notifications and summary information to a user.

[0011] FIG. 3 is a block diagram illustrating the operation of a wearable device and an electronic device for providing notifications and summary information to a user.

[0012] Figure 4a is a flowchart illustrating the operation of a wearable device for providing user identification information.

[0013] FIG. 4b illustrates a flowchart of a method of a wearable device for providing notifications.

[0014] Figure 5 illustrates an example of providing a notification to a user.

[0015] Figure 6 is a flowchart illustrating the operation of an electronic device for providing summary information.

[0016] Figure 7 illustrates signaling between a wearable device and an electronic device to provide summary information.

[0017] Figures 8a to 8c illustrate examples of providing summary information.

[0018] Figure 9 illustrates an example of signaling between a wearable device and an electronic device to provide notification and summary information for a user call.

[0019] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0020] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0021] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled, but this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." A condition described as "more than" may be replaced with "more than," a condition described as "less than" may be replaced with "less than," and a condition described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of elements from A (including A) to B (including B). hereinafter, "C" and / or "D" mean at least one of "C" or "D," that is, including {"C", "D", "C" and "D"}.

[0022] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0023] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0024] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0025] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0026] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0027] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0028] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0029] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0030] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0031] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0032] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0033] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0034] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0036] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0037] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0038] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0039] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0040] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0041] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0042] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0043] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0044] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0045] Figure 2a illustrates components of a wearable device and an electronic device.

[0046] FIG. 2A illustrates an exemplary block diagram of an electronic device (101) and a wearable device (103). For example, the wearable device (103) may be worn on a body part of a user. For example, the body part may include an ear part or an external auditory canal part of the ear of the user. For example, the wearable device (103) may be referred to as earbuds, earphones, a headset, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (XR) device, or a true wireless stereo (TWS). The wearable device (103) of FIG. 2A may be an example of the electronic device (102) of FIG. 1 connected to the electronic device (101).

[0047] Referring to FIG. 2A, the wearable device (103) may be connected to the electronic device (101) based on a wired network and / or a wireless network. For example, the wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), or a combination thereof. For example, the wireless network may include a network such as long term evolution (LTE), 5G new radio (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low-energy (BLE), or a combination thereof. For example, the wearable device (103) may be directly connected to the electronic device (101), or may be indirectly connected via one or more routers and / or access points (APs).

[0048] Components of a wearable device (103) are described with reference to FIG. 2A. In one embodiment, the wearable device (103) may include a processor (201), an audio output module (202), an input module (203), a sensor module (204), a memory (205), a communication module (206), an artificial intelligence module (207), and a battery (208). For example, the processor (201), the audio output module (202), the input module (203), the sensor module (204), the memory (205), the communication module (206), the artificial intelligence module (207), and the battery (208) may be electronically and / or operably coupled with each other by a communication bus.

[0049] Hereinafter, the hardware components being operatively coupled may mean that a direct connection or an indirect connection is established between the hardware components, either wired or wireless, so that a second hardware component is controlled by a first hardware component among the hardware components. The hardware components illustrated in FIG. 2A are illustrated based on different blocks, but the present disclosure is not limited thereto. For example, some of the hardware components illustrated in FIG. 2A (e.g., at least a portion of the processor (201), the memory (205), and the communication module (206)) may be included in a single integrated circuit such as a system on chip (SoC) or a system in package (SIP). The type and / or number of hardware components included in the wearable device (103) is not limited to those illustrated in FIG. 2A. For example, the wearable device (103) may include only some of the hardware components illustrated in FIG. 2A.

[0050] In one embodiment, the processor (201) (e.g., application processor (AP)) of the wearable device (103) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microcontroller (MCU), and / or a neural processing unit (NPU). The number of processors (201) may be one or more. For example, the processor (201) may have a multi-core processor structure, such as a dual core, a quad core, or a hexa core. The processor (201) of FIG. 2A may be applied substantially identically to the processor (120) of FIG. 1.

[0051] In one embodiment, the processor (201) may include a noise cancellation module. For example, the noise cancellation module of the processor (201) may control a noise cancellation filter (e.g., the noise cancellation filter (210) of FIG. 2B) to generate a sound (or audio signal) for canceling out ambient noise based on an audio signal acquired through microphones (e.g., a main microphone and a sub microphone) of the input module (203).

[0052] In one embodiment, the processor (201) may include a detachment recognition module. For example, the detachment recognition module of the processor (201) may identify whether the wearable device (103) is detached or not based on sensor data obtained from the sensor module (204). In one example, the detachment recognition module of the processor (201) may identify contact between the wearable device (103) and a body part of the user based on sensor data obtained from the sensor module (204). In one example, the detachment recognition module of the processor (201) may identify release of contact between the wearable device (103) and a body part of the user based on sensor data obtained from the sensor module (204).

[0053] In one embodiment, the processor (201) may include a control unit. For example, the control unit of the processor (201) may perform settings for an application for listening to ambient sounds, settings for activating an artificial intelligence module (207), and / or settings for detecting emergency signals.

[0054] In one embodiment, the DSP of the processor (201) may include a keyword spotter (KWS). For example, the KWS may identify a keyword from an audio signal based on a feature vector of the keyword obtained from the electronic device (101). For example, the keyword may refer to a text referring to a user of the wearable device (103). For example, the feature vector of the keyword may represent a frequency characteristic (e.g., mel-frequency cepstral coefficient (MFCC)) of a voice signal corresponding to the keyword and / or energy by frequency band (e.g., filter bank energy) of a voice signal corresponding to the keyword.

[0055] In one embodiment, the processor (201) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to individually and / or collectively perform the various functions described below in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, as well as situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor is capable of executing program instructions to accomplish or perform various functions.

[0056] In one embodiment, the wearable device (103) may include an audio output module (202). For example, the audio output module (202) of the wearable device (103) may include a speaker for outputting an audio signal, an acoustic signal, and / or a sound. For example, the wearable device (103) may include a nozzle that serves as a path for the audio signal, the acoustic signal, and / or the sound output from the speaker. For example, the nozzle may be referred to as an acoustic port. For example, the nozzle may be a path through which the wearable device (103) is supported within a body part of a user when the wearable device (103) is worn on the body part, and through which sound output from the wearable device (103) passes. For example, the nozzle may be connected to an ear tip. For example, the ear tip may represent a member that comes into contact with a body part of a user. In FIG. 2A, a wearable device (103) is illustrated that includes an audio output module (202) for outputting audio signals, voice signals, and / or sounds, but embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may further include an actuator (or motor, haptic module) for providing haptic feedback based on vibration.

[0057] In one embodiment, the wearable device (103) may include an input module (203). For example, the input module (203) of the wearable device (103) may include a microphone. For example, the microphone may include a main microphone, a sub microphone, or a combination thereof. For example, the wearable device (103) may acquire a sound (or a voice signal, an audio signal, and / or ambient noise) input from an external source of the wearable device (103) using the microphone of the input module (203).

[0058] In one embodiment, the wearable device (103) may include a sensor module (204). For example, the sensor module (204) may include a proximity sensor, a Hall sensor, an acceleration sensor, a skin sensor, a gyro sensor, a force sensor, or a combination thereof. However, the present disclosure is not limited thereto. For example, the sensor module (204) of the wearable device (103) may further include a magnetometer, a barometric pressure sensor, a temperature sensor, a humidity sensor, an illuminance sensor, and / or an infrared (IR) sensor.

[0059] In one embodiment, the wearable device (103) may include a memory (205). For example, the memory (205) may include a hardware component for storing data and / or instructions input to and / or output from the processor (201). For example, the memory (205) may include a volatile memory, such as a random-access memory (RAM), and / or a non-volatile memory, such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multimedia card (eMMC). The specific details of the memory (205) of Fig. 2a can be applied substantially identically to the details of the memory (130) of Fig. 1.

[0060] In one embodiment, the memory (205) may include one or more buffers (e.g., buffer (205-1) and buffer (205-2) of FIG. 2B), although the present disclosure is not limited thereto. The memory (205) and the buffers may be configured separately within the wearable device (103). For example, the memory (205) may include a first buffer (e.g., buffer (205-1) of FIG. 2B) for temporarily storing keywords identified from an audio signal by the processor (201) and likelihood information about the keywords. For example, the memory (205) may include a second buffer (e.g., buffer (205-2) of FIG. 2B) for temporarily storing voice signals other than user calls that require analysis by the electronic device (101).

[0061] In one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by the processor (201) of the wearable device (103) may be stored within the memory (205) of the wearable device (103). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, when an application is installed within the wearable device (103), it may mean that one or more instructions provided in the form of an application are stored within the memory (205), and that the one or more applications are stored in a format executable by the processor (201) of the wearable device (103) (e.g., a file having an extension specified by the operating system of the wearable device (103).

[0062] In one embodiment, the wearable device (103) may include a communication module (206). For example, the communication module (206) may include communication circuitry for supporting transmission and / or reception of electrical signals between the wearable device (103) and the electronic device (101). The communication module (206) may include, for example, at least one of a modem (modulator and demodulator), an antenna, and an optical / electronic (O / E) converter. The communication module (206) may support transmission and / or reception of electrical signals based on various types of communication means, such as Ethernet, Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). Specific details regarding the communication module (206) of FIG. 2A may be substantially identical to the communication module (190) and / or antenna module (197) of FIG. 1.

[0063] In one embodiment, the wearable device (103) may include an artificial intelligence module (207). For example, the artificial intelligence module (207) may be referred to as an artificial intelligence model, an on-device AI core, an on-device AI module, a machine learning model, a deep learning model, a natural language processing model, or equivalent technical terms. For example, the artificial intelligence module (207) may include a language model (e.g., a large language model (LLM)) and / or an artificial intelligence model for converting text into a speech signal (e.g., text to speech (TTS)).

[0064] In one embodiment, the artificial intelligence module (207) of the wearable device (103) may be a unit (function code, separate device, circuit, or set of instructions) that determines whether a keyword identified by the processor (201) (e.g., DSP) is used for a user call. For example, the artificial intelligence module (207) may use a prompt generated based on information collected by the electronic device (101) and the wearable device (103) as input data. For example, the artificial intelligence module (207) may output output data indicating whether a keyword is used for a user call based on the input data.

[0065] In one embodiment, the artificial intelligence module (207) of the wearable device (103) may be a unit (function code, separate device, circuit, or set of instructions) that determines whether a gesture in response to a user call is detected. For example, the artificial intelligence module (207) may use a prompt generated based on information collected by the electronic device (101) and the wearable device (103) as input data. For example, the artificial intelligence module (207) may output output data indicating whether a gesture in response to a user call is detected based on the input data.

[0066] In one embodiment, the artificial intelligence module (207) of the wearable device (103) may be a unit (or a set of function codes, separate devices, circuits, or instructions) that generates a voice signal corresponding to summary information. For example, the artificial intelligence module (207) may use summary information received (or acquired) from the electronic device (101) as input data. For example, the artificial intelligence module (207) may output a voice signal corresponding to the summary information based on the input data.

[0067] In one embodiment, the wearable device (103) may include a battery (208) for power supply. Specific details regarding the battery (208) may be substantially identical to those regarding the battery (189) of FIG. 1.

[0068] Referring to FIG. 2A, components of an electronic device (101) are described. The electronic device (101) may include a processor (211), a memory (212), a communication module (213), a sensor module (214), an artificial intelligence module (215), and an output unit (216). For example, the processor (211), the memory (212), the communication module (213), the sensor module (214), the artificial intelligence module (215), and the output unit (216) may be electrically and / or operably coupled with each other by a communication bus. Hereinafter, the operative coupling of hardware components may mean that a direct or indirect connection is established between the hardware components, either wired or wireless, so that a second hardware component is controlled by a first hardware component among the hardware components. The artificial intelligence module (215) illustrated in FIG. 2A is illustrated as a hardware component, but the present disclosure is not limited thereto. For example, the artificial intelligence module (215) may correspond to a software component. Although the hardware components illustrated in FIG. 2A are illustrated based on different blocks, the present disclosure is not limited thereto. For example, some of the hardware components of the electronic device (101) illustrated in FIG. 2A (e.g., at least a portion of the processor (211), the memory (212), the communication module (213), the sensor module (214), the artificial intelligence module (215), and the output unit (216)) may be included in a single integrated circuit such as a system on chip (SoC) or a system in package (SIP). The type and number of hardware components included in the electronic device (101) are not limited to those illustrated in FIG. 2A. For example, the electronic device (101) may include only some of the hardware components illustrated in FIG. 2A.

[0069] In one embodiment, the processor (211) of the electronic device (101) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microcontroller (MCU), and / or a neural processing unit (NPU). The number of processors (211) may be one or more. For example, the processor (211) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core. The processor (211) of FIG. 2A may be substantially identical to the content of the processor (120) of FIG. 1.

[0070] In one embodiment, the processor (211) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to individually and / or collectively perform the various functions described below in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, as well as situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor is capable of executing program instructions to accomplish or perform various functions.

[0071] In one embodiment, the memory (212) of the electronic device (101) may include hardware components for storing data and / or instructions input to and / or output from the processor (211). For example, the memory (212) may include volatile memory, such as random-access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multimedia card (eMMC).

[0072] In one embodiment, one or more instructions (or commands) representing operations and / or actions performed by the processor (211) of the electronic device (101) may be stored within the memory (212) of the electronic device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, being installed within the electronic device (101) may mean that one or more instructions provided in the form of an application are stored within the memory (212), and that one or more applications are stored in a format executable by the processor (211) of the electronic device (101). The specific details of the memory (212) of FIG. 2A may be substantially identically applied to the details of the memory (130) of FIG. 1.

[0073] In one embodiment, the communication module (213) of the electronic device (101) may include a communication circuit to support transmission and / or reception of electrical signals between the electronic device (101) and an external electronic device (e.g., a wearable device (103)) different from the electronic device (101). The communication module (213) may include at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication module (213) may support transmission and / or reception of electrical signals based on various types of communication means, such as Ethernet, Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). Specific details of the communication module (213) of FIG. 2A may be substantially identically applied to the communication module (109) and / or the antenna module (197) of FIG. 1.

[0074] In one embodiment, the electronic device (101) may include a sensor module (214). For example, the sensor module (214) may include an inertial measurement unit (IMU) sensor, a global positioning system (GPS) sensor, a proximity sensor, a hall sensor, a skin sensor, a force sensor, a barometric pressure sensor, a temperature sensor, a humidity sensor, an illuminance sensor, and / or an IR sensor. The specific details of the sensor module (214) of FIG. 2A may be substantially identical to the details of the sensor module (176) of FIG. 1.

[0075] In one embodiment, the electronic device (101) may include an artificial intelligence module (215). For example, the artificial intelligence module (215) may be referred to as an on-device AI core, an on-device AI module, a machine learning model, a deep learning model, a natural language processing model, or equivalent technical terms. For example, the artificial intelligence module (215) may include a language model (e.g., a large language model (LLM)), automatic speech recognition (ASR) (or speech to text (STT)), a generative artificial intelligence model, or a combination thereof.

[0076] In one embodiment, the artificial intelligence module (215) of the electronic device (101) may be a unit (a set of function codes, separate devices, circuits, or instructions) that identifies keywords indicating a user based on information collected by the electronic device (101) and the wearable device (103). For example, the artificial intelligence module (215) may use a prompt generated based on information (e.g., personalized parameters) collected by the electronic device (101) and the wearable device (103) as input data. For example, the artificial intelligence module (215) may output information on feature vectors of keywords, probability information of keywords, or a combination thereof based on the input data.

[0077] In one embodiment, the artificial intelligence module (215) of the electronic device (101) may be a unit (or a set of function codes, separate devices, circuits, or instructions) that generates text corresponding to a voice signal other than a user call received (or acquired) from the wearable device (103). For example, the artificial intelligence module (215) may use the voice signal other than a user call received (or acquired) from the wearable device (103) as input data. For example, the artificial intelligence module (215) may output text corresponding to the voice signal other than a user call based on the input data.

[0078] In one embodiment, the artificial intelligence module (215) of the electronic device (101) may be a unit (or a set of function codes, separate devices, circuits, or instructions) that generates summary information based on valuable information for the user among the contents of text corresponding to a voice signal other than a user call. For example, the artificial intelligence module (215) may use user status information representing the user's surroundings as input data. For example, the artificial intelligence module (215) may output the summary information based on the input data.

[0079] In one embodiment, the electronic device (101) may include an output unit (216). For example, the output unit (216) may include a screen output unit (e.g., screen output unit (216-1) of FIG. 2B), an audio output unit (e.g., audio output unit (216-2) of FIG. 2B), or a combination thereof. For example, the screen output unit may include a display for displaying summary information. For example, the audio output unit may include a speaker for outputting voice information.

[0080] FIG. 2B illustrates components of a wearable device and an electronic device for providing notifications and summary information to a user. The electronic device (101) of FIG. 2B may correspond to the electronic device (101) of FIGS. 1 and 2A. For example, the electronic device (101) may include a battery (217). The battery (217) may include a module for supplying power to the electronic device (101). The specific details of the battery (217) may be substantially identical to those of the battery (189) of FIG. 1. The wearable device (103) of FIG. 2B may correspond to the wearable device (103) of FIG. 2A.

[0081] Referring to FIG. 2B, in one embodiment, the electronic device (101) may generate user identification information based on user information (218). For example, the user information (218) may include information for identifying users of devices (e.g., the electronic device (101) and the wearable device (103)). For example, the user information (218) may include information acquired by the electronic device (101) and information acquired by the wearable device (103).

[0082] In one embodiment, the information acquired by the electronic device (101) may include sensor data acquired by the sensor module (214) of the electronic device (101), information on an application being executed by the electronic device (101), information on the user's name, information on the user's location, information on the user's status (e.g., exercise status), information on the user's schedule, information on the user's car number, information on the user's performance seat, information on the user's weather, information on the user's clothing, information on the user's characteristic characteristics analyzed from photos stored in the memory (212) in association with a photo application of the electronic device (101), information on the user's previous response to a call, or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the electronic device (101) may acquire additional information for identifying the user by monitoring (or, in real time, analyzing) an application (e.g., a phone application, a message application) being executed by the electronic device (101).

[0083] In one embodiment, the information acquired by the wearable device (103) may include information for identifying a user among information acquired using the microphone of the input module (203), sensor data acquired by the sensor module (204), or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the information acquired by the wearable device (103) may further include other information that can be used to identify a user.

[0084] In one embodiment, the electronic device (101) may generate user identification information based on user information (218). For example, the electronic device (101) may generate user identification information based on user information (218) by using a language model (e.g., a large language model (LLM)) of the artificial intelligence module (215). For example, the user identification information may include information on feature vectors of keywords (e.g., user name, job title, clothing), probability information of keywords, or a combination thereof. For example, a keyword may mean a text referring to a user of the wearable device (103). For example, information on a feature vector of a keyword may include information for identifying the keyword from an audio signal. The feature vector may represent frequency characteristics (e.g., mel-frequency cepstral coefficients (MFCC)) of a voice signal corresponding to the keyword and / or frequency band-specific energy (e.g., filter bank energy) of a voice signal corresponding to the keyword. For example, the likelihood information for a keyword may indicate the likelihood that the keyword will be used to summon a user of a device. For example, the likelihood information for a keyword may include numerical information regarding the likelihood.

[0085] In one embodiment, the wearable device (103) may acquire an audio signal using the microphone of the input module (203). For example, the audio signal may refer to a signal for sound acquired from outside the wearable device (103) using the microphone of the input module (203).

[0086] In one embodiment, the DSP (201-1) of the wearable device (103) can identify keywords from an audio signal acquired using a microphone of the input module (203) based on user identification information. The keyword spotter (KWS) of the DSP (201-1) can be used to identify keywords. For example, the DSP (201-1) can identify keywords from the audio signal using KWS based on information about the feature vector of the keyword. For example, the DSP (201-1) can temporarily store the identified keywords and the likelihood information of the keywords in a buffer (205-1).

[0087] In one embodiment, the DSP (201-1) of the wearable device (103) may detect a trigger condition for transmitting information about keywords stored in a buffer (205-1) (e.g., keywords and likelihood information about the keywords) to the artificial intelligence module (207). For example, the trigger condition may include a trigger condition based on likelihood information, a trigger condition based on buffer capacity, or a combination thereof.

[0088] In one embodiment, the DSP (201-1) of the wearable device (103) may identify a matching score based on the likelihood information of the keyword. For example, the DSP (201-1) of the wearable device (103) may detect a trigger condition when the matching score for the keyword exceeds a threshold score. For example, the DSP (201-1) of the wearable device (103) may control the buffer (205-1) to transmit information about the keyword to the artificial intelligence module (207) based on detecting the trigger condition.

[0089] In one embodiment, the DSP (201-1) of the wearable device (103) may monitor the remaining capacity of the buffer (205-1). For example, the wearable device (103) may detect a trigger condition when the remaining capacity of the buffer (205-1) is below a threshold capacity. For example, the DSP (201-1) of the wearable device (103) may control the buffer (205-1) to transmit information about a keyword to the artificial intelligence module (207) based on detecting the trigger condition.

[0090] In one embodiment, the DSP (201-1) of the wearable device (103) may generate a prompt indicating the likelihood that a keyword will be used in a user call, a prompt associated with the accessibility (or orientation) of a speaker, a prompt associated with a repeat call, a prompt indicating whether the user is using the electronic device (101), or a combination thereof. For example, the prompt indicating the likelihood that a keyword will be used in a user call may be generated based on likelihood information about the keyword in user identification information for the keyword. For example, the prompt associated with the accessibility of the speaker may be generated based on acoustic information (e.g., amplitude, frequency, pitch) of an audio signal corresponding to the keyword. For example, the prompt associated with a repeat call may be generated based on voiceprint analysis.

[0091] In one embodiment, the input data of the artificial intelligence module (207) may include a prompt indicating the likelihood that a keyword will be used to call a user, a prompt associated with the speaker's accessibility, a prompt associated with repeated calls, a prompt indicating whether the user is using an electronic device (101), or a combination thereof. The artificial intelligence module (207) may output output data based on the input data. For example, the output data may indicate whether the keyword will be used to call a user of a wearable device (103).

[0092] In one embodiment, the artificial intelligence module (207) of the wearable device (103) may determine whether a user gesture in response to a call to the user is detected when a keyword is used to call the user of the wearable device (103). For example, the input data of the artificial intelligence module (207) may include a prompt associated with the release of contact between the wearable device (103) and a body part of the user, a prompt associated with the rotation of a body part of the user wearing the wearable device (103), a prompt associated with a user input for launching an application for listening to ambient sounds, a prompt indicating whether a noise-removing application is running, a prompt associated with a user input for terminating the execution of a noise-removing application, or a combination thereof. The artificial intelligence module (207) may output output data based on the input data. For example, the output data of the artificial intelligence module (207) may indicate whether a user gesture in response to a user call is detected.

[0093] In one embodiment, the control unit (201-2) of the wearable device (103) may execute an application for listening to ambient sounds if a user gesture in response to a user call is not detected. For example, the control unit (201-2) of the wearable device (103) may control the noise canceling filter (210) to not perform noise cancellation if a user gesture in response to a user call is not detected.

[0094] In one embodiment, the wearable device (103) may provide a call notification for the user using the audio output module (202) if a user gesture in response to the user call is not detected. The wearable device (103) may provide a notification of a user call for a user who is unaware of the call to the user.

[0095] In one embodiment, the buffer (205-2) of the wearable device (103) may temporarily store a voice signal other than a user call. For example, the wearable device (103) may periodically transmit the voice signal other than a user call stored in the buffer (205-2) to the electronic device (101). In another example, the wearable device (103) may transmit the voice signal other than a user call stored in the buffer (205-2) to the electronic device (101) in response to a detection of an event that calls the user.

[0096] In one embodiment, the electronic device (101) may generate text corresponding to a voice signal other than a user call. For example, the electronic device (101) may generate text corresponding to a voice signal other than a user call using an automatic speech recognition (ASR) model (or a speech-to-text (STT) model).

[0097] In one embodiment, the electronic device (101) can identify user status information. The user status information can indicate the user's surroundings (e.g., riding a bus). For example, the user status information can include the user's location information, sensor data acquired by the sensor module (214), user lifestyle pattern information, information about applications running on the electronic device (101), the user's schedule information, weather information, the user's call information, message information, sensor data acquired from the wearable device (103), or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the user status information can further include other information that can be collected by the electronic device (101) and the wearable device (103) to indicate the user's surroundings.

[0098] In one embodiment, the electronic device (101) may generate summary information based on text and user status information corresponding to a voice signal other than a user call. For example, the electronic device (101) may generate content information based on the user status information and text using a language model (e.g., a large language model (LLM)) of the artificial intelligence module (215). For example, the content information may include key content or valuable information to the user identified by the language model based on the context of the text. For example, the electronic device (101) may generate summary information based on the content information using a generative artificial intelligence model of the artificial intelligence module (215). For example, the electronic device (101) may display the summary information through the display of the screen output unit (216-1). For example, the electronic device (101) may transmit the summary information to the wearable device (103). For example, the wearable device (103) can generate a voice signal corresponding to summary information using TTS (text to speech). The wearable device (103) can output the voice signal through the speaker of the audio output module (202).

[0099] FIG. 3 is a block diagram illustrating the operation of a wearable device and an electronic device for providing notifications and summary information to a user. The electronic device (101) of FIG. 3 may correspond to the electronic device (101) of FIGS. 1, 2A, and 2B. The wearable device (103) of FIG. 3 may correspond to the wearable device (103) of FIGS. 2A and 2B.

[0100] At least some of the operations of FIG. 3 may be performed by the electronic device (101) or the wearable device (103) of FIG. 1, FIG. 2A, and FIG. 2B. In the following description, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed. For example, at least two operations may be performed in parallel.

[0101] Referring to FIG. 3, in operation 301, according to one embodiment, the wearable device (103) may obtain an audio signal using the microphone of the input module (203). For example, the audio signal may refer to a signal for a sound obtained from the outside of the wearable device (103) using the microphone of the input module (203).

[0102] In operation 302, according to one embodiment, the wearable device (103) may activate an application for listening to ambient sounds when an emergency situation is identified. For example, the wearable device (103) may use the artificial intelligence module (207) to identify whether an emergency situation is present from an audio signal. For example, input data of the artificial intelligence module (207) for identifying whether an emergency situation is present may include frequency, amplitude, tone, or a combination thereof of the audio signal. For example, output data of the artificial intelligence module (207) may indicate whether an emergency situation is present. When the wearable device (103) identifies an emergency situation from the audio signal, it may activate an application for listening to ambient sounds.

[0103] In operation 303, according to one embodiment, the wearable device (103) can identify a keyword from an audio signal obtained using a microphone of the input module (203) based on user identification information.

[0104] In one embodiment, the wearable device (103) may receive (or acquire) user identification information from the electronic device (101). For example, the user identification information may include information about a feature vector of a keyword and likelihood information about the keyword. For example, a keyword may mean text that refers to a user of devices (e.g., the wearable device (103) and the electronic device (101)). For example, the information about the feature vector of the keyword may include information for identifying the keyword from an audio signal. The feature vector may represent frequency characteristics (e.g., mel-frequency cepstral coefficients (MFCC)) of a voice signal corresponding to the keyword and / or frequency band-specific energy (e.g., filter bank energy) of a voice signal corresponding to the keyword. The likelihood information about the keyword may represent the likelihood that the keyword is used to call the user of the devices. In one example, the likelihood information about the keyword may include numerical information about the likelihood.

[0105] In one embodiment, the wearable device (103) can identify a keyword from an audio signal based on user identification information obtained from the electronic device (101). For example, the keyword spotter (KWS) of the DSP (201-1) can be used to identify the keyword. For example, the wearable device (103) can identify the keyword from the audio signal using the KWS based on information about the feature vector of the keyword. For example, the wearable device (103) can temporarily store the identified keyword and the likelihood information of the keyword in the buffer (205-1).

[0106] In one embodiment, the wearable device (103) may detect a trigger condition for transmitting information about a keyword stored in a buffer (205-1) (e.g., keyword and likelihood information about the keyword) to an artificial intelligence module (207). For example, the trigger condition may include a trigger condition based on likelihood information, a trigger condition based on buffer capacity, or a combination thereof.

[0107] In one embodiment, the wearable device (103) may identify a matching score based on the likelihood information of a keyword. For example, the wearable device (103) may detect a trigger condition when the matching score for the keyword exceeds a threshold score (e.g., 100 points). For example, the wearable device (103) may control the buffer (205-1) to transmit information about the keyword to the artificial intelligence module (207) based on detecting the trigger condition.

[0108] In one embodiment, the wearable device (103) may monitor the remaining capacity of the buffer (205-1). For example, the wearable device (103) may detect a trigger condition when the remaining capacity of the buffer (205-1) is below a threshold capacity. For example, the wearable device (103) may control the buffer (205-1) to transmit information about a keyword to the artificial intelligence module (207) based on detecting the trigger condition.

[0109] In operation 304, according to one embodiment, the wearable device (103) may perform user call verification and user recognition judgment using a language model (e.g., a large language model (LLM)) of the artificial intelligence module (207). For example, user call verification may mean identifying whether a keyword is used to call a user of the wearable device (103). For example, user recognition judgment may mean identifying whether a user gesture in response to the user call is detected when the keyword is identified as being used to call a user.

[0110] In one embodiment, the wearable device (103) may perform user call verification using a language model of the artificial intelligence module (207). For example, input data of the language model may include a prompt indicating the likelihood of a keyword being used to call a user, a prompt associated with the accessibility of the speaker, a prompt associated with repeated calls, a prompt indicating whether the user is using the electronic device (101), or a combination thereof. For example, a prompt indicating the likelihood of a keyword being used to call a user may be generated based on likelihood information about the keyword in user identification information for the keyword. For example, a prompt associated with the accessibility of the speaker may be generated based on acoustic information (e.g., amplitude, frequency, pitch) of an audio signal corresponding to the keyword. For example, a prompt associated with repeated calls may be generated based on voiceprint analysis. For example, output data of the language model may indicate whether a keyword is used to call a user of the wearable device (103).

[0111] In one embodiment, the wearable device (103) may store a voice signal other than a user call in the buffer (205-2). The voice signal other than a user call may correspond to an audio signal excluding a signal for a keyword identified by the wearable device (103) as being used for a user call among the audio signals. For example, the wearable device (103) may periodically transmit the voice signal other than a user call stored in the buffer (205-2) to the electronic device (101). In another example, the wearable device (103) may transmit the voice signal other than a user call stored in the buffer (205-2) to the electronic device (101) in response to detecting an event that calls a user.

[0112] In one embodiment, the wearable device (103) may perform user recognition judgment using a language model of the artificial intelligence module (207). For example, input data of the language model may include a prompt associated with the release of contact between the wearable device (103) and a user's body part, a prompt associated with the rotation of a user's body part wearing the wearable device (103), a prompt associated with a user input for executing an application for listening to ambient sounds, a prompt indicating whether a noise-removing application is running, a prompt associated with a user input for terminating the execution of a noise-removing application, or a combination thereof. For example, output data of the artificial intelligence module (207) may indicate whether a user gesture in response to a user call is detected.

[0113] In operation 305, according to one embodiment, the wearable device (103) may provide a notification for a user call if a user gesture in response to the user call is not detected. The wearable device (103) may provide a notification for a user who does not recognize the user call. For example, providing a notification for a user call may include launching an application for listening to ambient sounds, terminating an application for noise cancellation, providing a voice notification through a speaker of the audio output module (202), or a combination thereof.

[0114] In operation 306, according to one embodiment, the electronic device (101) may generate user identification information based on the user information. For example, operation 306 may be performed in parallel with operation 303. For example, the user information may include information for identifying the user of the devices (e.g., the electronic device (101) and the wearable device (103)). For example, the user information may include information acquired by the electronic device (101) and information acquired by the wearable device (103).

[0115] In one embodiment, the information acquired by the electronic device (101) may include sensor data acquired by the sensor module (214) of the electronic device (101), information on an application being executed by the electronic device (101), information on the user's name, information on the user's location, information on the user's status (e.g., exercise status), information on the user's schedule, information on the user's car number, information on the user's performance seat, information on the user's weather, information on the user's clothing, information on the user's characteristic characteristics analyzed from photos stored in the memory (212) in association with a photo application of the electronic device (101), information on previous responses to user calls, or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the electronic device (101) may acquire additional information for identifying the user by monitoring (or, in real time, analyzing) an application (e.g., a phone application, a message application) being executed by the electronic device (101).

[0116] In one embodiment, the information acquired by the wearable device (103) may include information for identifying the user among information acquired using the microphone of the input module (203), sensor data acquired by the sensor module (204), or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the user information acquired by the wearable device (103) may further include other information that can be used to identify the user.

[0117] In one embodiment, the electronic device (101) may generate user identification information based on user information. For example, the electronic device (101) may generate user identification information based on the user information using a language model (e.g., LLM) of the artificial intelligence module (215). For example, the user identification information may include information on feature vectors of keywords (e.g., user name, job title, clothing), likelihood information of the keywords, or a combination thereof. For example, a keyword may refer to text that refers to a user of the wearable device (103). For example, information on the feature vector of a keyword may include information for identifying the keyword from an audio signal. The feature vector may represent frequency characteristics of a voice signal corresponding to the keyword (e.g., mel-frequency cepstral coefficient (MFCC)) and / or frequency band-specific energy of a voice signal corresponding to the keyword (e.g., filter bank energy). For example, likelihood information of a keyword may represent the likelihood that the keyword is used to call the user of the devices. For example, the likelihood information of a keyword may include numerical information about the likelihood.

[0118] In one embodiment, the electronic device (101) can transmit user identification information to the wearable device (103).

[0119] In operation 307, according to one embodiment, the electronic device (101) may generate text corresponding to a voice signal other than a user call using an automatic speech recognition (ASR) model (or a speech-to-text (STT) model). For example, the electronic device (101) may receive (or acquire) the voice signal other than a user call from the buffer (205-2) of the wearable device (103). For example, the electronic device (101) may generate text corresponding to the voice signal other than a user call using the ASR model of the artificial intelligence module (215). Although FIG. 3 illustrates an ASR model, this is merely an example, and the present disclosure is not limited thereto. For example, the electronic device (101) may generate text corresponding to the voice signal other than a user call using another artificial intelligence model that converts a voice signal into text.

[0120] In operation 308, according to one embodiment, the electronic device (101) may obtain user status information. The user status information may include information indicating the user's surrounding environment (e.g., boarding a bus). For example, the user status information may include the user's location information, sensor data obtained by the sensor module (214), user lifestyle pattern information, information about an application running on the electronic device (101), the user's schedule information, weather information, the user's call information, message information, sensor data obtained from the wearable device (103), information about the control unit settings of the wearable device (103), or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the user status information may further include other information that may be collected by the electronic device (101) and the wearable device (103) to indicate the user's surrounding environment.

[0121] In operation 309, according to one embodiment, the electronic device (101) may generate content information from a voice signal other than a user call using a language model of the artificial intelligence module (215). For example, the content information may mean valuable information for a user identified by the language model from text corresponding to the voice signal other than a user call, or important content in the context of the text. For example, input data of the language model for generating the content information may include text corresponding to the voice signal other than a user call, user status information, or a combination thereof. For example, output data of the language model may include content information.

[0122] In operation 310, according to one embodiment, the electronic device (101) can generate summary information based on content information using a generative artificial intelligence model.

[0123] In operation 311, according to one embodiment, the electronic device (101) may transmit summary information to the wearable device (103). In one example, the electronic device (101) may transmit the summary information to the wearable device (103) when the user does not use the electronic device (101) but only uses the wearable device (103). In one example, when a game application is running on the electronic device (101), the electronic device (101) may refrain from displaying the summary information through the display of the screen output unit (216-1) and transmit the summary information to the wearable device (103). However, the present disclosure is not limited thereto. For example, the wearable device (103) may generate a voice signal corresponding to the received summary information using TTS (text to speech). For example, the wearable device (103) can output the generated voice signal through the speaker of the audio output module (202).

[0124] In operation 312, according to one embodiment, the electronic device (101) may display summary information through the display of the screen output unit (216-1). In one example, when an application for real-time streaming is executed on the electronic device (101), the electronic device (101) may display summary information in the form of a message through the display of the screen output unit (216-1).

[0125] FIG. 4a illustrates a flowchart of a method of a wearable device for providing user identification information.

[0126] At least some of the operations of FIG. 4A may be performed by the electronic device (101). For example, at least some of the operations may be controlled by the processor (211) of the electronic device (101). In the following description, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed. For example, at least two operations may be performed in parallel.

[0127] Referring to FIG. 4A, in operation 401, according to one embodiment, the electronic device (101) may obtain user information. For example, the user information may include information for identifying users of devices (e.g., the electronic device (101) and the wearable device (103)). For example, the user information may include information obtained by the electronic device (101) and information obtained by the wearable device (103).

[0128] In one embodiment, the information acquired by the electronic device (101) may include sensor data acquired by the sensor module (214), information on an application being executed by the electronic device (101), information on the user's name, information on the user's location, information on the user's status (e.g., exercise status), information on the user's schedule, information on the user's car number, information on the user's performance seat, information on the user's weather, information on the user's clothing, information on the user's characteristic characteristics analyzed from photos stored in the memory (212) in association with a photo application of the electronic device (101), information on the user's previous response to a call, or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the electronic device (101) may acquire additional information for identifying the user by monitoring an application (e.g., a phone application, a message application) being executed by the electronic device (101).

[0129] In one embodiment, the information acquired by the wearable device (103) may include information for identifying the user acquired through the microphone of the input module (203), sensor data acquired by the sensor module (204), or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the user information acquired by the wearable device (103) may further include other information that can be used to identify the user.

[0130] In operation 402, according to one embodiment, the electronic device (101) may generate user identification information based on user information. For example, the electronic device (101) may generate user identification information based on user information using a language model of an artificial intelligence (AI) module (215). For example, the language model of the AI ​​module (215) may include a large language model (LLM).

[0131] In one embodiment, the user identification information may include information about feature vectors of keywords (e.g., user name, job title, clothing), likelihood information about the keywords, or a combination thereof. For example, the information about the feature vector of a keyword may include information for identifying the keyword from an audio signal. The feature vector may represent frequency characteristics of a voice signal corresponding to the keyword (e.g., mel-frequency cepstral coefficient (MFCC)) and / or frequency band-specific energy of a voice signal corresponding to the keyword (e.g., filter bank energy). For example, the likelihood information of a keyword may represent the likelihood that the keyword is used to call a user of a device. For example, the likelihood information of a keyword may include numerical information about the likelihood.

[0132] For example, a keyword may mean a text for identifying a user of a device (e.g., an electronic device (101) and a wearable device (103)). However, the present disclosure is not limited thereto. A keyword may further include a call word (e.g., "excuse me" or "wait a moment") for calling any user other than the user of the device.

[0133] In operation 403, according to one embodiment, the electronic device (101) may transmit the generated user identification information to the wearable device (103).

[0134] FIG. 4b illustrates a flowchart of a method of a wearable device for providing notifications.

[0135] At least some of the operations of FIG. 4B may be performed by the wearable device (103). For example, at least some of the operations may be controlled by the processor (201) of the wearable device (103). In the following description, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations (e.g., operations 411 and 412) may be changed. For example, at least two operations (e.g., operations 411 and 412) may be performed in parallel.

[0136] Referring to FIG. 4B, in operation 411, according to one embodiment, the wearable device (103) may receive user identification information from the electronic device (101). For example, the user identification information may include information about a feature vector of keywords, probability information of keywords, or a combination thereof.

[0137] In one embodiment, a keyword may refer to text identified by the electronic device (101) as referring to a user of the devices (e.g., the electronic device (101) and the wearable device (103)). However, the present disclosure is not limited thereto. For example, a keyword may further include a call word (e.g., "excuse me" or "wait a moment") for calling any user other than the user of the devices.

[0138] In one embodiment, information about a feature vector of a keyword may include information for identifying the keyword from an audio signal. The feature vector may represent frequency characteristics of a speech signal corresponding to the keyword (e.g., mel-frequency cepstral coefficient (MFCC)) and / or frequency band-specific energy of a speech signal corresponding to the keyword (e.g., filter bank energy). For example, the likelihood information of the keyword may represent the likelihood that the keyword is used to call a user of a device. For example, the likelihood information of the keyword may include numerical information about the likelihood.

[0139] In operation 412, according to one embodiment, the wearable device (103) can identify at least one keyword from an audio signal based on user identification information. In one embodiment, the wearable device (103) can obtain an audio signal using a microphone of the input module (203). For example, the audio signal may refer to a signal of sound obtained from outside the wearable device (103) using the microphone of the input module (203). For example, information about a feature vector of keywords can be used to identify at least one keyword from the audio signal. For example, the wearable device (103) can identify at least one keyword from the audio signal using a keyword spotter (KWS) based on information about the feature vector of keywords.

[0140] In operation 413, according to one embodiment, the wearable device (103) may determine whether at least one keyword is used to call a user of the wearable device (103). For example, the wearable device (103) may determine whether at least one keyword is used to call a user of the wearable device (103) using a language model of an artificial intelligence (AI) module (207). For example, the language model of the AI ​​model may include a large language model (LLM).

[0141] In one embodiment, the wearable device (103) may detect a trigger condition for performing call recognition. For example, in response to detecting the trigger condition for performing call recognition, the wearable device (103) may determine whether at least one keyword is used to call the user. For example, the trigger condition for performing call recognition may include a trigger condition based on probability information, a trigger condition based on buffer capacity, or a combination thereof.

[0142] In one embodiment, the wearable device (103) may identify a matching score based on the likelihood information of keywords. For example, the wearable device (103) may detect a trigger condition when the matching score for at least one keyword exceeds a threshold score. For example, in response to detecting the trigger condition, the wearable device (103) may determine whether at least one keyword is used to call a user.

[0143] In one embodiment, the wearable device (103) may monitor the remaining capacity of a buffer (e.g., buffer (205-1) of FIG. 2B) in which at least one keyword is temporarily stored. For example, the wearable device (103) may detect a trigger condition when the remaining capacity of the buffer is below a threshold capacity. For example, in response to detecting the trigger condition, the wearable device (103) may determine whether at least one keyword is used to call a user.

[0144] In one embodiment, the wearable device (103) may generate a prompt indicating the likelihood of at least one keyword being used to call a user. For example, the wearable device (103) may identify likelihood information for at least one keyword identified from an audio signal among likelihood information of keywords. For example, the wearable device (103) may generate a prompt associated with the likelihood of at least one keyword being used to call a user based on the identification. In one example, if the at least one keyword includes 'glasses,' the prompt may indicate 20 percent (%). The prompt may be structured as, 'There is a 20 percent chance that the user will be called.' In one example, if the at least one keyword includes 'yellow clothes,' the prompt may indicate 30 percent. The prompt may be structured as, 'There is a 30 percent chance that the user will be called.' In one example, if at least one keyword includes "glasses" and "yellow clothes," the prompt may indicate a 50 percent chance. The prompt may be structured as, "There is a 50 percent chance that the user will be called." In one example, if at least one keyword includes the user's name, "Kim Sam-seong," the prompt may indicate a 100 percent chance. The prompt may be structured as, "There is a 100 percent chance that the user will be called." However, these are merely examples, and the present disclosure is not limited thereto.

[0145] In one embodiment, the wearable device (103) may generate a prompt associated with accessibility (or orientation). For example, the wearable device (103) may generate a prompt associated with accessibility based on acoustic information (e.g., amplitude, frequency, pitch) of an audio signal corresponding to at least one keyword.

[0146] In one embodiment, the wearable device (103) may identify that a speaker of an audio signal is approaching a user of the wearable device (103) if the amplitude (e.g., decibel) of a signal corresponding to at least one keyword of the audio signal increases over time. For example, the wearable device (103) may identify that a speaker of the audio signal is approaching a user of the wearable device (103) based on the Doppler effect if a frequency of a signal corresponding to at least one keyword of the audio signal increases over time. For example, a prompt generated based on the identification may indicate that a speaker of the audio signal is approaching a user of the wearable device (103). In one example, the prompt may be configured as, 'The speaker is approaching the user.' However, this is merely an example, and the present disclosure is not limited thereto.

[0147] In one embodiment, the wearable device (103) may identify that a speaker of the audio signal is moving away from the user of the wearable device (103) if the amplitude of a signal corresponding to at least one keyword in the audio signal decreases over time. For example, the wearable device (103) may identify that a speaker of the audio signal is moving away from the user of the wearable device (103) based on the Doppler effect if the frequency of a signal corresponding to at least one keyword in the audio signal decreases over time. For example, a prompt generated based on the identification may indicate that the speaker of the audio signal is moving away from the user of the wearable device (103). In one example, the prompt may be configured as, 'The speaker is moving away from the user.' However, this is merely an example, and the present disclosure is not limited thereto.

[0148] In one embodiment, the wearable device (103) may generate a prompt associated with a repeated call. For example, the wearable device (103) may perform a voiceprint analysis based on acoustic information of an audio signal. For example, the wearable device (103) may identify, based on the voiceprint analysis, whether a signal corresponding to at least one keyword is obtained from a single user. In one example, upon identifying that the signal corresponding to at least one keyword is obtained from a single user, the prompt may be structured as, "The call is from the same user." In one example, upon identifying that the signal corresponding to at least one keyword is obtained from different users, the prompt may be structured as, "The call is from different users." However, this is merely an example, and the present disclosure is not limited thereto.

[0149] In one embodiment, the input data of an artificial intelligence model for determining whether at least one keyword is used for a user call may include a prompt indicating the likelihood that at least one keyword is used for a user call, a prompt associated with accessibility, a prompt associated with repeated calls, or a combination thereof. However, the present disclosure is not limited thereto. For example, the input data of the artificial intelligence model may further include a prompt associated with the role (e.g., subject, verb, object) of at least one keyword within a sentence, and a prompt indicating whether the user is using the electronic device (101). In one example, if the identified keyword functions as an object within a sentence, the probability that at least one keyword is identified as being used for a user call may be reduced.

[0150] In one embodiment, the wearable device (103) may generate output data from the input data using the language model of the artificial intelligence module (207). For example, the output data may indicate whether at least one keyword is used to call a user of the wearable device (103). In one example, the output data may indicate that at least one keyword is used to call a user. In another example, the output data may indicate that at least one keyword is not used to call a user. The wearable device (103) may identify whether at least one keyword is used to call a user of the wearable device (103) based on the output data of the language model of the artificial intelligence module (207).

[0151] In operation 414, according to one embodiment, the wearable device (103) may determine whether a user gesture in response to a call to the user is detected. For example, the wearable device (103) may determine whether a user gesture in response to a call to the user is detected based on a determination that at least one keyword is used to call the user. For example, the wearable device (103) may use a language model of the artificial intelligence module (207) to determine whether a user gesture in response to a call to the user is detected. For example, the language model of the artificial intelligence model may include an LLM.

[0152] In one embodiment, the wearable device (103) may generate a prompt associated with the release of contact between the wearable device (103) and a body part of the user. For example, the wearable device (103) may obtain sensor data from the sensor module (204). For example, the wearable device (103) may identify whether contact between the wearable device (103) and the body part of the user is released based on the sensor data obtained from the sensor module (204). For example, the wearable device (103) may generate a prompt associated with the release of contact between the wearable device (103) and the body part of the user based on the identification. In one example, the prompt may indicate the release of contact between the wearable device (103) and the body part of the user. In another example, the prompt may indicate contact between the wearable device (103) and the body part of the user.

[0153] In one embodiment, the wearable device (103) may generate a prompt associated with the rotation of a body part (e.g., head) of a user wearing the wearable device (103). For example, the wearable device (103) may identify the rotation angle of the body part of the user wearing the wearable device (103) based on sensor data acquired from the sensor module (204). For example, the wearable device (103) may generate a prompt associated with the rotation of the body part of the user based on the identification. For example, the prompt may indicate the rotation angle of the body part of the user.

[0154] In one embodiment, the wearable device (103) may generate a prompt associated with user input for launching an application for listening to ambient sound. For example, the wearable device (103) may identify whether user input for launching the application has been obtained after receiving an audio signal. For example, the wearable device (103) may generate a prompt associated with launching the application based on the identification. In one example, the prompt may indicate acquisition of user input for launching the application. In another example, the prompt may indicate that user input for launching the application has not been obtained.

[0155] In one embodiment, the wearable device (103) may generate a prompt associated with whether a noise cancellation application is running. For example, the wearable device (103) may identify whether the application is running while an audio signal is being received. For example, the wearable device (103) may generate a prompt indicating whether the application is running based on the identification. In one example, the prompt may indicate that the noise cancellation application is running while the audio signal is being received. In another example, the prompt may indicate that the noise cancellation application is not running while the audio signal is being received.

[0156] In one embodiment, the wearable device (103) may identify whether user input for terminating execution of an application for noise removal has been obtained after receiving an audio signal. For example, the wearable device (103) may generate a prompt associated with the user input for terminating execution of the application based on the identification. In one example, the prompt may indicate acquisition of user input for terminating execution of the application. In another example, the prompt may indicate that user input for terminating execution of the application has not been obtained.

[0157] In one embodiment, input data of a language model of an artificial intelligence module (207) for detecting a user gesture in response to a user call may include a prompt associated with a release of contact between a wearable device (103) and a user's body part, a prompt associated with a rotation of a user's body part wearing the wearable device (103), a prompt associated with a user input for executing an application for listening to ambient sounds, a prompt indicating whether an application for noise removal is running, a prompt associated with a user input for terminating the execution of an application for noise removal, or a combination thereof.

[0158] In one embodiment, the wearable device (103) may generate output data from the input data via the language model of the artificial intelligence module (207). For example, the output data may indicate whether a user gesture in response to a user call is detected. In one example, the output data may indicate the detection of a user gesture. In another example, the output data may indicate that a user gesture was not detected.

[0159] In one embodiment, while the noise removal application is running, the weights of the AI ​​model for detecting user gestures may be changed. For example, if the application's execution interferes with the user's perception of external sounds, the weights of the AI ​​model may be changed to reduce the likelihood of detecting user gestures.

[0160] In operation 415, according to one embodiment, the wearable device (103) may refrain from providing a notification to the user. For example, the wearable device (103) may refrain from providing a notification to the user based on a determination that a user gesture in response to a user call has been detected. For example, by refraining from providing a notification to the user, unnecessary notifications may be prevented from being provided to the user who has recognized the user call.

[0161] In operation 416, according to one embodiment, the wearable device (103) may provide a notification for the user. For example, the wearable device (103) may provide a notification for the user of the wearable device (103) based on a determination that a user gesture in response to a call to the user is not detected. For example, the wearable device (103) may provide a notification for a user call for a user who is unaware of the call to the user.

[0162] For example, the wearable device (103) can output a voice signal to call the user of the wearable device (103) using the speaker of the audio output module (202). For example, the wearable device (103) can control the audio output module (202) to reduce the volume of the sound output by the speaker of the audio output module (202). For example, the wearable device (103) can control the haptic module to provide a mechanical stimulus (e.g., vibration or movement) that the user can perceive through tactile or kinesthetic senses. For example, the wearable device (103) can execute an application for listening to ambient sounds. For example, the wearable device (103) can terminate the execution of an application for noise removal. For example, the wearable device (103) can transmit a message to the electronic device (101) to provide a notification for the user. The electronic device (101) may, in response to receiving a message to provide a notification for a user, display information indicating a call to the user using a display.

[0163] Figure 5 illustrates an example of providing a notification to a user.

[0164] In FIG. 5, an example is described in which a wearable device (103) provides a notification for a user (501) when a user (502) provides a voice signal (510) (e.g., Manager, Mr. Cheolsu, come to conference room A117) to call a user (501).

[0165] For example, the wearable device (103) may obtain (or receive) user identification information from the electronic device (101). For example, the user identification information may include a feature vector for a keyword (511), a feature vector for a keyword (512), likelihood information for a keyword (511), likelihood information for a keyword (512), or a combination thereof.

[0166] For example, the wearable device (103) can identify keywords from a voice signal (510). For example, the wearable device (103) can obtain (or receive) a voice signal (510) of a user (502) using a microphone of the input module (203). For example, the wearable device (103) can identify a keyword (511) from the voice signal (510) based on a feature vector of the keyword (511). For example, the wearable device (103) can identify a keyword (512) from the voice signal (510) based on a feature vector of the keyword (512). For the identification, a keyword spotter (KWS) can be used.

[0167] For example, the wearable device (103) can detect a trigger condition for performing call recognition. For example, the trigger condition may include a trigger condition based on probability information, a trigger condition based on buffer capacity, or a combination thereof.

[0168] In one example, at a first time point (514), the matching score (e.g., 30) of the keyword (511) may be less than a threshold score (e.g., 100). At a second time point (515), the matching scores (e.g., 130) of the keyword (511) and the keyword (512) may exceed the threshold score (e.g., 100). The wearable device (103) may detect a trigger condition at the second time point (515) when the matching scores exceed the threshold score. The wearable device (103) may perform call recognition by transmitting information about the keyword (511) and information about the keyword (512) stored in the buffer (205-1) to the artificial intelligence module (207).

[0169] In one example, at a first time point (514), the remaining capacity of the buffer (205-1) in which the keyword (511) is stored may exceed the threshold capacity. At a second time point (515), the remaining capacity of the buffer (205-1) in which the keyword (511) and the keyword (512) are stored may be less than the threshold capacity. The wearable device (103) may detect a trigger condition at the second time point (515) when the remaining capacity of the buffer (205-1) becomes less than the threshold capacity. The wearable device (103) may perform call recognition by transmitting information about the keyword (511) and information about the keyword (512) stored in the buffer (205-1) to the artificial intelligence module (207).

[0170] For example, the wearable device (103) can determine whether a combination of keywords (511) and keywords (512) are used to call the user (501). For example, a language model (e.g., a large language model (LLM)) of the artificial intelligence module (207) can be used for the determination. For example, input data of the language model can include a prompt indicating a possibility that the combination of keywords (511) and keywords (512) is used to call the user (501), a prompt associated with the accessibility of the user (502), a prompt indicating whether the keywords (511) and keywords (512) were generated by the same user (502), or a combination thereof. For example, the language model can output output data indicating that the keywords (511) and keywords (512) are used to call the user (501) based on the input data.

[0171] For example, the wearable device (103) may determine whether a user gesture in response to a call to the user (501) is detected. For example, the wearable device (103) may determine whether a user gesture is detected in a time interval between a second time point (515) and a third time point (516). The time interval between the second time point (515) and the third time point (516) may correspond to a time interval designated for detecting a user gesture. For example, a language model (e.g., LLM) of the artificial intelligence module (207) may be utilized to determine whether a user gesture is detected. For example, the input data of the language model may include a prompt associated with the release of contact between the wearable device (103) and a body part (e.g., an ear) of the user (501), a prompt associated with the rotation angle of the body part (e.g., a head) of the user (501), a prompt associated with a user input for launching an application for listening to ambient sound, a prompt indicating whether a noise cancellation application is running, a prompt associated with a user input for terminating the execution of a noise cancellation application, or a combination thereof. For example, the language model may output output data indicating whether a user gesture is detected based on the input data. In the example of FIG. 5, the wearable device (103) may identify that a user gesture is not detected.

[0172] For example, the wearable device (103) may provide a notification for the user (501) at a third time point (516). For example, the wearable device (103) may provide a notification for the user (501) based on a determination that no user gesture in response to a user call is detected during a time interval between the second time point (515) and the third time point (516).

[0173] For example, the wearable device (103) may output a voice signal for calling the user of the wearable device (103) using the speaker of the audio output module (202) from the third time point (516). For example, the wearable device (103) may control the audio output module (202) from the third time point (516) to reduce the volume of the sound output by the speaker of the audio output module (202). For example, the wearable device (103) may control the haptic module from the third time point (516) to provide a mechanical stimulus (e.g., vibration or movement) that the user can perceive through tactile or kinesthetic senses. For example, the wearable device (103) may execute an application for listening to ambient sounds from the third time point (516). For example, the wearable device (103) may terminate the execution of an application for noise removal at a third time point (516). For example, the wearable device (103) may transmit a message for providing a notification for the user to the electronic device (101) at a third time point (516). In response to receiving the message for providing a notification for the user, the electronic device (101) may display information indicating a call to the user through the display of the screen output unit (216-1).

[0174] As described above, the wearable device (103) can notify the user (501) of a call made by the user (502) that the user (501) is unaware of. However, a portion (513) of the voice signal (510) may not be heard by the user (501). Below, a method for providing a portion (513) of the voice signal (510) to the user (501) is described.

[0175] Figure 6 is a flowchart of a method of operation of an electronic device for providing summary information.

[0176] At least some of the operations of FIG. 6 may be performed by the electronic device (101). For example, at least some of the operations may be controlled by the processor (211) of the electronic device (101). In the following description, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed. For example, at least two operations may be performed in parallel.

[0177] In operation 601, according to one embodiment, the electronic device (101) may generate a text based on a voice signal obtained from the wearable device (103). For example, the voice signal may correspond to an audio signal excluding a signal for at least one keyword identified as being used for a user call by the wearable device (103) among the audio signals. For example, the voice signal may be referred to as a voice signal other than a user call. In one example, the voice signal obtained from the wearable device (103) may correspond to a portion (513) of the voice signal (510) of FIG. 5. For example, the electronic device (101) may periodically receive a voice signal from the wearable device (103). In another example, the electronic device (101) may receive a voice signal from the wearable device (103) when an event for a user call is detected by the wearable device (103). In one embodiment, the electronic device (101) can generate text based on a speech signal. For example, the electronic device (101) can generate text corresponding to the speech signal using an automatic speech recognition (ASR) model (or a speech-to-text (SST) model). In another example, the electronic device (101) can generate text corresponding to the speech signal using an ASR model and a language model (e.g., a large language model (LLM)).

[0178] In operation 602, according to one embodiment, the electronic device (101) may identify user status information. For example, the user status information may indicate the user's surrounding environment (e.g., boarding a bus, at a supermarket, or attending a lecture). For example, the user status information may include the user's location information, sensor data acquired by the sensor module (214), user lifestyle pattern information, information about applications running on the electronic device (101), the user's schedule information, weather information, the user's call information, message information, sensor data of the wearable device (103), or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the user status information may further include other information that may be collected by the electronic device (101) and the wearable device (103) to indicate the user's surrounding environment.

[0179] In operation 603, according to one embodiment, the electronic device (101) may generate summary information based on the text of the voice signal and user status information. The electronic device (101) may generate summary information about the text based on the text of the voice signal and user status information.

[0180] For example, a language model (e.g., LLM), a generative AI model, or a combination thereof of the artificial intelligence module (215) may be used to generate summary information. For example, the electronic device (101) may generate content information based on user status information and text using the language model. For example, the content information may represent key content or valuable information to the user identified by the language model based on the context of the text. For example, the electronic device (101) may generate summary information based on content information, user information, configuration information, or a combination thereof using the generative AI model. For example, the configuration information may include configuration information regarding the word level for summarizing, configuration information related to sentence structure, or a combination thereof. In a non-limiting example, if a portion of the text is omitted or implied, the electronic device (101) may generate text corresponding to the omitted or implied content using the generative AI model. The summary information may include text corresponding to the omitted or implied content. In one example, the electronic device (101) may generate information corresponding to a portion (513) of a voice signal (510) (e.g., there is a request to attend conference room A117).

[0181] In operation 604, according to one embodiment, the electronic device (101) may provide summary information to the user. For example, the electronic device (101) may display the summary information using a display. For example, the electronic device (101) may transmit the summary information to the wearable device (103). For example, the wearable device (103) may output a voice signal corresponding to the summary information using the speaker (209).

[0182] Figure 7 illustrates signaling between a wearable device and an electronic device to provide summary information.

[0183] At least some of the operations of FIG. 7 may be performed by the electronic device (101) of FIG. 1. For example, at least some of the operations may be controlled by the processor (120) of the electronic device (101). In the following description, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed. For example, at least two operations may be performed in parallel.

[0184] In operation 701, according to one embodiment, the electronic device (101) may receive a voice signal from the wearable device (103). For example, the voice signal may correspond to an audio signal excluding a signal for at least one keyword identified as being used for a user call by the wearable device (103) among the audio signals. In one example, the voice signal may correspond to a portion (513) of the voice signal (510) of FIG. 5 . However, the present disclosure is not limited thereto. For example, the electronic device (101) may receive an audio signal corresponding to a buffer capacity among the audio signals from the wearable device (103).

[0185] For example, the electronic device (101) may periodically receive a voice signal from the wearable device (103). In another example, the electronic device (101) may receive a voice signal from the wearable device (103) when an event for a user call is detected by the wearable device (103).

[0186] In operation 702, according to one embodiment, the electronic device (101) may generate text corresponding to the speech signal. For example, the electronic device (101) may generate text corresponding to the speech signal using an automatic speech recognition (ASR) model. In another example, the electronic device (101) may generate text corresponding to the speech signal using an ASR model and a language model (e.g., a large language model (LLM)).

[0187] In operation 703, according to one embodiment, the electronic device (101) may receive sensor data from the wearable device (103).

[0188] In operation 704, according to one embodiment, the electronic device (101) may identify user status information. For example, the user status information may indicate the user's surroundings (e.g., boarding a bus, at a supermarket, or attending a lecture). For example, the user status information may include the user's location information, sensor data acquired by the sensor module (214), user lifestyle pattern information, information about applications running on the electronic device (101), the user's schedule information, weather information, the user's call information, message information, sensor data of the wearable device (103), or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the user status information may further include other information that may be collected by the electronic device (101) and the wearable device (103) to indicate the user's surroundings.

[0189] In operation 705, according to one embodiment, the electronic device (101) may generate summary information. For example, the electronic device (101) may generate summary information based on text and user status information.

[0190] For example, a language model (e.g., LLM), a generative artificial intelligence model, or a combination thereof may be used to generate summary information. For example, the electronic device (101) may generate content information based on user state information and text using the language model. For example, the content information may represent valuable information for the user identified by the language model from text corresponding to the speech signal, or key content identified by the language model based on the context of the text. For example, the electronic device (101) may generate summary information based on content information, user information, configuration information, or a combination thereof using the generative artificial intelligence model. For example, the configuration information may include configuration information regarding the word level for summarization, configuration information related to sentence structure, or a combination thereof. In a non-limiting example, if a portion of the text is omitted or implied, the electronic device (101) may generate text corresponding to the omitted or implied content using the generative artificial intelligence model. Summary information may include text corresponding to omitted or implied content. In one example, the electronic device (101) may generate information corresponding to a portion (513) of a voice signal (510), such as "There is a request for attendance for conference room A117."

[0191] In operation 706, according to one embodiment, the electronic device (101) may transmit summary information to the wearable device (103). For example, the wearable device (103) may generate a voice signal corresponding to the summary information using TTS (text to speech). The wearable device (103) may output the voice signal corresponding to the summary information through the speaker of the audio output module (202).

[0192] In operation 707, according to one embodiment, the electronic device (101) may display summary information through a display.

[0193] Figures 8a to 8c illustrate examples of providing summary information.

[0194] In FIG. 8A, an example of an electronic device (101) generating summary information for a user (801) is described. For example, in FIG. 8A, an example of generating summary information for a user (801) who cannot hear a voice signal (810) of an external sound by wearing a wearable device (103) is described.

[0195] For example, the electronic device (101) can receive a voice signal (810) from the wearable device (103). The electronic device (101) can generate text corresponding to the voice signal (810) (e.g., the next stop is Gangnam Station, Gangnam Station). For example, the electronic device (101) can generate text corresponding to the voice signal (810) using an automatic speech recognition (ASR) model (or a speech to text (STT) model). In another example, the electronic device (101) can generate text corresponding to the voice signal (810) using an ASR model and a language model (e.g., a large language model (LLM)).

[0196] For example, the electronic device (101) can identify user status information. The user status information can indicate the user's surroundings (e.g., boarding a bus). For example, the user status information can include the user's location information, sensor data acquired by the sensor module (214), user lifestyle pattern information, information about applications running on the electronic device (101), the user's schedule information, weather information, the user's call information, message information, sensor data of the wearable device (103), or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the user status information can further include other information that can be collected by the electronic device (101) and the wearable device (103) to indicate the user's surroundings. In one example, the user status information can indicate the user's status (e.g., boarding a bus) based on analysis of a message application. In one example, user status information may indicate the user's (801) destination (e.g., Gangnam Station) based on analysis of the messaging application.

[0197] In one embodiment, the electronic device (101) may generate summary information based on text and user status information corresponding to the speech signal (810). For example, a language model (e.g., LLM), a generative artificial intelligence model, or a combination thereof may be used to generate the summary information. For example, the electronic device (101) may generate content information based on the user status information and text corresponding to the speech signal (810) using the language model. For example, the content information may represent key content identified by the language model based on the context of the text or valuable information to the user (801). For example, the electronic device (101) may generate summary information based on content information, user information, setting information, or a combination thereof using the generative artificial intelligence model. In one example, the summary information may include information (821) associated with the content of the voice signal (810) and information (822) for guiding based on the content of the voice signal (810).

[0198] In Fig. 8b, an example of displaying summary information using a display is described. Referring to Fig. 8b, the electronic device (101) can display objects (821), objects (822), objects (823), objects (824), and objects (825) using the display of the screen output unit (216-1). For example, the object (821) can represent information associated with the content of the voice signal (810). For example, the object (822) can represent information associated with the content of the voice signal (810).

[0199] Based on the associated information and user information, content generated by the generative artificial intelligence model can be displayed. For example, object (823) may provide guidance for terminating a summary. For example, object (824) may be an object for obtaining user input for terminating a summary. For example, object (825) may be an object for obtaining user input for not terminating a summary. However, the screen structure illustrated in FIG. 8B is merely an example, and the present disclosure is not limited thereto. For example, the electronic device (101) may display summary information on a portion of the screen displayed on the display while an application requiring user input is running.

[0200] In FIG. 8C, an example of providing summary information using a wearable device (103) is described. The wearable device (103) can receive summary information from an electronic device (101). For example, the wearable device (103) can generate a voice signal for the summary information based on text-to-speech (TTS). For example, the wearable device (103) can output the voice signal through an audio output module (202).

[0201] Figure 9 illustrates an example of signaling between a wearable device and an electronic device to provide notification and summary information for a user call.

[0202] At least some of the operations of FIG. 9 may be performed by the wearable device (103) or the electronic device (101). For example, at least some of the operations may be controlled by the processor (201) of the wearable device (103) or the processor (211) of the electronic device (101). In the following description, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed. For example, at least two operations (e.g., operations 912 and 931) may be performed in parallel.

[0203] Referring to FIG. 9, operations performed by the wearable device (103) are described. In operation 911, according to one embodiment, the wearable device (103) may obtain an audio signal (e.g., audio signal (510)). For example, the wearable device (103) may obtain an audio signal for a sound external to the wearable device (103) using a microphone of the input module (203).

[0204] In operation 931, according to one embodiment, the wearable device (103) may receive user identification information from the electronic device (101). For example, the user identification information may include information about a feature vector of keywords, probability information of keywords, or a combination thereof.

[0205] For example, a keyword may refer to a text identified by the artificial intelligence module (215) of the electronic device (101) as referring to a user of the devices (e.g., the electronic device (101) and the wearable device (103)). However, the present disclosure is not limited thereto. For example, a keyword may further include a call word (e.g., “excuse me” or “wait a moment”) for calling any user other than the user of the devices.

[0206] For example, information about a feature vector of a keyword may include information for identifying the keyword from an audio signal. The feature vector may represent frequency characteristics of a voice signal corresponding to the keyword (e.g., mel-frequency cepstral coefficient (MFCC)) and / or frequency band-specific energy of a voice signal corresponding to the keyword (e.g., filter bank energy). For example, likelihood information of a keyword may represent the likelihood that the keyword is used for a user call. For example, the likelihood information of a keyword may include numerical information about the likelihood.

[0207] In operation 912, according to one embodiment, the wearable device (103) can identify at least one keyword using a keyword spotter (KWS). For example, the wearable device (103) can identify at least one keyword from an audio signal. For example, the wearable device (103) can identify at least one keyword (e.g., keyword (511) and keyword (512)) from the audio signal based on information about feature vectors of the keywords.

[0208] In one embodiment, the wearable device (103) may detect a trigger condition. For example, the wearable device (103) may detect a trigger condition for performing call recognition. For example, in response to detecting a trigger condition for performing call recognition, the wearable device (103) may determine whether at least one keyword is used to call the user. For example, the trigger condition for performing call recognition may include a trigger condition based on likelihood information, a trigger condition based on buffer capacity, or a combination thereof. In one embodiment, the wearable device (103) may identify a matching score based on likelihood information of keywords. For example, the wearable device (103) may detect a trigger condition when a matching score for at least one keyword exceeds a threshold score. In one embodiment, the wearable device (103) may monitor the remaining capacity of a buffer (e.g., buffer (205-1) of FIG. 2B) in which at least one keyword is temporarily stored. For example, the wearable device (103) may detect a trigger condition when the remaining capacity of the buffer is below a threshold capacity.

[0209] In operation 913, according to one embodiment, the wearable device (103) can identify whether the user has called and whether the user has recognized the call.

[0210] In one embodiment, the wearable device (103) may determine whether at least one keyword is a user call. For example, the wearable device (103) may use a language model of an artificial intelligence (AI) module (207) to determine whether at least one keyword is used for a user call. For example, the language model of the AI ​​model may include a large language model (LLM).

[0211] In one embodiment, the wearable device (103) may generate a prompt indicating the likelihood of at least one keyword being used to call a user. For example, the wearable device (103) may identify likelihood information for at least one keyword identified from an audio signal among likelihood information of keywords. For example, the wearable device (103) may generate a prompt associated with the likelihood of at least one keyword being used to call a user based on the identification. In one example, if the at least one keyword includes 'glasses,' the prompt may indicate 20 percent (%). The prompt may be structured as, 'There is a 20 percent chance that the user will be called.' In one example, if the at least one keyword includes 'yellow clothes,' the prompt may indicate 30 percent. The prompt may be structured as, 'There is a 30 percent chance that the user will be called.' In one example, if at least one keyword includes "glasses" and "yellow clothes," the prompt may indicate a 50 percent chance. The prompt may be structured as, "There is a 50 percent chance that the user will be called." In one example, if at least one keyword includes the user's name, "Kim Sam-seong," the prompt may indicate a 100 percent chance. The prompt may be structured as, "There is a 100 percent chance that the user will be called." However, these are merely examples, and the present disclosure is not limited thereto.

[0212] In one embodiment, the wearable device (103) may generate a prompt associated with accessibility (or orientation). For example, the wearable device (103) may generate a prompt associated with accessibility based on acoustic information (e.g., amplitude, frequency, pitch) of an audio signal corresponding to at least one keyword.

[0213] In one embodiment, the wearable device (103) may identify that a speaker of an audio signal is approaching a user of the wearable device (103) if the amplitude (e.g., decibel) of a signal corresponding to at least one keyword of the audio signal increases over time. For example, the wearable device (103) may identify that a speaker of the audio signal is approaching a user of the wearable device (103) based on the Doppler effect if a frequency of a signal corresponding to at least one keyword of the audio signal increases over time. For example, a prompt generated based on the identification may indicate that a speaker of the audio signal is approaching a user of the wearable device (103). In one example, the prompt may be configured as, 'The speaker is approaching the user.' However, this is merely an example, and the present disclosure is not limited thereto.

[0214] In one embodiment, the wearable device (103) may identify that a speaker of the audio signal is moving away from the user of the wearable device (103) if the amplitude of a signal corresponding to at least one keyword in the audio signal decreases over time. For example, the wearable device (103) may identify that a speaker of the audio signal is moving away from the user of the wearable device (103) based on the Doppler effect if the frequency of a signal corresponding to at least one keyword in the audio signal decreases over time. For example, a prompt generated based on the identification may indicate that the speaker of the audio signal is moving away from the user of the wearable device (103). In one example, the prompt may be configured as, 'The speaker is moving away from the user.' However, this is merely an example, and the present disclosure is not limited thereto.

[0215] In one embodiment, the wearable device (103) may generate a prompt associated with a repeated call. For example, the wearable device (103) may perform a voiceprint analysis based on acoustic information of an audio signal. For example, the wearable device (103) may identify, based on the voiceprint analysis, whether a signal corresponding to at least one keyword is obtained from a single user. In one example, upon identifying that the signal corresponding to at least one keyword is obtained from a single user, the prompt may be structured as, "The call is from the same user." In one example, upon identifying that the signal corresponding to at least one keyword is obtained from different users, the prompt may be structured as, "The call is from different users." However, this is merely an example, and the present disclosure is not limited thereto.

[0216] In one embodiment, the input data of the language model of the artificial intelligence module (207) for determining whether at least one keyword is used for a user call may include a prompt indicating the likelihood that at least one keyword is used for a user call, a prompt associated with accessibility, a prompt associated with repeated calls, or a combination thereof. However, the present disclosure is not limited thereto. For example, the input data of the artificial intelligence model may further include a prompt associated with the role of at least one keyword within a sentence (e.g., subject, verb, object) and / or a prompt indicating whether the user is using the electronic device (101). In one example, if the identified keyword plays the role of an object within a sentence, the probability that at least one keyword is identified as being used for a user call may be reduced.

[0217] In one embodiment, the wearable device (103) may generate output data from the input data through the language model of the artificial intelligence module (207). For example, the output data may indicate whether at least one keyword is used to call a user of the wearable device (103). In one example, the output data may indicate that at least one keyword is used to call a user. In another example, the output data may indicate that at least one keyword is not used to call a user. The wearable device (103) may identify whether at least one keyword is used to call a user of the wearable device (103) based on the output data of the language model of the artificial intelligence module (207).

[0218] In one embodiment, the wearable device (103) may determine whether a user is recognized. For example, the wearable device (103) may determine whether a user gesture in response to a user call is detected based on a determination that at least one keyword is used for the user call. For example, the wearable device (103) may use a language model of the artificial intelligence module (207) to determine whether a user gesture in response to a user call is detected. For example, the language model of the artificial intelligence model may include an LLM.

[0219] In one embodiment, the wearable device (103) may generate a prompt associated with the release of contact between the wearable device (103) and a body part of the user. For example, the wearable device (103) may obtain sensor data from the sensor module (204). For example, the wearable device (103) may identify whether contact between the wearable device (103) and the body part of the user is released based on the sensor data obtained from the sensor module (204). For example, the wearable device (103) may generate a prompt associated with the release of contact between the wearable device (103) and the body part of the user based on the identification. In one example, the prompt may indicate the release of contact between the wearable device (103) and the body part of the user. In another example, the prompt may indicate contact between the wearable device (103) and the body part of the user.

[0220] In one embodiment, the wearable device (103) may generate a prompt associated with the rotation of a body part (e.g., head) of a user wearing the wearable device (103). For example, the wearable device (103) may identify the rotation angle of the body part of the user wearing the wearable device (103) based on sensor data acquired from the sensor module (204). For example, the wearable device (103) may generate a prompt associated with the rotation of the body part of the user based on the identification. For example, the prompt may indicate the rotation angle of the body part of the user.

[0221] In one embodiment, the wearable device (103) may generate a prompt associated with user input for launching an application for listening to ambient sound. For example, the wearable device (103) may identify whether user input for launching the application has been obtained after receiving an audio signal. For example, the wearable device (103) may generate a prompt associated with launching the application based on the identification. In one example, the prompt may indicate acquisition of user input for launching the application. In another example, the prompt may indicate that user input for launching the application has not been obtained.

[0222] In one embodiment, the wearable device (103) may generate a prompt indicating whether a noise cancellation application is running. For example, the wearable device (103) may identify whether the application is running while receiving an audio signal. For example, the wearable device (103) may generate a prompt indicating whether the application is running based on the identification. In one example, the prompt may indicate that the noise cancellation application is running while receiving an audio signal. In another example, the prompt may indicate that the noise cancellation application is not running while receiving an audio signal.

[0223] In one embodiment, the wearable device (103) may identify whether user input for terminating execution of an application for noise removal has been obtained after receiving an audio signal. For example, the wearable device (103) may generate a prompt associated with the user input for terminating execution of the application based on the identification. In one example, the prompt may indicate acquisition of user input for terminating execution of the application. In another example, the prompt may indicate that user input for terminating execution of the application has not been obtained.

[0224] In one embodiment, input data of an artificial intelligence model for detecting a user gesture in response to a user call may include a prompt associated with a release of contact between a wearable device (103) and a user's body part, a prompt associated with a rotation of a user's body part wearing the wearable device (103), a prompt associated with a user input for executing an application for listening to ambient sounds, a prompt indicating whether an application for noise removal is running, a prompt associated with a user input for terminating the execution of an application for noise removal, or a combination thereof.

[0225] In one embodiment, while the noise removal application is running, the weights of the AI ​​model for detecting user gestures may be changed. For example, if the application's execution interferes with the user's perception of external sounds, the weights of the AI ​​model may be changed to reduce the likelihood of detecting user gestures.

[0226] In one embodiment, the wearable device (103) may generate output data from the input data via the language model of the artificial intelligence module (207). For example, the output data may indicate whether a user gesture in response to a user call is detected. In one example, the output data may indicate the detection of a user gesture. In another example, the output data may indicate that a user gesture was not detected.

[0227] In operation 914, according to one embodiment, the wearable device (103) may provide a notification for the user. For example, the wearable device (103) may provide a notification for the user of the wearable device (103) based on a determination that a user gesture in response to a call to the user is not detected. For example, the wearable device (103) may provide a notification for a user call for a user identified as not recognizing the user call. For example, the wearable device (103) may output an audio signal for calling the user of the wearable device (103) using the speaker of the audio output module (202). For example, the wearable device (103) may control the audio output module (202) to reduce the volume of the sound output by the speaker of the audio output module (202). For example, the wearable device (103) can control the haptic module to provide a mechanical stimulus (e.g., vibration or movement) that the user can perceive through tactile or kinesthetic senses. For example, the wearable device (103) can execute an application for listening to ambient sounds. For example, the wearable device (103) can terminate the execution of an application for noise removal. For example, the wearable device (103) can transmit a message to the electronic device (101) for providing a notification to the user. In response to receiving the message for providing a notification to the user, the electronic device (101) can display information indicating a call to the user using a display.

[0228] Although not illustrated in FIG. 9, the wearable device (103) may refrain from providing a notification to the user based on a determination that a user gesture in response to a user call has been detected. For example, by refraining from providing a notification to the user, unnecessary notifications may be prevented from being provided to the user who has recognized the user call.

[0229] In operation 932, according to one embodiment, the wearable device (103) may transmit a voice signal other than a user call to the electronic device (101). For example, the voice signal other than a user call may correspond to an audio signal excluding a signal for at least one keyword identified by the wearable device (103) as being used for a user call among the audio signals. In one example, the voice signal may correspond to a portion (513) of the voice signal (510) of FIG. 5. For example, the wearable device (103) may periodically transmit the voice signal other than a user call to the electronic device (101). In another example, the wearable device (103) may transmit the voice signal other than a user call to the electronic device (101) when an event for a user call is detected.

[0230] In operation 933, according to one embodiment, the wearable device (103) may transmit sensor data and control unit setting information to the electronic device (101) via the communication module (206). For example, the control unit setting information may indicate whether an application for noise removal is running on the wearable device (103).

[0231] In operation 934, according to one embodiment, the wearable device (103) may receive summary information from the electronic device (101). In one example, the summary information may include information associated with portion (513) of FIG. 5.

[0232] In operation 915, according to one embodiment, the wearable device (103) may provide summary information to the user. For example, the wearable device (103) may generate voice information based on the summary information, based on text to speech (TTS). For example, the wearable device (103) may output the voice information using the speaker of the audio output module (202).

[0233] Operations performed by the electronic device (101) are described with reference to FIG. 9. In operation 921, according to one embodiment, the electronic device (101) may obtain user information. For example, the user information may include information for identifying a user of devices (e.g., the electronic device (101) and the wearable device (103)). For example, the user information may include information obtained by the electronic device (101) and information obtained by the wearable device (103).

[0234] In one embodiment, the information acquired by the electronic device (101) may include sensor data acquired by the sensor module (214), information about an application being executed by the electronic device (101), information about the user's name, information about the user's location, information about the user's status (e.g., exercise status), information about the user's schedule, information about the user's car number, information about the user's performance seat, information about the user's weather, information about the user's clothing, information about the user's characteristics analyzed from photos stored in the memory (212) in association with a photo application of the electronic device (101), information about the user's previous response to a call, or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the electronic device (101) may acquire additional information for identifying the user by monitoring an application being executed by the electronic device (e.g., a phone application, a message application).

[0235] In one embodiment, the user information acquired by the wearable device (103) may include information for identifying the user among information acquired through the microphone of the input module (203), sensor data acquired by the sensor module (204), or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the user information acquired by the wearable device (103) may further include other information that can be used to identify the user.

[0236] In operation 922, according to one embodiment, the electronic device (101) may analyze keywords referring to the user based on user information. For example, the keywords may refer to texts for identifying the users of the devices (e.g., the electronic device (101) and the wearable device (103)). However, the present disclosure is not limited thereto. The keywords may further include call words (e.g., "excuse me" or "wait a moment") for calling any user other than the users of the devices.

[0237] In operation 931, according to one embodiment, the electronic device (101) may transmit user identification information to the wearable device (103) via the communication module (213).

[0238] In one embodiment, the electronic device (101) may generate user identification information based on user information. The user identification information may include information on feature vectors of keywords (user name, job title, clothing), likelihood information of the keywords, or a combination thereof. For example, the information on the feature vector of the keyword may include information for identifying the keyword from an audio signal. For example, the information on the feature vector of the keyword may include information representing the frequency characteristics of a voice signal corresponding to the keyword (e.g., MFCC) and / or information representing the energy of each frequency band of the voice signal corresponding to the keyword (e.g., filter bank energy). For example, the likelihood information of the keyword may indicate the likelihood that the keyword is used to call the user of the devices. For example, the likelihood information of the keyword may include numerical information regarding the likelihood. For example, the electronic device (101) may transmit the generated user identification information to the wearable device (103) via the communication module (213).

[0239] In operation 932, according to one embodiment, the electronic device (101) may receive a voice signal other than a user call from the wearable device (103) through the communication module (213). The voice signal other than a user call may mean a voice signal other than a voice signal for a keyword identified by the wearable device (103) by user identification information.

[0240] In operation 923, according to one embodiment, the electronic device (101) may generate text corresponding to a voice signal other than a user call using an automatic speech recognition (ASR) model (or a speech to text (STT) model) of the artificial intelligence module (215). FIG. 9 illustrates ASR, but the present disclosure is not limited thereto. The electronic device (101) may generate text corresponding to a voice signal other than a user call using an artificial intelligence model that converts a voice signal into text.

[0241] In operation 933, according to one embodiment, the electronic device (101) may receive information about sensor data and control unit settings from the wearable device (103) via the communication module (213). For example, the control unit setting information may indicate whether an application for noise removal is running by the wearable device (103).

[0242] In operation 924, according to one embodiment, the electronic device (101) may identify (or obtain) user status information. For example, the user status information may represent the user's surrounding environment. For example, the user status information may include the user's location information, sensor data obtained by the sensor module (214), user lifestyle pattern information, information about applications running on the electronic device (101), the user's schedule information, weather information, the user's call information, message information, sensor data obtained from the wearable device (103), or a combination thereof. However, this is merely an example, and the present disclosure is not limited thereto. For example, the user status information may further include other information that may be collected by the electronic device (101) and the wearable device (103) to represent the user's surrounding environment.

[0243] In operation 925, according to one embodiment, the electronic device (101) may generate summary information.

[0244] In one embodiment, the electronic device (101) may generate summary information based on text and user status information corresponding to a voice signal other than a user call. For example, the electronic device (101) may generate content information based on the user status information and text using a language model (e.g., a large language model (LLM)) of the artificial intelligence module (215). For example, the content information may represent key content or valuable information to the user identified by the language model based on the context of the text. For example, the electronic device (101) may generate summary information based on the content information using a generative artificial intelligence model of the artificial intelligence module (215).

[0245] In operation 934, according to one embodiment, the electronic device (101) may transmit summary information to the wearable device (103) via the communication module (213). For example, the wearable device (103) may generate a voice signal corresponding to the summary information using TTS (text to speech). The wearable device (103) may output the voice signal via the speaker of the audio output module (202).

[0246] In operation 926, according to one embodiment, the electronic device (101) may provide summary information to the user. The electronic device (101) may display the summary information through the display of the screen output unit (216-1).

[0247] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0248] The wearable device (103) as described above may include at least one sensor. The wearable device (103) may include a microphone. The wearable device (103) may include a speaker. The wearable device (103) may include a memory (205) that stores instructions and includes one or more storage media. The wearable device (103) may include at least one processor (201) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify at least one keyword from an audio signal obtained from an electronic device connected to the wearable device and based on the audio signal obtained using the microphone. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine, using a language model, whether the at least one keyword in the audio signal is used to call a user. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device, upon determining that the at least one keyword is used to call the user, to determine, using the language model, whether a gesture responsive to a call to the user is detected. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device, upon determining that the gesture is not detected, to provide a notification for the user.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to refrain from providing a notification to the user based on a determination that the gesture is detected.

[0249] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate a first prompt indicating a likelihood that the at least one keyword is used in a call to the user, based on user identification information obtained from the electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate a second prompt indicating, based on acoustic information of the audio signal, whether a second user of the audio signal is approaching the user. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate a third prompt indicating, based on acoustic information of the audio signal, whether the audio signal is associated with a single user.

[0250] For example, input data of the language model for identifying whether the at least one keyword is used to call the user may include the first prompt, the second prompt, and the third prompt, and output data of the language model responsive to the input data may indicate whether the at least one keyword is used to call the user.

[0251] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, based on sensor information obtained from the at least one sensor, whether contact between the wearable device and at least a portion of the user's body is released. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate a fourth prompt indicating whether contact between the wearable device and at least a portion of the user's body is released.

[0252] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a rotation angle of a body part of the user wearing the wearable device based on sensor information obtained from the at least one sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate a fifth prompt indicating the rotation angle of the body part of the user.

[0253] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether user input for execution of a first application for listening to ambient sound is obtained. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate a sixth prompt indicating whether user input for execution of the first application is obtained.

[0254] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether user input for terminating execution of a second application for noise cancellation is obtained. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to generate a seventh prompt indicating whether user input for terminating execution of the second application is obtained.

[0255] For example, input data of the language model for identifying whether a gesture responding to a call to the user is detected may include a fourth prompt indicating whether contact between the wearable device and at least a part of the user's body is released, a fifth prompt indicating a rotation angle of the body part of the user wearing the wearable device, a sixth prompt indicating whether a user input for executing a first application for listening to ambient sound is obtained, and a seventh prompt indicating whether a user input for terminating the execution of a second application for noise removal is obtained, and output data of the language model responsive to the input data may indicate whether the gesture responding to a call to the user is detected.

[0256] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether a remaining capacity of a buffer of the wearable device is less than a threshold capacity, and, based on the identification that the remaining capacity of the buffer is less than the threshold capacity, determine whether the at least one keyword is used to call the user.

[0257] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to change weights of the language model for determining whether the gesture is detected while the second application for noise removal is running.

[0258] A method performed by a wearable device including at least one sensor, a microphone, and a speaker as described above, the method may include an operation of identifying at least one keyword from an audio signal obtained from an electronic device connected to the wearable device and an audio signal obtained using the microphone. The method may include an operation of determining, using a language model, whether the at least one keyword in the audio signal is used to call a user. The method may include an operation of determining, using the language model, whether a gesture responding to a call to the user is detected, based on a determination that the at least one keyword is used to call the user. The method may include an operation of providing a notification for the user, based on a determination that the gesture is not detected. The method may include an operation of refraining from providing a notification for the user, based on a determination that the gesture is detected.

[0259] For example, the method may include generating a first prompt based on user identification information obtained from the electronic device, indicating a possibility that the at least one keyword is used to make a call to the user. The method may include generating a second prompt based on acoustic information of the audio signal, indicating whether a second user of the audio signal is approaching the user. The method may include generating a third prompt based on acoustic information of the audio signal, indicating whether the audio signal is associated with a single user.

[0260] For example, input data of the language model for identifying whether the at least one keyword is used to call the user may include the first prompt, the second prompt, and the third prompt, and output data of the language model responsive to the input data may indicate whether the at least one keyword is used to call the user.

[0261] For example, the method may include an operation of identifying whether contact between the wearable device and at least a portion of the user's body is released based on sensor information obtained from the sensor. The method may include an operation of generating a fourth prompt indicating whether contact between the wearable device and at least a portion of the user's body is released.

[0262] For example, the method may include an operation of identifying a rotation angle of a body part of the user wearing the wearable device based on sensor information obtained from the sensor. The method may include an operation of generating a fifth prompt indicating the rotation angle of the body part of the user.

[0263] For example, the method may include an operation of identifying whether user input for executing a first application for listening to ambient sound has been obtained. The method may include an operation of generating a sixth prompt indicating whether user input for executing the first application has been obtained.

[0264] For example, the method may include an operation for identifying whether a user input for terminating execution of a second application for noise cancellation is obtained. The method may include an operation for generating a seventh prompt indicating whether a user input for terminating execution of the second application is obtained.

[0265] For example, input data of the language model for identifying whether a gesture responding to a call to the user is detected may include a fourth prompt indicating whether contact between the wearable device and at least a part of the user's body is released, a fifth prompt indicating a rotation angle of the body part of the user wearing the wearable device, a sixth prompt indicating whether a user input for executing a first application for listening to ambient sound is obtained, and a seventh prompt indicating whether a user input for terminating the execution of a second application for noise removal is obtained, and output data of the language model responsive to the input data may indicate whether the gesture responding to a call to the user is detected.

[0266] For example, the method may include an operation of identifying whether the remaining capacity of the buffer of the wearable device is less than a threshold capacity. The method may include an operation of determining whether the at least one keyword is used to call the user based on the identification that the remaining capacity of the buffer is less than the threshold capacity.

[0267] For example, the method may include changing weights of the language model for determining whether the gesture is detected while a second application for noise removal is running.

[0268] As described above, the wearable device (103) and electronic device (101) according to the present disclosure can improve the response speed of the devices to external sounds by implementing an on-device artificial intelligence model. Accordingly, the present disclosure can provide a more convenient user experience (UX) by providing a notification to a user wearing and using the wearable device (103) to react to external sounds.

[0269] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0270] The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0271] When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors in an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specifications of the present disclosure. The one or more programs may be provided as included in a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read only memory (CD-ROM)) or an application store (e.g., Play Store). ™ ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0272] These programs (software modules, software) may be stored in random access memory, non-volatile memory including flash memory, read only memory (ROM), electrically erasable programmable read only memory (EEPROM), magnetic disc storage devices, compact disc-ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, magnetic cassettes, or may be stored in memories formed by a combination of some or all of these. In addition, each configuration memory may include multiple copies.

[0273] Additionally, the program may be stored on an attachable storage device that is accessible via a communication network, such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a storage area network (SAN), or a combination thereof. Such a storage device may be connected to a device implementing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device implementing an embodiment of the present disclosure.

[0274] In the specific embodiments of the present disclosure described above, components included in the disclosure are expressed singularly or plurally, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in plural may be composed of singular elements, or components expressed in singular may be composed of plural elements.

[0275] According to embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0276] Meanwhile, although the detailed description of the present disclosure has described specific embodiments, it is obvious that various modifications are possible within the scope of the present disclosure.

Claims

1. As a wearable device, At least one sensor; mike; speaker; A memory storing instructions and including one or more storage media; and At least one processor comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on at least one keyword obtained from an electronic device connected to the wearable device and an audio signal obtained using the microphone, identifying the at least one keyword from the audio signal, Using a language model, determining whether at least one keyword in the audio signal is used to call a user, Based on a determination that at least one keyword is used to call the user, using the language model, determining whether a gesture responding to a call to the user is detected, Upon determining that the above gesture is not detected, a notification is provided for the user, and causing the user to refrain from providing notifications for the user based on a determination that the gesture is detected; Wearable devices.

2. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on user identification information obtained from the electronic device, generating a first prompt indicating a possibility that the at least one keyword is used in a call to the user; Based on the acoustic information of the audio signal, a second prompt is generated indicating whether a second user of the audio signal is approaching the user, Causing a third prompt to be generated based on the acoustic information of the audio signal, indicating whether the audio signal is associated with a single user. Wearable devices.

3. In paragraph 2, The input data of the language model for identifying whether the at least one keyword is used to call the user includes the first prompt, the second prompt, and the third prompt, and The output data of the language model responding to the input data indicates whether the at least one keyword is used to call the user. Wearable devices.

4. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on sensor information obtained from at least one sensor, identifying whether contact between the wearable device and at least a part of the user's body is released, and causing a fourth prompt to be generated indicating whether contact between the wearable device and at least a part of the user's body is released; Wearable devices.

5. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on sensor information obtained from at least one sensor, identifying a rotation angle of a body part of the user wearing the wearable device, and causing a fifth prompt to be generated, indicating the rotation angle of the body part of the user; Wearable devices.

6. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Identify whether user input is obtained for launching a first application for listening to ambient sound, and Causing a sixth prompt to be generated, indicating whether user input for executing the first application is obtained; Wearable devices.

7. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Identify whether user input is obtained to terminate execution of a second application for noise cancellation, and Causes a seventh prompt to be generated indicating whether user input is obtained to terminate execution of the second application. Wearable devices.

8. In paragraph 1, The input data of the language model for identifying whether a gesture responding to a call to the user is detected includes a fourth prompt indicating whether contact between the wearable device and at least a part of the user's body is released, a fifth prompt indicating a rotation angle of the body part of the user wearing the wearable device, a sixth prompt indicating whether a user input for executing a first application for listening to ambient sound is obtained, and a seventh prompt indicating whether a user input for terminating the execution of a second application for noise removal is obtained, and The output data of the language model responding to the input data indicates whether the gesture responding to the call to the user is detected. Wearable devices.

9. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Identifying whether the remaining capacity of the buffer of the wearable device is less than the threshold capacity, and Causing to determine whether the at least one keyword is used to call the user, based on the identification that the remaining capacity of the buffer is less than the threshold capacity; Wearable devices.

10. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: While the second application for noise removal is running, causing the weights of the language model to be changed to determine whether the gesture is detected. Wearable devices.

11. A method performed by a wearable device including at least one sensor, a microphone, and a speaker, An operation of identifying at least one keyword from an audio signal based on at least one keyword obtained from an electronic device connected to the wearable device and an audio signal obtained using the microphone; An operation of using a language model to determine whether at least one keyword in the audio signal is used to call a user; An action of determining, using the language model, whether a gesture responding to a call to the user is detected, based on a determination that at least one keyword is used to call the user; An action to provide a notification to the user based on a determination that the above gesture is not detected; and Including an action to refrain from providing a notification to the user based on a determination that the above gesture is detected. method.

12. In paragraph 11, An action of generating a first prompt indicating a possibility that the at least one keyword is used in a call to the user based on user identification information obtained from the electronic device; An operation of generating a second prompt indicating whether a second user of the audio signal is approaching the user based on acoustic information of the audio signal; and Further comprising an action of generating a third prompt indicating whether the audio signal is associated with a single user based on acoustic information of the audio signal. method.

13. In paragraph 12, The input data of the language model for identifying whether the at least one keyword is used to call the user includes the first prompt, the second prompt, and the third prompt, and The output data of the language model responding to the input data indicates whether the at least one keyword is used to call the user. method.

14. In paragraph 11, An operation of identifying whether contact between the wearable device and at least a part of the user's body is released based on sensor information obtained from the sensor; and Further comprising an action of generating a fourth prompt indicating whether contact between the wearable device and at least a part of the user's body is released; method.

15. In paragraph 11, An operation of identifying a rotation angle of a body part of a user wearing the wearable device based on sensor information obtained from the sensor; and Further comprising an action of generating a fifth prompt indicating the rotation angle of the body part of the user; method.

Images