Wearable electronic device for providing auditory information and method therefor

Abstract

A wearable electronic device according to various embodiments disclosed in the present document may comprise an image sensor, a speaker, a memory for storing instructions, and at least one processor including processing circuitry, wherein the instructions, when executed individually or collectively by the at least one processor, cause the wearable electronic device to: identify an action event of a user; analyze an action intent on the basis of the action event; identify a target object of the action event and an attribute of the target object on the basis of the action intent; identify, in correspondence to the attribute of the target object, a body part of the user and at least a partial area of the target object corresponding to the action intent; identify spatial information on the basis of the attribute of the target object, the body part, and the at least partial area of the target object; set, on the basis of the spatial information, a path through which the body part reaches the at least partial area of the target object; and provide, via the speaker, sound information including a first sound and a second sound for guiding the execution of the action event on the basis of the path.

Description

Wearable electronic device providing auditory information and method thereof

[0001] Various embodiments disclosed in this document relate to a wearable electronic device that provides auditory information, for example, a wearable electronic device that provides auditory information to guide interaction with an external object, and a method thereof.

[0002] With technological advancements, the need to use wearable electronic devices to recognize external objects and provide services based on these findings is increasing. For example, wearable electronic devices can extract and identify external objects from images captured through a camera, and provide various services based on these objects.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] Various embodiments disclosed in this document may disclose a wearable electronic device including an image sensor, a speaker, a memory for storing instructions, and at least one processor including processing circuitry. In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to identify a user's action event. In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to analyze an action intent based on the action event and identify an object of the action event and an attribute of the object based on the action intent. In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to identify a body part of the user corresponding to the action intent and at least a portion of the object in response to an attribute of the object. The instructions according to one embodiment, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to determine spatial information based on an attribute of the object, the body part, or at least a portion of the object. The instructions according to one embodiment, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to set a path for the body part to reach at least a portion of the object based on the spatial information.The instructions according to one embodiment, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to provide sound information including a first sound and a second sound guiding performance of the action event based on the path through the speaker.

[0005] Various embodiments disclosed in this document may disclose a method of a wearable electronic device having an image sensor. In one embodiment, the method may include an operation of identifying a user's action event. In one embodiment, the method may include an operation of analyzing an action intent based on the action event. In one embodiment, the method may include an operation of identifying an object of the action event and an attribute of the object based on the action intent. In one embodiment, the method may include an operation of identifying a body part of the user corresponding to the action intent and at least a portion of the object in response to the attribute of the object. In one embodiment, the method may include an operation of identifying spatial information based on the attribute of the object, the body part, and at least a portion of the object. In one embodiment, the method may include an operation of setting a path for the body part to reach at least a portion of the object based on the spatial information. In one embodiment, the method may include an operation of providing sound information, including a first sound and a second sound that guide the performance of the action event based on the path, through the speaker.

[0006] The technical tasks, technical features, and effects to be achieved in the present disclosure are not limited to the technical tasks, technical features, and effects mentioned above, and other technical tasks, technical features, and effects not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present invention pertains from the description below.

[0007] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0008] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment of the present disclosure.

[0009] FIGS. 2A and 2B are drawings illustrating an example of a configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0010] FIG. 3 is a block diagram illustrating an integrated intelligence system of a wearable electronic device according to one embodiment of the present disclosure.

[0011] FIG. 4 is a flowchart illustrating an example of the operation of a wearable electronic device according to one embodiment of the present disclosure.

[0012] FIGS. 5, 6, 7, and 8 are drawings illustrating an example of the operation of a wearable electronic device according to one embodiment of the present disclosure.

[0013] FIG. 9 is a diagram illustrating an example of a hierarchical operation structure of a wearable electronic device according to one embodiment of the present disclosure.

[0014] FIG. 10 is a diagram illustrating examples of an intent confirmation operation of a wearable electronic device according to one embodiment of the present disclosure.

[0015] FIGS. 11 and 12 are drawings for explaining examples of risk factor identification operations of a wearable electronic device according to one embodiment of the present disclosure.

[0016] FIGS. 13 and 14 are drawings for explaining examples of a path confirmation operation of a wearable electronic device according to one embodiment of the present disclosure.

[0017] FIG. 15 is a diagram illustrating examples of a user interface according to the operation of a wearable electronic device according to one embodiment of the present disclosure.

[0018] FIG. 16, FIG. 17, FIG. 18, FIG. 19, and FIG. 20 are drawings for explaining an example of operation of a wearable electronic device to which one embodiment of the present disclosure is applied.

[0019] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to one embodiment of the present disclosure.

[0020] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0021] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0022] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0023] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0024] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0025] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0026] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0027] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0028] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0029] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0030] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0031] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0032] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0033] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0034] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0035] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0036] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0037] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0038] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0039] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0040] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0041] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0042] FIGS. 2A and 2B are drawings illustrating an example of the configuration of a wearable electronic device (200) according to one embodiment of the present disclosure. FIG. 3 is a block diagram illustrating an integrated intelligence system of a wearable electronic device (200) according to one embodiment of the present disclosure.

[0043] Referring to FIGS. 2A, 2B, and 3, a wearable electronic device (200) (e.g., the electronic device (101) of FIG. 1) can acquire external images through an external sensor (210) (e.g., the sensor module (176) and / or the camera module (180) of FIG. 1) including one or more depth cameras (211), one or more side cameras (212), one or more upward or downward cameras (213), and / or one or more IR sensors (infra-red) (214). The wearable electronic device (200) can detect whether the wearable electronic device (200) is being worn and a user's gaze direction through an eye recognition sensor (220) (e.g., the sensor module (176) and / or the camera module (180) of FIG. 1) including one or more IR cameras (221), one or more LED rings (222), and / or a wearing detection sensor (223). The wearable electronic device (200) can obtain information such as the position and / or direction of the wearable device through various sensors such as a position recognition sensor (230) (e.g., the sensor module (176) of FIG. 1) including an inertial measurement unit (IMU) sensor (231) and / or an ultra wide band (UWB) sensor (232). The IMU sensor (231) can include an acceleration sensor, a gyroscope, and / or a geomagnetic sensor, and can obtain sensor information such as a posture angle, a posture angular velocity, and / or an azimuth angle of a user wearing the wearable electronic device (200).

[0044] The wearable electronic device (200) may include a sound module (240) including one or more speakers (241) (e.g., the sound output module (155) of FIG. 1) and / or one or more array microphones (242) (e.g., the input module (150) of FIG. 1), through which various external sounds including the user's voice may be received or various sounds may be output.

[0045] A wearable electronic device (200) can provide virtual or real image information through an image output device (250) (e.g., a display module (160) of FIG. 1) including one or more displays (251) and lenses (252).

[0046] A wearable electronic device (200) is worn on a user's head and can provide the user with images related to an augmented reality service. The wearable electronic device (200) may include a form that allows the wearer to be immersed in an image displayed on a display (251), as illustrated in FIG. 2A, or a form that allows at least a portion of external light to be detected by the wearer's eyes, as illustrated in FIG. 2B, in the form of glasses. Although not illustrated, the wearable electronic device (200) may also be provided in the form of, for example, earbuds equipped with a camera (e.g., including a microphone and an audio output device), or a form that includes earbuds equipped with a camera and a wearable device that can be mounted on the user's head or front (e.g., in front of the eyes).

[0047] Referring to FIG. 2B, the wearable electronic device (200) may be composed of a main body (273) and a support (e.g., a first support (275) and / or a second support (276)), and the main body (273) and the support (275, 276) may be operatively connected. The main body (273) may be at least partially mounted on a user's nose and may include a display module and a camera module. The support (275, 276) may include a support member mounted on the user's ear, and may include a first support (275) mounted on the left ear and / or a second support (276) mounted on the right ear. The display module may include a first glass (271) and / or a second glass (272), and may provide visual information to the user through the first glass (271) and the second glass (272). The display module may include a display panel and / or a lens (e.g., glass). For example, the display panel may include a transparent material such as glass or plastic. The wearable electronic device (200) processes various data through a data processing transmission / reception module (260) including one or more processors (261) (e.g., the processor (120) and / or the communication module (190) of FIG. 1) and may transmit and receive data with, for example, an external electronic device (e.g., the electronic devices (102, 104) of FIG. 1) or a server (e.g., the server (108) of FIG. 1).

[0048] Referring to FIG. 3, a wearable electronic device (200) may include a multimodal large language model (270) (e.g., memory (130) of FIG. 1). The multimodal large language model (270) may include a text awareness module, an audio awareness module, an image awareness module, a context awareness module, and / or a user intent awareness module. The wearable electronic device (200) may identify an action intent based on a user action event through the text recognition module, the audio recognition module, the image recognition module, the context awareness module, and / or the user intent awareness module included in the multimodal model (270), and guide a user's body part to perform the action intent targeting at least a portion of an object.

[0049] The configuration of the wearable electronic device (200) described above is an example and is not limited thereto, and various configurations are possible. For example, at least some of the components of the wearable electronic device (200) may be omitted, two or more components may be integrated, or additional components may be arranged.

[0050] FIG. 4 is a flowchart illustrating an example of the operation of a wearable electronic device (200) according to an embodiment of the present disclosure. FIGS. 5, 6, 7, and 8 are drawings illustrating an example of the operation of a wearable electronic device (200) according to an embodiment of the present disclosure.

[0051] Hereinafter, one embodiment will be described with reference to FIGS. 4 to 8.

[0052] According to one embodiment, a wearable electronic device (e.g., wearable electronic device (200) of FIGS. 2A and 2B) may include an image sensor (e.g., camera module (180) of FIG. 1, external sensor (210) of FIG. 2A, FIG. 2B, or FIG. 3, or eye recognition sensor (220)), a speaker (e.g., audio output module (155) of FIG. 1, speaker (241) of FIG. 2A, FIG. 2B, or FIG. 3), a memory (e.g., memory (130) of FIG. 1, multi-modal model (270) of FIG. 2A, FIG. 2B, or FIG. 3), and at least one processor (e.g., processor (120) of FIG. 1, data processing transmit / receive module (260) of FIG. 2A, FIG. 2B, or FIG. 3). The wearable electronic device (200) can be turned on and calibrated to components and image sensors as the device becomes active to allow images to be recognized more accurately.

[0053] Referring to FIG. 4, a wearable electronic device (200) according to one embodiment can, in operation 401, check a user's action event, and, in operation 403, analyze an action intent based on the action event.

[0054] According to one embodiment, the wearable electronic device (200) can detect a user's action through an image sensor and determine whether an action event has occurred. The wearable electronic device (200) can obtain an image through an external sensor (210) including an image sensor, for example, one or more depth cameras (211), one or more side cameras (212), one or more upward or downward cameras (213), and / or one or more IR sensors (infra-red) (214) of FIG. 2A, FIG. 2B, or FIG. 3. The wearable electronic device (200) can obtain sensor information, such as a current location and movement information of the user, through a location recognition sensor (230) including an IMU (inertial measurement unit) sensor (231) of FIG. 2A, FIG. 2B, or FIG. 3, and / or a GPS or UWB (ultra wide band) sensor (232).

[0055] The wearable electronic device (200) can determine whether an action event has occurred by checking the occurrence of a user's action event, such as an arm or hand extension motion, a hand waving motion, a stopping motion, a direction change, and / or an increase in speed, through the acquired image and / or sensor information. The wearable electronic device (200) can determine whether an action event has occurred based on object information analysis, such as tactile information recognition, such as Braille, destination recognition, such as a door, object recognition, such as a handle, or disability facility recognition, such as a voice inducer, through the acquired image and / or sensor information.

[0056] According to one embodiment, the wearable electronic device (200) can additionally determine whether an action event occurs through a voice input (e.g., a name of an object, a name of an action) via a microphone (e.g., an input module (150) of FIG. 1, a microphone (242) of FIG. 2A, FIG. 2B, or FIG. 3). Various data for determining whether the above-described action event occurs can be used in combination or individually. For example, a user can indicate an object to be searched for (e.g., "elevator button", "entrance") through a voice input, and the wearable electronic device (200) can confirm the occurrence of an action event accordingly. For example, a user can directly generate an action event by uttering "Where is the TV remote?" on a living room sofa through a voice input, thereby causing the wearable electronic device (200) to confirm the action intent.

[0057] In one embodiment, the wearable electronic device (200) can identify an action event through contextual information. For example, a user's behavior of standing in front of an elevator can predict that the user will soon be able to use the elevator, and accordingly, the wearable electronic device (200) can inform the user of the location of the elevator button. For example, AI reinforcement learning can be used to proactively discover the user's habits or mistakes, and when the user's frequent contextual action intents are learned or when the user's frequent mistakes are detected, the wearable electronic device (200) can detect an action event and confirm the action intent based on contextual information without any specific user action. For example, if the user is identified as approaching the door of a frequently visited store, the wearable electronic device (200) can confirm that the user will enter the store (e.g., confirm the action intent) and inform the user of the location of the door handle. For example, if the location of the user's tagging action on the bus transportation terminal is always detected to be to the left of the target location, the wearable electronic device (200) can guide the user in advance to tag to the right.

[0058] According to one embodiment, the wearable electronic device (200) may, upon confirming the occurrence of an action event, identify an action intent in operation 403. For example, the wearable electronic device (200) may extract the action intent from acquired images and / or sensor information. For example, the wearable electronic device (200) may identify user action information, object-related information, and / or situational information from acquired images and / or sensor information. In this case, the acquired images and / or sensor information may include images and / or sensor information acquired at or before the occurrence of the action event. The user action information may include actions such as reaching out, approaching, and shaking the head (up and down or left and right) and postures (e.g., lying down, sitting, standing, walking, stopping while walking, dozing). Object-related information may include object property information such as the type of thing (e.g., building (house, company, movie theater), place (elevator, stairs, bus stop, taxi stop, subway platform, object), color, area, text and / or symbol, shape (e.g., sphere, cube, column, height, parts). Object-related information may include object medium information (e.g., metal, glass, plastic, fabric, liquid), object context information (e.g., a knife is placed under a knife, a ladle is placed behind a glass water glass), object object motion information (e.g., door handle motion information such as PUSH / PULL, turn, and lower) (e.g., object object grip information such as one hand, two hands, grasp, and put on finger), object hazard information (e.g., sharp, slippery, rough, hot), or object surrounding hazard information (e.g., an object has fallen on the floor, a sandy beach is hot). Context information may include surrounding information acquired through the user's forward image and / or sensor information. May include environmental information.For example, contextual information may include information about the surrounding environment that can be used to infer the user's action intent, such as in front of a bus stop, in front of a door, or in front of stairs.

[0059] According to one embodiment, the wearable electronic device (200) can detect the user's surroundings based on acquired images and / or sensor information and confirm the user's action intent. If the user's action intent is confirmed, the target object can be confirmed. If the user's action intent is not confirmed, the wearable electronic device can check whether multiple action intents are possible. For example, if the user is in front of a door, the action intent may be grabbing a doorknob or ringing a bell. If multiple action intents are possible and it is difficult to confirm a single action intent, the wearable electronic device can also determine the action intent by asking the user for confirmation.

[0060] Referring to FIG. 5, the wearable electronic device (200) can extract an action intent of cooking by detecting a user's motion of extending a hand (501) in front of a sink, or more specifically, an action intent of holding a knife and cutting food based on object analysis including a knife, a cutting board, and food ingredients.

[0061] According to one embodiment, the wearable electronic device (200) can identify an action intent based on user action information and / or object-related information obtained from acquired images and / or sensor information.

[0062] For example, the wearable electronic device (200) may extract an action intent including an expected action of the user based on user action information and / or object-related information, targeting objects within a distance that the user can reach out and touch based on the current user location, i.e., the location of the wearable electronic device (200). The action intent may include a user action (e.g., exploring, touching, gripping, tagging, walking) toward a target object. For example, to confirm an action intent to grab an object, an action of reaching out, user input, and / or information learned from the user's current action and previous actions may be utilized. Confirmation of an action intent based on a user input (e.g., utterance of the name of a target object) may also be performed auxiliaryly, for example, when confirmation of the action intent through analysis of other information fails.

[0063] According to one embodiment, the wearable electronic device (200) may, based on the action intent, identify the object of the action event and the attributes of the object in operation 405, and identify risk factors according to the attributes of the object in operation 407. In response to the attributes of the object, the wearable electronic device (200) may identify a body part of the user corresponding to the action intent and at least a portion of the object in operation 409. The wearable electronic device (200) may identify spatial information based on the attributes of the object, risk factors, body parts, and at least a portion of the object in operation 411, and may set a reference path based on these information in operation 413. The execution order of these operations may be changed, specific operations may be omitted, or two or more operations may be integrated.

[0064] According to one embodiment, the wearable electronic device (200) can identify objects based on the identified action intent when the action intent is identified. The wearable electronic device (200) can provide feedback to the user regarding the location and / or status of the object, for example, through sound and / or voice.

[0065] According to one embodiment, the wearable electronic device (200) may use artificial intelligence such as AI (artificial intelligence), generative AI, a multi-modal model (e.g., the multi-modal model (270) of FIG. 3), or a large-scale multi-modal model to extract action intents as described above and perform operations such as identifying a target, attributes of the target, risk factors, body parts, at least a portion of the target, and spatial information. The multi-modal model may simultaneously process and integrate various types of data, including text, images, voice, and video. The multi-modal model may include multiple modal models, and may combine information produced by each modal model to enable more detailed data processing and integration. For example, by using the multi-modal model, text may be utilized in addition to the results of analyzing visual information such as images to provide specific explanations and / or context, and more reasonable answers or results may be derived by combining two or more types of information. A multimodal model may include, for example, a natural language processing (NLP) model for processing text data, and a computer vision model for processing images. The wearable electronic device (200) may perform multimodal fusion operations by combining data generated by each modal model through the multimodal model, thereby understanding and combining relationships between different data types, such as text and images, or different modal models to provide integrated information.

[0066] Referring to FIG. 6, the wearable electronic device (200) can identify a knife (502) on a cutting board as a target and a hand (501) as a user body part, and based on the properties (metal and plastic) of the knife as a target, identify a risk factor (e.g., a blade) and at least some area (e.g., a knife handle) as an interaction point of the target.

[0067] The wearable electronic device (200) can identify at least a portion of the target object of the action intent, i.e., the interaction target point or location of the object (e.g., a knife handle) and the user's interaction body part (e.g., a fingertip, a palm, a foot), and calculate the spatial location and distance between the body part and the portion of the object to generate one or more virtual movement paths. When two or more virtual movement paths are generated, the wearable electronic device (200) can select the final movement path by considering the operation method and / or risk factors of the object.

[0068] When determining an object's interaction point, contextual information can be additionally considered, such as whether the object can be directly grasped and used with bare hands, whether it is dangerous to grasp directly (e.g., requires gloves), or whether it can be grasped using a specific tool (e.g., a key). For example, a pot handle on a shelf or sink can be classified as an object that can be grasped and used directly, while a pot on a lit stove can be classified as an object that can be grasped using a tool. The action intent can then be analyzed to determine the object's interaction point. In this case, guidance (e.g., voice guidance) can be provided to further avoid the risk.

[0069] Risk factors associated with an action intent may include environmental hazards, human hazards, physical hazards, and / or health-related hazards. For example, environmental hazards may include risks associated with interactions with various objects (e.g., objects, people), such as collisions with obstacles like walls or furniture or people, falls from stairs or slopes, ground conditions such as slippery surfaces or puddles, and roadway hazards. Physical hazards may include risks such as fire, smoke, or noxious gases. Health-related hazards may include excessive noise, abnormal heart rate, or vital signs.

[0070] According to one embodiment, the wearable electronic device (200) can determine spatial information about a space including objects between a user's body part and the target based on the properties of the target, risk factors, body parts, and at least a portion of the target area.

[0071] According to one embodiment, the wearable electronic device (200) may, based on spatial information, set a path for a user's body part to reach at least a portion of a target area in operation 413. For example, after setting two or more paths, one path may be selected based on criteria such as user preference, habit, or shortest distance.

[0072] Referring to FIG. 7, the wearable electronic device (200) can set a path (R) on the X, Y, and Z axes so that the blade, which is a risk factor of the target knife, is positioned in the right direction, and accordingly, the hand, which is a body part, avoids the blade and reaches the knife handle.

[0073] The wearable electronic device (200) can determine whether a target object of an action intent has been detected, and if the target object has been detected, can provide guidance so that the user can find the object and interact with it. To this end, the wearable electronic device (200) can continuously track the spatial relationship with the object based on the user's position and / or direction through an image sensor. The wearable electronic device (200) can analyze the relative position and / or direction between the user and the object and provide sound including three-dimensional sound to enable recognition of the direction of the object, and can enable recognition of the distance to the object through changes in the frequency and / or amplitude of the sound.

[0074] According to one embodiment, the wearable electronic device (200) may provide sound information including a first sound and a second sound that guide the performance of an action event based on a set path in operation 415.

[0075] According to one embodiment, the wearable electronic device (200) may provide a first sound when a user body part approaches a set path, and may provide a second sound when the user body part deviates from the path.

[0076] Referring to FIG. 8, the wearable electronic device (200) can provide a first sound when the user's hand is directed toward the knife handle along a path (R), and can provide a second sound when the hand deviates from the path (O).

[0077] The wearable electronic device (200) can generate a first sound or a second sound by controlling the volume, frequency, peak, waveform of sound waves, and / or output interval of the sound. For example, the first sound can be generated and output by controlling the volume, frequency, peak, waveform of sound waves, and / or output interval based on relative distance information from the user's body part to the target, relative direction information, and / or distance or change in distance from a reference path. For example, the first sound can include a three-dimensional sound. For example, the first sound can convert one or more of the volume, peak, waveform of sound waves, and / or sound output interval, which are properties of the sound signal domain, in response to a change in the distance on the X, Y, and Z axes in the spatial domain between the user's body part and the target, thereby allowing the user to recognize a change in the spatial domain.

[0078] The wearable electronic device (200) may generate the first sound by changing the properties of the first sound and / or the second sound when the spatial information with respect to the target object changes as the user's body part moves. For example, the wearable electronic device (200) may generate the first sound by adjusting the volume or frequency of the sound according to the distance between the user's body part and the target object. For example, the wearable electronic device (200) may change the properties of the first sound so that the volume or frequency of the first sound increases as the target object gets closer along a reference path. For example, the wearable electronic device (200) may change the properties of the second sound so that the volume or frequency of the second sound increases as the wearable electronic device (200) deviates from the reference path or gets farther from the target object. For example, when the user's body part moves along the X, Y, and Z axes in the spatial domain, the wearable electronic device (200) may generate the first sound and the second sound by changing the properties by reflecting the change trend along the X, Y, and Z axes with respect to the target point of the object.

[0079] In one embodiment, the first sound and the second sound may be generated to have different sound colors. For example, the first sound may be generated to represent a positive, comfortable tone or color, and the second sound may be generated to represent a negative, uncomfortable tone or color.

[0080] In one embodiment, the wearable electronic device (200) may generate a first sound by including a sound indicating the relative direction of the object from the user's body part. For example, the first sound may include a three-dimensional sound to indicate the direction of a nearby object. Accordingly, the wearable electronic device (200) may provide information about the direction of the object through the first sound. For example, the wearable electronic device (200) may generate the first sound as a direction recognition sound so that the volume of the first sound increases or becomes sharper as the direction in which the user's body part is facing matches the direction in which the object is located.

[0081] According to one embodiment, the wearable electronic device (200) may provide a third sound when a user's body part approaches a hazard. The third sound may include voice guidance. The voice guidance may include, for example, the name, location, direction, operation or use method of an object, and guidance regarding hazards.

[0082] In one embodiment, to prevent collision risk as an environmental hazard, multiple third sounds may be provided. For example, when a vehicle is approaching, the wearable electronic device (200) may provide a sound (e.g., a voice) such as "A vehicle is approaching. Stop and look around" as a voice guidance, and may provide a vehicle approach warning sound (e.g., "bang bang") as a sound effect. For example, to prevent collision risk with a fixed object, the wearable electronic device (200) may provide a voice guidance such as "There is a fixed object ahead. Take a detour" as a voice guidance, and may provide a collision avoidance warning sound (e.g., "ding ding") as a sound effect. For example, for an obstacle on the road, the wearable electronic device (200) may provide a voice guidance such as "There is an obstacle on the road. Be careful to avoid it" as a voice guidance, and may provide an obstacle warning sound (e.g., "tta-ra-ran") as a sound effect. For example, the wearable electronic device (200) may provide an obstacle warning sound (e.g., "tta-ran") as a sound effect along with a voice guidance such as "There is a tree ahead. Turn." for natural obstacles. For example, the wearable electronic device (200) may provide a stair warning sound (e.g., "boom boom boom") as a sound effect along with a voice guidance such as "There are stairs ahead. Go down slowly." for stairs and slopes. For example, when there is a difference in elevation on the road, the wearable electronic device (200) may provide a ground warning sound (e.g., "rattling") as a sound effect along with a voice guidance such as "The road is bumpy. Walk slowly." For example, when the ground is slippery, the wearable electronic device (200) may provide a slippery warning sound (e.g., "beep beep") as a sound effect along with a voice guidance such as "The road is slippery. Walk carefully." For example, the wearable electronic device (200) may provide voice guidance when there is a problem with the road pavement condition, saying, "There is a crack in the road."For example, the wearable electronic device (200) may provide a collision prevention warning sound (e.g., "ding ding") as a sound effect along with a voice guidance saying "There is a person ahead. Move with caution" when there is a risk of collision with a person in front. For example, the wearable electronic device (200) may provide a collision prevention warning sound (e.g., "ding ding") as a sound effect along with a voice guidance saying "There is a person ahead. Move with caution" when there is a risk of collision with a person in front. For example, the wearable electronic device (200) may provide a congestion warning sound (e.g., "scratching") as a sound effect along with a voice guidance saying "This is a crowded space. Move slowly" when in a crowded space. For example, the wearable electronic device (200) may recognize traffic lights and signs when moving on a road and provide a traffic light warning sound (e.g., "red light") as a voice guidance saying "The traffic light is red. Stop." when there is a risk of collision with a person in front. For example, the wearable electronic device (200) may provide a vehicle approaching as a voice guidance according to detection of vehicle movement. A vehicle approach warning sound (e.g., "be careful") can be provided with a voice and sound effect saying "be careful".

[0083] In one embodiment, the wearable electronic device (200) may provide multiple third sounds to avoid danger from physical threats. For example, the wearable electronic device (200) may provide an emergency warning sound (e.g., "woo-woo") as a sound effect along with a voice prompt such as "Smoke detected. Evacuate immediately" for fire and smoke. For example, the wearable electronic device (200) may provide an emergency warning sound (e.g., "woo-woo") as a sound effect along with a voice prompt such as "Harmful gas detected. Ventilate immediately" for hazardous gas.

[0084] In one embodiment, the wearable electronic device (200) may provide a noise warning sound (e.g., "beep beep") along with a voice guidance such as "Excessive noise. Please wear earplugs" for excessive noise to avoid risks related to health hazards. For example, the wearable electronic device (200) may provide a health warning sound (e.g., "doo doo") along with a voice guidance such as "High heart rate. Please rest" based on heart rate and vital signs.

[0085] In one embodiment, the wearable electronic device (200) may guide the user to perform an action intent (e.g., opening a door, pressing a button) on the target object when the user's body part reaches the target object. The wearable electronic device (200) may determine whether the action intent has been successfully performed, determine whether additional work is required, and repeat the above-described process if necessary.

[0086] FIG. 9 is a diagram illustrating an example of a hierarchical operation structure of a wearable electronic device (200) according to one embodiment of the present disclosure.

[0087] According to one embodiment, a wearable electronic device (e.g., the wearable electronic device (200) of FIGS. 2A and 2B) can preprocess forward image information (e.g., text, images), environmental sounds, user commands (e.g., gestures, voice), and / or sensor data (e.g., IMU sensor data, GPS sensor data, depth sensor data) into input values ​​of a multimodal model through a multimodal input processing module (910).

[0088] According to one embodiment, the wearable electronic device (200) can perform a preprocessing operation based on a user behavior pattern on input values ​​generated by the multi-modal input processing module (910) through the prompt generation module (920), and generate a multi-modal prompt based thereon.

[0089] According to one embodiment, the wearable electronic device (200) analyzes input values ​​through a text recognition module, an audio recognition module, an image recognition module, a context recognition module and / or a user intent recognition module of a multimodal large language model (930) based on a multimodal prompt, thereby identifying an action intent based on a user action event as described above, and thereby causing a body part of the user to perform the action intent targeting at least a part of an object.

[0090] The context-aware module of the multimodal model (930) can generate more sophisticated and integrated responses through the interaction between various modalities (text, audio, and image recognition modules). It can process each type of information individually to understand its characteristics and context. The context-aware module can analyze text such as user-entered questions, conversation contexts, and descriptions using natural language processing (NLP) technology, and can analyze and extract information from current environmental information such as objects, colors, arrangements, and scenes in images using computer vision technology. The context-aware module can identify emotional states through sentiment analysis of audio that has been converted to text using speech recognition technology and then processed. The context-aware module can simultaneously analyze visual scene changes and audio information in videos containing dynamic information, including images and audio. In addition to the aforementioned, the context-aware module can supplement the context by processing various sensor data such as location, acceleration, and temperature. The context-aware module can independently process each input piece of information and then integrate it to understand the overall context. For example, the context-aware module can understand the context and content of text through semantic analysis, syntactic analysis, intent identification, and sentiment analysis. It can also identify connections between previously given text and the currently input text to maintain the flow of conversation and analyze continuous context by making appropriate connections. For example, the context-aware module can recognize key objects or people in an image, understand their interactions or situations, and analyze the context of the entire image to understand the events or environment occurring within the image. For example, the context-aware module can find correlations between images and text and determine how the image complements or reflects the text.

[0091] The user intent recognition module of the multimodal model (930) can identify the user's intent based on various types of input data (e.g., audio data including text, images, and voice) and generate an appropriate response accordingly. The user intent recognition module can analyze input coming from multiple modalities and infer the user's intent or request by combining the features of each modality. The input data can include text such as user commands, questions, and conversation content; visual information that indirectly indicates the user's intent; the way the user speaks (e.g., speed, intonation, and tone) and content; user gestures, behavioral data; and contextual data such as how the user interacts with the touchscreen or sensors. In addition, the input data can include success or failure records based on previous interactions, user contextual response data, user feedback on the input data, or reinforcement learning information. The user intent recognition module can infer the user's intent or request based on the input data. The user intent recognition module can process the input information from each modality and then synthesize it to ultimately infer the user's intent. For example, a user intent recognition module can identify the core intent of a user's command or question through the process of sentence structure analysis, intent classification, keyword extraction, and sentiment analysis from text. Furthermore, intent classification can classify the purpose of the user's question or command (e.g., requesting information, executing a command, expressing emotion). For example, a user intent recognition module can recognize specific objects in an image and infer user intent. Furthermore, it can analyze the overall context of the image and consider interactions or visual patterns within the image to determine the user's intent.For example, the user intent recognition module can convert voice data into text through automatic speech recognition (ASR) and combine it with natural language processing. Furthermore, it can identify the user's emotional state through voice intonation, tone, and speed, and use this to infer more accurate intent. For example, if the user's tone is angry or urgent, the user intent recognition module can infer that the user's request is more urgent. For example, the user intent recognition module can analyze the user's actions through action recognition to infer intent. For example, if the user reaches out to a specific object, the user intent recognition module can indicate interest or a need for that object. For example, the user intent recognition module can analyze events or scenes occurring in video data to identify the user's goals or situations, and can analyze the user's gestures or physical interactions on the screen.

[0092] The multi-modal model (930) comprehensively analyzes information processed in each module by combining it, and can cross-reference and supplement information from each modality, for example, by using cross-modal analysis and data fusion techniques. The user intent recognition module ultimately infers the user's intention based on the information obtained from each modality, determines what action the user requests or what information the user requests, and can reflect the user's emotional state, such as urgency, satisfaction, and dissatisfaction, in intent inference. It can also reflect contextual information, such as the user's previous input, current conversation context, and surrounding circumstances.

[0093] According to one embodiment, the wearable electronic device (200) may guide a user to perform an action intent by providing various information, such as three-dimensional spatial sound, augmented reality display, and / or haptics, through the output module (940) based on analysis of the multi-modal model (930).

[0094] FIG. 10 is a diagram for explaining examples of intent confirmation operations of a wearable electronic device (200) according to one embodiment of the present disclosure.

[0095] According to one embodiment, a wearable electronic device (e.g., the wearable electronic device (200) of FIGS. 2A and 2B) can identify an action intent upon confirming the occurrence of an action event.

[0096] According to one embodiment, the wearable electronic device (200) may extract an action intent from previously or currently acquired images and / or sensor information when an action event occurs. For example, the wearable electronic device (200) may identify user action information, object-related information, and / or contextual information from the acquired images and / or sensor information.

[0097] According to one embodiment, the wearable electronic device (200) may extract an action intent including an expected action of the user based on user behavior information and / or object-related information, targeting objects within a specified range, for example, a distance within which the user can reach out and reach, based on the current user location, i.e., the location of the wearable electronic device (200). For example, if a cutting board, a knife, and food ingredients are identified on a sink from the user's front image (1001), the action intent may be identified as cooking, or more specifically, chopping food ingredients with a knife. For example, if a door is identified from the user's front image (1002), the action intent may be identified as opening the door. For example, if items inside a shelf are identified from the user's front image (1003), the action intent may be identified as finding and taking out a specific item. For example, if a door and a doorknob are identified from the user's front image (1004), the action intent may be identified as grabbing the doorknob to open the door. For example, if stairs are identified from the user's front image (1005), the user's action intent may be identified as climbing or descending stairs.

[0098] FIG. 11 and FIG. 12 are drawings for explaining examples of risk factor identification operations of a wearable electronic device (200) according to one embodiment of the present disclosure.

[0099] According to one embodiment, a wearable electronic device (e.g., the wearable electronic device (200) of FIGS. 2A and 2B) can identify a target based on an action intent and identify at least a portion of an interaction point of the target and a risk factor based on the properties of the target. At least a portion of the target, i.e., an interaction target point or location, can be determined based on the action type of the action intent, the properties of the target, and the location and / or direction of the risk factor. For example, the interaction target location of the target can be set based on information such as the material of the target (e.g., metal, glass, plastic, fabric, liquid), the direction and / or shape in which the target is placed, the operation method of the target, situational information, and risk factors related to the target.

[0100] Referring to FIG. 11, if the target object is a knife, the knife's properties include plastic and metal parts, and in the case of the metal part, one end is formed as a sharp blade. The wearable electronic device (200) can identify a risk factor (e.g., a knife blade) of the target object of the action intent and identify the interaction target point or location of the object (e.g., a knife handle).

[0101] The wearable electronic device (200) can check the spatial information of the risk factors of the target object. In a situation where a knife on a cutting board is determined to be the target object based on the spatial information, and the handle of the knife is positioned in a left / right or reverse direction rather than in the user's forward direction, the wearable electronic device (200) can determine at least a portion of the target object as an interaction point by considering the direction of use and the risk factors so as to safely target the target object (e.g., a direction in which the handle can be touched first while avoiding the blade).

[0102] In Fig. 11, it can be seen that the blade of the knife (1101) is pointed to the left, the blade of the knife (1102) is pointed to the lower right, and the blade of the knife (1103) is pointed to the lower left. Accordingly, the wearable electronic device (200) can select a path that avoids the direction in which the blade is pointed when determining a path toward the knife handle, which is the target point of the object.

[0103] Referring to FIG. 12, if the target is a pot boiling on a gas range, the handle (1201, 1202), which is part of the pot, can become the target area of ​​the target. In contrast, if the pot has been placed on the sink for a certain period of time before the user's action event occurs, the lid (1213) area can also become the target area of ​​the target, as it is confirmed that there is no risk factor.

[0104] FIG. 13 and FIG. 14 are drawings for explaining examples of a path confirmation operation of a wearable electronic device (200) according to one embodiment of the present disclosure.

[0105] Referring to FIGS. 13 and 14, a wearable electronic device (e.g., the wearable electronic device (200) of FIGS. 2A and 2B) can identify a target by searching for objects that are recognized as pointed at by, for example, a user's body part (e.g., a hand) moving while hovering in close proximity when multiple objects of similar or identical types are searched. For example, when the user's hand remains in close proximity to a specific object for a certain period of time, the wearable electronic device (200) can provide information about the object to the user as voice information.

[0106] When multiple objects of the same type are detected, the wearable electronic device (200) may provide additional information for differentiation. For example, location information such as the upper left pot (1341), the left middle pot (1342), the lower left pot (1342), and the upper right pot (1302) in FIG. 13 may be provided. For example, to distinguish between similar or identical types of objects, the wearable electronic device (200) may register voice tags for the recognized objects.

[0107] The wearable electronic device (200) can identify at least a portion of an area that is an interaction point (e.g., a handle (1311, 1313) or an upper area (1312)) between a user's body part (1301) and the object (1302), when the object is determined to be the upper right pot (1302) among the objects.

[0108] The wearable electronic device (200) can identify at least a portion of a target object of an action intent, i.e., an interaction target point or location of the object and a user's interaction body part (e.g., a hand (1301)), calculate a spatial location and distance between the body part and a portion of the object, and generate one or more virtual movement paths (1321, 1322, 1323). When two or more virtual movement paths are generated, the wearable electronic device (200) can select a final movement path by considering the operation method and / or risk factors of the object. For example, the final path can be selected based on criteria such as user preference, habit, and shortest distance for two or more paths. For example, if a path (1323) from the user's hand (1301) to the handle (1313) is selected as the optimal path, the wearable electronic device (200) may generate a virtual tunnel (1410) based on the path (1323), as illustrated in FIG. 14, and provide a guide so that the user's hand approaches a selected area (1313) of the target through the virtual tunnel (1410).

[0109] FIG. 15 is a diagram illustrating examples of a user interface according to the operation of a wearable electronic device (200) according to one embodiment of the present disclosure.

[0110] According to one embodiment, a hand (1501) and / or a current gaze direction (1503) of a user wearing a wearable electronic device (e.g., the wearable electronic device (200) of FIGS. 2A and 2B) may not be directed toward a location of a target (1502). The wearable electronic device (200) may generate a three-dimensional stereoscopic sound to guide a direction corresponding to where the target is located so that the hand (1501) and / or the current gaze direction (1503) of the user may be directed toward the location of the target (1502). Accordingly, the hand (1531) and / or the gaze direction (1533) of the user may be turned toward the location of the target.

[0111] According to one embodiment, the wearable electronic device (200) can detect the direction of the user's gaze and / or the direction and / or position of the current hand through an image sensor, and calculate a direction change angle and / or distance by comparing it with the position of the target.

[0112] The wearable electronic device (200) can generate three-dimensional stereoscopic sound by adjusting the output sound levels of, for example, a left sound (1511) generated through a left speaker and a right sound (1512) generated through a right speaker in response to a direction change angle and / or distance toward a location and / or direction of a target (1502) based on the user's gaze direction (1503) and / or the direction and / or position of the current hand (1501), and guide the user's hand to target the target. For example, the wearable electronic device (200) can adjust the volume of the left sound (1511) to be larger in proportion to the change angle compared to the volume of the right sound (1512). For example, the wearable electronic device (200) can be adjusted so that the difference between the volume of the left sound (1511) and the volume of the right sound (1512) becomes smaller as the switching angle becomes smaller by the user moving the direction of the gaze and / or the direction and / or position of the hand.

[0113] According to one embodiment, the wearable electronic device (200) can adjust the volume of the left sound (1541) and the volume of the right sound (1542) to be the same as the user's hand (1531) and / or gaze direction (1533) changes to the position and / or direction of the object (1502).

[0114] According to one embodiment, the wearable electronic device (200) may provide a sound that becomes louder or sharper as the direction in which the object is located matches the forward direction of the user, as a sound for guiding direction recognition, in order to provide a search guide for an object based on the forward direction of the user.

[0115] According to one embodiment, the wearable electronic device (200) may guide the user's hand along a path toward the target point of the object by changing the sound to reflect the change in distance along the X, Y, and Z axes from the target point of the object when the user's hand moves along the X, Y, and Z axes, thereby allowing the user to recognize the change in distance along the X, Y, and Z axes from the target point of the object according to the movement of the hand. For example, the wearable electronic device (200) may provide a first sound having a positive tone when the user's hand is moving along the path toward the target point of the object. For example, the wearable electronic device (200) may provide a second sound having a negative tone when the user's hand is deviating from the path toward the target point of the object. For example, the wearable electronic device (200) can directly guide the name, location, direction, operation or usage method, and risk factors of an object by providing a third sound including a voice (e.g., “Door handle is on the left, please push the door forward”).

[0116] FIG. 16, FIG. 17, FIG. 18, FIG. 19, and FIG. 20 are drawings for explaining an example of operation of a wearable electronic device (200) to which one embodiment of the present disclosure is applied.

[0117] According to one embodiment, a user's action event and action intent may be in a series or may include a plurality of action events and corresponding action intents.

[0118] According to one embodiment, a wearable electronic device (e.g., wearable electronic device (200) of FIGS. 2A and 2B) may include an image sensor (e.g., camera module (180) of FIG. 1, external sensor (210) or eye recognition sensor (220) of FIG. 2A, FIG. 2B, or FIG. 3), a speaker (e.g., audio output module (155) of FIG. 1, speaker (241) of FIG. 2A, FIG. 2B, or FIG. 3), a memory (e.g., memory (130) of FIG. 1, multi-modal model (270) of FIG. 2A, FIG. 2B, or FIG. 3), and / or at least one processor (e.g., processor (120) of FIG. 1, data processing transmit / receive module (260) of FIG. 2A, FIG. 2B, or FIG. 3). The wearable electronic device (200) can acquire images through an external sensor (210) including an image sensor, for example, one or more depth cameras (211), one or more side cameras (212), one or more upward or downward cameras (213), and / or one or more IR sensors (infra-red) (214) of FIG. 2A, FIG. 2B, or FIG. 3. The wearable electronic device (200) can acquire sensor information, such as current location and user movement information, through a location recognition sensor (230) including an IMU (inertial measurement unit) sensor (231) of FIG. 2A, FIG. 2B, or FIG. 3, and / or a GPS or UWB (ultra wide band) sensor (232).

[0119] Referring to FIG. 16, the wearable electronic device (200) can detect a user's motion, for example, a motion of extending a hand (1601), through an image sensor and determine that an action event has occurred.

[0120] According to one embodiment, the wearable electronic device (200) can identify an action intent upon confirming the occurrence of an action event. For example, the wearable electronic device (200) can extract an action intent from acquired images and / or sensor information. For example, the wearable electronic device (200) can identify user action information, object-related information, and / or contextual information from acquired images and / or sensor information. For example, the wearable electronic device (200) can extract an action intent of boiling ramen based on user action information, object-related information, and contextual information of reaching out toward ramen placed on the sink in front of the sink. Accordingly, the wearable electronic device (200) can provide an acoustic guide so that the user can grab the ramen. For example, the wearable electronic device (200) may provide guidance such as “Move in the direction of the ramen bag” as a voice guide and provide a soft warning sound (e.g., “ding ding”) in the direction of the ramen bag as a 3D sound effect. For example, the wearable electronic device (200) may recognize the user’s hand (1601), detect the distance between the hand and the ramen bag, and adjust the sound so that the sound becomes louder as the hand gets closer to the ramen bag, or provide a sound as a voice guide such as “You are getting closer to the ramen bag. Reach out and pick it up.”

[0121] Referring to FIG. 17, the wearable electronic device (200) can detect a knife (1701), a green onion (1702) placed on a cutting board in front of a sink, and a pot (1703) placed on a range (1704) from the user's front image.

[0122] According to one embodiment, the wearable electronic device (200) may predict the next action intent for objects detected from front images by referring to contextual information, such as recipe information for boiling ramen, based on the action intent of boiling ramen. For example, the wearable electronic device (200) may provide an acoustic guide to guide the user's hand to the location of a knife (1701) and a green onion (1702). For example, if the wearable electronic device (200) extracts "cut green onion" and "boil water in a pot" as multiple action intents, it may recommend an appropriate action intent based on contextual information, such as a recipe, or receive a command (e.g., green onion) through user feedback (e.g., "Would you like to chop green onion or boil water in a pot?").

[0123] According to one embodiment, in order to perform an action intent of preparing a pot and water, the wearable electronic device (200) may provide a voice guide such as “Find the pot and place it on the kitchen counter” and a sound (e.g., “Tantan”) indicating the direction of the pot as a 3D sound effect. For example, the wearable electronic device (200) may recognize the user’s hand and detect the distance between the hand and the pot, and may change the sound so that the sound gets louder as the hand gets closer to the pot. In addition, the wearable electronic device (200) may provide a voice guide such as “You are getting closer to the pot. Reach out and pick it up.” For example, the wearable electronic device (200) may additionally provide a guidance sound such as “The pot is on the counter. Add water” or “Be careful not to overflow when adding water.”

[0124] In one embodiment, to perform an action intent of boiling water, the wearable electronic device (200) may provide a voice guide "Turn on the gas range" to instruct turning on the gas range, and may provide a sound (e.g., "doo doo") generated in the direction of the gas range as a three-dimensional sound effect. For example, the wearable electronic device (200) may recognize the user's hand and detect the distance between the hand and the pot, and may change the sound so that the sound becomes louder as the hand gets closer to the gas range handle. Additionally, the wearable electronic device (200) may provide a sound as a voice guide, "You have come close to the gas range handle. Turn it to turn it on."

[0125] In one embodiment, to perform an action intent of chopping a wave, the wearable electronic device (200) may provide a sound such as "Find the wave and place it on the chopping board" as a voice guide, and may generate a soft sound (e.g., "ding ding") at the location of the wave as a 3D sound effect. For example, the wearable electronic device (200) may recognize the user's hand, detect the distance between the hand and the wave, and generate a sound change so that the sound becomes louder as the hand gets closer to the wave. For example, the wearable electronic device (200) may provide a sound such as "You are getting close to the wave. Reach out and pick it up" as a voice guide. Additionally, the wearable electronic device (200) may provide a voice guidance such as "Place the wave on the chopping board and use the knife."

[0126] Referring to FIG. 18, the wearable electronic device can identify an action intent and determine the properties of the user's body part, the hand (1801), and the target object, the knife (1802), for executing the action intent. The wearable electronic device can identify the position and direction of the handle and blade of the target object, the knife (1802), from the front image, and guide the movement of the hand to avoid a risk factor in which the hand approaches the blade.

[0127] Referring to FIG. 18, the wearable electronic device (200) may provide a positive sound when it is determined that the user's hand (1801) is approaching the knife in a direction opposite to the direction of the blade.

[0128] Referring to FIG. 19, the wearable electronic device (200) may provide a negative sound when it is determined that the user's hand (1901) is approaching the knife in the direction of the blade.

[0129] The wearable electronic device (200) may provide a sound guide such as "When holding a knife, hold the handle." Additionally, the wearable electronic device (200) may provide a soft sound (e.g., "Ttararan") at the position or direction of the knife handle as a three-dimensional sound effect.

[0130] Referring to FIG. 20, the wearable electronic device (200) may provide a sound indicating the position or direction of the knife handle, along with a corresponding sound among positive and negative sounds, so that the user's hand (2001) approaches the knife handle (2002) rather than the blade. For example, the wearable electronic device (200) may indicate that the user is getting closer to an object by making the sound louder as the user's hand approaches the knife handle. The wearable electronic device (200) may provide a sound as a voice guide, such as "You are getting closer to the handle. Hold it carefully," and may additionally provide a sound as a voice guide, such as "Be careful not to hurt your fingers when cutting green onions."

[0131] The wearable electronic device (200) can check the boiling state of water and provide, for example, a voice guidance sound such as "When the water starts to boil, add ramen", and additionally provide a soft sound (e.g., "boom boom boom") along with the sound of water boiling as a 3D sound effect. Additionally, the wearable electronic device (200) can provide a voice guidance such as "The water is boiling. Add ramen and check the cooking time."

[0132] The wearable electronic device (200) can provide a voice guidance saying, "Please add the ramen and soup," and a three-dimensional sound effect making a soft positive sound (e.g., "ding ding") in the direction of the ramen and soup. The wearable electronic device (200) can recognize the hand to detect the distance between the hand and the ramen bag and the soup, and change the sound according to the distance so that the sound becomes louder as the hand gets closer to the ramen bag and the soup. For example, the wearable electronic device (200) can provide a voice guidance saying, "You are close to the ramen bag and the soup. Reach out and pick it up," and additionally a voice saying, "Carefully put the ramen and the soup into the pot."

[0133] The wearable electronic device (200) can then provide a soft sound (e.g., "tta-ra-ran") at the location of the wave with a 3D sound effect and a sound such as "put the sliced ​​green onion in" as a voice guide. The wearable electronic device (200) can recognize the hand to detect the distance between the hand and the wave, and change the sound according to the distance so that the sound becomes louder as the hand gets closer to the wave. The wearable electronic device (200) can provide a sound such as "You are getting close to the sliced ​​green onion. Reach out and pick it up" as a voice guide, and additionally provide a voice such as "Carefully put the sliced ​​green onion in the pot."

[0134] Thereafter, the wearable electronic device (200) can check the elapsed time or the status of the ramen based on the recipe. Based on the check result, if the wearable electronic device (200) confirms that the ramen is finished boiling, it can check the fire-off action intent and provide a sound such as "The ramen is finished boiling. Turn off the fire" as a voice guide and provide a positive sound (e.g., "dududu") generated from the direction of the gas range as a 3D sound effect. The wearable electronic device (200) can recognize the hand to detect the distance between the hand and the gas range handle and change the sound according to the distance so that the sound becomes louder as the hand gets closer to the gas range handle. The wearable electronic device (200) can provide a sound such as "You have gotten close to the gas range handle. Turn it to turn off the fire" as a voice guide.

[0135] The wearable electronic device (200) can then confirm the hot pot lifting action intent and provide a voice guide saying "Lift hot pot carefully" at a volume level, and additionally provide guidance on the risk factors through a voice saying "Put on kitchen gloves before handling the pot." The wearable electronic device (200) can provide a voice guide guiding the user to put on gloves and a voice guide guiding the user to lift the pot. In addition, the wearable electronic device (200) can provide a positive sound (e.g., "ding ding") generated at the glove position or a positive sound (e.g., "beep beep") generated at the pot handle position as a 3D sound effect. For example, the wearable electronic device (200) can adjust the sound to become louder as the user's hand gets closer to the glove position or the pot handle, and can provide a voice guide saying "You are getting closer to the glove. Put on your gloves by reaching out" or "You are getting closer to the pot handle. Lift with both hands." The wearable electronic device (200) may provide a 3D sound effect along with a voice saying “Move towards the table” to guide the user to perform the action intent of bringing ramen to the table, or may additionally provide a voice saying “When you get close to the table, slowly put the pot down.”

[0136] A wearable electronic device (200) can guide a user to safely perform an action intent that the user wishes to perform by avoiding risk factors by utilizing a three-dimensional sound including a plurality of sounds including a first sound and a second sound and a voice guide.

[0137] A wearable electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, or wearable electronic device (200) of FIG. 2A or 2B) may include an image sensor (e.g., external sensor (210) of FIG. 2A or 2B and / or eye recognition sensor (220)), a speaker (e.g., audio output module (155) of FIG. 1, speaker (241) of FIG. 2A or 2B), a memory for storing instructions (e.g., memory (130) of FIG. 1), and at least one processor including processing circuitry (e.g., processor (120) of FIG. 1). The instructions may store instructions that, when individually or collectively executed by the at least one processor, cause the wearable electronic device to identify an action event of a user, analyze an action intent based on the action event, identify an object of the action event and an attribute of the object based on the action intent, identify a body part of the user corresponding to the action intent and at least a portion of the object in response to the attribute of the object, identify spatial information based on the attribute of the object, the body part, and at least a portion of the object, set a path for the body part to reach at least a portion of the object based on the spatial information, and provide sound information including a first sound and a second sound that guide the performance of the action event through the speaker based on the path.

[0138] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to provide the first sound when the body part approaches the path and to provide the second sound when the body part departs from the path.

[0139] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to generate the first sound provided when the body part approaches the path or the second sound provided when the body part departs from the path according to adjustments in volume, frequency, peak, waveform or output interval of the sound.

[0140] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to generate the first sound to indicate a relative direction of the object from the body part.

[0141] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to generate the first sound by adjusting the volume or frequency of the sound as the body part approaches the object.

[0142] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to identify a risk factor according to an attribute of the object and to provide a third sound by adjusting the volume or frequency of the sound when the body part is in proximity to the risk factor.

[0143] In one embodiment, the third sound may include voice guidance.

[0144] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to extract the action intent based on user action information, voice input, or information about detected image objects in response to the action event.

[0145] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to determine the spatial information based on information about image objects detected in response to the action event.

[0146] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable electronic device to analyze the action intent through user input if the action intent analysis fails.

[0147] According to one embodiment, a method of a wearable electronic device having an image sensor may include an operation of confirming an action event of a user, an operation of analyzing an action intent based on the action event, an operation of confirming an object of the action event and an attribute of the object based on the action intent, an operation of confirming a body part of the user corresponding to the action intent and at least a portion of the object in response to the attribute of the object, an operation of confirming spatial information based on the attribute of the object, the body part, and at least a portion of the object, an operation of setting a path for the body part to reach at least a portion of the object based on the spatial information, and an operation of providing sound information including a first sound and a second sound that guides performance of the action event based on the path through the speaker.

[0148] According to one embodiment, a method of a wearable electronic device having an image sensor may further include providing the first sound when the body part approaches the path and providing the second sound when the body part deviates from the path.

[0149] According to one embodiment, a method of a wearable electronic device having an image sensor may further include an operation of generating the first sound provided when the body part approaches the path or the second sound provided when the body part deviates from the path according to adjustment of a volume, a frequency, a peak, a waveform of a sound wave, or an output interval of the sound.

[0150] According to one embodiment, a method of a wearable electronic device having an image sensor may further include generating the first sound to indicate a relative direction of the object from the body part.

[0151] According to one embodiment, a method of a wearable electronic device having an image sensor may further include an operation of generating the first sound by adjusting the volume or frequency of the sound as the body part approaches the object.

[0152] According to one embodiment, a method of a wearable electronic device having an image sensor may further include an operation of identifying a risk factor according to an attribute of the object, and providing a third sound by adjusting the volume or frequency of the sound when the body part approaches the risk factor.

[0153] In one embodiment, the third sound may include voice guidance.

[0154] According to one embodiment, a method of a wearable electronic device having an image sensor may further include an operation of extracting the action intent based on user action information, voice input, or information about detected image objects in response to the action event.

[0155] According to one embodiment, a method of a wearable electronic device having an image sensor may further include an operation of verifying the spatial information based on information about image objects detected according to the action event.

[0156] According to one embodiment, a method of a wearable electronic device having an image sensor may further include an operation of analyzing the action intent through a user input when the action intent analysis fails.

[0157] The embodiments disclosed in this document are merely examples presented to facilitate easy explanation and understanding of the technical content, and are not intended to limit the scope of the technology disclosed in this document. Therefore, the scope of the technology disclosed in this document should be interpreted to include all modifications or variations derived based on the technical concepts of the various embodiments disclosed in this document, in addition to the embodiments disclosed herein.

[0158] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable electronic devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0159] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0160] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0161] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0162] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0163] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In wearable electronic devices, image sensor; speaker; memory that stores instructions; and At least one processor comprising processing circuitry, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable electronic device to: Check the user's action events, Based on the above action event, the action intent is analyzed, Based on the above action intent, the object of the action event and the properties of the object are checked, Identifying the user's body part corresponding to the action intent and at least a portion of the object corresponding to the properties of the object, Identify spatial information based on the properties of the object, the body part, and at least a portion of the object, Based on the above spatial information, a path is set for the body part to reach at least a portion of the target area, and A wearable electronic device that provides sound information including a first sound and a second sound that guide the performance of the action event through the speaker based on the above path.

2. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable electronic device to: When the body part approaches the path, the first sound is provided, and A wearable electronic device configured to provide the second sound when the body part deviates from the path.

3. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that generates the first sound provided when the body part approaches the path or the second sound provided when the body part departs from the path according to adjustment of the volume, frequency, peak, waveform of the sound wave, or output interval of the sound.

4. In paragraph 3, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that generates the first sound to indicate the relative direction of the object from the body part.

5. In paragraph 3, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that generates the first sound by adjusting the volume or frequency of the sound as the body part approaches the target.

6. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable electronic device to: Identify risk factors according to the properties of the above objects, and When the above body part is close to the above risk factor, the volume or frequency of the sound is adjusted to provide a third sound, The third sound is a wearable electronic device including voice guidance.

7. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that extracts the action intent based on information about the user's behavior, voice input, or detected image objects according to the above action event.

8. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that verifies spatial information based on information about image objects detected according to the above action event.

9. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that analyzes the action intent through user input when the above action intent fails to be analyzed.

10. A method for a wearable electronic device having an image sensor, Action to check user action events; An action of analyzing an action intent based on the above action event; An action of checking the object of the action event and the properties of the object based on the action intent; An action of identifying a body part of the user corresponding to the action intent and at least a portion of the object in response to the properties of the object; An operation of identifying spatial information based on the properties of the object, the body part, or at least a portion of the object; An operation of setting a path for the body part to reach at least a portion of the target area based on the spatial information; and A method comprising an action of providing sound information including a first sound and a second sound guiding the performance of the action event based on the path through a speaker.

11. In paragraph 10, A method further comprising providing the first sound when the body part approaches the path and providing the second sound when the body part deviates from the path.

12. In paragraph 10, An action of generating the first sound provided when the body part approaches the path or the second sound provided when the body part departs from the path according to adjustment of the volume, frequency, peak, waveform or output interval of the sound; An action of generating the first sound to indicate the relative direction of the object from the body part; or A method further comprising an action of generating the first sound by adjusting the volume or frequency of the sound as the body part approaches the target.

13. In paragraph 10, Further comprising an action of checking risk factors according to the properties of the above object and providing a third sound by adjusting the volume or frequency of the sound when the body part is close to the risk factor. The third sound is a method including voice guidance.

14. In paragraph 10, An operation of extracting the action intent based on the user's action information, voice input, or information about detected image objects according to the above action event; and A method further comprising an action of verifying the spatial information based on information about image objects detected according to the above action event.

15. In paragraph 10, A method further comprising an action of analyzing the action intent through user input if analyzing the action intent fails.

Images