Wearable electronic device for recognizing object, and control method thereof

Abstract

A wearable electronic device is provided. The wearable electronic device comprises: at least one sensor; a display; at least one processor comprising the at least one sensor and a processing circuit; and a memory storing instructions which, when executed by the at least one processor, cause the wearable electronic device to: detect a plurality of objects included within a first field of view (FOV) of the wearable electronic device, the plurality of objects being detectable through the at least one sensor of the wearable electronic device; recognize the type of a first object among the plurality of objects; identify whether the first object is a real object or a virtual object; determine a first risk level of the first object on the basis of the recognized type of the first object and whether the first object is the real object or the virtual object; determine a first distance related to the first object on the basis of the first risk level of the first object; and on the basis of the distance between the first object and the user's body being less than or equal to the first distance, provide feedback on the basis of the first risk level of the first object.

Description

Wearable electronic device for recognizing objects and method for controlling the same

[0001] The present disclosure relates to an electronic device for recognizing an object and a method for controlling the same.

[0002] The variety of services and additional features offered through electronic devices, such as smartphones, is steadily increasing. To enhance the utility of these devices and satisfy the diverse needs of users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices to offer a variety of features and differentiate themselves from competitors. Consequently, the various functions offered through electronic devices are also becoming increasingly sophisticated.

[0003] The variety of services and additional features offered through wearable electronic devices, such as augmented reality glasses (AR glasses), virtual reality glasses (VR glasses), and head-mounted displays (HMDs), is steadily increasing. To enhance the utility of these devices and satisfy the needs of diverse users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices that offer diverse features and differentiate themselves from competitors. Consequently, the various functions offered through wearable electronic devices are also becoming increasingly sophisticated.

[0004] AR glasses or VR glasses, worn on the user's head, can provide a realistic experience by displaying virtual images. AR glasses or VR glasses can replace smartphones in various fields, such as gaming, entertainment, education, and social networking services (SNS). Users can access smartphone content and / or content similar to reality through the AR glasses or VR glasses worn on their head.

[0005] The above information is provided solely as background information to aid in understanding the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0006] Aspects of the present disclosure address at least the problems and / or disadvantages mentioned above, and provide at least the advantages described below. Accordingly, various embodiments of the present disclosure provide an electronic device for recognizing an object and a method for controlling the same.

[0007] Additional aspects will be partly explained in the following description, partly will be obvious from the description, or may be learned by practice of the presented embodiments.

[0008] According to one aspect of the present disclosure, a wearable electronic device is provided. The wearable electronic device includes at least one sensor, at least one processor including a processing circuit, and a memory storing instructions.

[0009] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to detect a plurality of objects of the wearable electronic device detectable via the at least one sensor.

[0010] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to recognize a type of a first object among the plurality of objects.

[0011] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to determine whether the first object is a real object or a virtual object.

[0012] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to determine a first risk level of the first object based on the recognized type of the first object and whether the first object is the real object or the virtual object.

[0013] In one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to determine a first distance associated with the first object based on a first risk level of the first object.

[0014] According to one embodiment, the instructions, when executed by the at least one processor, may cause the wearable electronic device to provide feedback based on a first risk level of the first object based on a distance between the first object and the user's body being less than or equal to the first distance.

[0015] According to one aspect of the present disclosure, a method for controlling a wearable electronic device is provided. The method for controlling a wearable electronic device may include an operation of detecting a plurality of objects of the wearable electronic device that are detectable through at least one sensor of the wearable electronic device or an external electronic device.

[0016] According to one embodiment, a method for controlling a wearable electronic device may include an operation of recognizing a type of a first object among the plurality of objects.

[0017] According to one embodiment, a method for controlling a wearable electronic device may include an operation of determining whether the first object is a real object or a virtual object.

[0018] According to one embodiment, a control method of a wearable electronic device may include an operation of determining a first risk level of the first object based on the recognized type of the first object and whether the first object is the real object or the virtual object.

[0019] According to one embodiment, a method of controlling a wearable electronic device may include an operation of determining a first distance related to the first object based on a first risk level of the first object.

[0020] According to one embodiment, a method for controlling a wearable electronic device may include providing feedback based on a first risk level of the first object based on a distance between the first object and the user's body being less than or equal to the first distance.

[0021] According to one aspect of the present disclosure, one or more non-transitory computer-readable storage media are provided, which store computer-executable instructions that, when executed by a wearable electronic device, cause the wearable electronic device to perform operations. The operations may include detecting a plurality of objects of the wearable electronic device that are detectable through at least one sensor of the electronic device or an external electronic device.

[0022] In one embodiment, the operations may include recognizing a type of a first object among the plurality of objects.

[0023] In one embodiment, the operations may include an operation of the electronic device determining whether the first object is a real object or a virtual object.

[0024] In one embodiment, the operations may include determining a first risk level of the first object based on the recognized type of the first object and whether the first object is the real object or a virtual object.

[0025] In one embodiment, the actions may include determining a first distance associated with the first object based on a first risk of the first object.

[0026] In one embodiment, the actions may include providing feedback based on a first risk level of the first object based on a distance between the first object and the user's body being less than or equal to the first distance.

[0027] Other aspects, advantages and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments of the present disclosure taken in conjunction with the accompanying drawings.

[0028] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0029] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0030] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0031] FIG. 3 is a drawing showing the front of a wearable electronic device according to one embodiment.

[0032] FIG. 4 is a drawing showing the back of a wearable electronic device according to one embodiment.

[0033] FIG. 5 is another perspective view of a wearable electronic device according to one embodiment.

[0034] FIG. 6 is a drawing schematically illustrating an object recognition operation of an electronic device according to one embodiment.

[0035] FIG. 7 is a drawing schematically illustrating the configuration of an electronic device according to one embodiment.

[0036] FIG. 8 is a flowchart illustrating an operation of recognizing an object in an electronic device and providing feedback related to the object, according to one embodiment.

[0037] FIG. 9 is a drawing for explaining the recognition range of a real object and a virtual object according to one embodiment.

[0038] FIG. 10 is a diagram for explaining an operation of identifying the risk of an object of an electronic device according to one embodiment.

[0039] FIG. 11 is a diagram illustrating a boundary area based on the risk level of an object according to one embodiment.

[0040] FIG. 12 is a flowchart for explaining an operation of changing a boundary area according to the characteristics of an object, according to one embodiment.

[0041] FIG. 13 is a drawing for explaining a boundary area changed according to the characteristics of an object according to one embodiment.

[0042] FIG. 14A is a diagram for explaining a feedback operation of an electronic device when a user's body approaches an object according to one embodiment.

[0043] FIG. 14b is a diagram for explaining a feedback operation of an electronic device when a user's body approaches an object according to one embodiment.

[0044] FIG. 15 is a flowchart illustrating an operation of changing a boundary area according to a user's approach speed according to one embodiment.

[0045] FIG. 16 is a drawing for explaining a boundary area changed according to a user's approach speed according to one embodiment.

[0046] FIG. 17a is a drawing for explaining boundary areas of a real object and a virtual object based on the real object according to one embodiment.

[0047] FIG. 17b is a drawing for explaining boundary areas of a real object and a virtual object based on the real object according to one embodiment.

[0048] The same reference numbers are used to indicate identical components throughout the drawing.

[0049] The following description, with reference to the attached drawings, is provided to facilitate a comprehensive understanding of various embodiments of the present disclosure as defined by the claims and their equivalents. While it includes numerous specific details to aid understanding, these should be considered merely illustrative. Accordingly, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. Furthermore, descriptions of known functions and configurations may be omitted for clarity and brevity.

[0050] The terms and words used in the following description and claims are not to be construed as limited to their bibliographic meanings, but rather have been used by the inventors solely to facilitate a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the present invention as defined by the appended claims and their equivalents.

[0051] The singular forms "a," "an," and "the" should be understood to include plural referents unless the context clearly indicates otherwise. Thus, for example, a reference to "a surface of a part" includes a reference to one or more such surfaces.

[0052] It should be understood that each block of the flowchart and the combination of flowcharts can be performed by one or more computer programs containing computer-executable instructions. The one or more computer programs may be stored entirely in a single memory device, or the one or more computer programs may be divided into different parts stored in multiple different memory devices.

[0053] Any function or operation described in the present disclosure may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing, and may include an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (e.g., a GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a wireless-fidelity (Wi-Fi) chip, or a Bluetooth chip. TM It includes circuits such as chips, Global Positioning System (GPS) chips, Near Field Communication (NFC) chips, connectivity chips, sensor controllers, touch controllers, fingerprint sensor controllers, display driver integrated circuits (ICs), audio codec chips, universal serial bus (USB) controllers, camera controllers, image processing ICs, microprocessor units (MPUs), systems on chips (SoCs), and ICs.

[0054] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0055] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0056] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0057] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0058] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0059] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0060] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0061] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0062] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0063] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0064] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0065] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0066] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0067] The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0068] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0069] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0070] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0071] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0072] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0073] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0074] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0075] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0076] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0077] Referring to FIG. 2, a wearable electronic device (200) according to one embodiment may include at least one of a light output module (211), a display member (201), and a camera module (250).

[0078] According to one embodiment, the light output module (211) may include a light source capable of outputting an image and a lens for guiding the image to the display member (201). According to one embodiment, the light output module (211) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0079] According to one embodiment, the display member (201) may include an optical waveguide (e.g., a waveguide). According to one embodiment, an output image of the optical output module (211) incident on one end of the optical waveguide may be propagated inside the optical waveguide and provided to a user. According to one embodiment, the optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide the output image of the optical output module (211) to the user's eyes by using at least one diffractive element or reflective element.

[0080] According to one embodiment, the camera module (250) can capture still images and / or moving images. According to one embodiment, the camera module (250) is disposed within the lens frame and can be disposed around the display member (201).

[0081] According to one embodiment, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. According to one embodiment, the first camera module (251) can periodically or aperiodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0082] According to one embodiment, the second camera module (253) can capture an external image.

[0083] According to one embodiment, the third camera module (255) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. According to one embodiment, the third camera module (255) can be used for 3 degrees of freedom (3DoF), 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. According to one embodiment, the second camera module (253) can also be used for hand detection and tracking, and user gesture recognition. According to one embodiment, at least one of the first camera module (251) to the third camera module (255) can be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module can include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0084] FIG. 3 is a drawing showing the front of a wearable electronic device according to one embodiment.

[0085] FIG. 4 is a drawing showing the back of a wearable electronic device according to one embodiment.

[0086] Referring to FIGS. 3 and 4, in one embodiment, camera modules (311, 312, 313, 314, 315, 316) and / or depth sensors (317) for obtaining information related to the surrounding environment of the wearable electronic device (300) may be arranged on the first surface (310) of the housing.

[0087] In one embodiment, the camera modules (311, 312) can acquire images related to the environment surrounding the wearable electronic device.

[0088] In one embodiment, the camera modules (313, 314, 315, 316) can acquire images while the wearable electronic device is worn by a user. The camera modules (313, 314, 315, 316) can be used for hand detection, tracking, and recognition of user gestures (e.g., hand movements). The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. In one embodiment, the camera modules (311, 312) can also be used for hand detection and tracking, and user gestures.

[0089] In one embodiment, a depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (217), camera modules (213, 214, 215, 216) may determine the distance to an object.

[0090] According to one embodiment, a camera module (325, 326) for facial recognition and / or a display (321) (and / or a lens) may be arranged on the second side (320) of the housing.

[0091] In one embodiment, a face recognition camera module (325, 326) adjacent to the display may be used to recognize a user's face, or may recognize and / or track both eyes of the user.

[0092] In one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). In one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among the plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIGS. 3 and 4, the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2.

[0093] As described above, according to one embodiment, the wearable electronic device (300) may have a form factor for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured on a body part of the user. The wearable electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0094] FIG. 5 is another perspective view of an electronic device according to one embodiment.

[0095] Referring to FIG. 5, the electronic device (400) may be a head-mounted device (HMD) capable of providing images in front of the user's eyes. The configuration of the electronic device (400) of FIG. 5 may be entirely or partially identical to the configuration of the wearable electronic device (200) of FIG. 2.

[0096] According to one embodiment, the electronic device (400) may include a first housing (410), a second housing (420), and a third housing (430) that may form an exterior of the electronic device (400) and provide a space in which components of the electronic device (400) may be placed.

[0097] According to one embodiment, the electronic device (400) may include a first housing (410) capable of surrounding at least a portion of a user's head. According to one embodiment, the first housing (410) may include a first side (400a) facing the exterior of the electronic device (400) (e.g., in the +Z direction).

[0098] According to one embodiment, the first housing (410) can surround at least a portion of the internal space (I). For example, the first housing (410) can include a second side (400b) facing the internal space (I) of the electronic device (400) and a third side (400c) opposite the second side (400b). According to one embodiment, the first housing (410) can be combined with the third housing (430) to form a closed curve shape surrounding the internal space (I).

[0099] According to one embodiment, the first housing (410) can accommodate at least some of the components of the electronic device (400). For example, an optical output module and a circuit board can be placed within the first housing (410).

[0100] According to one embodiment, the electronic device (400) may include one display member (440) corresponding to the left and right eyes. The display member (440) may be disposed in the first housing (410). The configuration of the display member (440) of FIG. 5 may be all or part of the same as the configuration of the display member (201) of FIG. 2.

[0101] According to one embodiment, the electronic device (400) may include a second housing (420) that can be mounted on a user's face. According to one embodiment, the second housing (420) may include a fourth surface (400d) that can at least partially face the user's face. According to one embodiment, the fourth surface (400d) may be a surface facing the internal space (I) of the electronic device (400) (e.g., -Z direction). According to one embodiment, the second housing (420) may be coupled with the first housing (410).

[0102] According to one embodiment, the electronic device (400) may include a third housing (430) that can be mounted on the back of the user's head. According to one embodiment, the third housing (430) may be coupled to the first housing (410). According to one embodiment, the third housing (430) may accommodate at least some of the components of the electronic device (400). For example, a battery (e.g., battery (189) of FIG. 1) may be placed within the third housing (430).

[0103] FIG. 6 is a drawing schematically illustrating an object recognition operation of an electronic device according to one embodiment.

[0104] Referring to FIG. 6, an electronic device (or wearable electronic device) (101) (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3, the wearable electronic device (300) of FIG. 4, or the electronic device (400) of FIG. 5) can recognize a plurality of objects (610, 620, 630, 640) arranged in space. For example, the electronic device (101) can recognize a plurality of objects (610, 620, 630, 640) included in an area detectable by the electronic device (101) and / or a field of view displayed on a display of the electronic device (101) (e.g., the display module (160) of FIG. 1).

[0105] According to one embodiment, the plurality of objects (610, 620, 630, 640) may include virtual objects (610, 620) and real objects (630, 640).

[0106] According to one embodiment, the electronic device (101) can identify the risk level based on the characteristics of the recognized plurality of objects (610, 620, 630, 640). For example, virtual objects (610, 620) that pose no risk to the user may be identified as having a low risk level, and real objects (630, 640) that pose a risk to the user may be identified as having a high risk level. According to one embodiment, the risk level of an object may be adjusted based on the context with the user, such as the object's purpose.

[0107] According to one embodiment, the electronic device (101) may set a boundary area surrounding each object based on the risk level of the plurality of objects (610, 620, 630, 640).

[0108] According to one embodiment, the electronic device (101) may provide feedback (e.g., visual UI (e.g., highlight), sound, vibration) when the user's body approaches within the boundary area.

[0109] This can prevent confusion between real and virtual objects and reduce risks that may arise from user actions within the virtual environment.

[0110] The object recognition and feedback provision operations of the electronic device are described below with reference to FIGS. 7 to 13, 14a, 14b, 15, 16, 17a, and 17b.

[0111] FIG. 7 is a drawing schematically illustrating the configuration of an electronic device according to one embodiment.

[0112] Referring to FIG. 7, a wearable electronic device (101) (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3, the wearable electronic device (300) of FIG. 4, or the electronic device (400) of FIG. 5)) includes a processor (120) (e.g., the processor (120) of FIG. 1), a display module (160) (e.g., the display module (160) of FIG. 1), a sensor module (176) (e.g., the sensor module (176) of FIG. 1, the camera module (180) of FIG. 1, the second camera module (253) of FIG. 2, the third camera module (255) of FIG. 2, the camera modules (311, 312, 313, 314, 315, 316) of FIG. 3, and a depth sensor (317) of FIG. 3) and a communication module (190) (e.g., the communication module (190) of FIG. 1).

[0113] According to one embodiment, the wearable electronic device (101) can communicate with external electronic devices through a communication module (190) under the control of the processor (120). For example, the wearable electronic device (101) can communicate with at least one of a second wearable electronic device (104-1) (e.g., the electronic device (104) of FIG. 1) and a third wearable electronic device (104-2) (e.g., the electronic device (104) of FIG. 1) worn by a user of the wearable electronic device (101), an electronic device (104-3) (e.g., the electronic device (104) of FIG. 1) being used by the user of the wearable electronic device (101) (e.g., a smartphone or tablet PC), a server (108) (e.g., the server (108) of FIG. 1), or a wearable electronic device (104-4) of another user (e.g., the electronic device (104) of FIG. 1).

[0114] According to one embodiment, the wearable electronic device (101) can identify a real object included in a space where the wearable electronic device (101) is placed through the sensor module (176) under the control of the processor (120). For example, the sensor module (176) can include an image sensor and / or a proximity sensor. For example, the wearable electronic device (101) can recognize a real object included in an image based on an image acquired through a camera including an image sensor. For example, the wearable electronic device (101) can transmit the image acquired through the camera to the server (108) and receive information about the real object included in the image from the server (108). According to one embodiment, the wearable electronic device (101) can detect the presence of a real object and / or identify a distance to the real object through the proximity sensor.

[0115] According to one embodiment, the wearable electronic device (101) may obtain information on an actual object contained in a space from another user's wearable electronic device (104-4) via the communication module (190). In this way, even if the wearable electronic device (101) does not detect a space, it may obtain information on an actual object based on sensing information about the space received from another wearable electronic device (104-4).

[0116] According to one embodiment, the wearable electronic device (101) may display an image of a real space acquired through a camera (e.g., the camera module (180) of FIG. 1) under the control of the processor (120) on the display module (160). According to one embodiment, the wearable electronic device (101) may display a virtual object on the image based on a user input or a manufacturer setting. According to one embodiment, the wearable electronic device (101) may obtain information on a virtual object included in the image displayed on the display module (160).

[0117] According to one embodiment, the wearable electronic device (101) can identify the risk level of each of a real object and a virtual object. According to one embodiment, the wearable electronic device can obtain a boundary area for each of the real object and the virtual object based on the risk level. For example, the wearable electronic device (101) can receive the risk level for each of the real object and the virtual object from the server (108). The risk level can be the degree of risk that may be caused by the object when the user moves. The boundary area can be in a form that surrounds the object. For example, the boundary area can be in a form that surrounds the object by contacting the surface of the object or surrounds the object at a set distance from the surface of the object. For example, the set distance can be a larger value as the risk level of the object increases. According to one embodiment, the boundary area for each object may or may not be displayed on the display module (160).

[0118] According to one embodiment, the wearable electronic device (101) may provide feedback based on the level of risk when the user's body approaches the boundary area. For example, the feedback may include at least one of visual feedback displaying a visual UI (e.g., highlighting) on ​​an object, auditory feedback providing a warning sound, and haptic feedback such as vibration.

[0119] According to one embodiment, the wearable electronic device (101) can determine whether the user's body approaches the boundary area through the sensor module (176). For example, the wearable electronic device (101) can determine whether the user's body, such as a hand, approaches the boundary area through a camera including an image sensor and / or a proximity sensor. For example, the wearable electronic device (101) can determine whether the user's body approaches the boundary area based on the movement of the wearable electronic device (101) through a GPS sensor, a gyro sensor, and / or an acceleration sensor.

[0120] According to one embodiment, the wearable electronic device (101) can determine whether a person wearing the second wearable electronic device (104-1) or the third wearable electronic device (104-2), holding the electronic device (104-3), or a body in close proximity to the electronic device (104-3) (e.g., when a smartphone is placed in a pocket) approaches a boundary area based on the location information (or movement information) of the second wearable electronic device (104-1), the third wearable electronic device (104-2), and / or the electronic device (104-3) received through the communication module (190).

[0121] In one embodiment, the intensity of the feedback provided may increase as the risk level of the object approaching the user's body increases. For example, as the risk level of the object approaching the user's body increases, the brightness of the visual UI may increase, the size of the visual UI may increase, the volume of the warning sound may increase, the pitch of the warning sound may increase, the intensity of the vibration may increase, or the vibration pattern may become faster. In one embodiment, as the risk level increases, the number of types of feedback provided may increase.

[0122] According to one embodiment, the wearable electronic device may transmit a feedback provision command to the second wearable electronic device (104-1), the third wearable electronic device (104-2), and / or the electronic device (104-3) via the communication module (190). According to one embodiment, the second wearable electronic device (104-1), the third wearable electronic device (104-2), and / or the electronic device (104-3) that has received the feedback provision command may provide visual, auditory, and / or tactile feedback. For example, the second wearable electronic device (104-1), the third wearable electronic device (104-2), and / or the electronic device (104-3) may sequentially provide feedback to provide a direction in which an object is located.

[0123] FIG. 8 is a flowchart illustrating an operation of recognizing an object in an electronic device and providing feedback related to the object, according to one embodiment.

[0124] Referring to FIG. 8, in operation 810, an electronic device (or a wearable electronic device) (e.g., an electronic device (101) of FIG. 1, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3, a wearable electronic device (300) of FIG. 4, or an electronic device (400) of FIG. 5) may detect a plurality of objects included in a field of view (FOV) of the wearable electronic device that can be detected through at least one sensor (e.g., a sensor module (176) of FIG. 1, a camera module (180) of FIG. 1, a second camera module (253) of FIG. 2, a third camera module (255) of FIG. 2, camera modules (311, 312, 313, 314, 315, 316) of FIG. 3, and a depth sensor (317) of FIG. 3).

[0125] According to one embodiment, the electronic device may detect at least one real object included in a detectable area (or a field of view (FOV) of the real object) through at least one sensor in a real space. For example, the detectable area through the at least one sensor may be wider than a field of view displayed by a display (e.g., a display module (160) of FIG. 1). For example, the detectable area through the at least one sensor may be the entire area surrounding the electronic device.

[0126] According to one embodiment, the electronic device can detect at least one virtual object included in a second FOV corresponding to an area displayed on the display.

[0127] According to one embodiment, information about a virtual object included in an image is stored in an electronic device, so that the electronic device can obtain information about a virtual object included in a field of view (or, a field of view of the virtual object, a second field of view) displayed on a display. For example, the information about the virtual object may include information about the type of the virtual object and / or information about the location of the virtual object. According to one embodiment, the information about the virtual object may further include at least a portion of an area detectable through at least one sensor in addition to the field of view displayed on the display.

[0128] According to one embodiment, the recognition range of a real object and a virtual object is described below with reference to FIG. 9.

[0129] According to one embodiment, an electronic device can receive information about a space from another user's electronic device (or wearable electronic device). For example, the information about the space may include structural information about the space and recognition information about real objects acquired by the other user's electronic device. For example, the recognition information about the real object may include information about the type and / or location of the real object. In this way, by acquiring information about the space through an external electronic device, the electronic device can acquire information about the space and the real objects placed in the space without directly sensing the space.

[0130] In one embodiment, in operation 820, the electronic device may recognize the type of a first object among a plurality of objects. For example, the electronic device may recognize the types of the plurality of objects through image recognition. For example, information about the type of object may include information about what the object is (e.g., sofa, hot coffee, cold coffee, light).

[0131] According to one embodiment, in operation 830, the electronic device can determine whether the first object is a real object or a virtual object. For example, the electronic device can determine that the first object is a real object if it is detected by at least one sensor. The electronic device can detect the first object through information about a virtual object included in an image, or can determine that the first object is a virtual object if the first object is detected through image recognition.

[0132] According to one embodiment, the electronic device can identify risk levels of a plurality of objects, each of which includes at least one real object and at least one virtual object displayed on a display, based on set conditions.

[0133] In one embodiment, in operation 840, the electronic device may determine a first risk level of the first object based on the recognized type of the first object and whether the first object is a real object or a virtual object.

[0134] According to one embodiment, the electronic device can differentially set the risk level of each object based on the characteristics of the objects.

[0135] For example, an electronic device may identify an object as being more dangerous depending on whether it is real or virtual. If the object is real, it may be at risk of being hit by the user's movements, and so the device may identify sharp, breakable, or hot objects as being more dangerous than other real objects, as they could potentially cause injury to the user.

[0136] According to one embodiment, the operation of identifying risk based on the characteristics of an object is described below with reference to FIG. 10.

[0137] According to one embodiment, the electronic device may determine the risk level of a virtual object based on a user's usage environment related to the virtual object, among virtual objects.

[0138] For example, if the virtual object is a sofa, the virtual sofa may be identified as more dangerous than other virtual objects, as there is a risk of injury if a user sits on it, rather than on a real object. For example, if the virtual object is a wall, the virtual wall may be identified as more dangerous than other virtual objects, as there is a risk of injury if a user leans on it, rather than on a real object.

[0139] According to one embodiment, the operation of changing the risk level depending on the type of virtual object will be described below with reference to FIGS. 12 and 13.

[0140] In one embodiment, in operation 850, the electronic device may determine a first distance associated with a first object based on a first risk level of the first object. For example, the first distance may be a distance from a surface of the first object. In one embodiment, a set of points that are a specific distance (e.g., the first distance) from the surface of the body may be referred to as a boundary region.

[0141] According to one embodiment, the electronic device can obtain boundary regions surrounding a plurality of objects based on risk levels.

[0142] In one embodiment, the electronic device may acquire a boundary area such that the distance between the surface of the object and the boundary increases as the risk level of the object increases. For example, the electronic device may determine that the value of the first distance increases as the risk level of the first object increases. In one embodiment, a large distance between the surface of the object and the boundary may indicate a deep boundary depth.

[0143] For example, in the case of virtual objects, the risk of physical collision is low, so the surface of the object and the boundary area may come into contact, or the distance between the surface of the object and the boundary area may be relatively short. In one embodiment, in the case of real objects, collisions may occur depending on the user's movement or displacement, so the distance between the surface of the object and the boundary area may be relatively long.

[0144] In one embodiment, the electronic device may determine the number of boundaries based on the risk level of the object. For example, if the number of boundaries is two or more, the two or more boundaries may be located at different distances from the surface of the object. For example, the electronic device may further determine a second distance that is greater than the first distance based on the first risk level of the first object.

[0145] In the above, it has been described that the boundary area is obtained based on the risk level obtained by the characteristics of the objects, but it is not limited to this, and the boundary area can also be obtained based on the characteristics of the objects without an operation to identify the risk level.

[0146] According to one embodiment, an operation of setting a boundary area based on the characteristics of an object is described below with reference to FIG. 11.

[0147] In one embodiment, in operation 860, the electronic device may provide feedback based on the first risk level of the first object based on whether the distance between the first object and the user's body is less than or equal to the first distance. For example, when the user's body approaches the first object, the electronic device may provide feedback (e.g., visual UI, sound, vibration) if the distance between the first object and the user's body is less than or equal to the first distance.

[0148] According to one embodiment, the electronic device may provide feedback based on a first risk level corresponding to the first boundary area, based on identification of a user's body approaching a first boundary area among the boundary areas.

[0149] According to one embodiment, the electronic device can identify that a user's body approaches a first boundary area corresponding to a first object among a plurality of objects through at least one sensor and / or a camera of the electronic device. For example, the electronic device can identify that a user's body, such as a hand, approaches within the first boundary area through a camera and / or a proximity sensor. According to one embodiment, the electronic device can detect a movement of the electronic device through a GPS sensor, a gyro sensor, and / or an acceleration sensor, and identify that a user's body wearing the electronic device approaches within the first boundary area based on the movement of the electronic device.

[0150] According to one embodiment, the electronic device can identify the approach of the user's body to the first boundary area based on a detection value received from an external electronic device connected via a communication module. For example, the electronic device can receive location information and / or movement information of the external electronic device from an external electronic device including a smartwatch worn on a wrist, a smart ring worn on a hand, and / or a smartphone held in a hand or placed in a pocket, and identify whether the user's body approaches within the first boundary area based on the location information and / or movement information of the external electronic device.

[0151] According to one embodiment, the electronic device may determine the number of feedbacks based on the risk level of the object. For example, as shown in [Table 1] below, the higher the risk level, the more types of feedback may be provided.

[0152] LayervisualhapticsoundLevel 1Not providedOLevel 21 depth layerOOLevel 32 depth layerOOO

[0153] Referring to [Table 1], as the risk level increases, the depth of the boundary area may increase. For example, as the risk level increases, the distance between the surface of the object and the boundary area may increase. For example, an object at level 1 may have the same surface and boundary area (not provided). An object at level 2 may have a first distance (e.g., depth layer 1) between the surface of the object and the boundary area. An object at level 3 may have a second distance (e.g., depth layer 2) between the surface of the object and the boundary area that is greater than the first distance. In one embodiment, as the risk level increases, the types of feedback may increase. For example, an electronic device may provide visual feedback at level 1, visual and vibration feedback at level 2, and visual, vibration, and auditory feedback at level 3.

[0154] In one embodiment, the electronic device may determine the intensity of the feedback based on the perceived risk of the object. For example, as the perceived risk of the object increases, the brightness of the visual UI may increase, the size of the visual UI may increase, the volume of the warning sound may increase, the pitch of the warning sound may increase, the intensity of the vibration may increase, or the vibration pattern may become faster.

[0155] For example, electronic devices can increase the intensity of feedback as the risk (level) increases, as shown in [Table 2] below.

[0156] visualhapticsoundLevel 1O(soft visual)Level 2O(strong visual)Level 3O(strong visual)O(light haptic)Level 4O(strong visual)O(medium haptic)O(light sound)Level 5O(strong visual)O(strong haptic)O(strong sound)

[0157] [Table 2] describes that as the level of risk increases, not only the intensity of the feedback but also the number of feedbacks increases, but this is not limited thereto. In one embodiment, even if the level of risk increases, the number of feedbacks may be maintained and only the intensity of the feedback may increase. In one embodiment, the electronic device may change the boundary area for an object based on the speed at which the user's body approaches the object. For example, if the user's body approaches a real or virtual object at a speed exceeding a set value, the electronic device may change the boundary area acquired for the object to correspond to a higher level of risk. For example, if the user's body approaches a real or virtual object at a speed exceeding a set value, the electronic device may modify the boundary area so that the distance between the surface of the object and the boundary area becomes wider, or may acquire one more boundary area with a wider distance between the surface of the object and the boundary area.

[0158] In one embodiment, the electronic device can change the boundary area of ​​an object in real time when it detects that the user's body is approaching the object at a speed exceeding a certain speed.

[0159] For example, an electronic device may change the risk level of an object based on the object's existing risk level (or risk weight) and the user's approach speed, as shown in Table 3 below. In one embodiment, the electronic device may change the boundary area based on the changed risk level.

[0160] Risk weight 1 (low) Risk weight 2 (medium) Risk weight 3 (high) Speed ​​1 (slow) Level 1 Level 2 Level 3 Speed ​​2 (medium) Level 2 Level 3 Level 4 Speed ​​3 (fast) Level 3 Level 4 Level 5

[0161] Referring to [Table 3], if the electronic device identifies the risk of an object as Level 1, Level 2, and Level 3, respectively, when the user approaches at a first slow speed, the electronic device can maintain the risk at Level 1, Level 2, and Level 3, respectively. According to one embodiment, when the user approaches at a second speed that is higher than the set speed, the risk can be increased by one level to Level 2, Level 3, and Level 4, respectively. The electronic device can change the boundary area of ​​each object based on the changed risk.

[0162] In one embodiment, when a user approaches at a third speed that is higher than the set speed and faster than the second speed, the risk level can be increased by one level to level 3, level 4, and level 5, respectively. The electronic device can change the boundary area of ​​each object based on the changed risk level.

[0163] According to one embodiment, an embodiment of changing the boundary area of ​​an object based on a user's approach speed will be described with reference to FIGS. 15 and 16 below.

[0164] In one embodiment, when the electronic device provides two or more types of feedback, it may provide visual feedback, vibration feedback, and auditory feedback in that order.

[0165] According to one embodiment, the electronic device may transmit a command to provide feedback to a connected external device via a communication module. For example, the electronic device may transmit a command to provide feedback (e.g., visual, vibrational, or audible) to a connected smartwatch, smart ring, and / or smartphone via the communication module. According to one embodiment, the electronic device may synchronize the timing of providing feedback with the external electronic device.

[0166] In one embodiment, the electronic device may interact with an external electronic device to provide location information (e.g., direction and / or distance) of an object through feedback. For example, the electronic device and the external electronic device may sequentially provide vibration feedback to provide information on the location of the object. The electronic device and the external electronic device may sequentially provide auditory feedback to provide information on the location of the object.

[0167] In one embodiment, the electronic device may provide feedback when moving a virtual object based on user input. For example, feedback (e.g., visual, vibration, or auditory) may be provided when grabbing a virtual object, lifting it from the surface of another object, attaching it to the surface of another object, dragging it within a specific surface, or releasing the hand after completing the movement of the virtual object.

[0168] In one embodiment, the electronic device may provide a realistic experience by providing feedback based on the weight of the actual object corresponding to the virtual object, with stronger feedback being provided as the weight increases. In another embodiment, the electronic device may determine the strength of the feedback based on the characteristics of the surface on which the virtual object moves (e.g., friction, slope, unevenness).

[0169] According to one embodiment, when a virtual object is attached to a real object, the electronic device may provide feedback corresponding to the risk level of the real object when the virtual object approaches within a boundary area of ​​the real object, thereby indicating that the object being approached by the virtual object is a real object.

[0170] FIG. 9 is a drawing for explaining the recognition range of a real object and a virtual object according to one embodiment.

[0171] Referring to FIG. 9, an electronic device (or, wearable electronic device) (101) (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3, the wearable electronic device (300) of FIG. 4, or the electronic device (400) of FIG. 5) can identify a real object in a real space and a virtual object in a virtual environment.

[0172] Depending on the embodiment, the detection range of the real object and the detection range of the virtual object may be different.

[0173] For example, the electronic device (101) can identify at least one real object included in a real space through at least one sensor (e.g., the sensor module (176) of FIG. 1, the camera module (180) of FIG. 1, the second camera module (253) of FIG. 2, the third camera module (255) of FIG. 2, the camera modules (311, 312, 313, 314, 315, 316) of FIG. 3, and the depth sensor (317) of FIG. 3).

[0174] According to one embodiment, the electronic device (101) can detect at least one real object included in a detectable area (910) by at least one sensor. For example, the detectable area (910) by at least one sensor may be the entire surrounding area surrounding the electronic device (101). For example, the electronic device (101) can set a boundary area based on the risk level even for a real object that is not displayed on the display. According to one embodiment, the electronic device (101) can provide feedback when the user's body approaches the boundary area even for a real object that is not displayed on the display.

[0175] According to one embodiment, the electronic device (101) may set a boundary area based on the risk level of an actual object included in the field of view (920) corresponding to an image displayed on the display. According to one embodiment, even if the boundary area disappears from the field of view (920), the electronic device (101) may provide feedback when the user's body approaches the boundary area.

[0176] According to one embodiment, the electronic device (101) can identify at least one virtual object included in an image displayed on a display (e.g., the display module (160) of FIG. 1). For example, the electronic device (101) can identify at least one object included in a field of view (920) corresponding to the image displayed on the display. According to one embodiment, since the electronic device (101) stores information on all virtual objects arranged in a virtual space, it can identify virtual objects arranged in a space that is not displayed as well as a space that is displayed on the display. For example, the electronic device (101) can set a boundary area based on a risk level for a virtual object that is not displayed on the display. According to one embodiment, even for a virtual object that is not displayed on the display, the electronic device (101) can provide feedback when the user's body approaches the boundary area.

[0177] According to one embodiment, the electronic device (101) may set a boundary area based on the risk level of a virtual object included in the field of view (920) corresponding to an image displayed on the display. According to one embodiment, even if the boundary area disappears from the field of view (920), the electronic device (101) may provide feedback when the user's body approaches the boundary area.

[0178] According to one embodiment, the field of view (920) corresponding to the image displayed on the display may be included in an area (910) detectable by at least one sensor.

[0179] While real objects require the user to recognize them even if they aren't displayed, virtual objects don't physically impact the user, so they don't need to be identified. By identifying only virtual objects that are displayed, the resources consumed in identifying virtual objects can be reduced.

[0180] FIG. 10 is a diagram for explaining an operation of identifying the risk of an object of an electronic device according to one embodiment.

[0181] FIG. 11 is a diagram illustrating a boundary area based on the risk level of an object according to one embodiment.

[0182] Referring to FIG. 10, in operation 1010, an electronic device (or wearable electronic device) (101) (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3, the wearable electronic device (300) of FIG. 4, or the electronic device (400) of FIG. 5) can identify an object within the FOV. For example, the electronic device can identify at least one real object included in a detectable area through at least one sensor (e.g., the sensor module (176) of FIG. 1, the camera module (180) of FIG. 1, the second camera module (253) of FIG. 2, the third camera module (255) of FIG. 2, the camera modules (311, 312, 313, 314, 315, 316) of FIG. 3, and the depth sensor (317) of FIG. 3). According to one embodiment, the electronic device can identify at least one virtual object included in a virtual space displayed on a display (e.g., display module (160) of FIG. 1).

[0183] In one embodiment, in operation 1020, the electronic device can determine whether the object is a real object. In one embodiment, if the object is not a real object (operation 1020 - No), in operation 1030, the electronic device can determine the risk level of the object as level 1. For example, as illustrated in FIG. 11, level 1 may be a risk level that only provides visual feedback when the surface of the object (1110) and the boundary area (1111) coincide, and the user's body approaches within the boundary area.

[0184] In one embodiment, if the object is a real object (operation 1020 - yes), then in operation 1040, the electronic device may determine whether the real object is a hazardous object upon user contact. For example, if the object is sharp, fragile, or hot, the electronic device may determine that the object is a hazardous object upon user contact.

[0185] According to one embodiment, if the object is determined not to be a dangerous object when the user comes into contact with it (operation 1040 - No), in operation 1050, the electronic device may determine the danger level of the object to be level 2. For example, as illustrated in FIG. 11, level 2 may be a danger level in which the distance between the surface of the object (1120) and the boundary area (1121) is a first distance (e.g., d1), and when the user's body approaches within the boundary area, the provided feedback may be increased in intensity and / or number compared to level 1. For example, level 2 may be a danger level in which visual feedback and vibration feedback are provided when the user's body approaches within the boundary area.

[0186] According to one embodiment, if an object is determined to be a dangerous object upon user contact (operation 1040 - Yes), in operation 1060, the electronic device may determine the danger level of the object to be level 3. For example, as illustrated in FIG. 11, level 3 may be a second distance (e.g., d2) greater than the first distance between the surface of the object (1130) and the second boundary area (1132), and when the user's body approaches within the boundary area, the provided feedback may be increased in intensity and / or number compared to level 2. For example, level 3 may be a danger level that provides visual feedback, vibration feedback, and auditory feedback when the user's body approaches within the boundary area.

[0187] According to one embodiment, Level 3 may be configured to have two or more boundary areas set for the object (1130). For example, Level 3 may be a risk level that sets a first boundary area (1131) having a first distance from the surface of the object (1130) and a second boundary area (1132) having a second distance greater than the first distance. According to one embodiment, when the user's body approaches within the second boundary area (1132), the electronic device may provide visual feedback and vibration feedback. According to one embodiment, when the user's body approaches within the first boundary area (1131), the electronic device may provide visual feedback, vibration feedback, and auditory feedback.

[0188] In this way, by providing differential feedback when the user's body approaches the object according to its characteristics, the user can understand the object's characteristics and reduce the possibility of danger that may occur upon contact.

[0189] FIG. 12 is a flowchart for explaining an operation of changing a boundary area according to the characteristics of an object, according to one embodiment.

[0190] FIG. 13 is a drawing for explaining a boundary area changed according to the characteristics of an object according to one embodiment.

[0191] Referring to FIG. 12, in operation 1210, an electronic device (or wearable electronic device) (101) (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3, the wearable electronic device (300) of FIG. 4, or the electronic device (400) of FIG. 5) can identify risk levels of a plurality of objects, each including at least one real object and at least one virtual object.

[0192] In one embodiment, in operation 1220, the electronic device may obtain context data between each of the plurality of objects and the user.

[0193] In one embodiment, contextual data between an object and a user may be data related to the user's usage environment associated with the object. For example, in the case of a virtual sofa, which is a virtual object, contextual data may indicate the possibility that the user will perform a sitting action on the sofa. For example, in the case of a virtual wall, which is a virtual object, contextual data may indicate the possibility that the user will perform a leaning action against the wall.

[0194] In one embodiment, in operation 1230, the electronic device may change the risk level of at least some of the plurality of objects based on context data.

[0195] For example, as illustrated in FIG. 13, in the case of a virtual sofa (1310), since a dangerous situation such as falling over may occur when a person sits on it, the electronic device may change the risk level of the virtual sofa (1310) to be higher than that of other virtual objects. According to an embodiment, as the risk level of the virtual sofa (1310) increases, the electronic device may acquire a boundary area (1311) so as to have a first distance (e.g., d1) from the surface of the virtual sofa (1310). According to an embodiment, when it is detected that the user's body approaches within the boundary area (1311) of the virtual sofa (1310), the electronic device may provide both visual feedback and vibration feedback, unlike other virtual objects that only provide visual feedback.

[0196] For example, in the case of a virtual wall, since a dangerous situation such as falling may occur if a person leans against it, the electronic device may change the risk level of the virtual wall to be higher than that of other virtual objects. According to one embodiment, as the risk level of the virtual wall increases, the electronic device may acquire a boundary area to have a first distance (e.g., d1) from the surface of the virtual wall. According to one embodiment, when it is detected that the user's body approaches within the boundary area of ​​the virtual wall, the electronic device may provide both visual feedback and vibration feedback, unlike other virtual objects that only provide visual feedback.

[0197] FIG. 14A is a diagram for explaining a feedback operation of an electronic device when a user's body approaches an object according to one embodiment.

[0198] Referring to FIG. 14A, an electronic device (or wearable electronic device) (101) (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3, the wearable electronic device (300) of FIG. 4, or the electronic device (400) of FIG. 5) can obtain a first boundary area (1411) and a second boundary area (1412) for a virtual object (1410) that is a hot object. For example, the first boundary area (1411) can have a first distance (d1) from a surface of the virtual object (1410), and the second boundary area (1412) can have a second distance (d2) from a surface of the virtual object (1410).

[0199] According to one embodiment, the electronic device (101) may provide a first feedback corresponding to the second boundary area (1412) when it detects that the user's body approaches within the second boundary area (1412) through at least one sensor (e.g., the sensor module (176) of FIG. 1, the camera module (180) of FIG. 1, the second camera module (253) of FIG. 2, the third camera module (255) of FIG. 2, the camera modules (311, 312, 313, 314, 315, 316) of FIG. 3, and the depth sensor (317) of FIG. 3). For example, the electronic device (101) may detect whether the position of the user's hand approaches within the second boundary area (1412) through a camera or a proximity sensor. According to one embodiment, the first feedback may include visual feedback and vibration feedback.

[0200] In one embodiment, the electronic device (101) may transmit a command to provide a first feedback to an external electronic device (104) (e.g., the electronic device (104) of FIG. 1) with which it is connected. In one embodiment, the external electronic device (104) may provide visual feedback and vibration feedback.

[0201] According to one embodiment, when the electronic device (101) detects that the user's body approaches the first boundary area (1411) through at least one sensor, the electronic device (101) may provide a second feedback corresponding to the first boundary area (1411). For example, the second feedback may include visual feedback, vibration feedback, and auditory feedback. For example, the second feedback may be stronger than the first feedback.

[0202] In one embodiment, the electronic device (101) can transmit a command to an external electronic device (104) to which it is connected to provide a second feedback. In one embodiment, the external electronic device (104) can provide visual feedback, vibration feedback, and auditory feedback.

[0203] FIG. 14b is a diagram for explaining a feedback operation of an electronic device when a user's body approaches an object according to one embodiment.

[0204] Referring to FIG. 14B, an electronic device (or, wearable electronic device) (101) can acquire a first boundary area (1411) and a second boundary area (1412) for a virtual object (1410) that is a hot object. For example, the first boundary area (1411) can have a first distance (d1) from the surface of the virtual object (1410), and the second boundary area (1412) can have a second distance (d2) from the surface of the virtual object (1410).

[0205] According to one embodiment, when the electronic device (101) detects that the user's body approaches within the second boundary area (1412), the electronic device (101) may provide a first feedback corresponding to the second boundary area (1412). For example, when a first external electronic device (104-1) (e.g., a smart ring) and a second external electronic device (104-2) (e.g., a smart watch) are worn on the user's hand and approach a virtual object (1410), the first external electronic device (104-1) and the second external electronic device (104-2) may obtain location information and / or movement information through a GPS sensor, a gyro sensor, and / or an acceleration sensor included in the first external electronic device (104-1) and the second external electronic device (104-2), respectively. The first external electronic device (104-1) and the second external electronic device (104-2) can transmit location information and / or movement information to the electronic device (101).

[0206] According to one embodiment, the electronic device (101) can determine whether the body of the user wearing the first external electronic device (104-1) and the second external electronic device (104-2) approaches within the second boundary area (1412) based on location information and / or movement information received from the first external electronic device (104-1) and the second external electronic device (104-2).

[0207] In one embodiment, the electronic device (101) may transmit a command to provide a first feedback to an external electronic device (104-5) (e.g., the electronic device (104) of FIG. 1) to which it is connected. In one embodiment, the external electronic device (104-5) may provide visual feedback and vibration feedback.

[0208] According to one embodiment, when the electronic device (101) detects that the user's body approaches within the first boundary area (1411), the electronic device (101) may provide a second feedback corresponding to the first boundary area (1411). For example, the electronic device (101) may determine whether the user's body wearing the first external electronic device (104-1) and the second external electronic device (104-2) approaches within the first boundary area (1411) based on location information and / or movement information received from the first external electronic device (104-1) and the second external electronic device (104-2). For example, the second feedback may include visual feedback, vibration feedback, and auditory feedback. For example, the second feedback may be stronger than the first feedback.

[0209] In one embodiment, the electronic device (101) may transmit a command to an external electronic device (104-5) to provide second feedback. In one embodiment, the external electronic device (104-5) may provide visual feedback, vibration feedback, and auditory feedback.

[0210] FIG. 15 is a flowchart illustrating an operation of changing a boundary area according to a user's approach speed according to one embodiment.

[0211] FIG. 16 is a drawing for explaining a boundary area changed according to a user's approach speed according to one embodiment.

[0212] Referring to FIG. 15, in operation 1510, an electronic device (or wearable electronic device) (e.g., electronic device (101) of FIG. 1, wearable electronic device (200) of FIG. 2, wearable electronic device (300) of FIG. 3, wearable electronic device (300) of FIG. 4, or electronic device (400) of FIG. 5) can confirm that a user is moving in a virtual environment in which real objects and virtual objects with boundary areas set exist.

[0213] For example, the electronic device can determine that the electronic device is moving through at least one sensor (e.g., the sensor module (176) of FIG. 1, the camera module (180) of FIG. 1, the second camera module (253) of FIG. 2, the third camera module (255) of FIG. 2, the camera modules (311, 312, 313, 314, 315, 316) of FIG. 3, and the depth sensor (317) of FIG. 3). For example, the electronic device can obtain information related to the movement of the electronic device through a GPS sensor, a gyro sensor, and / or an acceleration sensor. According to one embodiment, the electronic device can determine that the electronic device is moving based on a change in the size of an object included in an actual space due to the movement and / or a distance to the object through a camera and / or a proximity sensor.

[0214] In one embodiment, at operation 1520, the electronic device may determine whether the user is approaching a particular boundary area at a particular speed or greater.

[0215] According to one embodiment, if the user's moving speed is greater than a certain speed (operation 1520 - Yes), in operation 1530, the electronic device may change the boundary area where contact is predicted based on the user's moving direction and speed. For example, as illustrated in FIG. 16, if the user's (10) moving speed is greater than a certain speed, the electronic device may increase the risk level of a specific object (1610) toward which the user is facing, and change the previously set boundary area (1611) to a wider boundary area (1612).

[0216] In one embodiment, if the user's movement speed is less than a certain speed (action 1520 - No), the electronic device may proceed to action 840 to determine whether the user's body has contacted the changed boundary area.

[0217] FIG. 17a is a drawing for explaining boundary areas of a real object and a virtual object based on the real object according to one embodiment.

[0218] Referring to FIG. 17A, an electronic device (or a wearable electronic device) (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3, the wearable electronic device (300) of FIG. 4, or the electronic device (400) of FIG. 5) may set a boundary area (1711) having a set distance from an actual object (1710). In one embodiment, the boundary area (1711) is illustrated as being a hexahedron, but is not limited thereto.

[0219] In one embodiment, when the real object (1710) is a TV, a virtual screen (1720), which is a virtual object larger than the display of the real object (1710), may be placed based on a user input, and an image may be displayed through the virtual screen (1720). In one embodiment, a boundary area (1721) that coincides with the surface of the virtual screen (1720) may be set.

[0220] In one embodiment, the electronic device may display an indicator (e.g., a visual cue) on the real object (1710) or the virtual screen (1720) to distinguish between the real object (1710) and the virtual screen (1720).

[0221] According to one embodiment, if the speed at which the user (10) approaches the real object (1710) is greater than or equal to a set value, the electronic device may increase the risk level for the real object (1710) by widening the boundary area (1721) and increasing the types and number of feedbacks. For example, if the types of feedback are increased, the electronic device may provide visual feedback, vibration feedback, and auditory feedback in that order. According to one embodiment, if the speed at which the user (10) approaches the real object (1710) is greater than or equal to a set value, the electronic device may increase the intensity of the feedbacks.

[0222] FIG. 17b is a drawing for explaining boundary areas of a real object and a virtual object based on the real object according to one embodiment.

[0223] Referring to FIG. 17b, an electronic device (or a wearable electronic device) can set a boundary area (1711) having a set distance from an actual object (1710).

[0224] In one embodiment, when the real object (1710) is a TV, a virtual screen (1720), which is a virtual object larger than the display of the real object (1710), may be placed based on a user input, and an image may be displayed through the virtual screen (1720). In one embodiment, a boundary area (1721) that coincides with the surface of the virtual screen (1720) may be set.

[0225] In one embodiment, when a dangerous real object (1730) (e.g., a hot object, a sharp object, a breakable object) is detected in addition to a real object (1710) which is a TV, the electronic device may set a first boundary area (1731) having a first distance from the dangerous real object (1730) and a second boundary area (1732) having a second distance greater than the first distance for the dangerous real object (1730). In one embodiment, the first boundary area (1731) and the second boundary area (1732) are illustrated as being hexahedrons, but are not limited thereto.

[0226] According to one embodiment, when the speed at which the user (10) approaches the dangerous real object (1730) is less than a set value, the electronic device may provide vibration feedback corresponding to a second boundary area (1732) of the dangerous real object (1730) to indicate that the user is approaching the dangerous real object (1730).

[0227] According to one embodiment, if the speed at which the user (10) approaches the dangerous real object (1730) is greater than a set value, the electronic device may increase the risk level for the dangerous real object (1730) by widening the first boundary area (1731) and / or the second boundary area (1732), adding boundary areas, and increasing the type, number, and intensity of feedback. For example, if the type of feedback is increased, the electronic device may provide visual feedback, vibration feedback, and auditory feedback in that order.

[0228] In this way, if there is a possibility that a danger may be caused by the user's movement, the size and / or number of boundary areas, the type, number and / or intensity of feedback may be adjusted in real time to provide a notification to the user to inform them of the presence of the object and reduce the danger.

[0229] According to one embodiment, a wearable electronic device may include at least one sensor, a display, and at least one processor electrically connected to the at least one sensor and the display.

[0230] According to one embodiment, the at least one processor can detect a plurality of objects included within a first field of view (FOV) of the wearable electronic device detectable via the at least one sensor.

[0231] In one embodiment, the at least one processor can recognize a type of a first object among the plurality of objects.

[0232] In one embodiment, the at least one processor can determine whether the first object is a real object or a virtual object.

[0233] According to one embodiment, the at least one processor may determine a first risk level of the first object based on the recognized type of the first object and whether the first object is the real object or a virtual object.

[0234] In one embodiment, the at least one processor may determine a first distance associated with the first object based on a first risk of the first object.

[0235] In one embodiment, the at least one processor may provide feedback based on a first risk level of the first object based on a distance between the first object and the user's body being less than or equal to the first distance.

[0236] In one embodiment, the at least one processor can detect at least one real object included within the first FOV via the at least one sensor.

[0237] In one embodiment, the at least one processor is capable of detecting at least one virtual object included within the first FOV and included within a second FOV corresponding to an area displayed on the display.

[0238] In one embodiment, the at least one processor may determine that the value of the first distance is greater as the risk of the first object is higher.

[0239] In one embodiment, the at least one processor may determine the strength of the feedback based on the first risk.

[0240] In one embodiment, the at least one processor may determine the number of feedbacks based on the first risk.

[0241] In one embodiment, the at least one processor may further determine a second distance that is greater than the first distance based on the first risk.

[0242] In one embodiment, the at least one processor may change the first distance based on a speed at which the user's body approaches the first object.

[0243] According to one embodiment, the at least one processor may determine the risk level of the first object based on a user's usage environment related to the first object, if the first object is a virtual object.

[0244] According to one embodiment, the wearable electronic device may further include a communication module.

[0245] In one embodiment, the at least one processor may transmit a command to provide the feedback to an external device connected via the communication module.

[0246] According to one embodiment, the at least one processor can identify a distance between the first object and the user's body based on a detection value received from an external device connected via the communication module.

[0247] According to one embodiment, a method of controlling a wearable electronic device may include detecting a plurality of objects included within a first FOV of the wearable electronic device detectable via at least one sensor of the wearable electronic device.

[0248] According to one embodiment, a method for controlling a wearable electronic device may include an operation of recognizing a type of a first object among the plurality of objects.

[0249] According to one embodiment, a method for controlling a wearable electronic device may include an operation of determining whether the first object is a real object or a virtual object.

[0250] According to one embodiment, a control method of a wearable electronic device may include an operation of determining a first risk level of the first object based on the recognized type of the first object and whether the first object is the real object or the virtual object.

[0251] According to one embodiment, a method of controlling a wearable electronic device may include an operation of determining a first distance related to the first object based on a first risk level of the first object.

[0252] According to one embodiment, a method for controlling a wearable electronic device may include providing feedback based on a first risk level of the first object based on a distance between the first object and the user's body being less than or equal to the first distance.

[0253] In one embodiment, the operation of detecting the plurality of objects may detect at least one real object included within the first FOV via the at least one sensor.

[0254] According to one embodiment, the operation of detecting the plurality of objects may detect at least one virtual object included within the first FOV and included within a second FOV corresponding to an area displayed on a display of the wearable electronic device.

[0255] According to one embodiment, the operation of determining the first distance may determine that the value of the first distance is greater as the risk of the first object is higher.

[0256] In one embodiment, the action of providing the feedback may determine the strength of the feedback based on the first risk level.

[0257] In one embodiment, the action of providing the feedback may determine the number of feedbacks based on the first risk level.

[0258] According to one embodiment, the operation of determining the first distance may further determine a second distance that is a greater value than the first distance based on the first risk.

[0259] According to one embodiment, the control method of the wearable electronic device may further include an operation of changing the first distance based on a speed at which the user's body approaches the first object.

[0260] According to one embodiment, the operation of determining the first risk level of the first object may determine the risk level of the first object based on a user's usage environment related to the first object, if the first object is a virtual object.

[0261] According to one embodiment, the method for controlling the wearable electronic device may further include an operation of transmitting a command to provide the feedback to an external device connected through a communication module of the wearable electronic device.

[0262] According to one embodiment, the action of providing the feedback may identify a distance between the first object and the user's body based on a detection value received from an external device connected via a communication module of the wearable electronic device.

[0263] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may include instructions for causing an electronic device to detect a plurality of objects included within a first field of view (FOV) of the wearable electronic device detectable through at least one sensor.

[0264] According to one embodiment, the one or more programs may include instructions for the electronic device to recognize a type of a first object among the plurality of objects.

[0265] In one embodiment, the one or more programs may include instructions for the electronic device to determine whether the first object is a real object or a virtual object.

[0266] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine a first risk level of the first object based on the recognized type of the first object and whether the first object is the real object or a virtual object.

[0267] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine a first distance associated with the first object based on a first risk level of the first object.

[0268] In one embodiment, the one or more programs may include instructions for the electronic device to provide feedback based on a first risk level of the first object based on a distance between the first object and the user's body being less than or equal to the first distance.

[0269] In one embodiment, the one or more programs may include instructions for the electronic device to detect at least one real object contained within the first FOV via the at least one sensor.

[0270] According to one embodiment, the one or more programs may include instructions for the electronic device to detect at least one virtual object included within the first FOV and included within a second FOV corresponding to an area displayed on the display.

[0271] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine that the value of the first distance is greater as the risk of the first object becomes higher.

[0272] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine the intensity of the feedback based on the first risk level.

[0273] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine a number of feedbacks based on the first risk level.

[0274] In one embodiment, the one or more programs may include instructions that cause the electronic device to further determine a second distance that is greater than the first distance based on the first risk.

[0275] In one embodiment, the one or more programs may include instructions for the electronic device to change the first distance based on a speed at which the user's body approaches the first object.

[0276] In one embodiment, the one or more programs may include instructions for the electronic device to determine a risk level of the first object based on a user's usage environment related to the first object, if the first object is a virtual object.

[0277] According to one embodiment, the wearable electronic device may further include a communication module.

[0278] In one embodiment, the one or more programs may include instructions for the electronic device to transmit a command to an external device connected via the communication module to provide the feedback.

[0279] In one embodiment, the one or more programs may include instructions for the electronic device to identify a distance between the first object and the user's body based on a detection value received from an external device connected via the communication module.

[0280] Electronic devices according to the embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0281] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0282] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0283] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0284] According to one embodiment, the method according to one embodiment disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0285] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0286] It will be appreciated that the various embodiments of the present disclosure, as described in the claims and specification, may be implemented in the form of hardware, software, or a combination of hardware and software.

[0287] Such software may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules), and the one or more computer programs include computer-executable instructions that, when executed by one or more processors of the electronic device, cause the electronic device to perform the following tasks:

[0288] Any such software may be stored in the form of volatile or non-volatile storage, for example, a storage device (whether erasable or rewritable), such as read-only memory (ROM), or in the form of memory, for example, random access memory (RAM), a memory chip, a device or integrated circuit, or a compact disk (CD), a digital versatile disk (DVD), a magnetic disk, or an optically or magnetically readable medium, such as magnetic tape. It will be appreciated that the storage device and the storage medium are various embodiments of a computer program or a non-transitory machine-readable storage device suitable for storing a computer program, which includes instructions that, when executed, implement various embodiments of the present disclosure. Accordingly, various embodiments provide a program comprising code for implementing an apparatus or method as recited in any of the claims of the present disclosure, and a non-transitory machine-readable storage storing such a program.

[0289] While the present disclosure has been illustrated and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. In wearable electronic devices, At least one sensor; At least one processor comprising a processing circuit; and When executed by at least one processor, causing the wearable electronic device to: Detecting a plurality of objects of the wearable electronic device detectable through at least one sensor or external electronic device, Recognize the type of the first object among the above multiple objects, Check whether the first object above is a real object or a virtual object, Based on the recognized type of the first object and whether the first object is the real object or the virtual object, a first risk level of the first object is determined, Determine a first distance related to the first object based on the first risk of the first object, A wearable electronic device comprising a memory storing instructions for providing feedback based on a first risk level of the first object, based on a distance between the first object and the user's body being less than or equal to the first distance.

2. In paragraph 1, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: Detecting at least one real object contained within the first FOV via the at least one sensor, A wearable electronic device, which detects at least one virtual object included within the first FOV and included within a second FOV corresponding to an area displayed on the display.

3. In paragraph 1 or 2, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that determines that the value of the first distance increases as the first risk of the first object increases.

4. In any one of paragraphs 1 to 3, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that determines the intensity of feedback based on the first risk level.

5. In any one of paragraphs 1 to 4, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that determines the number of feedbacks based on the first risk level.

6. In any one of paragraphs 1 to 5, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that further determines a second distance that is greater than the first distance based on the first risk level.

7. In any one of paragraphs 1 to 6, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that changes the first distance based on the speed at which the user's body approaches the first object.

8. In any one of paragraphs 1 to 7, The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that determines the risk level of the first object based on a user's usage environment related to the first object, when the first object is a virtual object.

9. In any one of paragraphs 1 to 8, further comprising a communication module; The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that transmits a command to provide said feedback to an external device connected via said communication module.

10. In any one of paragraphs 1 to 9, further comprising a communication module; The above instructions, when executed by the at least one processor, cause the wearable electronic device to: A wearable electronic device that identifies the distance between the first object and the user's body based on a detection value received from an external device connected through the communication module.

11. A method for controlling a wearable electronic device, An operation of detecting a plurality of objects of the wearable electronic device detectable via at least one sensor of the wearable electronic device or an external electronic device; An action of recognizing the type of a first object among the plurality of objects; An action to determine whether the first object is a real object or a virtual object; An operation of determining a first risk level of the first object based on the recognized type of the first object and whether the first object is the real object or the virtual object; An operation of determining a first distance related to the first object based on a first risk level of the first object; and A method for controlling a wearable electronic device, comprising: providing feedback based on a first risk level of the first object, based on a distance between the first object and the user's body being less than or equal to the first distance.

12. In paragraph 11, The operation of detecting the above multiple objects is as follows: Detecting at least one real object contained within the first FOV via the at least one sensor, A control method for a wearable electronic device, comprising: detecting at least one virtual object included in the first FOV and included in a second FOV corresponding to an area displayed on a display of the wearable electronic device.

13. In paragraph 11 or 12, The operation of determining the first distance above is: A control method for a wearable electronic device, wherein the value of the first distance is determined to be larger as the first risk of the first object is higher.

14. In any one of paragraphs 11 to 13, The action of providing the above feedback is: A method for controlling a wearable electronic device, wherein the intensity of feedback is determined based on the first risk level.

15. In any one of paragraphs 11 to 14, The action of providing the above feedback is: A method for controlling a wearable electronic device, wherein the number of feedbacks is determined based on the first risk level.

Images