Electronic device, method, and storage medium for augmenting object on basis of user operation

Abstract

Provided are an electronic device, a method, and a storage medium for augmenting an object on the basis of a user operation. The electronic device comprises: a camera for capturing an image including an object; a display; at least one processor including a processing circuit; and a memory for storing instructions individually or collectively executed by the at least one processor. The instructions stored in the memory are configured to cause the electronic device to display a captured image on a display. The instructions stored in the memory are configured to cause the electronic device to determine an object contacted by a user motion as an object operated according to a preset condition. The instructions stored in the memory are configured to cause the electronic device to store the operated object in an object list. The instructions stored in the memory are configured to cause the electronic device to augment the last position of the object and display same on the display when determining the intention of a user to find the object included in the object list.

Description

Electronic devices, methods, and storage media for augmenting objects based on user manipulation

[0001] Embodiments of the present document relate to electronic devices, methods and storage media, for example, electronic devices, methods and storage media for managing information of an object and augmenting an object based on user operations.

[0002] When a user loses an object (e.g., a smartphone, earphones, a remote control, etc.) or forgets its location, there are two ways to find it.

[0003] The first method involves objects transmitting their location information to a user's electronic device. For example, if a smart tag is attached to an object, the tag can transmit location information. The user can then use the device's augmented reality (AR) function to check the distance and / or direction of the object.

[0004] The second method involves having the user's electronic device recognize objects and identify their locations. Both methods facilitate object navigation by providing the user with the necessary information. For example, the user's electronic device could capture images of its surroundings and recognize objects based on their characteristics from the captured images. The device would then display the recognized objects on a display, allowing the user to confirm their location.

[0005] The above information may be provided solely as background information to aid in understanding the present disclosure. None of the above-described matters are claimed as prior art related to the present disclosure or can be used in determining prior art.

[0006] An electronic device according to various embodiments of the present document may include a camera for capturing an image including an object, a display, at least one processor including a processing circuit, and a memory for storing instructions to be executed individually or collectively by the at least one processor. The instructions stored in the memory may be configured to cause the electronic device to display the captured image on the display. The instructions may be configured to cause the electronic device to determine that the object touched by a user motion is a manipulated object according to a preset condition. The instructions may be configured to cause the electronic device to store the manipulated object in an object list. The instructions may be configured to cause the electronic device to augment the last position of the object and display it on the display when it is determined that the user intends to find an object included in the object list.

[0007] A method for augmenting an object based on a user manipulation according to various embodiments of the present document may include an operation of displaying an image including the object. The method may include an operation of determining that the object touched by the user motion is a manipulated object according to preset conditions. The method may include an operation of storing the manipulated object in an object list. The method may include an operation of augmenting and displaying the last location of the object if it is determined that the user intends to find an object included in the object list.

[0008] A non-transitory computer-readable storage medium having recorded thereon a program for performing a method of augmenting an object based on a user operation according to various embodiments of the present document may include an operation of displaying an image including an object. The storage medium may include an operation of determining that the object touched by a user motion is a manipulated object according to a preset condition. The storage medium may include an operation of storing the manipulated object in an object list. The storage medium may include an operation of augmenting and displaying the last position of the object when determining that the user intends to find an object included in the object list.

[0009] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0010] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0011] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to various embodiments.

[0012] FIG. 3 is a block diagram illustrating a configuration that provides object augmentation operations according to various embodiments.

[0013] FIG. 4 is a flowchart illustrating an object augmentation operation according to various embodiments.

[0014] FIG. 5 is a flowchart illustrating an operation for determining a manipulation object according to various embodiments.

[0015] FIGS. 6A, 6B, and 6C are diagrams illustrating examples of determining and managing manipulation objects according to various embodiments.

[0016] FIG. 7 is a diagram illustrating an example of determining an intention to find an object according to various embodiments.

[0017] FIGS. 8A and 8B are diagrams illustrating examples of determining an intention to find an object based on an object manipulation motion according to various embodiments.

[0018] FIG. 9 is a drawing explaining an operation of displaying an object by augmenting it according to various embodiments.

[0019] FIG. 10 is a diagram illustrating an object list according to various embodiments.

[0020] FIG. 11 and FIG. 12 are diagrams illustrating an operation of updating an object list according to various embodiments.

[0021] FIG. 13 is a diagram illustrating an example of interacting with a virtual object according to various embodiments.

[0022] FIG. 14 is a flowchart illustrating a method of augmenting an object based on user manipulation according to various embodiments.

[0023] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0024] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0025] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0026] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0027] The memory (130) can store various data used by at least one component (e.g., the processor (120) or the sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., the program (140)) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a non-volatile memory (134). The non-volatile memory (134) can include at least one internal memory (136) and an external memory (138).

[0028] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0029] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0031] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0032] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0033] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0034] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0035] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0036] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0037] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0039] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0040] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0041] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0042] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0043] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0044] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0045] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0047] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to various embodiments.

[0048] Referring to FIG. 2, the electronic device (200) may include a camera (210), a display (220), a memory (230), and a processor (240).

[0049] For example, the electronic device (200) (e.g., the electronic device (101) of FIG. 1) may be a visual wearable device. A visual wearable device may include a device that is worn in the eye area of ​​a user and displays content, images, and / or information through a display. In one example, the electronic device (200) may include a glasses-type wearable device, a head-mounted display (HMD), or a visual see-through (VST).

[0050] For example, a camera (210) (e.g., a camera module (180) of FIG. 1) can capture (or photograph) an image of the surroundings of an electronic device (200). The captured image may include an object. For example, the object may include various types of objects, such as a cup, a mouse, a pen, a remote control, a smartphone, earphones, a switch, and / or a virtual object. As an example, the electronic device (200) may include one or more cameras (210).

[0051] The display (220) (e.g., the display module (160) of FIG. 1) can display content, images, and / or information. For example, the display (220) can be implemented as a general display (e.g., a Liquid Crystal Display (LCD), an Organic Light Emitting Diode (OLED), etc.) or a transparent display. As an example, if the electronic device (200) is a glasses-type wearable device, the electronic device (200) can include a transparent display. The environment surrounding the electronic device (200) can be shown to the user through the transparent display and captured by the camera (210). According to an example, the electronic device (200) can display an image captured by the camera (210) on the transparent display. The electronic device (200) can identify an object included in the captured image and display data related to the identified object on the transparent display, thereby implementing an augmented reality (AR) environment. For example, if the electronic device (200) is an HMD, the electronic device (200) may include a general display. The electronic device (200) may display an image captured by the camera (210) on the general display. The electronic device (200) may identify an object included in the captured image and display data related to the identified object on the general display, thereby implementing an AR environment.

[0052] For example, the memory (230) (e.g., the memory (130) of FIG. 1) can store data, algorithms, programs, commands, etc. that perform functions of the electronic device (200). The commands, etc. stored in the memory (230) can be loaded into the processor (230) and executed by the processor (240) (e.g., the processor (120) of FIG. 1).

[0053] For example, the processor (240) can control each component of the electronic device (200). The electronic device (200) can include one or more processors (240) including processing circuits. The processor (240) can correspond to a plurality of processors that collectively perform a plurality of functions by dividing them among the processors.

[0054] For example, the processor (240) may display an image captured by the camera (210) on the display (220). In one example, the captured image may include an object. The processor (240) may determine an object contacted by a user motion as a manipulated object according to preset conditions. As an example, the electronic device (200) may be worn by a user, and the camera (210) may capture a certain range of an area in front of the user. An object may exist in the area in front of the user, and the user may manipulate the object. The camera (210) may capture the object and the user motion manipulating the object. The processor (240) may determine the object and the user motion interacting with the object (e.g., contact, manipulation). If the preset conditions are satisfied, the processor (240) may determine the determined object as an object manipulated by the user.

[0055] For example, the preset condition may include detection of a user motion in contact with an object for a preset period of time or longer, a change in the state of the object, and / or a change in the position of the object. For example, if the preset period of time is 5 seconds and the user holds the cup for 5 seconds or longer, the processor (240) may determine that the cup is an object manipulated by the user. For example, if the user changes a switch from an off state to an on state, the processor (240) may determine that the switch is an object manipulated by the user. For example, if the user moves a remote control located on a living room table to a TV stand, the processor (240) may determine that the remote control is an object manipulated by the user.

[0056] For example, the processor (240) may store the manipulated object in an object list. The object list may include information about the manipulated object. In one example, the information about the object may include an image of the object, location information about the object, information about the time at which the object was manipulated, and / or information about the operation type (or manipulation method) of the object. In one example, the location information about the object may be information about the most recent location of the object or information about the last location of the object. The processor (240) may generate different object lists depending on the location. In one example, the processor (240) may generate a first object list corresponding to a car, a second object list corresponding to a living room, and / or a third object list corresponding to a bedroom. The processor (240) may store the generated object list(s) in the memory (230).

[0057] When the processor (240) determines that the user intends to find an object included in the object list, the processor (240) may augment the last location of the object and display it on the display (220). For example, the electronic device (200) may include a microphone (e.g., the input module (150) of FIG. 1). As an example, when the processor (240) receives the user's voice searching for an object through the microphone, the processor may determine that the user intends to find the object. As an example, when the processor (240) identifies a movement that is repeated more than a preset number of times in an image acquired from the camera (210), the processor may determine that the user intends to find the object. For example, the electronic device (200) may include a sensor (e.g., the sensor module (176) of FIG. 1). As an example, when the processor (240) identifies a movement of the electronic device (200) that is repeated more than a preset number of times, the processor may determine that the user intends to find the object. As an example, if the processor (240) identifies a user motion corresponding to a motion manipulating an object, it can determine that the user intends to find the object.

[0058] For example, if the processor (240) determines the user's intention to find an object, it can display an object included in the object list through the display (220). The processor (240) can receive a user input for selecting one of the objects included in the displayed object list. The processor (240) can augment the last position of the selected object and display it on the display (220).

[0059] For example, an object list may include one or more objects. The processor (240) may store information about objects stored in the object list in the memory (230). The information about the object may include information about the operation type of the object. When the processor (240) identifies a user motion corresponding to a motion for manipulating an object, the processor (240) may determine the similarity between the operation method (or operation method) of an object included in the object list and the user motion. The processor (240) may determine an object that includes a similarity ratio higher than a preset ratio as the object that the user is looking for. The processor (240) may augment the last position of the object determined to be the object that the user is looking for and display it on the display (220).

[0060] For example, the object list may include multiple objects. The processor (240) may determine the similarity between the user's motion and each of the multiple objects. When displaying the objects included in the object list through the display (220), the processor (240) may also provide the similarity of each of the multiple objects.

[0061] For example, the processor (240) may identify a user wearing the electronic device (200) and perform an action based on the motion and / or input of the user wearing the electronic device (200). As an example, the processor (240) may store characteristic information of a body part (e.g., a hand) of the user and / or characteristic information of a voice in the memory (230).

[0062] For example, when the processor (240) detects a user motion, it can determine whether the body part performing the user motion substantially matches the stored feature information of the body part. If the body part performing the user motion substantially matches the stored feature information of the body part, the processor (240) can determine that the motion is a motion of a user wearing the electronic device (200). If the user motion corresponds to a motion for manipulating an object, the processor (240) can determine that the user motion is an intention to find the object. If the body part performing the user motion does not match the feature information of the stored body part, the processor (240) can ignore the user motion.

[0063] For example, when the processor (240) receives a user's voice, it can determine whether the received user's voice substantially matches the characteristic information of the stored voice. If the characteristic information of the received user's voice and the stored voice substantially match, the processor (240) can determine that the user's voice is the voice of the user wearing the electronic device (200). If the user's voice is a voice searching for an object, the processor (240) can determine that the user's voice is intended to search for an object. If the characteristic information of the user's voice and the stored voice do not match, the processor (240) can ignore the user's voice.

[0064] For example, if an object related to an application being executed and / or a device on which the application is executed is included in the object list, the processor (240) may provide a visual effect to the object related to the application being executed and / or the device on which the application is executed. In one example, the processor (240) may identify content displayed on a TV and / or a TV displaying content. A TV remote control may be included in the object list, and the TV remote control may be an object related to the TV and / or content displayed on the TV. The processor (240) may provide a visual effect to the remote control included in the object list. In one example, the visual effect may include an indicator, a color, a shade, a border, and / or a blink.

[0065] For example, the processor (240) can update information of an object included in an object list. As an example, the processor (240) can receive location information of the object. According to an example, if the object is an electronic device (e.g., a smartphone) including a communication function, the processor (240) can receive location information directly from the object. According to an example, the processor (240) can receive location information of the object from an external electronic device through a communication interface (e.g., the communication module (190) of FIG. 1). The processor (240) can update the location information of the object based on the received location information of the object.

[0066] The electronic device, method, and storage medium of this document can determine the user's intention and easily check information about an object the user is looking for.

[0067] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0068] FIG. 3 is a block diagram illustrating a configuration that provides object augmentation operations according to various embodiments.

[0069] Referring to FIG. 3, the electronic device (200 of FIG. 2) may include a sensor data manager (310), an object history list (320), a detection manager (330), and a display (340). For example, the sensor data manager (310), the object history list (320), and the detection manager (330) may be stored in a memory (230 of FIG. 2) or loaded into a processor (240 of FIG. 2) for operation.

[0070] For example, the sensor data manager (310) may be a device that manages a camera module (311) (e.g., the camera module (180) of FIG. 1 or the camera (210) of FIG. 2) and a sensor module (312) (e.g., the sensor module (176) of FIG. 1) of an electronic device (200 of FIG. 2). The camera module (311) may acquire images and / or depth information. In one example, the camera module (311) may capture an image including an object and acquire depth information of the object based on a distance from the object. The sensor module (312) may acquire location information of the electronic device (200) using a gyro sensor and / or a global position system (GPS).

[0071] For example, the object history list (320) may store information about objects with which the user has interacted in chronological order. As an example, the object information may include an image of the object (321), location information of the object (322), information about the time at which the object was manipulated (323), and / or information about the object's operation type (324).

[0072] For example, the detection manager (330) can receive image and location information from the sensor data manager (310) and extract necessary information. As an example, the object detection module (331) can identify objects within the user's field of view. The motion detection module (332) can determine the user's intention based on a preset body part of the user (e.g., a hand). As an example, the user's intention can be determined by identifying an interaction such as grabbing or manipulating an object based on a captured image and / or received voice. The detection manager (330) can determine the user's intention by integrating object detection information and motion detection information. According to an example, the electronic device (200 of FIG. 2) can determine the user's intention using deep learning, classical image processing, and / or signal processing methods using the detected data.

[0073] For example, when the user's intention to find an object is determined, the electronic device (200 in FIG. 2) can augment the information of the object based on the information stored in the object history list (320) and display it on the display (340).

[0074] FIG. 4 is a flowchart illustrating an object augmentation operation according to various embodiments.

[0075] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0076] According to one embodiment, steps 410 to 480 may be understood to be performed in a processor (e.g., processor (240) of FIG. 2) of an electronic device (e.g., electronic device (200) of FIG. 2).

[0077] Referring to FIG. 4, an electronic device (200 of FIG. 2) may be worn by a user (410). As an example, the electronic device (200 of FIG. 2) may be a visual wearable device such as a glasses-type wearable device, a head-mounted display (HMD), or a visual see-through (VST).

[0078] For example, an electronic device (200 of FIG. 2) can detect a user's motion (420). When the electronic device (200 of FIG. 2) is worn by a user, the electronic device (200 of FIG. 2) can continuously detect the user's motion. The electronic device (200 of FIG. 2) can determine whether the user's motion manipulates (or interacts with) an object (430). For example, the electronic device (200 of FIG. 2) can determine whether the object is manipulated based on preset conditions. The preset conditions for determining whether the object is manipulated may include the user's motion coming into contact with the object for a preset period of time or longer, a change in the state of the object, and / or a change in the position of the object.

[0079] For example, if it is determined that an object has been manipulated (430-Yes), the electronic device (200 of FIG. 2) can add the object to the object list (440). The electronic device (200 of FIG. 2) can add the object and continuously detect the user's motion (420). For example, if it is not determined that the object has been manipulated (430-No), the electronic device (200 of FIG. 2) can determine whether there is an intention to find the object (450). For example, the electronic device (200 of FIG. 2) can determine whether there is an intention to find the object based on preset conditions. As an example, the preset conditions for determining the user's intention may include reception of the user's voice to find the object, movement of an image repeated more than a preset number of times, movement of the electronic device (200 of FIG. 2) repeated more than a preset number of times, and / or identification of a user motion corresponding to a motion of manipulating the object.

[0080] For example, if the determined intent is not an intent to find an object (450-No), the user's motion can be continuously detected (420). For example, if the determined intent is an intent to find an object (450-Yes), the electronic device (200 of FIG. 2) can display a list of objects (460). For example, the list of objects can include one or more objects. The electronic device (200 of FIG. 2) can select an object based on the user's selection input (470) and display the location of the selected object in an augmented manner (480).

[0081] When the electronic device (200 of FIG. 2) determines the user's intention by identifying the user's motion corresponding to the motion of manipulating an object, the electronic device (200 of FIG. 2) may omit displaying the object list and selecting the object. According to one example, when the user's motion is similar to the motion type information of the stored object by a preset ratio or more, the electronic device (200 of FIG. 2) may determine an object that includes a similarity higher than the preset ratio as the object that the user is looking for. According to one example, the object list may include a plurality of objects. The electronic device (200 of FIG. 2) may determine the similarity between the user's motion and the motion type information of each of the plurality of objects. When the electronic device (200 of FIG. 2) displays the objects included in the object list through the display (220), the similarity of each of the plurality of objects may be provided together. According to one example, when the electronic device is removed, the operation of the electronic device (200 of FIG. 2) may be terminated.

[0082] FIG. 5 is a flowchart illustrating an operation for determining a manipulation object according to various embodiments.

[0083] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0084] According to one embodiment, steps 510 to 560 may be understood to be performed in a processor (e.g., processor (240) of FIG. 2) of an electronic device (e.g., electronic device (200) of FIG. 2).

[0085] Referring to FIG. 5, an electronic device (200 of FIG. 2) can track a user's hand (or body part) (510). When the electronic device (200 of FIG. 2) is worn by a user, the electronic device (200 of FIG. 2) can continuously detect the user's hand. The electronic device (200 of FIG. 2) can determine whether the user's hand is in contact with an object (520).

[0086] For example, if the user's hand does not come into contact with the object (520-No), the electronic device (200 of FIG. 2) can continuously detect and track the user's hand (510). The user's hand in contact with the object can be released from contact after a certain period of time. If the user's hand comes into contact with the object (520-Yes), the electronic device (200 of FIG. 2) can determine whether the user's hand is released from contact with the object (530). The electronic device (200 of FIG. 2) can determine the time that the user's hand and the object maintain contact. The electronic device (200 of FIG. 2) can determine whether the time that the contact is maintained is longer than a preset time (540).

[0087] For example, if the time that the user's hand and the object maintain contact is longer than a preset time (540-Yes), the electronic device (200 of FIG. 2) can determine that the object that came into contact with the user's hand is a manipulation object (550). If the time that the user's hand and the object maintain contact is shorter than the preset time (540-No), the electronic device (200 of FIG. 2) can determine whether the position and / or state of the object that came into contact with the user's hand has changed (560).

[0088] For example, if the position and / or state of an object changes (560-Yes), the electronic device (200 of FIG. 2) may determine that the object in contact with the user's hand is a manipulation object (550). If the position and / or state of the object does not change (560-No), the electronic device (200 of FIG. 2) may not determine that the object in contact with the user's hand is a manipulation object. The electronic device (200 of FIG. 2) may continuously detect and track the user's hand (510).

[0089] FIGS. 6A, 6B, and 6C are diagrams illustrating examples of determining and managing manipulation objects according to various embodiments.

[0090] Referring to FIG. 6A, an electronic device (200) is illustrated. The electronic device (200) can display an image captured by a camera (210 of FIG. 2) on a display (220 of FIG. 2). As an example, as illustrated in FIG. 6A, when a user holds a cup (21), the electronic device (200) can display the user's hand (22) holding the cup (21) on the display (220 of FIG. 2).

[0091] For example, the electronic device (200) can identify an object included in an image. In one example, the electronic device (200) can identify a cup (21) included in the image. The electronic device (200) can continuously detect the user's hand (22) based on the captured image. The electronic device (200) can determine whether the user's hand (22) is in contact (or interacts) with the object. In one example, the electronic device (200) can determine an area (21a) where the cup (21) is located and an area (22a) where the user's hand (22) is located. If the area (21a) where the cup (21) is located and the area (22a) where the user's hand (22) is located overlap by a certain ratio or more, the electronic device (200) can determine that the user's hand (22) is in contact with the cup (21).

[0092] For example, when the contact between the user's hand (22) and the cup (21) is released, the electronic device (200) can determine the contact time between the user's hand (22) and the cup (21). If the contact time is longer than a preset time, the electronic device (200) can determine that the cup (21) is an object manipulated by the user.

[0093] For example, if the cup (21) is determined to be a manipulated object, the electronic device (200) may store the cup (21) in the object list (10). The electronic device (200) may store information about the cup (21) in the object list (10). As an example, the information about the cup (21) stored in the object list (10) may include an image of the cup (21), location information about the cup (21), information about the time at which the cup (21) was manipulated, and / or information about the operation type of the cup (21). The electronic device (200) may store objects in chronological order. According to an example, the electronic device (200) may store the most recently contacted cup (21) at the top of the object list (10), and display information (26) (e.g., image, time) about the cup at the top of the object list (10).

[0094] For example, if an object that has come into contact with the user's hand (22) is already included in the object list (10), the electronic device (200) can update information (e.g., location, time) of the object included in the object list (10). In one example, if a cup (21) that has come into contact with the user's hand (22) is already included in the object list (10), the electronic device (200) can update information (26) of the cup. If the object that has come into contact with the user's hand (22) has a similar shape to an object stored in the object list (10) but is of a different type, the electronic device (200) can separately add the object that has come into contact with the user's hand (22). In one example, if a cup of a first type is included in the object list (10) and the object that has come into contact with the user's hand (22) is a cup of a second type, the electronic device (200) can add the cup of the second type to the object list (10).

[0095] For example, the electronic device (200) can add objects to the object list (10) up to a preset number. If the electronic device (200) adds objects exceeding the preset number, the electronic device (200) can delete the object stored first among the objects included in the object list (10) and add the object.

[0096] Referring to Fig. 6b, an object whose position has changed is illustrated. The electronic device (200 of Fig. 2) can determine that an object that comes into contact with the user and whose position has changed is a manipulated object. For example, the user can move a cup (21) located on a table (1) onto a TV stand (3). The electronic device (200 of Fig. 2) can identify the cup (21) on the table (1). The electronic device (200 of Fig. 2) can detect that the cup (21) comes into contact with the user's hand, moves onto the TV stand (3), and then releases contact with the user's hand. After the cup (21) is released from the user's hand, the electronic device (200 of Fig. 2) can determine that the cup (21) has moved from the table (1) to the TV stand (3). Since the position of the cup (21) has changed, the electronic device (200 in FIG. 2) can determine the cup (21) as a manipulated object and store it in the object list.

[0097] Referring to FIG. 6c, an object whose state has changed is illustrated. The electronic device (200 of FIG. 2) can determine an object that has come into contact with a user and whose state has changed as a manipulated object. For example, a switch may be in an off state (23a). The user may manipulate the switch in the off state (23a) to an on state (23b). The electronic device (200 of FIG. 2) can identify the switch in the off state (23a). The electronic device (200 of FIG. 2) can detect that the switch in the off state (23a) comes into contact with the user's hand, changes to an on state (23b), and then releases the contact with the user's hand. Since the switch in the off state (23a) has changed to an on state (23b), the electronic device (200 of FIG. 2) can determine the switch as a manipulated object and store it in the object list.

[0098] FIG. 7 is a diagram illustrating an example of determining an intention to find an object according to various embodiments.

[0099] Referring to Figure 7, various methods for determining the intent to locate an object are illustrated. For example, an electronic device (200 of Figure 2) can analyze data collected from a sensor data manager (310 of Figure 3) and learn patterns in advance.

[0100] For example, a camera (210 of FIG. 2) can capture an image. A user wearing an electronic device (200 of FIG. 2) can look around to find an object. As the user looks around, the captured image can move up, down, left, and right repeatedly a predetermined number of times. The electronic device (200 of FIG. 2) can learn the user's pattern using a classical image processing method and / or an image-based deep learning method. The electronic device (200 of FIG. 2) can determine the user's intention to find an object based on the movement of the image that is repeated a predetermined number of times or more.

[0101] In one example, a microphone can receive a user's voice. A user wearing an electronic device (200 of FIG. 2) can utter specific words and / or sentences when searching for an object. The electronic device (200 of FIG. 2) can learn the user's speech using classical audio processing and / or audio-based deep learning. The electronic device (200 of FIG. 2) can determine the user's intention to search for an object based on the user's speech containing specific words and / or sentences.

[0102] According to an example, an electronic device (200 of FIG. 2) may include a sensor. The sensor may detect location information of the electronic device (200 of FIG. 2). For example, the sensor may include a gyro sensor. A user wearing the electronic device (200 of FIG. 2) may look around when searching for an object. When the user looks around, the electronic device (200 of FIG. 2) may move in an up, down, left, and right direction repeatedly a predetermined number of times or more. The electronic device (200 of FIG. 2) may learn the user's pattern using a support vector machine (SVM) motion classification method and / or a gyro sensor-based deep learning method. The electronic device (200 of FIG. 2) may determine the user's intention to search for an object based on the movement of the electronic device (200 of FIG. 2) repeated a preset number of times or more.

[0103] FIGS. 8A and 8B are diagrams illustrating examples of determining an intention to find an object based on an object manipulation motion according to various embodiments.

[0104] For example, an electronic device (200 of FIG. 2) can determine an intention to find an object by identifying a user motion corresponding to a motion of manipulating an object.

[0105] Referring to FIG. 8A, as an example, an electronic device (200 of FIG. 2) can display a computer (30) on a display (220 of FIG. 2) and identify the computer (30) as an object. The electronic device (200 of FIG. 2) can detect and track a user's hand (32). The user's hand (32) can move in a motion that operates a mouse (36). As an example, the user's hand (32) can move left and right or make a click motion. The electronic device (200) can store one or more objects in an object list (10). As an example, the object list (10) can store a mouse (36). The object list (10) can store information about the mouse (36). According to an example, the object list (10) may include an image of a mouse (36), position information of the mouse (36), time information at which the mouse (36) was operated, and / or operation type information of the mouse (36).

[0106] The electronic device (200 of FIG. 2) can determine the similarity between the motion type information of the object stored in the object list (10) and the user's motion. For example, since the user's motion is a motion of operating a mouse (36), it may be similar to the motion type information of the mouse (36) by a preset ratio or more. The electronic device (200 of FIG. 2) can determine that the mouse (36) including the motion type information that is similar to the user's motion by a preset ratio or more is the object the user is looking for.

[0107] Referring to FIG. 8B, as an example, an electronic device (200 of FIG. 2) can display a tablet PC (40) on a display (220 of FIG. 2) and identify the tablet PC (40) as an object. The electronic device (200 of FIG. 2) can detect and track a user's hand (42). The user's hand (42) can move in a motion that manipulates a stylus pen (46). As an example, the user's hand (32) can make a writing motion. The electronic device (200) can store one or more objects in an object list (10). As an example, the object list (10) can store a stylus pen (46). The object list (10) can store information about the stylus pen (46). According to an example, the object list (10) may include an image of a stylus pen (46), location information of the stylus pen (46), time information at which the stylus pen (46) was operated, and / or operation type information of the stylus pen (46).

[0108] The electronic device (200 of FIG. 2) can determine the similarity between the motion type information of the object stored in the object list (10) and the user's motion. For example, since the user's motion is a motion of operating a stylus pen (46), the motion type information of the stylus pen (46) may be similar to the motion type information of the stylus pen (46) by a preset ratio or more. The electronic device (200 of FIG. 2) can determine that the stylus pen (46) including the motion type information that is similar to the user's motion by a preset ratio or more is the object that the user is looking for.

[0109] For example, when the object list (10) includes multiple objects, the electronic device (200 of FIG. 2) can determine the similarity between the user's motion and the motion type information of each of the multiple objects, and provide the similarity of each of the multiple objects. Based on the provided similarity, the electronic device (200 of FIG. 2) can determine the object selected by the user as the object the user is looking for.

[0110] FIG. 9 is a drawing explaining an operation of displaying an object by augmenting it according to various embodiments.

[0111] Referring to FIG. 9, a drawing is shown in which an object that a user is searching for is augmented and displayed. The electronic device (200) can augment and display an object determined to be the object to be searched among one or more objects included in the object list (10).

[0112] For example, the object list (10) may include a plurality of objects. A remote control (56) may be selected from among the plurality of objects. As an example, when a received user's voice, a movement of an image that is repeated more than a preset number of times, and / or a movement of the electronic device (200) that is repeated more than a preset number of times are detected, the electronic device (200) may determine that the user's voice and / or movement is an intention of the user to find an object. The electronic device (200) may display an object list (10) that includes a plurality of objects. The electronic device (200) may determine that an object selected by the user from among the plurality of objects is an object to be found.

[0113] As illustrated in FIG. 9, according to an example, when a remote control (56) is selected from the object list (10), the electronic device (200) may determine that the remote control (56) is an object that the user is looking for. As an example, the object list (10) may include an image of the remote control (56) and / or location information of the remote control (56). The electronic device (200) may augment the remote control (56) and display it on the display (220 of FIG. 2) based on the image of the remote control (56) and / or the location information of the remote control (56). If the last location of the remote control (56) is the TV stand (3), the electronic device (200) may augment and display the image (59) of the augmented remote control on the TV stand (3).

[0114] As an example, to enable the user to easily recognize the object they are looking for and the last location of the object, the electronic device (200) may display a first indicator (57a) around the remote control (56) of the object list (10) and display a second indicator (57b) around the image (59) of the remote control. For example, the indicators (57a, 57b) may include symbols, shapes, shades, colors, and / or highlights.

[0115] As an example, the electronic device (200) can determine the user's intention to find an object based on a user motion corresponding to the object's operation method. When the electronic device (200) identifies a user motion corresponding to the object's operation method, the electronic device (200) can determine the similarity between the operation method of an object included in the object list (10) and the user motion. According to an example, when a user motion including a button click operation that is identical or similar to the operation method of a remote control (56) is identified, the electronic device (200) can determine an object including a similarity greater than or equal to a preset ratio as the object the user is trying to find. The electronic device (200) can determine a remote control (56) including operation type information that is similar to the user's motion greater than or equal to a preset ratio as the object the user is trying to find. The electronic device (200) can augment the image (59) of the remote control at the last position of the remote control (56) and display it on the display (220 of FIG. 2). The electronic device (200) can display a second indicator (57b) around the image (59) of the remote control.

[0116] FIG. 10 is a diagram illustrating an object list according to various embodiments.

[0117] Referring to FIG. 10, a plurality of object lists (10a, 10b, 10c) are illustrated. The object lists (10a, 10b, 10c) may be generated for each location. For example, the electronic device (200) may generate a first object list (10a) corresponding to a house (61), a second object list (10b) corresponding to a car (63), and a third object list (10c) corresponding to a workroom (65), respectively. For example, locations where the object lists are generated separately may be divided into each area within a single location, such as a living room, a room, a kitchen, and / or a porch.

[0118] For example, an electronic device (200 in FIG. 2) can obtain location information of the electronic device (200 in FIG. 2) using an access point (AP) for Wi-Fi communication, a Bluetooth location beacon, a base station for cellular communication, and / or GPS. The electronic device (200 in FIG. 2) can display a list of corresponding objects based on the obtained location information and store the objects.

[0119] FIG. 11 and FIG. 12 are diagrams illustrating an operation of updating an object list according to various embodiments.

[0120] The electronic device (200) can receive location information from an external device and update information of objects stored in the object list based on the received location information.

[0121] Referring to FIG. 11, an electronic device (200) that receives location information from an object is illustrated. For example, if the object includes a communication function, the electronic device (200) can receive information from the object. In one example, the object may be a smartphone (71). The object list of the electronic device (200) may include information (73) of the smartphone. For example, the smartphone (71) may periodically calculate location information of the device, and when the location information changes, transmit the location information of the device. The electronic device (200) may receive location information from the smartphone (71). When location information is received from the smartphone (71), the electronic device (200) may update information (73) of the smartphone included in the object list. For example, the electronic device (200) may update the received location and / or reception time.

[0122] Referring to FIG. 12, an electronic device (200b) receiving location information from an external electronic device (200a) is illustrated. For example, the external electronic device (200a) may be a visual wearable device such as a glasses-type wearable device, an HMD, or a VST. Each of the external electronic device (200a) and the electronic device (200b) may include the same object in their object lists. In one example, the object list of the external electronic device (200a) may include information (86) about a cup, and the object list of the electronic device (200b) may also include information (87) about the same cup.

[0123] According to one example, the external electronic device (200a) can detect contact between the user and the cup (81) and determine that the cup (81) is a manipulated object. For example, if the contact between the user and the cup (81) is maintained for a certain period of time and / or the position of the cup (81) changes, the external electronic device (200a) can determine that the cup (81) is a manipulated object. If the cup (81) is determined to be a manipulated object, the external electronic device (200a) can update the information (86) of the cup in the object list. For example, the external electronic device (200a) can update the position and / or manipulation time of the cup (81). The external electronic device (200a) can transmit data related to the changed cup (81) to the electronic device (200b). When receiving data related to a changed cup (81) from an external electronic device (200a), the electronic device (200b) can update information (87) about the cup included in the object list. For example, the electronic device (200b) can update the position and / or operation time of the cup (81).

[0124] FIG. 13 is a diagram illustrating an example of interacting with a virtual object according to various embodiments.

[0125] Referring to FIG. 13, an electronic device (200) including a virtual object in an object list (10) is illustrated. The electronic device (200) can store not only objects included in a captured image but also generated virtual objects in the object list.

[0126] According to one example, the electronic device (200) can create a dog (91) as a virtual object and display the dog (91) created in the AR environment on the display (220 of FIG. 2). When the user interacts with (e.g., makes contact with) the dog (91), the electronic device (200) can determine whether to store the dog (91) in the object list. For example, when the electronic device (200) detects a user motion that makes contact with the dog (91) and / or a change in the location of the dog (91), the electronic device (200) can store the dog (91) in the object list. For example, the electronic device (200) can store an image of the dog (91), location information of the dog (91), time information of contact with the dog (91), and / or information on the method of contacting the dog (91) in the object list. According to one example, the information on the method of contacting the dog (91) can include information such as stroking and / or feeding.

[0127] For example, if it is determined that the user intends to find a dog (91) based on the user's voice and / or movement, the electronic device (200) can display a list of objects. If a dog (91) included in the object list is selected, the electronic device (200) can augment and display the last location information of the dog (91). As an example, if a user motion that is the same or similar to the information on the way of contacting the dog (91) is detected, the electronic device (200) can determine the similarity between the information on the way of contacting the dog (91) included in the object list and the information on the way of contacting the dog (91). If the similarity between the detected user motion and the information on the way of contacting the dog (91) is the highest, the electronic device (200) can select the dog (91) as the object to be found and augment and display the last location of the dog (91).

[0128] FIG. 14 is a flowchart illustrating a method of augmenting an object based on user manipulation according to various embodiments.

[0129] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0130] According to one embodiment, steps 1410 to 1440 may be understood to be performed in a processor (e.g., processor (240) of FIG. 2) of an electronic device (e.g., electronic device (200) of FIG. 2).

[0131] Referring to FIG. 14, an electronic device (200 of FIG. 2) can display an image including an object (1410). For example, the electronic device (200 of FIG. 2) can capture an image of its surroundings using a camera (210 of FIG. 2). The captured image can include an object. For example, the object can include various types of objects, such as a cup, a mouse, a pen, a remote control, a smartphone, earphones, a switch, and / or a virtual object.

[0132] For example, an electronic device (200 of FIG. 2) can determine an object touched by a user motion as a manipulated object according to preset conditions (1420). The electronic device (200 of FIG. 2) can determine an object and a user motion interacting with the object (e.g., touching, manipulating). If the preset conditions are satisfied, the electronic device (200 of FIG. 2) can determine the determined object as an object manipulated by the user. For example, the preset conditions may include detection of a user motion touching the object for a preset period of time or longer, a change in the state of the object, and / or a change in the position of the object.

[0133] For example, an electronic device (200 of FIG. 2) may store a manipulated object in an object list (1430). The object list may include information about the manipulated object. For example, the information about the object may include an image of the object, location information about the object, information about the time the object was manipulated, and / or information about the operation type of the object. For example, the location information about the object may be the most recent location information or the last location information of the object.

[0134] For example, the electronic device (200 of FIG. 2) can display the last position of the object by augmenting it (1440). For example, if the electronic device (200 of FIG. 2) receives the voice of a user searching for an object through a microphone, the electronic device can determine that the user intends to search for the object. For example, if the electronic device (200 of FIG. 2) identifies the movement of an image that is repeated more than a preset number of times, the electronic device can determine that the user intends to search for the object. For example, if the electronic device (200 of FIG. 2) identifies the movement of the electronic device (200 of FIG. 2) that is repeated more than a preset number of times, the electronic device can determine that the user intends to search for the object. For example, if the electronic device (200 of FIG. 2) identifies a user motion corresponding to a motion of manipulating an object, the electronic device can determine that the user intends to search for the object. For example, if the electronic device (200 of FIG. 2) determines that the user intends to search for an object, the electronic device can display an object included in an object list. An electronic device (200 of FIG. 2) can receive a user input for selecting one of the objects included in a displayed object list. The electronic device (200 of FIG. 2) can display the last position of the selected object by augmenting it.

[0135] As an example, an electronic device (200) may include a camera (210) for capturing an image including an object, a display (220), at least one processor (240) including a processing circuit, and a memory (230) for storing instructions that are individually or collectively executed by the at least one processor. The instructions stored in the memory (230) may be configured to cause the electronic device (200) to display the captured image on the display (220). The instructions may be configured to cause the electronic device (200) to determine that the object contacted by a user motion is a manipulated object according to preset conditions. The instructions may be configured to cause the electronic device (200) to store the manipulated object in an object list. The instructions may be configured to cause the electronic device (200) to augment the last position of the object and display it on the display (220) when it is determined that the user intends to find an object included in the object list.

[0136] As an example, the preset condition may include at least one of detection of the user motion contacting the object for a preset period of time or longer, a change in the state of the object, and a change in the position of the object.

[0137] As an example, the electronic device (200) may further include a microphone. The command may be configured to cause the electronic device (200) to determine an intention to find the object based on at least one of: reception of the user's voice searching for the object through the microphone; movement of the image repeated a preset number of times or more; movement of the electronic device repeated a preset number of times or more; and identification of the user's motion corresponding to a motion manipulating the object.

[0138] As an example, the command may be configured to cause the electronic device (200) to determine the similarity between the operation modes of the plurality of objects and the user motion when the object list includes a plurality of objects including operation modes corresponding to the user motion, and to provide the similarity of each of the plurality of objects.

[0139] As an example, the command may be configured to cause the electronic device (200) to store at least one of characteristic information of a body part including the user's hand and characteristic information of the user's voice. The command may be configured to cause the electronic device (200) to determine whether the user motion including the characteristic information of the body part and the voice including the characteristic information of the user's voice are intended to find the object when identifying the user motion.

[0140] As an example, the command may be configured to cause the electronic device (200) to display an object included in the object list through the display (220) when determining that the user intends to find the object. The command may be configured to cause the electronic device (200) to display an object included in the object list on the display (220) by augmenting the last position of the object when receiving a user input selecting the object included in the object list.

[0141] As an example, the command may be configured to cause the electronic device (200) to store a plurality of object lists corresponding to different locations in the memory (230).

[0142] As an example, the command may be configured to cause the electronic device (200) to receive location information of the object included in the object list from an external device and update the location information of the object included in the object list.

[0143] As an example, the object list may include at least one of an image of the object, location information of the object, time information at which the object was manipulated, and operation type information of the object.

[0144] As an example, the command may be configured to cause the electronic device (200) to provide a visual effect to an object related to an application being executed, if the object is included in the object list.

[0145] As an example, a method for augmenting an object based on a user operation in an electronic device (200) may include an operation of displaying an image including the object. The method may include an operation of determining that the object contacted by the user motion is a manipulated object according to preset conditions. The method may include an operation of storing the manipulated object in an object list. The method may include an operation of augmenting and displaying the last location of the object when it is determined that the user intends to find an object included in the object list.

[0146] As an example, the preset condition may include at least one of detection of the user motion contacting the object for a preset period of time or longer, a change in the state of the object, and a change in the position of the object.

[0147] As an example, the method may further include an operation of determining an intention to find the object based on at least one of: receiving a voice of the user searching for the object, a movement of the image repeated a first or more preset number of times, a movement of the electronic device repeated a second or more preset number of times, and identification of a user motion corresponding to a motion of manipulating the object.

[0148] As an example, the method may further include an operation of determining a similarity between the motion modes of the plurality of objects and the motion of the user, if the object list includes a plurality of objects having motion modes corresponding to the user motion. The method may further include an operation of providing the similarity of each of the plurality of objects.

[0149] As an example, the method may further include an operation of storing at least one of characteristic information of a body part including the user's hand and characteristic information of the user's voice. The method may further include an operation of determining whether the user motion including the characteristic information of the body part and the voice including the characteristic information of the user's voice are intended to find the object when identifying the user motion.

[0150] As an example, the operation of displaying the last position of the object by augmenting it may be performed by displaying an object included in the object list when determining that the user intends to find the object, and by displaying the last position of the object by augmenting it when receiving a user input for selecting an object included in the object list.

[0151] As an example, the method may further include the operation of storing a plurality of lists of objects corresponding to different locations.

[0152] As an example, the method may further include an operation of receiving location information of the object included in the object list from an external device. The method may further include an operation of updating the location information of the object included in the object list.

[0153] As an example, the object list may include at least one of an image of the object, location information of the object, time information at which the object was manipulated, and operation type information of the object.

[0154] As an example, a non-transitory computer-readable storage medium having recorded thereon a program for performing a method of augmenting an object based on a user operation may include an operation of displaying an image including the object. The storage medium may include an operation of determining that the object touched by the user motion is a manipulated object according to a preset condition. The storage medium may include an operation of storing the manipulated object in an object list. The storage medium may include an operation of augmenting and displaying the last position of the object when it is determined that the user intends to find an object included in the object list.

[0155] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0156] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0157] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0158] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0159] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0160] The effects of this document are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the above description.

Claims

1. In electronic devices, A camera that captures images containing objects; display; At least one processor comprising a processing circuit; and A memory for storing instructions to be executed individually or collectively by at least one processor; The instructions stored in the above memory cause the electronic device to: Displaying the captured image on the display, The object contacted by the user motion is judged as a manipulated object according to preset conditions, Save the above manipulated object to the object list, An electronic device configured to augment the last location of the object and display it on the display when the user determines that the user intends to find an object included in the object list.

2. In paragraph 1, The above-mentioned conditions are: An electronic device comprising at least one of: detecting a user motion in contact with the object for a preset period of time, changing the state of the object, and changing the position of the object.

3. In paragraph 1, Including Mike; The instructions stored in the above memory cause the electronic device to: An electronic device configured to determine an intention to find the object based on at least one of: reception of the user's voice searching for the object through the microphone; movement of the image repeated a preset number of times or more; movement of the electronic device repeated a preset number of times or more; and identification of the user's motion corresponding to a motion manipulating the object.

4. In paragraph 3, The instructions stored in the above memory cause the electronic device to: An electronic device configured to determine the similarity between the operation modes of the plurality of objects and the user motion when a plurality of objects including operation modes corresponding to the user motion are included in the object list, and to provide the similarity of each of the plurality of objects.

5. In paragraph 3, The instructions stored in the above memory cause the electronic device to: Store at least one of the characteristic information of the body part including the hand of the user and the characteristic information of the voice of the user, An electronic device configured to determine whether there is an intention to find the object by identifying the user motion including characteristic information of the body part and the voice including characteristic information of the user's voice.

6. In paragraph 1, The instructions stored in the above memory cause the electronic device to: An electronic device that is set to display objects included in the object list through the display when determining that the user intends to find the object, and to augment the last position of the object and display it on the display when receiving a user input for selecting an object included in the object list.

7. In paragraph 1, The instructions stored in the above memory cause the electronic device to: An electronic device configured to store multiple lists of objects corresponding to different locations in said memory.

8. In paragraph 1, The instructions stored in the above memory cause the electronic device to: An electronic device configured to receive location information of the objects included in the object list from an external device and update the location information of the objects included in the object list.

9. In paragraph 1, The above list of objects is: An electronic device comprising at least one of an image of the object, location information of the object, time information at which the object was manipulated, and operation type information of the object.

10. In paragraph 1, The instructions stored in the above memory cause the electronic device to: An electronic device configured to provide a visual effect to an object related to a running application when the object is included in the object list.

11. A method for augmenting an object based on user manipulation in an electronic device, The action of displaying an image containing an object; An action of judging an object touched by a user motion as a manipulated object according to preset conditions; An action of storing the above-mentioned manipulated object in an object list; and A method comprising: an operation of displaying the last position of the object by augmenting it when determining that the user intends to find an object included in the object list.

12. In paragraph 11, The above-mentioned conditions are: A method comprising at least one of detecting a user motion in contact with the object for a preset period of time, changing the state of the object, and changing the position of the object.

13. In paragraph 11, A method further comprising: an operation of determining an intention to find the object based on at least one of: receiving a voice of the user searching for the object; movement of the image repeated a preset number of times or more; movement of the electronic device repeated a preset number of times or more; and identification of a user motion corresponding to a motion of manipulating the object.

14. In paragraph 13, When a plurality of objects including a motion method corresponding to the user motion are included in the object list, an operation of determining the similarity between the motion method of the plurality of objects and the user's motion; and A method further comprising: providing the similarity of each of the plurality of objects; 15. A non-transitory computer-readable storage medium having recorded thereon a program for performing a method of augmenting an object based on user operation, The action of displaying an image containing an object; An action of judging an object touched by a user motion as a manipulated object according to preset conditions; An action of storing the above-mentioned manipulated object in an object list; and A non-transitory computer-readable storage medium having recorded thereon a program for performing a method including: an operation of displaying the last position of the object by augmenting it when the user determines that the user intends to find an object included in the object list;

Images