Method for using real-world object as input tool, and electronic device for carrying out same

Abstract

Provided are a method for using a real-world object as an input tool, and an electronic device for carrying out same. The electronic device comprises: a camera for obtaining an image by photographing a real-world object in a real-world space and a user's hand; a memory for storing at least one instruction; and at least one processor for executing the at least one instruction stored in the memory, wherein the at least one processor may recognize the real-world object interacting with the user's hand from the obtained image, identify a pre-registered function using the recognized real-world object as an input tool, and carry out the identified pre-registered function.

Description

Method for using real objects as input tools and electronic devices for doing so

[0001] The present disclosure relates to an electronic device and method for using a real-world object as an input tool. More specifically, the present disclosure relates to an electronic device and method for detecting a real-world object that does not communicate with the electronic device and performing a predetermined function based on the detection result.

[0002] Augmented Reality (AR) is a technology that overlays virtual images onto the physical environment or real-world objects of the real world, while Virtual Reality (VR) is a technology that allows users to interact with or independently display virtual objects that do not actually exist within a virtual environment. Recently, AR devices (e.g., smart glasses) or VR devices that utilize AR and VR technologies are being used in useful ways in daily life, such as for information retrieval, route guidance, camera shooting, and games.

[0003] Since augmented reality devices or virtual reality devices do not allow touch operation due to their nature, the user's hand gestures or signal transmission and reception through a paired electronic device are used as input interfaces to provide services.

[0004] In one embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a camera that captures images of a real object and a user's hand in a real space. The electronic device may include a memory that stores at least one instruction. The electronic device may include at least one processor that executes the at least one instruction stored in the memory. The at least one processor may recognize the real object interacting with the user's hand from the captured image. The at least one processor may identify a pre-registered function that uses the recognized real object as an input tool. The at least one processor may perform the identified pre-registered function.

[0005] In one embodiment of the present disclosure, a method for utilizing a real-world object as an input tool of an electronic device may be provided. The method may include a step of capturing an image of a real-world object and a user's hand in a real-world space. The method may include a step of recognizing the real-world object interacting with the user's hand from the acquired image. The method may include a step of identifying a pre-registered function for using the recognized real-world object as an input tool. The method may include a step of performing the identified pre-registered function.

[0006] In one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing a method of using a real object as an input tool of an electronic device on a computer may be provided.

[0007] In one embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a camera that captures an image of a real object in a real space. The electronic device may include a memory that stores at least one instruction. The electronic device may include at least one processor that executes the at least one instruction stored in the memory. The at least one processor may recognize the real object from the acquired image. The at least one processor may register the recognized real object as an input tool. The at least one processor may map an interaction between the registered real object and a user's hand to a predefined function.

[0008] The present disclosure can be readily understood by reference to the following detailed description and the accompanying drawings, in which reference numerals may refer to structural elements.

[0009] FIG. 1 is a conceptual diagram illustrating an operation of using a real object as an input tool of an electronic device according to one embodiment of the present disclosure.

[0010] FIG. 2 is a block diagram showing components of an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 3 is a flowchart showing a method of using a real object as an input tool of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 4a is a conceptual diagram illustrating an operation of creating or deleting a virtual object using a real object according to one embodiment of the present disclosure.

[0013] FIG. 4b is a conceptual diagram illustrating an operation of creating or deleting a virtual object using a real object according to one embodiment of the present disclosure.

[0014] FIG. 5A is a conceptual diagram illustrating an operation of creating or deleting a virtual object using a real object according to one embodiment of the present disclosure.

[0015] FIG. 5b is a conceptual diagram illustrating an operation of creating or deleting a virtual object using a real object according to one embodiment of the present disclosure.

[0016] FIG. 6 is a conceptual diagram illustrating an operation of determining a color of a virtual object based on the color of at least a portion of a real object used as an input tool according to one embodiment of the present disclosure.

[0017] FIG. 7 is a conceptual diagram illustrating an operation of determining the color of a virtual object based on the color of a target object pointed by a real object according to one embodiment of the present disclosure.

[0018] FIG. 8 is a conceptual diagram illustrating an operation of determining the color of a virtual object based on the shape of the virtual object according to one embodiment of the present disclosure.

[0019] FIG. 9 is a conceptual diagram illustrating an operation of determining the size of a virtual object based on the size of at least a portion of a real object according to one embodiment of the present disclosure.

[0020] FIG. 10 is a flowchart illustrating a method of performing a function of creating or deleting a virtual object by tracking at least a portion of a real object according to one embodiment of the present disclosure.

[0021] FIG. 11 is a flowchart illustrating a method for determining at least one of a color and a size of a virtual object based on at least one of a color and a size of at least a portion of a real object according to one embodiment of the present disclosure.

[0022] FIG. 12 is a flowchart illustrating a method for determining the color, shape, and appearance of a virtual object based on the color, shape, and appearance of a target object pointed by a real object according to one embodiment of the present disclosure.

[0023] FIG. 13 is a flowchart showing a method for obtaining an image in which a virtual object is synthesized into a real image according to one embodiment of the present disclosure.

[0024] FIG. 14A is a conceptual diagram illustrating an operation of identifying a pre-registered function based on an occluded portion of a real object according to one embodiment of the present disclosure.

[0025] FIG. 14b is a conceptual diagram illustrating an operation of identifying a pre-registered function based on an occluded portion of a real object according to one embodiment of the present disclosure.

[0026] FIG. 14c is a conceptual diagram illustrating an operation of identifying a pre-registered function based on an occluded portion of a real object according to one embodiment of the present disclosure.

[0027] FIG. 14d is a conceptual diagram illustrating an operation of identifying a pre-registered function based on an occluded portion of a real object according to one embodiment of the present disclosure.

[0028] FIG. 15 is a flowchart showing a method for identifying a pre-registered feature based on an occluded portion of a real object according to one embodiment of the present disclosure.

[0029] FIG. 16 is a conceptual diagram illustrating an operation of identifying a pre-registered function based on the manner in which a user's hand grasps a real object according to one embodiment of the present disclosure.

[0030] FIG. 17 is a flowchart illustrating a method for identifying a pre-registered function based on the manner in which a user's hand grasps a real-world object according to one embodiment of the present disclosure.

[0031] FIG. 18 is a flowchart illustrating a method for identifying a pre-registered function based on the manner in which a user's hand grasps a real-world object according to one embodiment of the present disclosure.

[0032] FIG. 19 is a flowchart illustrating a method for identifying a pre-registered function based on the manner in which a user's hand grasps a real-world object according to one embodiment of the present disclosure.

[0033] FIG. 20 is a flowchart showing a method for determining whether to perform a pre-registered function based on the distance between a user's hand and a real object according to one embodiment of the present disclosure.

[0034] FIG. 21 is a conceptual diagram illustrating an operation of performing a function of determining a performance value of a virtual object according to the characteristics of a real object according to one embodiment of the present disclosure.

[0035] FIG. 22 is a flowchart showing a method for determining performance values ​​of a virtual object based on characteristics of a real object according to one embodiment of the present disclosure.

[0036] FIG. 23 is a conceptual diagram illustrating an operation of using a real object similar to a virtual object to be mapped as an input tool according to one embodiment of the present disclosure.

[0037] FIG. 24 is a conceptual diagram illustrating an operation of generating a virtual object based on a predetermined movement according to one embodiment of the present disclosure.

[0038] FIG. 25 is a conceptual diagram illustrating an operation of generating a virtual object based on a predetermined movement according to one embodiment of the present disclosure.

[0039] FIG. 26 is a flowchart illustrating a method for generating a virtual object based on a predetermined movement according to one embodiment of the present disclosure.

[0040] FIG. 27 is a conceptual diagram illustrating an operation of generating a virtual object based on a predetermined movement and personal authentication input according to one embodiment of the present disclosure.

[0041] FIG. 28 is a flowchart illustrating a method for generating a virtual object based on predetermined movements and personal authentication input according to one embodiment of the present disclosure.

[0042] FIG. 29 is a conceptual diagram illustrating an operation of performing a function of activating a predetermined sensor based on a detection result according to one embodiment of the present disclosure.

[0043] FIG. 30 is a block diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0044] FIG. 31 is a flowchart illustrating a method for performing a function of activating a predetermined sensor according to one embodiment of the present disclosure.

[0045] FIG. 32 is a flowchart showing a method for registering a real object as an input tool of an electronic device according to one embodiment of the present disclosure.

[0046] FIG. 33 is a conceptual diagram illustrating a method of performing a pre-registered function based on the movement of a user's ball or the shape of the user's ball according to one embodiment of the present disclosure.

[0047] FIG. 34A is a conceptual diagram illustrating a method for performing a function of controlling a virtual object based on the movement of a user's ball or the shape of the user's ball according to one embodiment of the present disclosure.

[0048] FIG. 34b is a conceptual diagram illustrating a method for performing a function of controlling a virtual object based on the movement of a user's ball or the shape of the user's ball according to one embodiment of the present disclosure.

[0049] FIG. 35A is a conceptual diagram illustrating a method for performing a function of controlling the size of a virtual screen based on the movement of a user's ball or the shape of the user's ball according to one embodiment of the present disclosure.

[0050] FIG. 35b is a conceptual diagram illustrating a method for performing a function of controlling the size of a virtual screen based on the movement of a user's ball or the shape of the user's ball according to one embodiment of the present disclosure.

[0051] FIG. 36A is a conceptual diagram illustrating a method of performing a function identified based on the movement of a user's ball or the shape of the user's ball, or performing a function of canceling a performed function, according to one embodiment of the present disclosure.

[0052] FIG. 36b is a conceptual diagram illustrating a method of performing a function identified based on the movement of a user's ball or the shape of the user's ball according to one embodiment of the present disclosure, or performing a function of canceling a performed function.

[0053] FIG. 37 is a flowchart illustrating a method for identifying a pre-registered function based on the distance between a real object and a user's mouth according to one embodiment of the present disclosure.

[0054] FIG. 38 is a flowchart showing a method of performing a pre-registered function based on the movement of a user's ball or the shape of the user's ball according to one embodiment of the present disclosure.

[0055] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0056] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0057] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0058] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0059] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0060] In the present disclosure, 'Augmented Reality' may mean displaying a virtual object (or virtual image) together within a real physical environment space or displaying a real object and a virtual object (or virtual image) together.

[0061] In the present disclosure, 'Virtual Reality' may mean interacting with a virtual object (or virtual image) that does not actually exist within a virtual environment or independently displaying a virtual object (or virtual image).

[0062] In the present disclosure, an 'augmented reality device' or a 'virtual reality device' may refer to a device capable of expressing augmented reality or virtual reality. For example, an 'augmented reality device' or a 'virtual reality device' may include augmented reality glasses in the shape of glasses worn by a user on the face, a head-mounted display apparatus (HMD) worn on the head, or an augmented reality helmet.

[0063] In the present disclosure, functions related to 'artificial intelligence' can be operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU or a VPU (Vision Processing Unit), or an AI-only processor such as an NPU. One or more processors may be controlled to process input data according to predefined operation rules or AI models stored in memory. Alternatively, if one or more processors are AI-only processors, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.

[0064] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0065] In the present disclosure, the 'artificial intelligence model' may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs neural network operations through operations between the operation results of the previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the plurality of weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network model may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0066] In the present disclosure, 'object recognition' may refer to image signal processing that inputs an image into an artificial intelligence model and detects an object from the input image, tracks the object, classifies the object into a specific category, or segments the object through inference using the artificial intelligence model. In one embodiment of the present disclosure, object recognition may refer to image processing that uses an artificial intelligence model to detect an object from an image captured by a camera, segment the object, track the object, and obtain location information of a plurality of feature points (e.g., joints) included in the object.

[0067] In the present disclosure, 'object detection' may mean the task of identifying the location and boundary of a specific object in an image or video using an artificial intelligence model, and distinguishing the object from other backgrounds.

[0068] In the present disclosure, 'object tracking' may mean a task of continuously detecting the movement of a specific object in a video using an artificial intelligence model, and identifying and tracking the object over time.

[0069] In the present disclosure, 'object segmentation' may mean a task of distinguishing multiple objects (or multiple parts of one object) at the pixel level in an image or video using an artificial intelligence model, and extracting the boundary of each object (or each part).

[0070] In the present disclosure, 'object classification' may mean a task of identifying the type or category of an object in an image or video using an artificial intelligence model and classifying the object into a predefined class or category.

[0071] In the present disclosure, 'object pose estimation' may refer to a task of estimating pose information about the position and motion of an object in an image or video using an artificial intelligence model. For example, 'object pose estimation' may refer to a task of identifying a skeleton or keypoint of a specific object to infer and estimate motions such as the pose, direction, and joints of the object. In the present disclosure, a 'joint' refers to a part of the human body where bones are connected to each other, and refers to one or more parts of the upper body such as the neck, arms, and shoulders, as well as hands such as fingers, wrists, and palms.

[0072] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0073] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0074] FIG. 1 is a conceptual diagram illustrating an operation of using a real object as an input tool of an electronic device according to one embodiment of the present disclosure.

[0075] Referring to FIG. 1, the electronic device (100) may be a device including a camera. However, the present disclosure is not limited thereto, and the camera's function may be performed by an external device. The electronic device (100) may be a device that acquires at least one image or video including a real-world object (110) through a camera, uses the real-world object (110) as an input tool of the electronic device (100), and performs a preset function using the real-world object (110). For example, the electronic device (100) may include, but is not limited to, an augmented reality device, a virtual reality device, a smart TV, a smart phone, a tablet PC, a laptop PC, and the like. In FIG. 1, the electronic device (100) is illustrated as an augmented reality device or a virtual reality device, but is not limited thereto. Hereinafter, for convenience of explanation, it is assumed that the electronic device (100) is an augmented reality device or a virtual reality device. The components of the electronic device (100) will be described in detail with reference to FIG. 2.

[0076] In one embodiment of the present disclosure, a camera of an electronic device (100) can capture (or photograph) a real object and a user's hand (10) within a real space to obtain at least one image or video. The electronic device (100) can detect a real object (110) through object recognition. The electronic device (100) can perform a preset function based on the detection result corresponding to the real object (110).

[0077] In one embodiment of the present disclosure, the real object (110) may include any physical object existing in reality that does not communicate or pair with the electronic device (100). The real object (110) may not have functions related to communication and connection with the electronic device (100). For example, the real object (110) may not have functions for transmitting and receiving electrical signals. For example, the real object (110) may not support a protocol for data exchange or connection with the electronic device (100).

[0078] In one embodiment of the present disclosure, the real object (110) may communicate with the electronic device (100), but in the process of performing the method according to one embodiment of the present disclosure, the real object (110) may not perform data exchange with the electronic device (100).

[0079] In one embodiment of the present disclosure, a real object (110) is a concept encompassing physical objects having various shapes and sizes. For example, the real object (110) may be an object of a size that a user can hold with his or her hand (e.g., a cup, a container, a cup holder, a pen, a pencil, a board marker, a hammer, a screwdriver, a saw, a drill, fruit, grain, a watch, a necklace, a wallet, glasses, sporting goods, a book, a toy, an electronic product, furniture, etc.), but the present disclosure is not limited thereto.

[0080] In one embodiment of the present disclosure, the electronic device (100) can recognize a real-world object interacting with the user's hand (10) from the acquired image. For example, the interaction between the user's hand (10) and the real-world object may indicate a situation in which the user's hand (10) grasps, holds, touches, or moves the real-world object. For example, the interaction between the user's hand (10) and the real-world object may indicate a situation in which the user's hand (10) approaches or moves away from the real-world object. However, the interaction between the user's hand (10) and the real-world object is not limited to the examples described above.

[0081] In one embodiment of the present disclosure, the electronic device (100) can detect a real object (110) and a user's hand (10) from a video or at least one image. For example, the electronic device (100) can detect the real object (110) and the user's hand (10) using an object detection model. For example, the electronic device (100) can recognize a real object interacting with the user's hand (10) based on the detected user's hand (10) and the detected real object.

[0082] In one embodiment of the present disclosure, the electronic device (100) can track a real object (110) or at least a portion of the real object (110). For example, the electronic device (100) can track at least a portion of the real object (110) using an object tracking model. For example, the electronic device (100) can track one portion (e.g., a pencil tip portion) of the real object (110) (e.g., a pencil).

[0083] In one embodiment of the present disclosure, the electronic device (100) can generate a virtual object (120) based on the movement of a tracked real object (110). For example, the electronic device (100) can generate a virtual object (120) based on the movement and position of one side portion (e.g., a pencil tip portion) of the real object (110) (e.g., a pencil).

[0084] In one embodiment of the present disclosure, an electronic device (100) may include a display. The electronic device (100) may display a virtual object (120) on the display.

[0085] In one embodiment of the present disclosure, the electronic device (100) can synthesize a virtual object (132) onto a real image (131). For example, the virtual object (132) can be generated based on the detection result and the tracking result of the real object (110). The electronic device (100) can display the synthesized image (130) on a display. Here, 'synthesis' can include combining the real image (131) and the virtual object (132) such that the virtual object (132) overlaps a portion of the real image (131).

[0086] FIG. 2 is a block diagram showing components of an electronic device (200) according to one embodiment of the present disclosure. The configuration, operation, and function of the electronic device (200) may correspond to the configuration, operation, and function of the electronic device (100) of FIG. 1. For convenience of explanation, any content overlapping with that described in FIG. 1 is omitted.

[0087] Referring to FIG. 2, an electronic device (200) according to an embodiment of the present disclosure may include a camera (210), a sensor (220), a communication interface (230), a user interface (240), a processor (250), and a memory (260). However, the components illustrated in FIG. 2 are not essential components, and the electronic device (200) may omit some of the components illustrated in FIG. 2 or further include additional components. For example, unlike the electronic device (200) according to an embodiment of the present disclosure, at least one of the camera (210) and the sensor (220) may be omitted.

[0088] In one embodiment of the present disclosure, the electronic device (200) may be implemented as a portable device, in which case the electronic device (200) may further include a battery that supplies power to the camera (210), sensor (220), communication interface (230), user interface (240), processor (250), and memory (260).

[0089] The camera (210) can obtain at least one image or video of a real object (e.g., a real object (FIG. 1, 110)) in a real space by photographing (or capturing) the real object. In one embodiment of the present disclosure, the camera (210) can obtain a two-dimensional image including the user's hand (10) by photographing (or capturing) the user's hand (10) that is holding the real object.

[0090] In one embodiment of the present disclosure, the camera (210) may include at least two cameras. For example, the camera (210) may include a first camera and a second camera. For example, the first camera may correspond to the left eye, and the second camera may correspond to the right eye, but the present disclosure is not limited thereto. The first camera and the second camera may constitute a stereo camera that obtains three-dimensional position coordinates of an object through triangulation based on a two-dimensional image obtained in an area where their fields of view overlap and a positional relationship between the cameras.

[0091] In one embodiment of the present disclosure, the camera (210) may be implemented in a small form factor so that it can be mounted on a virtual reality device or an augmented reality device, and may be a lightweight RGB camera that consumes low power. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the camera (210) may be implemented as any type of camera known in the art, such as an RGB-depth camera with a depth estimation function, a stereo fish-eye camera, a grayscale camera, or an infrared camera.

[0092] In one embodiment of the present disclosure, the camera (210) may include a lens module, an image sensor, and an image processing module. The camera (210) may acquire still images or video of a real object and a user's hand (10) by an image sensor (e.g., CMOS or CCD). The video may include a plurality of image frames acquired in real time by photographing a real scene (or real space) including the real object and the user's hand (10) through the camera (210). The image processing module may encode a still image composed of a single image frame acquired through the image sensor or video data composed of a plurality of image frames.

[0093] The sensor (220) can convert the measured or detected information into an electrical signal by measuring or detecting a physical quantity. The sensor (220) can transmit the converted electrical signal to the processor (250). The processor (250) can process the electrical signal to generate sensor data. For example, the sensor (220) can include at least one button for touch input, a microphone sensor, a gesture sensor, a gyroscope, a gyro sensor, a barometric pressure sensor, a magnetic sensor, a magnetometer, an acceleration sensor, an accelerometer, a grip sensor, a proximity sensor, an RGB sensor, a biophysical sensor, a temperature sensor, a humidity sensor, an illuminance sensor, an ultraviolet sensor, an electromyography sensor, an electroencephalography sensor, an electrocardiogram sensor, an infrared sensor, an ultrasonic sensor, an iris sensor, or a fingerprint sensor, but the present disclosure is not limited thereto.

[0094] In one embodiment of the present disclosure, the electronic device (200) can activate the sensor (220) based on the detection results corresponding to the real object and the user's hand (10). The electronic device (200) can obtain sensor data using the activated sensor (220). In one embodiment of the present disclosure, the electronic device (200) can modulate the sensor data based on at least some of the shape, form, color, material, and function of the real object.

[0095] The communication interface (230) may support the establishment of a wired or wireless communication channel between the electronic device (200) and another external electronic device (not shown) or server (not shown) and the performance of communication through the established communication channel. In one embodiment, the communication interface (230) may receive data from another external electronic device (not shown) or server (not shown) or transmit data to another external electronic device (not shown) or server (not shown) through wired or wireless communication.

[0096] In one embodiment of the present disclosure, the communication interface (230) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module), and may communicate with another external electronic device (not shown) or a server (not shown) via at least one network (e.g., a short-range communication network (e.g., Bluetooth, WiFi direct, or IrDA (infrared data association)) or a long-range communication network (e.g., a cellular network, the Internet, or a computer network (e.g., a LAN or WAN))) using any one of the communication modules.

[0097] The user interface (240) may include an input interface (241) and an output interface (243).

[0098] The input interface (241) is for receiving input from a user (hereinafter, user input). The input interface (241) may be at least one of a key pad, a dome switch, a touch pad (contact electrostatic capacitance type, pressure resistive film type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, piezo effect type, etc.), a jog wheel, and a jog switch, but is not limited thereto.

[0099] In one embodiment of the present disclosure, the input interface (241) may include a microphone (242). Although the microphone (242) and the sensor (220) are illustrated as separate blocks in FIG. 2 , the sensor (220) may include the microphone (242). In one embodiment of the present disclosure, the microphone (242) may receive sound. For example, the microphone (242) may receive a user's voice. The microphone (242) may convert the user's voice into an audio signal, which is an electrical signal.

[0100] In one embodiment of the present disclosure, the electronic device (200) can activate the microphone (242) based on the detection results corresponding to the real object and the user's hand (10). The electronic device (200) can obtain an audio signal corresponding to the user's voice using the activated microphone (242). In one embodiment of the present disclosure, the electronic device (200) can modulate the audio signal based on at least some of the shape, form, color, material, and function of the real object.

[0101] The output interface (243) is for outputting an audio signal or a video signal, and may include, for example, a display (244) or a speaker (245).

[0102] In one embodiment of the present disclosure, the electronic device (200) can display an image or a video through the display (244). For example, the electronic device (200) can display a virtual image including a virtual object through the display (244). For example, the electronic device (200) can display an image in which a virtual object is synthesized into an image corresponding to a real scene through the display (244). For example, the display (244) can include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, a light-emitting diode (LED), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an electrophoretic display. In addition, depending on the implementation form, the electronic device (200) may include two or more displays.

[0103] In one embodiment of the present disclosure, the electronic device (200) can display in real time on the display (244) at least one image or video of the real world acquired by the camera (210). For example, the at least one image or video of the real world can include at least one image frame acquired in real time by photographing (or capturing) a real scene (or real space) including a real object (110) and a user's hand (10) through the camera (210). The electronic device (200) can overlap at least one virtual object with the at least one image or video of the real world and display it on the display (244). For example, the electronic device (200) can overlap a virtual object (120) with a video of the real world including a real object (110) and a user's hand (10) and display it on the display (244). The speaker (245) can output an audio signal received from the communication interface (230) or stored in the memory (260).

[0104] The processor (250) may be implemented through a combination of software and a general-purpose processor, such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU). In the case of a dedicated processor, it may include memory for implementing embodiments of the present disclosure or a memory processing unit for utilizing external memory.

[0105] The processor (250) may be composed of one or more processors. In this case, it may be implemented as a combination of dedicated processors, or it may be implemented through a combination of multiple general-purpose processors, such as an AP, CPU, or GPU, and software.

[0106] In one embodiment, the processor (250) may be equipped with an artificial intelligence (AI) processor. The AI ​​processor may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (e.g., a CPU or application processor) or a graphics-only processor (e.g., a GPU) and equipped in the electronic device (200). For example, the AI ​​processor may perform data processing necessary for learning and / or inference related to at least one artificial intelligence model.

[0107] The artificial intelligence-related functions according to the present disclosure are operated through a processor (250) and a memory (260). The processor (250) may be composed of one or more processors. In this case, the one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models (e.g., deep neural network models) stored in the memory (260). Alternatively, when the one or more processors are artificial intelligence-only processors, the artificial intelligence-only processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0108] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned by a learning algorithm using a plurality of learning data, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed in the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above. The memory (260) may store a program for processing and controlling the processor (250), and may store input / output data. The memory (260) may store at least one artificial intelligence model.

[0109] The memory (260) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk. In addition, the electronic device (200) may also operate a web storage or cloud server that performs a storage function on the Internet.

[0110] In one embodiment of the present disclosure, the memory (260) may store data, firmware, software, process codes, etc. that are processed or to be processed by the processor (250). In one embodiment of the present disclosure, the memory (260) may store data and program codes corresponding to at least one of an object recognition module (261), an object detection module (262), an object tracking module (263), an object classification module (264), an object segmentation module (265), an object pose estimation module (266), a virtual object management module (267), and a function identification module (268).

[0111] In one embodiment of the present disclosure, the object recognition module (261) may include an object detection module (262), an object tracking module (263), an object classification module (264), an object segmentation module (265), and an object pose estimation module (266). The object recognition module (261) may use at least one artificial intelligence model that takes an image or video as input to output data corresponding to an object included in the image.

[0112] In one embodiment of the present disclosure, the object detection module (262) can detect an object included in at least one image using an artificial intelligence model (which may also be referred to as an object detection model) that takes at least one image as input. The object detection module (262) can estimate location information and boundary information of the detected object. For example, the object detection module (262) can receive an image including a real object and a user's hand (10). The object detection module (262) can estimate location information and boundary information of each of the real object and the user's hand (10) using an artificial intelligence model that takes the image as input. In the present disclosure, the 'detection result' can include values ​​including location information and boundary information. For example, the location information can indicate where an object is located in an image or video frame. For example, the boundary information can indicate a boundary line or outline of an object in an image or video frame. In one embodiment of the present disclosure, the object detection module (262) can recognize a real object interacting with the user's hand (10) based on the detection result.

[0113] In one embodiment of the present disclosure, the object tracking module (263) can track the movement of an object (or a part of an object) included in a video using an artificial intelligence model (which may also be referred to as an object tracking model) that takes at least one image as input. The object tracking module (263) can estimate continuous position information and a movement pattern of the object. For example, the object detection module (262) can receive a video or a continuous image sequence including a real object. The object detection module (262) can estimate continuous position information and a movement pattern of at least a part of the real object using an artificial intelligence model that takes the video or the continuous image sequence as input. In the present disclosure, the 'tracking result' can include values ​​including continuous position information and a movement pattern.

[0114] In one embodiment of the present disclosure, the object classification module (264) can classify at least one object included in at least one image into a predefined class using an artificial intelligence model (which may also be referred to as an object classification model) that takes at least one image as input. The object classification module (264) can estimate a class label of the object. For example, the object classification module (264) can receive at least one image including a real object. The object classification module (264) can classify the real object into a predefined class using an artificial intelligence model that takes at least one image as input.

[0115] In one embodiment of the present disclosure, the object segmentation module (265) can segment at least one image into multiple objects or multiple parts of one object using an artificial intelligence model (which may also be referred to as an object segmentation model) that takes at least one image as input. The object segmentation module (265) can estimate boundary and area information of the multiple objects or multiple parts of one object. For example, the object segmentation module (265) can receive at least one image including a real object. The object segmentation module (265) can segment the real object into multiple parts using an artificial intelligence model that takes at least one image as input.

[0116] In one embodiment of the present disclosure, the object pose estimation module (266) can infer pose information such as position, direction, and joint of an object included in at least one image by using an artificial intelligence model (which may also be referred to as an object pose estimation model) that takes at least one image as input. The object pose estimation module (266) can identify feature points of the object and infer pose, motion, and joint movement of the object. For example, the object pose estimation module (266) can receive at least one image including a user's hand (10) grasping a real object. The object segmentation module (265) can infer the position, direction, and method of the user's hand (10) grasping the real object by using an artificial intelligence model that takes at least one image as input.

[0117] In one embodiment of the present disclosure, the virtual object management module (267) may perform a function of creating or deleting a virtual object. The virtual object management module (267) may create or delete a virtual object according to an identified function of the electronic device (200). For example, in a first function, the electronic device (200) may create a virtual object according to a tracking result of a real object. For example, in a second function, the electronic device (200) may delete a virtual object according to a tracking result of a real object.

[0118] In one embodiment of the present disclosure, the function identification module (268) can identify a pre-registered function of an electronic device (200) that uses a real object as an input tool based on the object recognition result of the object recognition module (261). For example, the pre-registered function may include a function for creating a virtual object, a function for deleting a virtual object, a function for changing the characteristics of a virtual object, etc.

[0119] In one embodiment of the present disclosure, at least a part of the functions of at least one of an object recognition module (261), an object detection module (262), an object tracking module (263), an object classification module (264), an object segmentation module (265), an object pose estimation module (266), a virtual object management module (267), and a function identification module (268) may be executed by the processor (250), but the present disclosure is not limited thereto, and may be executed by an external server (not shown).

[0120] In one embodiment of the present disclosure, the electronic device (200) may be an augmented reality device or a virtual reality device. The augmented reality device or the virtual reality device may be implemented in the form of glasses worn on a person's face or a head-mounted device worn on a person's head, and may be designed with a small form factor for portability. Therefore, the storage capacity of the memory (260) of the augmented reality device or the virtual reality device, the computational processing speed of the processor (250), etc. may be limited compared to a server (not shown). Accordingly, the server (not shown) may transmit necessary data (e.g., detection results, tracking results, etc.) to the augmented reality device or the virtual reality device via a communication network after performing operations requiring storage of large amounts of data and large amounts of computation. In this way, the augmented reality device or virtual reality device can receive and use data corresponding to the detection result or tracking result (e.g., object location information and boundary information, continuous location information, movement pattern information, etc.) from a server (not shown) without a processor having a large memory and fast computing capability, thereby shortening the processing time required to process the image and implementing real-time object recognition.

[0121] FIG. 3 is a flowchart illustrating a method for using a real-world object as an input tool for an electronic device according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 and 2 are omitted. For convenience of explanation, FIG. 3 will be described with reference to FIG. 2.

[0122] Referring to FIG. 3, a method of using a real object as an input tool of an electronic device (200) may include operations S310 to S340. In one embodiment, operations S310 to S340 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The method of using a real object as an input tool by the electronic device (200) according to one embodiment of the present disclosure is not limited to that illustrated in FIG. 3, and any one of the operations illustrated in FIG. 3 may be omitted, or operations not illustrated in FIG. 3 may be further included.

[0123] In operation S310, the electronic device (200) can capture images by photographing a real object and the user's hand (10) within a real space. The camera (210) can capture images of the real space including the real object and the user's hand (10). The image processing module of the processor (250) or the camera (210) can generate a video based on the captured real scene.

[0124] In operation S320, the electronic device (200) can recognize a real object interacting with the user's hand (10) from the acquired image. The electronic device (200) can acquire a recognition result. For example, the recognition result can include at least one of position information (e.g., bounding box, etc.) and boundary information (e.g., polygon, set of points, pixel mask, etc.) of each of the real object and the user's hand (10). In one embodiment of the present disclosure, the electronic device (200) can detect the real object and the user's hand (10) using an artificial intelligence model that inputs at least one image.

[0125] In operation S330, the electronic device (200) may identify a pre-registered function that uses a recognized real object as an input tool. In one embodiment of the present disclosure, the pre-registered function may be predetermined by a user or manufacturer setting and stored in the memory (260). In one embodiment of the present disclosure, the pre-registered function may be a function of creating or deleting a virtual object. In one embodiment of the present disclosure, the pre-registered function may be a function of determining a performance value of a virtual object according to characteristics (e.g., color, material, shape, etc.) of the real object. In one embodiment of the present disclosure, the pre-registered function may be a function of creating a pre-determined virtual object according to a pre-determined movement. In one embodiment of the present disclosure, the pre-registered function may be a function of activating at least one pre-determined sensor.

[0126] In one embodiment of the present disclosure, the electronic device (200) can identify a pre-registered function mapped to a detected real object based on detection of a pre-registered (pre-stored) real object or an interaction between a real object and a user's hand (10). For example, the electronic device (200) can access a mapping table stored in the memory (260) (or storage) or an external device. The mapping table can include relationship information between the detection result and the function. For example, the mapping table can include relationship information between the type of the real object and the function, relationship information between the shape of the real object and the function, relationship information between the interaction between the user's hand (10) and the real object and the function, etc. In one embodiment of the present disclosure, the mapping table can be predetermined by a user or a manufacturer's setting. The electronic device (200) can identify a pre-registered function based on the mapping table.

[0127] In operation S340, the electronic device (200) may perform the identified pre-registered function. In one embodiment of the present disclosure, the electronic device (200) may perform a function of creating a virtual object. In one embodiment of the present disclosure, the electronic device (200) may perform a function of deleting a pre-created virtual object.

[0128] FIGS. 4A and 4B are conceptual diagrams illustrating an operation of creating or deleting a virtual object using a real object according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 3 are omitted. For convenience of explanation, FIGS. 4A and 4B will be described with reference to FIG. 2.

[0129] Referring to FIG. 4A, the electronic device (200) can recognize (or detect) a real object (410) and a user's hand (10) from an image. The electronic device (200) can track the movement of all or at least a portion of the recognized real object (410). For example, the electronic device (200) can track the movement of one side (411) of the real object (410) (e.g., the tip of a pencil). The movement of one side (411) of the real object (410) may occur according to the movement of the user's hand (10). For example, the one side (411) of the real object (410) may move along a path (20). However, the path (20) is merely an example that can be recognized (or detected) according to the movement of the user's hand (10), and the path along which the one side (411) of the real object (410) may move is not limited to the path (20) illustrated in FIG. 4A.

[0130] In one embodiment of the present disclosure, a portion of a real object (410) to be tracked may be determined in advance. In one embodiment of the present disclosure, the electronic device (200) may divide the real object (410) into a plurality of portions. For example, the electronic device (200) may divide the real object (410) into a plurality of portions using an artificial intelligence model that inputs a video. For example, as illustrated in FIG. 4A, if the real object (410) is a pencil, the electronic device (200) may divide one side of the pencil where the graphite lead is placed as a first portion, the other side of the pencil where the eraser is placed as a second portion, and the body of the pencil (e.g., a point where the distance from one side of the pencil and the distance from the other side of the pencil are the same) as a third portion.

[0131] The electronic device (200) can generate a virtual object (420) based on the tracked movement. For example, the electronic device (200) can recognize that the user's hand (10) is positioned toward one side (411) of the pencil where the graphite lead is placed based on the recognition result. The electronic device (200) can perform a function of generating a virtual object (or may also be referred to as a drawing function) based on the recognition result. The electronic device (200) can obtain a path (20) along which the one side (411) of the pencil moves based on the tracked movement. The electronic device (200) can generate a virtual object (420) along the path (20) along which the one side (411) of the pencil moves. In one embodiment of the present disclosure, the virtual object (420) can be displayed on the display (244) of the electronic device (200).

[0132] Referring to FIG. 4B, the electronic device (200) can recognize (or detect) a real object (410) and a user's hand (10) from an image. The electronic device (200) can track the movement of at least a portion of the recognized real object (410). For example, the electronic device (200) can track the movement of the other side (412) of the real object (410) (e.g., the part that is furthest from the eraser part of a pencil or the tip part of a pencil). The movement of the other side (412) of the real object (410) can occur according to the movement of the user's hand (10). For example, the other side (412) of the real object (410) can move along a path (30). However, the path (30) is only an example that can be recognized (or detected) according to the movement of the user's hand (10), and the path along which the other side (412) of the real object (410) can move is not limited to the path (30) illustrated in FIG. 4b.

[0133] The electronic device (200) can delete the virtual object (420) based on the tracked movement. For example, the electronic device (200) can recognize that the user's hand (10) is positioned toward the other end (412) of the pencil where the eraser is placed based on the recognition result. The electronic device (200) can perform a function of deleting the virtual object (or may also be referred to as an erasing function) based on the recognition result. The electronic device (200) can obtain a path (30) along which the other end (412) of the pencil moves based on the tracked movement. The electronic device (200) can delete the virtual object (420) along the path (30) along which the other end (412) of the pencil moves. In one embodiment of the present disclosure, a portion of the virtual object (420) corresponding to the path (30) along which the other end (412) of the pencil moves may be deleted, and only the remaining portion (421) may be displayed on the display (244).

[0134] Figures 5a and 5b are conceptual diagrams illustrating an operation of creating or deleting a virtual object using a real object according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 4b are omitted. For convenience of explanation, Figures 5a and 5b will be described with reference to Figure 2.

[0135] Referring to FIG. 5A, the electronic device (200) can recognize (or detect) a real object (510) and a user's hand (10) from an image. The electronic device (200) can track the movement of all or at least a portion of the recognized real object (510). Although the real object (510) is illustrated as a paper cup holder in FIGS. 5A and 5B , the real object (510) can be any object that includes at least one sharp edge or pointy edge. For example, the electronic device (200) can track the movement of an edge (511) of the real object (510) (e.g., one of the edges of the paper cup holder). The movement of the edge (511) of the real object (510) can occur according to the movement of the user's hand (10). For example, the edge (511) of the real object (510) can move along a path (40). However, the path (40) is only an example that can be recognized (or detected) according to the movement of the user's hand (10), and the path along which the corner (511) of the real object (510) can move is not limited to the path (40) illustrated in FIG. 5a.

[0136] The electronic device (200) can generate a virtual object (520) based on the tracked movement. For example, the electronic device (200) can recognize that the user's hand (10) is positioned toward the corner (511) of the paper cup holder based on the recognition result. The electronic device (200) can perform a function of generating a virtual object (or may also be referred to as a drawing function) based on the recognition result. The electronic device (200) can obtain a path (40) along which the corner (511) of the paper cup holder moves based on the recognition result. The electronic device (200) can generate a virtual object (520) along the path (40) along which the corner (511) of the paper cup holder moves. In one embodiment of the present disclosure, the virtual object (520) can be displayed on the display (244).

[0137] Referring to FIG. 5B, the electronic device (200) can recognize (or detect) a real object (410) and a user's hand (10) from an image. The electronic device (200) can track the movement of the entire or at least a portion of the recognized real object (510). For example, the electronic device (200) can track the movement of the entire area (512) of the real object (510) (e.g., the entire area of ​​a paper cup holder in the field of view looking at the paper cup holder). The movement of the entire area (512) of the real object (510) can occur according to the movement of the user's hand (10). For example, the entire area (512) of the real object (510) can move along a path (50). However, the path (50) is only an example that can be recognized (or detected) according to the movement of the user's hand (10), and the path along which the entire area (512) of the real object (510) can move is not limited to the path (50) illustrated in FIG. 5b.

[0138] The electronic device (200) can delete the virtual object (520) based on the tracked movement. For example, the electronic device (200) can recognize the shape of the real object (510) placed on the user's hand (10) based on the recognition result. The electronic device (200) can perform a function of deleting the virtual object (or may also be referred to as an erasing function) based on the recognition result. The electronic device (200) can obtain a path (30) along which the entire area (512) of the real object (510) moves based on the tracked movement. The electronic device (200) can delete the virtual object (520) along the path (50) along which the entire area (512) of the real object (510) moves. In one embodiment of the present disclosure, a portion corresponding to the path (50) along which the entire area (512) of the real object (510) of the virtual object (520) moves may be deleted, and only the remaining portion (521) may be displayed on the display (244).

[0139] Referring to FIGS. 4A to 5B , in one embodiment of the present disclosure, a real object (410, 510) may be pre-registered (or stored) as an input tool of an electronic device (200). For example, the electronic device (200) may extract features of the real object (410, 510). For example, the features of the real object (410, 510) may include at least one of the shape, form, color, and texture of the real object (410, 510), and may be any characteristic that can uniquely identify the real object (410, 510). In one embodiment of the present disclosure, the electronic device (200) may detect the real object (410, 510) from at least one image. The electronic device (200) may compare the features of the detected real object (410, 510) with the features of the pre-registered input tool. For example, the electronic device (200) can compare the features of the detected real object (410, 510) with the features of the previously registered input tool using distance measurement between features, Euclidean distance, cosine similarity, etc.

[0140] In one embodiment of the present disclosure, the electronic device (200) can pre-register which part of the real object (410, 510) to track. For example, the electronic device (200) can pre-register which part of the real object (410, 510) to track based on at least one of the shape of the real object (410, 510), the shape of the user's hand (10), and the characteristics of the interaction between the user's hand (10) and the real object (410, 510) (e.g., the shape of the user's hand (10) grasping the real object (410, 510), and the position of the user's hand (10) grasping the real object (410, 510)).

[0141] In one embodiment of the present disclosure, the electronic device (200) can pre-register the characteristics (e.g., color, thickness, etc.) of the virtual object (420, 520) based on the tracking result of the real object (410, 510). For example, the electronic device (200) can pre-register the characteristics of the virtual object (420, 520) based on the shape of the real object (410, 510), the shape in which the user's hand (10) grasps the real object (410, 510), and the position in which the user's hand (10) grasps the real object (410, 510).

[0142] According to one embodiment of the present disclosure, accurate and rapid detection and tracking operations can be performed by pre-registering a real object, a tracking target of the real object, and a function of the real object.

[0143] FIG. 6 is a conceptual diagram illustrating an operation for determining the color of a virtual object based on the color of at least a portion of a real-world object used as an input tool according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 5b are omitted. For convenience of explanation, FIG. 6 will be described with reference to FIG. 2.

[0144] Referring to FIG. 6, the electronic device (200) can detect a real object (610) and a user's hand (10). The electronic device (200) can track the movement of one side (611) of the real object (610). The electronic device (200) can generate a virtual object (620) based on the detection result and the tracking result. In one embodiment of the present disclosure, the electronic device (200) can determine the color of the virtual object (620) based on the color of the one side (611) of the real object (610). For example, as illustrated in FIG. 6, it is assumed that the color of the one side (611) of the real object (610) is black, but the present disclosure is not limited thereto, and the color of the one side (611) of the real object (610) can be any color. The electronic device (200) can recognize that the color of one side (611) of the real object (610) (e.g., the tip of a pencil) is black. The electronic device (200) can determine the color of the virtual object (620) to be black based on the color of the one side (611) of the real object (610) being black. The electronic device (200) can generate a virtual object of the determined color (e.g., black).

[0145] FIG. 7 is a conceptual diagram illustrating an operation of determining the color of a virtual object based on the color of a target object pointed to by a real object according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 5b are omitted. For convenience of explanation, FIG. 7 will be described with reference to FIG. 2.

[0146] Referring to FIG. 7, the electronic device (200) may acquire an image including a real object (710), a user's hand (10) holding the real object (710), and a target object (731). In one embodiment of the present disclosure, the target object (731) may be an object included in a separate image (730) displayed on the display (244). In this case, the image may include the separate image (730), the real object (710), and the user's hand (10).

[0147] The electronic device (200) can detect a real object (710), a user's hand (10), and a target object (731) from images acquired through a camera (210) of the electronic device (200). In one embodiment of the present disclosure, the electronic device (200) can track the movement of one side (711) of the real object (710). The electronic device (200) can generate a virtual object (720) based on the tracking result.

[0148] In one embodiment of the present disclosure, the electronic device (200) can recognize a target object (731) pointed by one side (711) of a real object (710) from an image. The electronic device (200) can determine at least one of a color, a shape, and a shape of a virtual object (720) based on at least one of a color, a shape, and a shape of the target object (731). The electronic device (200) can generate a virtual object (720) having at least one of the determined color, shape, and shape. For example, as illustrated in FIG. 7, it is assumed that the color of the target object (731) is red, but the present disclosure is not limited thereto, and the color of the target object (731) can be any color. The electronic device (200) can recognize that the color of the target object (731) is red. The electronic device (200) can determine the color of the virtual object (720) as red based on the fact that the color of the target object (731) is red. The electronic device (200) can generate a virtual object (720) having a determined color (e.g., red).

[0149] FIG. 8 is a conceptual diagram illustrating an operation for determining the color of a virtual object based on its shape, according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 5b are omitted. For convenience of explanation, FIG. 8 will be described with reference to FIG. 2.

[0150] Referring to FIG. 8, the electronic device (200) can recognize (or detect) a real object (810) and a user's hand (10) from an image acquired through a camera (210) of the electronic device (200). The electronic device (200) can track the movement of at least a portion (811) of the recognized real object (810). The electronic device (200) can generate a virtual object (831, 832) based on the tracking result. In one embodiment of the present disclosure, the virtual object (831, 832) can be synthesized onto an image (830) displayed on a display (244). Here, 'synthesis' can include combining the image (830) and the virtual object (831, 832) so that the virtual object (831, 832) overlaps a portion of the image (830).

[0151] In one embodiment of the present disclosure, the electronic device (200) can determine the type of the virtual object (831, 832). For example, the electronic device (200) can classify the virtual object (831, 832) using an artificial intelligence model that inputs a virtual image corresponding to the virtual object (831, 832). For example, the electronic device (200) can classify the virtual object (831, 832) into a predetermined class. For example, the artificial intelligence model can classify the virtual object (831) into a heart class. For example, the artificial intelligence model can classify the virtual object (832) into a crown class.

[0152] In one embodiment of the present disclosure, the electronic device (200) may determine a color mapped to a classification result (e.g., class information, which is an output of an artificial intelligence model) as the color of the virtual objects (831, 832). For example, the electronic device (200) may access a mapping table stored in the memory (260) (or storage) or an external device. For example, the mapping table may include relationship information between classes and colors. In one embodiment of the present disclosure, the mapping table may be predetermined by a user or manufacturer setting. The electronic device (200) may determine a color mapped to a class as the color of the virtual objects (831, 832) based on the mapping table. For example, the heart class may be mapped to red, and the crown class may be mapped to yellow. The electronic device (200) may determine the color of the virtual object (831) as red, and the color of the virtual object (832) as yellow. In one embodiment of the present disclosure, the color of the virtual object (831, 832) may represent at least one of the color of the outline of the virtual object (831, 832) and the color of the area occupied by the virtual object (831, 832). The electronic device (200) may generate the virtual object (831, 832) having the determined color.

[0153] FIG. 9 is a conceptual diagram illustrating an operation for determining the size of a virtual object based on the size of at least a portion of a real object according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 5b are omitted. For convenience of explanation, FIG. 9 will be described with reference to FIG. 2.

[0154] Referring to FIG. 9, the electronic device (200) can recognize (or detect) a real object (910) and a user's hand (10) from an image acquired through a camera (210) of the electronic device (200). The electronic device (200) can track the movement of at least a portion of the recognized real object (610). In FIG. 9, the real object (910) is illustrated as a brush, but the present disclosure is not limited thereto, and the real object (910) can be any object in which a portion to be tracked has a specific size (e.g., width). For example, the electronic device (200) can track the movement of one side (911) of the real object (910) (e.g., the head of the brush). The movement of one side (911) of the real object (910) can occur according to the movement of the user's hand (10). For example, one side (911) of a real object (910) can move along a path (60). However, the path (60) is merely an example that can be recognized (or detected) according to the movement of the user's hand (10), and the path along which one side (911) of the real object (910) can move is not limited to the path (60) illustrated in FIG. 9.

[0155] The electronic device (200) can generate a virtual object (920) based on the tracking result. For example, the electronic device (200) can recognize that the user's hand (10) is positioned toward one side (911) of the real object (910) (e.g., the head of the brush) based on the recognition result. The electronic device (200) can perform a virtual object generation function based on the recognition result. The electronic device (200) can obtain a path (60) along which one side (911) of the real object (910) (e.g., the head of the brush) moves based on the tracking result. The electronic device (200) can generate a virtual object (920) along the path (60) along which one side (911) of the real object (910) (e.g., the head of the brush) moves. In one embodiment of the present disclosure, the electronic device (200) can determine the size (e.g., width) of the virtual object (620) based on the size (e.g., width) of one side (911) (e.g., the head of a brush) of the real object (910). The electronic device (200) can determine the width of one side (911) (e.g., the head of a brush) of the real object (910) based on the recognition result. The electronic device (200) can generate a virtual object (920) having the width of one side (911) (e.g., the head of a brush) of the real object (910) based on the path (60) along which one side (911) (e.g., the head of a brush) of the real object (910) moves. In one embodiment of the present disclosure, the virtual object (920) can be displayed on the display (244).

[0156] FIG. 10 is a flowchart illustrating a method for creating or deleting a virtual object by tracking at least a portion of a real-world object according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 9 are omitted. For convenience of explanation, FIG. 10 will be described with reference to FIGS. 2 and 3.

[0157] Referring to FIG. 10, operation S340 of FIG. 3 may include operations S1010 and S1020. In one embodiment of the present disclosure, operations S1010 and S1020 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). Detailed steps of operation S340 according to the present disclosure are not limited to those illustrated in FIG. 10, and any one of the operations illustrated in FIG. 10 may be omitted, or operations not illustrated in FIG. 10 may be further included.

[0158] In operation S1010, the electronic device (200) can track the movement of at least a portion of a real object based on the acquired image. In one embodiment of the present disclosure, the electronic device (200) can track the movement of at least a portion of the real object using an artificial intelligence model that inputs at least one image. For example, at least a portion of the real object to be tracked may be preset. In this case, the electronic device (200) can additionally input information corresponding to at least a portion of the preset real object into the artificial intelligence model. For example, the information corresponding to at least a portion of the preset real object may be location information or boundary information of at least a portion of the real object. The location information or boundary information may be included in the recognition result. The tracking result may include at least one of the location, speed, direction, and movement pattern of the tracking target.

[0159] In operation S1020, the electronic device (200) may perform a function of generating or deleting a virtual object based on a tracking result (which may also be referred to as a tracked movement). For example, the electronic device (200) may generate a virtual object along a movement path (i.e., the tracking result) formed by at least a portion of a real object. For example, the electronic device (200) may delete an already generated virtual object along a movement path (i.e., the tracking result) formed by at least a portion of the real object. Whether the electronic device (200) performs a function of generating a virtual object or a function of deleting a virtual object may be determined based on a recognition result (e.g., an interaction between a recognized user's hand and the real object). For example, the recognition result may include at least one of a shape of the real object, a shape of the user's hand, and an interaction between the real object and the user's hand (e.g., a shape in which the user's hand grasps the real object, and a position in which the user's hand grasps the real object).

[0160] FIG. 11 is a flowchart illustrating a method for determining at least one of the color and size of a virtual object based on at least one of the color and size of at least a portion of a real object, according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 10 are omitted. For convenience of explanation, FIG. 11 will be described with reference to FIGS. 2, 3, and 10.

[0161] Referring to FIG. 11, operation S1110 may correspond to operation S1010 of FIG. 10. Operation S1120 may correspond to operation S1020 of FIG. 10. In one embodiment of the present disclosure, operations S1110 to S1150 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The detailed steps of operation S340 are not limited to those illustrated in FIG. 11, and any one of the steps illustrated in FIG. 11 may be omitted, or steps not illustrated in FIG. 11 may be further included.

[0162] In operation S1110, the electronic device (200) can track the movement of a real object based on the acquired image. Operation S1210 corresponds to operation S1010 of FIG. 10, and thus a detailed description thereof is omitted.

[0163] In operation S1120, the electronic device (200) may perform a function of creating or deleting a virtual object based on the tracked movement. Operation S1220 corresponds to operation S1020 of FIG. 10, and thus a detailed description thereof will be omitted.

[0164] In operation S1130, the electronic device (200) can recognize at least one of the color and size of at least a portion of the real object from the acquired image. In one embodiment of the present disclosure, the electronic device (200) can recognize the color of at least a portion of the real object from the acquired image. For example, the electronic device (200) can recognize the color of one side (e.g., the tip of a colored pencil) of the real object. In one embodiment of the present disclosure, the electronic device (200) can recognize the size of at least a portion of the real object from the acquired image. For example, the electronic device (200) can recognize the size of one side (e.g., the head of a brush) of the real object. In one embodiment of the present disclosure, at least a portion of the real object may be a portion of the real object corresponding to a tracking target.

[0165] In operation S1140, the electronic device (200) may determine at least one of a color and a size of a virtual object based on at least one of a color and a size of at least a portion of the real object. For example, if the color of at least a portion of the real object (i.e., the tracking target) is red, the electronic device (200) may determine the color of the virtual object to be red. For example, if the size of at least a portion of the real object (i.e., the tracking target) is a specific size, the electronic device (200) may determine the size of the virtual object to be the specific size.

[0166] In operation S1150, the electronic device (200) may generate a virtual object having at least one of the determined colors and sizes. In one embodiment of the present disclosure, the virtual object may be displayed on the display (244).

[0167] FIG. 12 is a flowchart illustrating a method for determining the color, shape, and form of a virtual object based on the color, shape, and form of a target object pointed to by a real object according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 11 are omitted. For convenience of explanation, FIG. 12 will be described with reference to FIGS. 2, 3, and 10.

[0168] Referring to FIG. 12, operation S1210 may correspond to operation S1010 of FIG. 10. Operation S1220 may correspond to operation S1020 of FIG. 10. In one embodiment of the present disclosure, operations S1210 to S1250 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The detailed steps of operation S340 are not limited to those illustrated in FIG. 12, and any one of the operations illustrated in FIG. 12 may be omitted, or operations not illustrated in FIG. 12 may be further included.

[0169] In operation S1210, the electronic device (200) can track the movement of a real object based on the acquired image. Operation S1210 corresponds to operation S1010 of FIG. 10, and thus a detailed description thereof is omitted.

[0170] In operation S1220, the electronic device (200) may perform a function of creating or deleting a virtual object based on the tracked movement. Operation S1220 corresponds to step S1020 of FIG. 10, and thus a detailed description thereof will be omitted.

[0171] In operation S1230, the electronic device (200) can recognize a target object pointed by a real object from the acquired image. The target object may be a real object that is included in the acquired image but is different from the real object interacting with the user's hand (10). The electronic device (200) can detect the target object from the image. The electronic device (200) can determine whether at least a portion of the real object points to the target object based on the tracking result. For example, the electronic device (200) can determine whether at least a portion of the real object is located at the location of the target object based on at least one of a position, a speed, and a direction of at least a portion of the real object. The electronic device (200) can determine that at least a portion of the real object points to the target object based on determining that at least a portion of the real object is located at the location of the target object. In one embodiment of the present disclosure, the target object may be an object displayed in an image displayed on the display (244). In one embodiment of the present disclosure, the target object may be an actual object within a real physical environment space.

[0172] In operation S1240, the electronic device (200) may determine one of the color, shape, and shape of the virtual object based on at least one of the color, shape, and shape of the target object. For example, if the color of the target object is red, the electronic device (200) may determine the color of the virtual object to be red. For example, if the color of the part of the target object corresponding to the location pointed by the real object is blue, the electronic device (200) may determine the color of the virtual object to be blue.

[0173] In operation S1250, the electronic device (200) may generate a virtual object having at least one of a determined color, shape, and shape. In one embodiment of the present disclosure, the virtual object may be displayed on a display (244).

[0174] Figure 13 is a flowchart illustrating a method for obtaining an image by synthesizing a virtual object onto a real image according to one embodiment of the present disclosure. Any details that overlap with those described in Figures 1 to 12 are omitted. For convenience of explanation, Figure 13 will be described with reference to Figure 2.

[0175] Referring to FIG. 13, a method for obtaining an image synthesized with a virtual object in a real image according to an embodiment of the present disclosure may include operations S1310 and S1320 after operation S1020. In one embodiment, operations S1310 and S1320 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The method for obtaining an image synthesized with a virtual object in a real image according to an embodiment of the present disclosure is not limited to that illustrated in FIG. 13, and any one of the operations illustrated in FIG. 13 may be omitted, or operations not illustrated in FIG. 13 may be further included.

[0176] In operation S1310, the electronic device (200) may acquire a second image by synthesizing a virtual object with a first image obtained by photographing a real space using a camera (210). For example, the first image may correspond to the real image (131) of FIG. 1. For example, the virtual object synthesized with the real image (131) may correspond to the virtual object (132). For example, the second image may correspond to the synthesized image (130) of FIG. 1.

[0177] In operation S1320, the electronic device (200) can display a second image on the display (244) or the display of an external device.

[0178] Figures 14a to 14d are conceptual diagrams illustrating an operation for identifying a pre-registered function based on an occluded portion of a real-world object according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 13 are omitted. For convenience of explanation, Figures 14a to 14d will be described with reference to Figure 2.

[0179] Referring to FIG. 14A, the electronic device (200) can detect a real object (1410) from an image. The electronic device (200) can divide the real object (1410) into a plurality of parts. For example, the electronic device (200) can divide the real object (1410) into a plurality of parts using an artificial intelligence model that inputs a video. For example, the electronic device (200) can divide one side of the real object (1410) (e.g., the head of the board marker) into a first part (1411), the other side of the real object (1410) (e.g., the tail of the board marker or the part that is the farthest from one side of the board marker) into a second part (1412), and the body part of the real object (1410) (e.g., the part where the distance from one side of the board marker and the distance from the other side of the board marker are the same) into a third part (1413). The electronic device (200) can store the result of dividing the real object (1410) into a plurality of parts (1411, 1412, 1413) in the memory (260). For example, the electronic device (200) can store location information or boundary information of the plurality of parts (1411, 1412, 1413) in the memory (260). In FIGS. 14A to 14D , an example in which the real object (1410) is divided into three parts is illustrated, but the present disclosure is not limited thereto. In one embodiment of the present disclosure, the electronic device (200) can obtain at least one location value corresponding to each of the plurality of divided parts. For example, the electronic device (200) can obtain at least one location value corresponding to each of the plurality of divided parts by determining the distance of each of the plurality of divided parts from one side of the real object (1410) (e.g., the head of a board marker). In one embodiment of the present disclosure, the electronic device (200) can divide a real object (1410) into a plurality of parts according to predetermined positions.

[0180] Referring to FIG. 14B, the electronic device (200) can detect a real object (1410) and a user's hand (10) holding the real object (1410) from an image. Based on the segmentation result of FIG. 14A, the electronic device (200) can detect at least one part among a plurality of parts (1411, 1412, 1413) that is occluded by the user's hand (10). For example, as illustrated in FIG. 14B, a second part (1412), which is the other side of the real object (1410) (e.g., the tail of the board marker or the part that is furthest from one side of the board marker), can be occluded by the user's hand (10) holding the real object (1410). Based on the detection that the second part (1412) of the real object (1410) is occluded, the electronic device (200) can identify a pre-registered function. For example, the pre-registered function may be a function for generating a virtual object. The electronic device (200) can track the movement of a first part (1411), which is one side (e.g., the head of a board marker) of a real object (1410). The electronic device (200) can generate a virtual object based on the tracking result.

[0181] Referring to FIG. 14C, based on the segmentation result of FIG. 14A, the electronic device (200) can detect at least one part among the plurality of parts (1411, 1412, 1413) that is occluded by the user's hand (10). For example, as illustrated in FIG. 14C, a first part (1411), which is one side (e.g., the head of a board marker) of a real object (1410), may be occluded by the user's hand (10) that holds the real object (1410). Based on the detection that the first part (1411) of the real object (1410) is occluded, the electronic device (200) can identify a pre-registered function. For example, the pre-registered function may be a function of deleting a virtual object. The electronic device (200) can track the movement of the second part (1412), which is the other side of the real object (1410) (e.g., the tail part of the board marker or the part that is the farthest from one side of the board marker). The electronic device (200) can delete a pre-generated virtual object based on the tracking result.

[0182] Referring to FIG. 14d, based on the segmentation result of FIG. 14a, the electronic device (200) can detect at least one part among the plurality of parts (1411, 1412, 1413) that is occluded by the user's hand (10). For example, as illustrated in FIG. 14d, a third part (1413), which is a body part of the real object (1410) (e.g., a part where the distance from one side of the board marker and the distance from the other side of the board marker are the same), can be occluded by the user's hand (10) that holds the real object (1410). The electronic device (200) can perform a preset function based on the detection that the third part (1413) of the real object (1410) is occluded. For example, the preset function can be a function of switching to a mode for setting the characteristics (e.g., color, thickness, etc.) of the virtual object. The electronic device (200) can detect a motion of a user's hand (10) touching a first portion (1411), which is one side of a real object (1410) (e.g., the head of a board marker), or a second portion (1412), which is the other side of the real object (1410) (e.g., the tail of a board marker or the portion that is furthest from one side of the board marker). In the present disclosure, a touch can refer to a motion of a user's hand (10) contacting a real object (1410).

[0183] In one embodiment of the present disclosure, the electronic device (200) may perform a function of switching to a mode for setting characteristics (e.g., color, thickness, etc.) of a virtual object based on a detected motion of the user's hand (10). For example, in the mode for setting characteristics of the virtual object, if a motion of the user's hand (10) continuously touching the first part (1411) of the real object (1410) a predetermined number of times is detected, the electronic device (200) may provide a user interface including a button or slider for determining the color of the virtual object through the display (244). In one embodiment of the present disclosure, the electronic device (200) may determine the color of the virtual object based on the position or movement of the real object (1410). For example, in a mode for setting the characteristics of a virtual object, if a motion is detected in which the user's hand (10) continuously touches the second part (1412) of the real object (1410) a predetermined number of times, the electronic device (200) may provide a user interface including a button or slider, etc. for determining the thickness of the virtual object through the display (244). In one embodiment of the present disclosure, the electronic device (200) may determine the thickness of the virtual object based on the position or movement of the real object (1410). For example, if a motion is detected in which the user's hand (10) touches the second part (1412) of the real object (1410) for a predetermined time, the electronic device (200) may obtain an image in which an image corresponding to a real scene and a virtual object displayed on the display are synthesized.

[0184] In the above example, the motion of the user's hand (10) was described as a motion of touching a segmented portion of the real object (1410) a predetermined number of times, but the motion of the user's hand (10) is not limited thereto. For example, the motion of the user's hand (10) may be modified in various ways, such as maintaining a touch on a segmented portion of the real object (1410) for a predetermined time, or touching the real object (1410) using a predetermined number of fingers. For example, when a motion of touching the real object (1410) with two fingers (e.g., the index and middle fingers) is detected, the electronic device (200) may provide a user interface including a button or slider for determining the color of the virtual object, and when a motion of touching the real object (1410) with three fingers (e.g., the index, middle, and ring fingers) is detected, the electronic device (200) may provide a user interface including a button or slider for determining the thickness of the virtual object.

[0185] Figure 15 is a flowchart illustrating a method for identifying a pre-registered feature based on an occluded portion of a real-world object according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 14d are omitted. For convenience of explanation, Figure 15 will be described with reference to Figures 2 and 14c.

[0186] Referring to FIG. 15, operation S320 of FIG. 3 may include operations S1510 to S1530. Operation S330 of FIG. 3 may include operation S1540. In one embodiment of the present disclosure, operations S1510 to S1540 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The detailed operations of operations S330 and S340 according to the present disclosure are not limited to those illustrated in FIG. 15, and any one of the operations illustrated in FIG. 15 may be omitted, and operations not illustrated in FIG. 15 may further be included.

[0187] In operation S1510, the electronic device (200) can detect a real object (1410) from the acquired image. In one embodiment of the present disclosure, the electronic device (200) can detect the real object (1410) using an artificial intelligence model that inputs at least one image.

[0188] In operation S1520, the electronic device (200) may divide the detected real object (1410) into a plurality of parts (1411, 1412, 1413). In one embodiment of the present disclosure, the number of the plurality of parts of the real object (1410) or the positions of the plurality of parts may be determined in advance by a user or manufacturer setting. In one embodiment of the present disclosure, the electronic device (200) may infer position information or boundary information of the plurality of parts (1411, 1412, 1413) of the real object (1410) using an artificial intelligence model that inputs an image including the real object (1410). In one embodiment of the present disclosure, the position information or boundary information of the plurality of parts (1411, 1412, 1413) may be stored in advance. In this case, operation S1510 may be omitted.

[0189] In operation S1530, the electronic device (200) can detect at least one part occluded by the user's hand (10) among the plurality of parts (1411, 1412, 1413). In one embodiment of the present disclosure, the electronic device (200) can identify the positions of the plurality of parts (1411, 1412, 1413) based on the segmentation result. The electronic device (200) can compare the identified position with the position of the user's hand (10). Based on the comparison result, the electronic device (200) can identify at least one part occluded by the user's hand (10) among the plurality of parts (1411, 1412, 1413). In one embodiment of the present disclosure, when location information or boundary information of a plurality of parts (1411, 1412, 1413) is stored in advance, the electronic device (200) can infer at least one part occluded by the user's hand (10) based on the location information or boundary information of the plurality of parts (1411, 1412, 1413) stored in advance.

[0190] In operation S1540, the electronic device (200) can identify a pre-registered function based on at least one occluded portion. For example, the electronic device (200) can identify a mapped function among a plurality of pre-registered functions based on at least one of the first portion (1411), the second portion (1412), and the third portion (1413) being occluded. For example, the pre-registered function may be one of a function for creating a virtual object, a function for deleting a virtual object, or a function for setting characteristics of a virtual object, but the present disclosure is not limited thereto.

[0191] FIG. 16 is a conceptual diagram illustrating an operation for identifying a pre-registered function based on the way a user's hand (10) grasps a real-world object, according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 14a are omitted. For convenience of explanation, FIG. 16 will be described with reference to FIG. 2.

[0192] Referring to FIG. 16, the electronic device (200) can detect a real object (1610) from an image. The electronic device (200) can divide the real object (1610) into a plurality of parts. For example, the electronic device (200) can divide the real object (1610) into a plurality of parts using an artificial intelligence model that inputs at least one image. For example, the electronic device (200) can divide one side of the real object (1610) (e.g., a tip of a pen) into a first side (1611), the other side of the real object (1610) (e.g., a side that is furthest from the tail of the pen or the tip of the pen) into a second side (1612), and the body side of the real object (1610) (e.g., a side where the distance from one side of the pen and the distance from the pen of a board marker are the same) into a third side (1613). In FIG. 16, an example of a real object (1610) being divided into three parts is illustrated, but the present disclosure is not limited thereto. In one embodiment of the present disclosure, the electronic device (200) can obtain at least one position value corresponding to each of the plurality of divided parts. In one embodiment of the present disclosure, the electronic device (200) can divide the real object (1610) into a plurality of parts according to predetermined positions.

[0193] The electronic device (200) can determine a position at which the user's hand (10) grasps the real object (1610) based on the segmentation result. The electronic device (200) can determine a relative position at which the user's hand (10) grasps the real object (1610) based on at least one position value corresponding to each of the plurality of segmented portions. For example, the electronic device (200) can determine that the user's hand (10) is grasping the real object (1610) at a position corresponding to between the first portion (1611) and the third portion (1613). For example, if the position at which the user's hand (10) grasps the real object (1610) is determined to be between the first portion (1611) and the third portion (1613), the electronic device (200) can perform a function of generating a virtual object. For example, if the location where the user's hand (10) grasps the real object (1610) is determined to be between the second part (1612) and the third part (1613), the electronic device (200) can perform a function of deleting the virtual object.

[0194] In one embodiment of the present disclosure, the electronic device (200) can determine a gripping direction indicating which of the plurality of parts (1611, 1612, 1613) the user's hand (10) is gripping the real object (1610). In one embodiment of the present disclosure, the electronic device (200) can estimate the posture of the user's hand (10) using an artificial intelligence model that inputs an image including the user's hand (10). For example, the electronic device (200) can detect the user's hand (10) and detect feature points for joints included in the user's hand (10) (for example, a portion where a plurality of bones included in the hand (10) are connected to each other, which may represent one or more parts included in the wrist, fingers, back of the hand, or palm). The feature point may mean a point within the image that is distinguishable from the surrounding background or easy to identify. For example, the feature points of the hand joint may include at least one of the feature points of the wrist joint, the feature points of the palm joint, the feature points of the dorsal joint of the hand, and the feature points of the fingers (thumb, index finger, middle finger, ring finger, little finger). For example, the feature points may include two-dimensional or three-dimensional coordinate values.

[0195] In one embodiment of the present disclosure, the electronic device (200) can determine the gripping direction of the user's hand (10) based on feature points. For example, the electronic device (200) can identify the position and shape of at least one finger (e.g., thumb (11a), index finger (11b), and middle finger (11c)) of the user's hand (10) based on feature points of the fingers. The electronic device (200) can determine the gripping direction of the user's hand (10) based on the position and shape of the fingers. For example, the electronic device (200) can identify that the thumb (11a), index finger (11b), and middle finger (11c) grip the real object (1610) from the third portion (1613) toward the first portion (1612).

[0196] The electronic device (200) can identify and perform a pre-registered function based on the gripping direction. For example, if the direction in which the user's hand (10) grips the real object (1610) is determined to be from the third part (1613) to the first part (1611), the electronic device (200) can perform a function of generating a virtual object. For example, if the direction in which the user's hand (10) grips the real object (1610) is determined to be from the third part (1613) to the second part (1612), the electronic device (200) can perform a function of deleting the virtual object.

[0197] According to one embodiment of the present disclosure, the interaction between a real object and a user's hand (10) can be more accurately determined by considering not only the grip position but also the grip direction.

[0198] FIG. 17 is a flowchart illustrating a method for identifying a pre-registered function based on the manner in which a user's hand (10) grasps a real-world object, according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 16 are omitted. For convenience of explanation, FIG. 17 will be described with reference to FIGS. 2, 15, and 16.

[0199] Referring to FIG. 17, operation S320 of FIG. 3 may include operations S1710 to S1730. Operation S330 of FIG. 3 may include operation S1740. In one embodiment of the present disclosure, operations S1710 to S1740 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The detailed operations of operations S330 and S340 according to the present disclosure are not limited to those illustrated in FIG. 17, and any one of the operations illustrated in FIG. 17 may be omitted, or operations not illustrated in FIG. 17 may be further included. Operation S1710 corresponds to operation S1510 of FIG. 15, and operation S1720 corresponds to operation S1520 of FIG. 15, and therefore, a detailed description thereof will be omitted.

[0200] In operation S1730, the electronic device (200) can determine a position at which the user's hand (10) grasps the real object (1610) based on the segmentation result. In one embodiment of the present disclosure, the electronic device (200) can identify positions of a plurality of parts (1611, 1612, 1613) based on the segmentation result. The electronic device (200) can compare the identified positions with the position of the user's hand (10). Based on the comparison result, the electronic device (200) can determine a relative position of the user's hand (10) with respect to the positions of the plurality of parts (1611, 1612, 1613). In one embodiment of the present disclosure, when position information or boundary information of a plurality of parts (1611, 1612, 1613) is stored in advance, the electronic device (200) can infer the relative position of the user's hand (10) based on the position information or boundary information of the plurality of parts (1611, 1612, 1613) stored in advance.

[0201] In operation S1740, the electronic device (200) can identify a pre-registered function based on the determined location. For example, the electronic device (200) can identify a mapped function among a plurality of pre-registered functions based on the user's hand (10) being located between at least two of the first part (1611), the second part (1612), and the third part (1613). For example, the pre-registered function may be one of a function for creating a virtual object, a function for deleting a virtual object, or a function for setting characteristics of a virtual object, but the present disclosure is not limited thereto.

[0202] FIG. 18 is a flowchart illustrating a method for identifying a pre-registered function based on the manner in which a user's hand (10) grasps a real-world object, according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 17 are omitted. For convenience of explanation, FIG. 18 will be described with reference to FIG. 2 and FIGS. 15 to 17.

[0203] Referring to FIG. 18, operation S320 of FIG. 3 may include operations S1810 to S1835. Operation S330 of FIG. 3 may include operation S1840. In one embodiment of the present disclosure, operations S1810 to S1840 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The detailed operations of operations S330 and S340 according to the present disclosure are not limited to those illustrated in FIG. 18, and any one of the operations illustrated in FIG. 18 may be omitted, or operations not illustrated in FIG. 18 may be further included. Operation S1810 corresponds to S1510 of FIG. 15, operation S1820 corresponds to operation S1520 of FIG. 15, and operation S1830 corresponds to operation S1730 of FIG. 17, and therefore, a detailed description thereof will be omitted.

[0204] In operation S1835, the electronic device (200) may determine a gripping direction indicating which of the plurality of parts (1611, 1612, 1613) the user's hand (10) is gripping the real object (1610). In one embodiment of the present disclosure, the electronic device (200) may perform pose estimation of the user's hand (10) using an artificial intelligence model. In one embodiment of the present disclosure, the electronic device (200) may determine a direction in which at least one finger of the user's hand (10) grips the real object (1610) based on the pose estimation result.

[0205] In operation S1840, the electronic device (200) can identify a pre-registered function based on the determined position and the determined gripping direction. For example, the electronic device (200) can identify a mapped function among a plurality of pre-registered functions based on the user's hand (10) being positioned between at least two of the first portion (1611), the second portion (1612), and the third portion (1613) and the user's hand (10) gripping from one portion toward the other portion. For example, the pre-registered function may be one of a function for creating a virtual object, a function for deleting a virtual object, or a function for setting characteristics of a virtual object, but the present disclosure is not limited thereto.

[0206] Figure 19 is a flowchart illustrating a method for identifying a pre-registered function based on the way a user's hand grasps a real-world object, according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 18 are omitted. For convenience of explanation, Figure 19 will be described with reference to Figures 2 and 15.

[0207] Referring to FIG. 19, operation S320 of FIG. 3 may include operations S1910 to S1930. Operation S330 of FIG. 3 may include operation S1940. In one embodiment of the present disclosure, operations S1910 to S1940 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The detailed operations of operations S330 and S340 according to the present disclosure are not limited to those illustrated in FIG. 19, and any one of the operations illustrated in FIG. 19 may be omitted, or operations not illustrated in FIG. 19 may be further included. Operation S1910 corresponds to operation S1510 of FIG. 15, and operation S1920 corresponds to operation S1520 of FIG. 15, and therefore, a detailed description thereof will be omitted.

[0208] In operation S1930, the electronic device (200) can recognize (or detect) a motion of the user's hand (10) touching at least one of a plurality of parts (1411, 1412, 1413) of a real object (1410). In one embodiment of the present disclosure, the electronic device (200) can track the motion of the user's hand (10). In one embodiment of the present disclosure, the electronic device (200) can perform pose estimation of the user's hand (10). The electronic device (200) can estimate the motion of the user's hand (10) based on the pose estimation result of the user's hand (10). For example, an action of the user's hand (10) touching at least one of the plurality of parts (1411, 1412, 1413) of the real object (1410) may include an action of the user's hand (10) touching at least one of the plurality of parts (1411, 1412, 1413) of the real object (1410) a predetermined number of times or a predetermined amount of time.

[0209] In operation S1940, the electronic device (200) can identify a preset function based on the recognized motion. For example, the electronic device (200) can recognize (or detect) a motion in which the user's hand (10) touches at least one of a plurality of parts (1411, 1412, 1413) of a real object (1410) a predetermined number of times. The electronic device (200) can perform a function of switching to a mode for setting a virtual object that changes the color or thickness of the virtual object based on the detected motion.

[0210] FIG. 20 is a flowchart illustrating a method for determining whether to perform a predetermined function based on the distance between a user's hand (10) and a real-world object, according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 19 are omitted. For convenience of explanation, FIG. 20 will be described with reference to FIGS. 2 and 3.

[0211] Referring to FIG. 20, operation S2010 may be included between operation S320 and operation S330 of FIG. 3. In one embodiment of the present disclosure, operation S2010 may be performed by the electronic device (200) or the processor (250) of the electronic device (200).

[0212] In operation S2010, the electronic device (200) can determine whether the distance between the user's hand (10) and the real object exceeds a threshold value. The threshold value may be preset by the user or manufacturer. If the threshold value is exceeded, the procedure ends. In this case, the electronic device (200) may no longer use the real object as an input tool. If the threshold value is not exceeded, the procedure moves to operation S330.

[0213] In one embodiment of the present disclosure, the electronic device (200) can measure the distance between the user's hand (10) and the real object. For example, the electronic device (200) can measure the distance between the user's hand (10) and the real object by calculating the pixel distance between the recognized user's hand (10) and the real object. Alternatively, the electronic device (200) can also measure the distance between the user's hand (10) and the real object using a sensor (220) capable of measuring distance (e.g., an ultrasonic sensor). The electronic device (200) can determine to use the real object as an input tool if the measured distance does not exceed a predefined threshold value. The electronic device (200) can determine not to use the real object as an input tool if the measured distance exceeds the predefined threshold value.

[0214] FIG. 21 is a conceptual diagram illustrating an operation for determining the performance value of a virtual object based on the characteristics of a real object according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 20 are omitted. For convenience of explanation, FIG. 21 will be described with reference to FIG. 2.

[0215] Referring to FIG. 21, the electronic device (200) can detect real objects (2110a, 2110b, 2110c) and a user's hand (10) from an image. For example, as illustrated in FIG. 21, the real objects (2110a, 2110b, 2110c) may be table tennis rackets, but the present disclosure is not limited thereto and may be any object that can be gripped by the user's hand (10). The electronic device (200) can detect at least one of a color, a material, a shape, a shape of the user's hand (10), and a position of the user's hand (10) of at least a portion of the real objects (2110a, 2110b_1, 2110b_2) from the image.

[0216] In one embodiment of the present disclosure, the electronic device (200) can classify the type of a real object (2110a, 2110b_1, 2110b_2). For example, the electronic device (200) can classify whether a table tennis racket, which is a real object (2110a, 2110b_1, 2110b_2), is a penholder racket (e.g., real object (2110a)) or a shakehand racket (e.g., real object (2110b_1, 2110b_2)). In one embodiment of the present disclosure, the electronic device (200) can classify the type of the real object (2110a, 2110b_1, 2110b_2) using an artificial intelligence model that inputs an image including the real object (2110a, 2110b_1, 2110b_2). In one embodiment of the present disclosure, the electronic device (200) can classify the type of the real object (2110a, 2110b_1, 2110b_2) based on the manner in which the user's hand (10) grasps the real object (2110a, 2110b_1, 2110b_2). For example, the electronic device (200) can identify the manner in which the user's hand (10) grasps the real object (2110a, 2110b_1, 2110b_2) by performing a pose estimation of the user's hand (10).

[0217] In one embodiment of the present disclosure, the electronic device (200) can detect (or classify) the material of at least a portion of a real object (2110a, 2110b_1, 2110b_2). For example, the electronic device (200) can classify whether the rubber of the other side of a table tennis racket, which is a real object (2110a, 2110b_1, 2110b_2), is a flat rubber or a protruding rubber. In one embodiment of the present disclosure, the electronic device (200) can classify the material of at least a portion of the real object (2110a, 2110b_1, 2110b_2) using an artificial intelligence model that inputs an image including the real object (2110a, 2110b_1, 2110b_2).

[0218] In one embodiment of the present disclosure, the electronic device (200) can detect (or classify) the color of at least a portion of a real object (2110a, 2110b_1, 2110b_2). For example, the electronic device (200) can classify whether the color of the other side of a table tennis racket, which is a real object (2110a, 2110b_1, 2110b_2), is red or black. For example, the first side of the table tennis racket (e.g., the real object (2110b_1)) can be red, and the second side (the real object (2110b_2)) can be black. In one embodiment of the present disclosure, the electronic device (200) can classify the color of at least a portion of the real object (2110a, 2110b_1, 2110b_2) using an artificial intelligence model that inputs an image including the real object (2110a, 2110b_1, 2110b_2).

[0219] In one embodiment of the present disclosure, the electronic device (200) can determine the performance values ​​of virtual objects (2131, 2132) corresponding to real objects (2110a, 2110b_1, 2110b_2) based on the detection results. For example, the predetermined performance value can be determined based on whether the table tennis racket, which is the real object (2110a), is a penholder racket or a shakehand racket. For example, the electronic device (200) can access a mapping table stored in the memory (260) (or storage) or an external device that is communicatively connected to the electronic device (200). For example, the mapping table can include relationship information between the type of the table tennis racket and the performance value. In one embodiment of the present disclosure, the mapping table can be predetermined by a user or a manufacturer's setting. The electronic device (200) can determine the performance value mapped to each type of racket as the performance value of the virtual object (2131) based on the mapping table. For example, a penholder racket may be mapped to a first spin value and a first power value, and a shakehand racket may be mapped to a second spin value and a second power value. The first spin value and the second spin value may be different, for example, the first spin value may be less than the second spin value. And, the first power value may be different from the second power value, for example, the first power value may be greater than the second power value.

[0220] For example, a predetermined performance value may be determined based on the material of at least a portion of a real object (2110a), which is a real object (2110a), (2110b_1, 2110b_2). For example, the electronic device (200) may access a mapping table stored in the memory (260) (or storage) or an external device that is communicatively connected to the electronic device (200). For example, the mapping table may include relationship information between the material of the other side of the table tennis racket and the performance value. In one embodiment of the present disclosure, the mapping table may be predetermined by a user or manufacturer's setting. The electronic device (200) may determine a performance value mapped to each material of the other side of the table tennis racket as a performance value of the virtual object (2131) based on the mapping table. For example, a protruding rubber may be mapped to a third power value, and a flat rubber may be mapped to a fourth power value. The third power value and the fourth power value may be different, for example, the third power value may be greater than the fourth power value.

[0221] For example, a predetermined performance value may be determined based on the color of at least a portion of a real object (2110a, 2110b_1, 2110b_2). For example, the electronic device (200) may access a mapping table stored in the memory (260) (or storage) or an external device that is communicatively connected to the electronic device (200). For example, the mapping table may include relationship information between the color of the other side of the table tennis racket and the performance value. In one embodiment of the present disclosure, the mapping table may be predetermined by a user or manufacturer's setting. The electronic device (200) may determine a performance value mapped to each color of the other side of the table tennis racket as a performance value of the virtual object (2131) based on the mapping table. For example, a red other side may be mapped to a third spin value and a fifth power value, and a black other side may be mapped to a fourth spin value and a sixth power value. The third spin value and the fourth spin value may be different, for example, the third spin value may be greater than the fourth spin value. And, the fifth power value may be different from the sixth power value, for example, the fifth power value may be less than the sixth power value.

[0222] In one embodiment of the present disclosure, the electronic device (200) can display a virtual reality game (2130) through a display (244) or a display of an external device that is communicatively connected to the electronic device (200). For example, as illustrated in FIG. 21, the virtual reality game (2130) may be a table tennis game, but the present disclosure is not limited thereto and may be a game that uses any object that can be grasped by a user's hand (10). The virtual reality game (2130) may include virtual objects (2131, 2132).

[0223] In one embodiment of the present disclosure, the electronic device (200) can track the movement of at least a portion of a real object (2110a, 2110b_1, 2110b_2). The electronic device (200) can display the movement of a virtual object (2131) corresponding to the movement of the real object (2110a, 2110b_1, 2110b_2) on the display (244) based on the tracking result corresponding to the movement of the real object (2110a, 2110b_1, 2110b_2). Depending on the movement of the virtual object (2131), the virtual object (2131) can collide with the virtual object (2132). The electronic device (200) can calculate a physical quantity after the collision of the virtual object (2132) based on at least one of the speed, direction, position of the movement of the real object (2110a, 2110b_1, 2110b_2), and the determined performance value of the virtual object (2131). The electronic device (200) can determine at least one of the speed, direction, position, and spin direction of the movement of the virtual object (2132) based on the calculated physical quantity.

[0224] Figure 22 is a flowchart illustrating a method for determining performance values ​​of a virtual object based on the characteristics of a real object, according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 21 are omitted. For convenience of explanation, Figure 22 will be described with reference to Figures 2, 3, and 21.

[0225] Referring to FIG. 22, operation S340 of FIG. 3 may include operation S2210. In one embodiment of the present disclosure, operation S2210 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). Detailed steps of operation S340 according to the present disclosure are not limited to those illustrated in FIG. 22, and any one of the operations illustrated in FIG. 22 may be omitted, or operations not illustrated in FIG. 22 may be further included.

[0226] In operation S2210, the electronic device (200) may perform a function of determining a performance value of a virtual object based on at least one of a color, a material, a shape of at least a portion of a real object (2110a, 2110b_1, 2110b_2), a shape of a user's hand (10), and a position of the user's hand (10). However, the present disclosure is not limited thereto, and the performance value of the virtual object (2131) may be determined based on any characteristic of the real object (2110a, 2110b_1, 2110b_2) that can be acquired from an image. In one embodiment of the present disclosure, relationship information between any characteristic of the real object (2110a, 2110b_1, 2110b_2) and the performance value may be previously included in a mapping table stored in a memory (260) (or storage) of the electronic device (200) or an external device that is communicatively connected to the electronic device (200).

[0227] In one embodiment of the present disclosure, the electronic device (200) can track the movement of at least a portion of a real object (2110a, 2110b_1, 2110b_2). The electronic device (200) can display the movement of a virtual object (2131) based on the tracking result on the display (244). The electronic device (200) can calculate a first physical quantity corresponding to the movement of the virtual object (2131) based on the determined performance value. The electronic device (200) can calculate a second physical quantity, which is a post-collision physical quantity of another virtual object (2132) colliding with the virtual object (2131), based on the first physical quantity. For example, the electronic device (200) can calculate the speed, rotational speed, or angle at the time of collision of the virtual object (2131) based on the determined performance value, and can calculate the speed, rotational speed, or direction of the virtual body (2132) after the collision based on the calculated speed, rotational speed, or angle at the time of collision of the virtual object (2131). The electronic device (200) can utilize known physical laws (e.g., conservation of momentum) to calculate the first physical quantity and the second physical quantity.

[0228] FIG. 23 is a conceptual diagram illustrating an operation of using a real object similar to a virtual object to be mapped as an input tool according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 22 are omitted. For convenience of explanation, FIG. 23 will be described with reference to FIGS. 2 and 21.

[0229] Referring to FIG. 23, the purpose and function of the real object (2310) are different from those of the virtual object (2131) of the virtual reality game (2130), but the shape and form of the real object (2310) may be similar to those of the virtual object (2131). For example, the real object (2310) may include a paddle face (2311) and a handle (2312) similar to the shape of the virtual object (2131). The real object (2310) may be utilized as an input tool of the electronic device (200) so as to be used for the same purpose as the virtual object (2131). For example, the electronic device (200) may obtain a video including the real object (2310). The electronic device (200) may detect the real object (2310) and the user's hand (10) holding the real object (2310) from the image. The electronic device (200) can track the movement of at least a portion of the real object (2310) (e.g., the surface (2311)) based on the detection result. The electronic device (200) can display the movement of the virtual object (2131) mapped to the real object (2310) on the display (244) based on the tracking result.

[0230] Figure 24 is a conceptual diagram illustrating an operation for generating a virtual object based on a predetermined movement according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 23 are omitted. For convenience of explanation, Figure 24 will be described with reference to Figure 2.

[0231] Referring to FIG. 24, the electronic device (200) can obtain a reality image (2430) including a third reality object (2431). For example, in FIG. 24, the third reality object (2431) is illustrated as a document, but the present disclosure is not limited thereto. The electronic device (200) can obtain an image including a fourth reality object (2410) and a user's hand (10) through a camera (210) of the electronic device (200). The electronic device (200) can detect the fourth reality object (2410) and the user's hand (10) from the obtained image. The electronic device (200) can detect a movement of the recognized fourth reality object (2410). The electronic device (200) can determine whether the detected movement is a predetermined movement. The electronic device (200) may perform a function of generating a predetermined virtual object (2420) based on determining that the detected movement is a predetermined movement. For example, the virtual object (2420) may be a user's signature or an arbitrary stamp. For example, the movement of the fourth real object (2410) that triggers a function of the electronic device (200) may be preset by a user or manufacturer's setting. For example, the electronic device (200) may trigger a function of the electronic device (200) based on at least one of the distance, direction, speed, and number of times the fourth real object (2410) moves. For example, the virtual object (2420) may be preset by a user or manufacturer's setting.

[0232] Figure 25 is a conceptual diagram illustrating an operation for generating a virtual object based on a predetermined movement according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 24 are omitted. For convenience of explanation, Figure 25 will be described with reference to Figure 2.

[0233] Referring to FIG. 25, the electronic device (200) can obtain a real image (2530). For example, although the real image (2530) in FIG. 25 is illustrated as an image taken of a real location, the present disclosure is not limited thereto. The electronic device (200) can obtain an image including a real object (2510) and a user's hand (10). The electronic device (200) can detect the real object (2510) and the user's hand (10) from the image. The electronic device (200) can detect the movement of the recognized real object (2510). The electronic device (200) can determine whether the detected movement is a predetermined movement. Based on determining that the detected movement is a predetermined movement, the electronic device (200) can perform a function of generating a predetermined virtual object (2520). For example, the virtual object (2520) can be a stamp corresponding to one's evaluation of a location. According to one embodiment of the present disclosure, a user can intuitively check his / her evaluation of a place he / she has already visited.

[0234] FIG. 26 is a flowchart illustrating a method for generating a virtual object based on predetermined movements according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 25 are omitted. For convenience of explanation, FIG. 26 will be described with reference to FIGS. 2, 3, and 24.

[0235] Referring to FIG. 26, operation S340 of FIG. 3 may include operations S2610 to S2630. In one embodiment of the present disclosure, operations S2610 to S2630 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The detailed steps of operation S340 according to the present disclosure are not limited to those illustrated in FIG. 26, and any one of the operations illustrated in FIG. 26 may be omitted, or operations not illustrated in FIG. 26 may be further included.

[0236] In operation S2610, the electronic device (200) can detect the movement of the recognized real object. In one embodiment of the present disclosure, the electronic device (200) can track the movement of at least a portion of the real object. In one embodiment of the present disclosure, the electronic device (200) can detect the movement of the object by matching feature points of the object within a plurality of image frames. In one embodiment of the present disclosure, the electronic device (200) can track the movement of at least one of the real object (2410) and the user's hand (10). The electronic device (200) can determine whether a predetermined movement has occurred based on the tracking result.

[0237] In operation S2620, the electronic device (200) can determine whether the detected movement is a predetermined movement. For example, the movement of the real object (2410) can be predetermined based on at least one of the distance, speed, direction, and number of movements of the real object (2410).

[0238] In operation S2630, the electronic device (200) may perform a function of generating a predetermined virtual object (2420) based on determining that the detected movement is a predetermined movement. The electronic device (200) may display the virtual object (2420) through a display (244) or a display of an external device. In one embodiment of the present disclosure, the electronic device (200) may generate the virtual object (2420) at a point (or coordinates) of a real image (2530) corresponding to a location where the movement of the real object (2410) occurs.

[0239] Figure 27 is a conceptual diagram illustrating an operation for creating a virtual object based on a predetermined movement and personal authentication input according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 26 are omitted. For convenience of explanation, Figure 27 will be described with reference to Figure 2.

[0240] Referring to FIG. 27, the electronic device (200) may display a security document screen (2730) requesting a user's signature through the display (244). The electronic device (200) may acquire an image including a real object (2710) and the user's hand (10). The electronic device (200) may detect the real object (2710) and the user's hand (10) from the image acquired through the camera (210) of the electronic device (200). The electronic device (200) may determine whether a predetermined movement of the real object (2710) has occurred. Based on the determination that the predetermined movement has occurred, the electronic device (200) may activate an authentication mode that requests personal authentication input. In one embodiment of the present disclosure, the authentication mode may be an iris authentication mode. For example, the electronic device (200) may recognize the user's iris (70) using a sensor (220) (e.g., an iris sensor). For example, if the electronic device (200) is a virtual reality device or an augmented reality device, the sensor (220) may include a left eye sensor and a right eye sensor. The left eye sensor may be positioned toward the iris of the user's left eye, and the right eye sensor may be positioned toward the iris of the user's right eye. The electronic device (200) may generate a predetermined virtual object (2720) in response to a personal authentication input. For example, the electronic device (200) may compare the iris recognition result with pre-stored user information. Based on a determination that the iris recognition result and the user information match, the electronic device (200) may generate the predetermined virtual object (2720). For example, the predetermined virtual object (2720) may be the user's signature, seal, or seal pre-stored by the user. The electronic device (200) may display the predetermined virtual object (2720) together with a security document screen (2730) on the display (244).

[0241] Figure 28 is a flowchart illustrating a method for generating a virtual object based on predetermined movements and personal authentication input according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 through 27 are omitted. For convenience of explanation, Figure 28 will be described with reference to Figures 2, 26, and 27.

[0242] Referring to FIG. 28, operation S2630 of FIG. 26 may include operations S2810 to S2840. In one embodiment of the present disclosure, operations S2810 to S2840 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The detailed operations of operation S2630 according to the present disclosure are not limited to those illustrated in FIG. 28, and any one of the operations illustrated in FIG. 28 may be omitted, and operations not illustrated in FIG. 28 may further be included.

[0243] In operation S2810, the electronic device (200) may activate an authentication mode requiring personal authentication input based on determining that the movement of the recognized real object (2710) is a predetermined movement.

[0244] In operation S2820, the electronic device (200) may obtain personal authentication input using the sensor (220). For example, the sensor (220) may be an iris recognition sensor, but the present disclosure is not limited thereto, and may be any sensor capable of verifying the user's identity. For example, the personal authentication input may be the user's iris recognition data, but the present disclosure is not limited thereto, and may correspond to any data sensed by the sensor (220).

[0245] In operation S2830, the electronic device (200) may generate a predetermined virtual object (2720) in response to a personal authentication input. The electronic device (200) may generate a virtual object (2720) stored in the memory (260). For example, the location of the generated virtual object (2720) may be a point (or coordinates) on the security document screen (2730) corresponding to the location where the movement of the real object (2710) occurred, but the location of the virtual object (2720) may vary depending on a user input.

[0246] In operation S2840, the electronic device (200) can display a predetermined virtual object (2720) on the display (244). The electronic device (200) can synthesize the virtual object (2720) on the security document screen (2730) displayed on the display (244).

[0247] Figure 29 is a conceptual diagram illustrating an operation for activating a predetermined sensor based on a detection result according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 28 are omitted. For convenience of explanation, Figure 29 will be described with reference to Figure 2.

[0248] Referring to FIG. 29, an electronic device (200) may be an augmented reality device worn by a user (1). A real object (2910) may be grasped by a hand (10) of the user (1). The electronic device (200) may obtain a video including a scene in which the hand (10) of the user (1) grasps the real object (2910) using a camera (210). The electronic device (200) may detect the real object (2910) and the hand (10) of the user (1) based on the video. In addition, the electronic device (200) may identify the type of the detected real object (2910). For example, the electronic device (200) may obtain the type of the real object (2910) using an artificial intelligence model (e.g., an object classification model) that inputs a video. For example, as illustrated in FIG. 29, the electronic device (200) can identify that the real object (2910) is a spoon. Additionally, the electronic device can identify that the hand (10) of the user (1) is holding the real object (2910).

[0249] The electronic device (200) may perform a function of activating at least one sensor (220) determined based on the detection result. For example, the electronic device (200) may activate the microphone (242) based on identifying that the hand (10) of the user (1) is holding a spoon, which is a real object (2910). The electronic device (200) may access a mapping table stored in the memory (260) (or storage) or an external device. For example, the mapping table may include relationship information between the type of the real object (2910) and the sensor to be activated. For example, a spoon may be mapped to a microphone (242). In one embodiment of the present disclosure, the mapping table may be predetermined by a user or manufacturer's settings. The electronic device (200) may determine to activate a sensor mapped to the type of the real object (2910) based on the mapping table. The electronic device (200) may receive a user's voice using the activated microphone (242).

[0250] FIG. 30 is a block diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure. Any details overlapping with those described in FIGS. 1 to 29 are omitted. The configuration, operation, and functions of the electronic device (3000) may correspond to those of the electronic device (200) of FIG. 2 .

[0251] Referring to FIG. 30, an electronic device (3000) according to one embodiment may include a microphone (3042), a camera (3010), a speaker (3045), and a processor (3050). However, the components illustrated in FIG. 30 are not essential components, and the electronic device (3000) may omit some of the components illustrated in FIG. 30 or further include additional components. For example, the electronic device (3000) according to one embodiment of the present disclosure may further include at least one of a camera (210), a sensor (220), a communication interface (230), a user interface (240), and a memory (260), as illustrated in FIG. 2. The configuration, operation, and function of the microphone (3042), camera (3010), speaker (3045), and processor (3050) correspond to the configuration, operation, and function of the microphone (242), camera (210), speaker (245), and processor (250) of FIG. 2, so overlapping content is omitted.

[0252] The camera (3010) can photograph (or capture) a scene including a user's hand (10) and a real-world object, and generate a corresponding image or video. The processor (3050) can detect the user's hand and the real-world object from the photographed (or captured) image or video. Based on the detection result, the processor (3050) can activate a microphone (3042) mapped to the real-world object.

[0253] An activated microphone (3042) can receive a user's voice. The microphone (3042) can sample the user's voice, which is an analog signal, and convert it into a digital signal. The microphone (3042) can transmit a first audio signal, which is the user's voice converted into a digital signal, to the processor (3050). The processor (3050) can modulate the first audio signal based on the type of the real object. For example, the processor (3050) can emphasize or suppress a specific frequency band of the first audio signal using a frequency band filter. For example, the processor (3050) can adjust the time scale or volume of the first audio signal. For example, the processor (3050) can add sound effects by performing sound synthesis on the first audio signal.

[0254] In one embodiment of the present disclosure, the processor (3050) may determine a modulation scheme of the first audio signal based on the characteristics of the real object. In one embodiment of the present disclosure, the processor (3050) may access a mapping table stored in memory (or storage) or an external device. For example, the mapping table may include relationship information between the type of the real object (2910) and the audio modulation scheme. For example, a spoon may be mapped to a grand and loud voice, a razor may be mapped to a sharp voice, a toy may be mapped to a cute voice, and a hot dog may be mapped to a fat character voice. In one embodiment of the present disclosure, the mapping table may be predetermined by a user or manufacturer's settings. The processor (3050) may determine to modulate the first audio signal using an audio modulation scheme mapped to the type of the real object (2910) based on the mapping table. The processor (3050) may generate a second audio signal by modulating the first audio signal.

[0255] The speaker (3045) can receive a second audio signal from the processor (3050). The speaker (3045) can convert the second audio signal into an analog signal. The speaker (3045) can output a modulated user voice, which is a converted analog signal, into space.

[0256] Figure 31 is a flowchart illustrating a method for performing a function of activating a predetermined sensor based on a detection result according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 27 are omitted. For convenience of explanation, Figure 31 will be described with reference to Figures 2, 3, and 30.

[0257] Referring to FIG. 31, operation S340 of FIG. 3 may include operation S3110. In one embodiment of the present disclosure, operation S3110 may be performed by an electronic device (200, 3000) or a processor (250, 3050) of the electronic device (200, 3000).

[0258] In operation S3110, the electronic device (3000) may perform a function of activating at least one predetermined sensor. In one embodiment of the present disclosure, the electronic device (3000) may acquire sensor data using the at least one activated sensor. In one embodiment of the present disclosure, the electronic device (3000) may modulate the sensor data based on at least some of the shape, form, color, material, and function of the real object.

[0259] Figure 32 is a flowchart illustrating a method for registering a real-world object as an input tool of an electronic device according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 and 31 are omitted. For convenience of explanation, Figure 32 will be described with reference to Figure 2.

[0260] Referring to FIG. 32, a method for registering a real object as an input tool of an electronic device (200) may include operations S3210 to S3240. In one embodiment of the present disclosure, operations S3210 to S3240 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). A method for an electronic device (200) according to one embodiment of the present disclosure to use a real object as an input tool is not limited to that illustrated in FIG. 32, and any one of the operations illustrated in FIG. 32 may be omitted, or operations not illustrated in FIG. 32 may be further included.

[0261] In operation S3210, the electronic device (200) may acquire at least one image including a real object (which does not communicate with the electronic device (200). In one embodiment of the present disclosure, the electronic device (200) may acquire at least one image including the real object using a camera (210).

[0262] In operation S3220, the electronic device (200) may recognize (or detect) a real object based on at least one image. In one embodiment of the present disclosure, the electronic device (200) may obtain data including location information and boundary information of the real object using an artificial intelligence model that inputs at least one image. In one embodiment of the present disclosure, the electronic device (200) may obtain data including the type of the real object using an artificial intelligence model that inputs at least one image. In one embodiment of the present disclosure, the electronic device (200) may obtain data including 3D rendering data of the real object using an artificial intelligence model that inputs at least one image. For example, the 3D rendering data may include shape data, structural data, color data, and / or texture (e.g., texture, etc.) data of the real object required to display the real object on the display (244).

[0263] In operation S3230, the electronic device (200) can register (or store) a real-world object as an input tool of the electronic device (200). The electronic device (200) can store data including information about the real-world object in the memory (260). In one embodiment of the present disclosure, when the electronic device (200) detects a registered real-world object, the electronic device (200) can perform a function mapped to the real-world object.

[0264] In operation S3240, the electronic device (200) may map the interaction between the registered real object and the user's hand (10) to a predefined function. In one embodiment of the present disclosure, at least one image may include the user's hand (10) grasping the object. The electronic device (200) may detect the user's hand (10) from the at least one image. The electronic device (200) may map the real object to a predefined function of using the real object as the input tool based on at least one of the shape of the real object, the shape of the user's hand (10), and the interaction between the user's hand (10) and the real object. The electronic device (200) may store a mapping table including relationship information between at least one of the shape of the real object, the shape of the user's hand (10), and the interaction between the user's hand (10) and the real object and the predefined function in the memory (260). In one embodiment of the present disclosure, when the electronic device (200) detects at least one of the shape of a real object corresponding to a registered pre-defined function, the shape of the user's hand (10), and the interaction between the user's hand (10) and the real object, the electronic device (200) can identify and perform the registered pre-defined function.

[0265] In one embodiment of the present disclosure, an electronic device (200) can divide a real object into a plurality of parts. The electronic device (200) can detect at least one part occluded by a user's hand (10) among the plurality of parts. Based on the at least one occluded part, the electronic device (200) can register a predefined function that uses the real object as an input tool. The electronic device (200) can store, in a memory (260), a mapping table including relationship information between at least one part occluded by the user's hand (10) among the plurality of parts of the real object and the predefined function. In one embodiment of the present disclosure, when the electronic device (200) detects at least one part occluded by the user's hand (10) among the plurality of parts corresponding to the registered predefined function, the electronic device (200) can identify and perform the registered predefined function.

[0266] In one embodiment of the present disclosure, an electronic device (200) can divide a real object into a plurality of parts. Based on the result of the division, the electronic device (200) can determine a position where a user's hand (10) grasps the real object. Based on the determined position, the electronic device (200) can register a predefined function that uses the real object as an input tool. The electronic device (200) can store a mapping table including relationship information between a position where the user's hand (10) grasps the real object and a predefined function in a memory (260). In one embodiment of the present disclosure, when the electronic device (200) detects a position where the user's hand (10) grasps the real object corresponding to a registered predefined function, the electronic device (200) can identify and perform the registered predefined function.

[0267] In one embodiment of the present disclosure, the electronic device (200) can determine a gripping direction indicating which part of a plurality of parts of the real object the user's hand (10) is gripping the real object toward. The electronic device (200) can register a predefined function based on the determined gripping direction. The electronic device (200) can store, in the memory (260), a mapping table including relationship information between at least one of the direction and the gripping position of the user's hand (10) gripping the real object and the predefined function. In one embodiment of the present disclosure, when the electronic device (200) detects at least one of the direction and the gripping position of the user's hand (10) gripping the real object corresponding to the registered predefined function, the electronic device (200) can identify and perform the registered predefined function.

[0268] In one embodiment of the present disclosure, the electronic device (200) can detect a motion of a user's hand (10) touching at least one of a plurality of parts. Based on the detected motion, the electronic device (200) can register a predefined function of using a real object as an input tool. The electronic device (200) can store a mapping table including relationship information between the motion of the user's hand (10) touching at least one of the plurality of parts and the predefined function in the memory (260). In one embodiment of the present disclosure, when the electronic device (200) detects a motion of the user's hand (10) touching at least one of the plurality of parts corresponding to the registered predefined function, the electronic device (200) can identify and perform the registered predefined function.

[0269] Figure 33 is a conceptual diagram illustrating a method for performing a pre-registered function based on the movement or shape of a user's ball, according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 32 are omitted. For convenience of explanation, Figure 33 will be described with reference to Figure 2.

[0270] Referring to FIG. 33, the electronic device (200) can capture an image of a real object (3310) and a hand (10) of a user (1) in a real space using a first camera (210a). The electronic device (200) can recognize a real object (3310) interacting with the hand (10) of the user (1) from the image. In one embodiment of the present disclosure, the electronic device (200) can recognize a real object (3310) held by the hand (10) of the user (1). In FIG. 33, the real object (3310) is illustrated as a straw, but the present disclosure is not limited thereto, and the real object (3310) may be an object of a size that a person can hold with a hand.

[0271] In one embodiment of the present disclosure, the electronic device (200) can obtain (measure or estimate) the distance between the real object (3310) and the mouth of the user (1). For example, the electronic device (200) can obtain the distance between the real object (3310) and the mouth of the user (1) by calculating the pixel distance between the real object (3310) and the mouth of the user (1) in the obtained image. Alternatively, the electronic device (200) can obtain the distance between the real object (3310) and the mouth of the user (1) by using a sensor (220) capable of measuring the distance (e.g., an ultrasonic sensor). The electronic device (200) can determine whether the distance between the real object (3310) (or one side of the real object (3310)) and the mouth of the user (1) is less than or equal to a pre-defined threshold value. Based on determining that the distance between the real object (3310) and the user's (1) mouth is less than or equal to a pre-defined threshold value, the electronic device (200) may determine to use the real object (3310) as an input tool for performing a pre-registered function.

[0272] In one embodiment of the present disclosure, based on a determination that a real object (3310) will be used as an input tool for performing a pre-registered function, the electronic device (200) may activate the second camera (210b). The electronic device (200) may acquire an image of the ball (2) of the user (1) using the second camera (210b). The second camera (210b) may be positioned on the electronic device (200) so as to have a field of view that captures the ball (2) of the user (1).

[0273] The electronic device (200) can detect a motion of the user (1) inhaling (inhaling) or exhaling (exhaling) while the user (1) brings the real object (3310) close to the user's (1) mouth. In one embodiment of the present disclosure, the electronic device (200) can detect a movement of the user's (1) ball (2) or a shape of the user's (1) ball (2) from an image. For example, the electronic device (200) can detect a movement of the user's (1) ball (2) expanding or a shape of the user's (1) ball (2) expanding. For example, the electronic device (200) can detect a movement of the user's (1) ball (2) contracting or a shape of the user's (1) ball (2) contracting.

[0274] In one embodiment of the present disclosure, the electronic device (200) can receive the breathing sound of the user (1) using a microphone. The microphone can convert the breathing sound of the user (1) into an audio signal, which is an electrical signal. The electronic device (200) can classify the audio signal corresponding to the breathing sound of the user (1) into either an inhalation sound or an exhalation sound. In this case, the second camera (210b) may be omitted.

[0275] In one embodiment of the present disclosure, the electronic device (200) may include a camera (210). The camera (210) may include a first camera (210a). The direction of the first camera (210a) may coincide with the direction in which the eyes of the user (1) look. The first camera (210a) may capture images of the hand (10) of the user (1) and a real object (3310) in front of the user (1). The camera (210) may include a second camera (210b). The second camera (210b) may be positioned on the electronic device (200) to have a field of view that captures the ball (2) of the user (1). The second camera (210b) may capture images of the ball (2) of the user (1).

[0276] FIGS. 34a and 34b are conceptual diagrams illustrating a method for performing a function of controlling a virtual object based on the movement of a user's (1) ball or the shape of the user's (1) ball according to one embodiment of the present disclosure. Any overlapping content described in FIGS. 1 to 33 is omitted. For convenience of explanation, FIGS. 34a and 34b will be described with reference to FIG. 2.

[0277] Referring to FIGS. 34a and 34b , the electronic device (200) can capture images of a real object (3410) and a hand (10) of a user (1) in a real space using a first camera (210a). The electronic device (200) can recognize a real object (3410) interacting with the hand (10) of the user (1) from the acquired images. For example, the electronic device (200) can recognize a hand (10) of the user (1) holding the real object (3410) from the acquired images. The electronic device (200) can generate a virtual object (3420) based on the recognition result. For example, the virtual object (3420) may be in the shape of a bubble, but the present disclosure is not limited thereto, and the shape of the virtual object (3420) may be determined in advance according to settings of a user or a manufacturer.

[0278] In one embodiment of the present disclosure, the electronic device (200) may activate the second camera (210b) based on the recognition result, but the present disclosure is not limited thereto, and the second camera (210b) may already be activated. The electronic device (200) may capture an image of the ball (2) of the user (1) using the second camera (210b). The electronic device (200) may detect the movement of the ball (2) of the user (1) or the shape of the ball (2) of the user (1) from the acquired image. The electronic device (200) may recognize the exhalation or inhalation of the user (1) by detecting the movement of the ball (2) of the user (1) or the shape of the ball (2) of the user (1).

[0279] Referring to FIG. 34A, a virtual object (3420) can be brought into contact with one side of a real object (3410). In one embodiment of the present disclosure, the electronic device (200) can detect an exhalation of the user (1). In one embodiment of the present disclosure, the electronic device (200) can detect a movement in which the ball (2) of the user (1) expands. In one embodiment of the present disclosure, the electronic device (200) can detect the expanded shape of the ball (2) of the user (1). Based on the detection result, the electronic device (200) can enlarge the size of the virtual object (3420).

[0280] In one embodiment of the present disclosure, the electronic device (200) can recognize the inhalation of the user (1). In one embodiment of the present disclosure, the electronic device (200) can detect a movement in which the ball (2) of the user (1) contracts. In one embodiment of the present disclosure, the electronic device (200) can detect the contracted shape of the ball (2) of the user (1). Based on the detection result, the electronic device (200) can reduce the size of the virtual object (3420).

[0281] Referring to FIG. 34B, a virtual object (3420) can be spaced apart from one side of a real object (3410) by a predefined distance. In one embodiment of the present disclosure, the electronic device (200) can detect an exhalation of the user (1). In one embodiment of the present disclosure, the electronic device (200) can detect a movement in which the ball (2) of the user (1) expands. In one embodiment of the present disclosure, the electronic device (200) can detect an expanded shape of the ball (2) of the user (1). Based on the detection result, the electronic device (200) can move the virtual object (3420) so that the distance between the virtual object (3420) and the real object (3410) increases. For example, the virtual object (3420) can move from a first position (L1) to a second position (L2). The first location (L1) may be closer to the real object (3410) than the second location (L2).

[0282] In one embodiment of the present disclosure, the electronic device (200) can detect the inhalation of the user (1). In one embodiment of the present disclosure, the electronic device (200) can detect the movement of the ball (2) of the user (1) contracting. In one embodiment of the present disclosure, the electronic device (200) can detect the contracted shape of the ball (2) of the user (1). Based on the detection result, the electronic device (200) can move the virtual object (3420) so that the distance between the virtual object (3420) and the real object (3410) becomes closer. For example, the virtual object (3420) can move from the third position (L3) to the fourth position (L4). The third position (L3) can be further from the real object (3410) than the fourth position (L4). In one embodiment of the present disclosure, the electronic device (200) can express the movement of the virtual object (3420) in virtual reality as an animation.

[0283] FIGS. 35a and 35b are conceptual diagrams illustrating a method for controlling the size of a virtual screen based on the movement of a ball (2) of a user (1) or the shape of the ball (2) of the user (1) according to one embodiment of the present disclosure. Any overlapping content described in FIGS. 1 to 34b is omitted. For convenience of explanation, FIGS. 35a and 35b will be described with reference to FIG. 2.

[0284] Referring to FIGS. 35a and 35b, the electronic device (200) can display a virtual screen (3530) in a virtual space. The electronic device (200) can detect the movement of the ball (2) of the user (1) or the shape of the ball (2) of the user (1). The electronic device (200) can adjust the size of the virtual screen (3530) based on the movement of the ball (2) of the user (1) or the shape of the ball (2) of the user (1).

[0285] Referring to FIG. 35A, the electronic device (200) can detect the exhalation of the user (1). In one embodiment of the present disclosure, the electronic device (200) can detect the movement of the ball (2) of the user (1) expanding. In one embodiment of the present disclosure, the electronic device (200) can detect the expanded shape of the ball (2) of the user (1). Based on the detection result, the electronic device (200) can enlarge the size of the virtual screen (3530).

[0286] Referring to FIG. 35b, the electronic device (200) can detect the inhalation of the user (1). In one embodiment of the present disclosure, the electronic device (200) can detect the contraction movement of the ball (2) of the user (1). In one embodiment of the present disclosure, the electronic device (200) can detect the contracted shape of the ball (2) of the user (1). Based on the detection result, the electronic device (200) can reduce the size of the virtual screen (3530).

[0287] Unlike as shown in FIGS. 35A and 35B, the electronic device (200) can adjust the position of the virtual screen (3530) based on the detection result. For example, the electronic device (200) can move the position of the virtual screen (3530) to a position farther away from the electronic device (200) based on the movement of the user's (1) ball (2) expanding and the shape of the user's (1) ball (2) expanding. For example, the electronic device (200) can move the position of the virtual screen (3530) to a position closer to the electronic device (200) based on the movement of the user's (1) ball (2) contracting and the shape of the user's (1) ball (2) contracting.

[0288] FIGS. 36A and 36B are conceptual diagrams illustrating a method for performing a pre-identified function or undoing a performed function based on the movement of a user's ball or the shape of the user's ball according to one embodiment of the present disclosure. Any overlapping content described in FIGS. 1 to 35B is omitted. For convenience of explanation, FIGS. 36A and 36B will be described with reference to FIG. 2.

[0289] Referring to FIG. 36A, the electronic device (200) can display an image editing window (3630) through the display (244). As illustrated in FIGS. 4A to 20, the electronic device (200) can capture images of a real object (3610) and a hand (10) of a user (1) in a real space. The electronic device (200) can recognize a real object (3610) interacting with the hand (10) of the user (1) from the acquired images. The electronic device (200) can identify a function for generating a virtual object (3620) using the recognized real object (3610) as an input tool. The electronic device (200) can generate the virtual object (3620) based on the movement of at least a portion of the real object (3610) (e.g., a tip portion of a pencil). In one embodiment of the present disclosure, the electronic device (200) can create a virtual object (3620) on a target image displayed in an image editing window (3630).

[0290] The electronic device (200) can obtain (measure or estimate) the distance between the real object (3610) and the user's mouth. The electronic device (200) can determine whether the distance between the real object (3610) and the user's mouth is less than or equal to a pre-defined threshold value. Based on determining that the distance between the real object (3610) and the user's mouth is less than or equal to the pre-defined threshold value, the electronic device (200) can determine to use the real object (3610) as an input tool for performing a pre-registered function.

[0291] The electronic device (200) can acquire an image of the user's (1) ball (2) using the second camera (210b). In one embodiment of the present disclosure, based on a determination to use a real object (3310) as an input tool for performing a pre-registered function, the electronic device (200) can activate the second camera (210b).

[0292] The electronic device (200) can detect a motion of the user (1) exhaling (breathing) while the user (1) brings the real object (3610) close to the user's (1) mouth. In one embodiment of the present disclosure, the electronic device (200) can detect the movement of the user's (1) ball (2) or the shape of the user's (1) ball (2) from an image. The electronic device (200) can detect the movement of the user's (1) ball (2) expanding or the shape of the user's (1) ball (2). The electronic device (200) can delete the virtual object (3620) based on the movement of the user's (1) ball (2) expanding or the shape of the user's (1) ball (2). For example, the electronic device (200) can delete all virtual objects (3620) created on the target image of the image editing window (3630) based on the movement of the user's (1) ball (2) expanding or the expanded shape of the user's (1) ball (2).

[0293] Referring to FIG. 36b, the electronic device (200) can display an image editing window (3630) through the display (244). The electronic device (200) can detect a motion of the user (1) inhaling (breathing) while the real object (3610) is brought close to the user's (1's) mouth. In one embodiment of the present disclosure, the electronic device (200) can detect a movement of the user's (1's) ball (2) or a shape of the user's (1's) ball (2) from an image. The electronic device (200) can detect a movement of the user's (1's) ball (2) contracting or a shape of the user's (1's) ball (2). The electronic device (200) can cancel a previously performed function based on the movement of the user's (1's) ball (2) contracting or the shape of the user's (1's) ball (2). For example, the electronic device (200) can cancel the function of deleting a virtual object (3620).

[0294] In one embodiment of the present disclosure, the electronic device (200) can generate a virtual object (3620) on a target image based on a cancellation. For example, when the electronic device (200) performs a function of deleting a virtual object (3620) generated in a previous step, the electronic device (200) can store information about the virtual object (3620) in the memory (260) (e.g., a buffer memory). The electronic device (200) can load information about the virtual object (3620) stored in the memory based on the contraction movement of the user's (1) ball (2) or the contraction shape of the user's (1) ball (2). The electronic device (200) can generate the virtual object (3620) on the target image based on the information about the loaded virtual object (3620).

[0295] Figure 37 is a flowchart illustrating a method for identifying a pre-registered function based on the distance between a real-world object and a user's mouth, according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 36b are omitted. For convenience of explanation, Figure 37 will be described with reference to Figures 2 and 3.

[0296] Referring to FIG. 37, operation S330 of FIG. 3 may include operations S3710 and S3720. In one embodiment of the present disclosure, operations S3710 and S3720 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). Detailed operations of operation S340 according to the present disclosure are not limited to those illustrated in FIG. 37, and any one of the operations illustrated in FIG. 37 may be omitted, and operations not illustrated in FIG. 37 may be further included.

[0297] In operation S3710, the electronic device (200) may determine whether the distance between the real object and the user's mouth is less than or equal to a predefined threshold value. In one embodiment of the present disclosure, the electronic device (200) may measure the distance between the real object and the user's mouth based on the acquired image and / or sensor data (e.g., depth value). If the distance between the real object and the user's mouth is greater than the predefined threshold value, the procedure ends. If the distance between the real object and the user's mouth is less than or equal to the predefined threshold value, the procedure moves to operation S3720.

[0298] In operation S3720, the electronic device (200) may identify a pre-registered function that uses the recognized real object as an input tool based on determining that the distance between the real object and the user's mouth is less than or equal to a pre-defined threshold value. For example, the pre-registered function may be a function that identifies and performs a pre-registered sub-function based on the shape of the user's ball or the movement of the user's ball. However, the present disclosure is not limited thereto, and the pre-registered function may include a first sub-function and a second sub-function. For example, the first sub-function may be a function that generates a virtual object. For example, the second sub-function may be a function that controls a virtual object generated by the first sub-function based on the shape of the user's ball or the movement of the user's ball.

[0299] Figure 38 is a flowchart illustrating a method for performing a pre-registered function based on the movement or shape of a user's ball, according to one embodiment of the present disclosure. Any details overlapping with those described in Figures 1 to 37 are omitted. For convenience of explanation, Figure 38 will be described with reference to Figures 2 and 3.

[0300] Referring to FIG. 38, operation S340 of FIG. 3 may include operations S3810 to S3850. In one embodiment of the present disclosure, operations S3810 to S3850 may be performed by the electronic device (200) or the processor (250) of the electronic device (200). The detailed steps of operation S340 according to the present disclosure are not limited to those illustrated in FIG. 38, and any one of the operations illustrated in FIG. 38 may be omitted, or operations not illustrated in FIG. 38 may be further included.

[0301] In operation S3810, the electronic device (200) may activate at least one predetermined sensor based on the identification result. For example, the electronic device (200) may include a first camera and a second camera. For example, the first camera may capture a real object and the user's hand (10) in a real space to obtain a first image. For example, the second camera may capture a user's ball to obtain a second image. The electronic device (200) may activate the second camera based on the identification result. However, the present disclosure is not limited thereto, and the second camera may be omitted and the first camera may capture the user's ball, or the second camera may already be activated. In one embodiment of the present disclosure, operation S3810 may be omitted.

[0302] In operation S3820, the electronic device (200) may acquire a second image of the user's ball. In one embodiment of the present disclosure, the electronic device (200) may capture the user's ball using at least one of the first camera and the second camera.

[0303] In operation S3830, the electronic device (200) can detect the movement of the user's ball from the second image. In one embodiment of the present disclosure, the electronic device (200) can detect the shape of the user's ball from the second image. For example, the electronic device (200) can detect the movement (or shape) of the user's ball expanding or contracting from the second image.

[0304] In operation S3840, the electronic device (200) may identify a pre-registered sub-function based on the movement of the user's ball or the shape of the user's ball. For example, the pre-registered sub-function may include at least one of a function for controlling a virtual object (e.g., controlling the size or position of the virtual object), a function for controlling a virtual screen (or may also be referred to as a virtual window) (e.g., controlling the size or position of the virtual screen), a function for deleting a virtual object, a function for undoing a previously performed function, and a function for redoing a canceled function. In one embodiment of the present disclosure, a sub-function corresponding to the movement of the user's ball or the shape of the user's ball may be pre-registered. In one embodiment of the present disclosure, a first sub-function corresponding to the movement (or shape) of the user's ball expanding and a second sub-function corresponding to the movement (or shape) of the user's ball contracting may be mapped as one function pair.

[0305] In operation S3850, the electronic device (200) can perform a pre-registered sub-function.

[0306] In one embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a camera that captures images of a real object and a user's hand in a real space. The electronic device may include a memory that stores at least one instruction. The electronic device may include at least one processor that executes the at least one instruction stored in the memory. The at least one processor may recognize the real object interacting with the user's hand from the captured image. The at least one processor may identify a pre-registered function that uses the recognized real object as an input tool. The at least one processor may perform the identified pre-registered function.

[0307] In one embodiment of the present disclosure, the at least one processor can detect the real-world object from the acquired image. The at least one processor can segment the detected real-world object into a plurality of regions. The at least one processor can detect at least one region among the plurality of regions that is occluded by the user's hand. The pre-registered function can be identified based on the at least one occluded region.

[0308] In one embodiment of the present disclosure, the at least one processor can detect the real-world object from the acquired image. The at least one processor can segment the detected real-world object into a plurality of parts. The at least one processor can determine the location where the user's hand grasps the real-world object based on the segmentation result. The at least one processor can identify the pre-registered function based on the determined location.

[0309] In one embodiment of the present disclosure, the at least one processor can determine a gripping direction indicating which of the plurality of regions the user's hand is gripping the real-world object toward. The at least one processor can identify the pre-registered function based on the determined gripping direction.

[0310] In one embodiment of the present disclosure, the at least one processor can detect the real-world object from the acquired image. The at least one processor can segment the detected real-world object into a plurality of parts. The at least one processor can recognize a motion of the user's hand touching at least one of the plurality of parts. The at least one processor can identify the pre-registered function based on the recognized motion.

[0311] In one embodiment of the present disclosure, the at least one processor can track the movement of the real object based on the acquired image. The at least one processor can perform a function of creating or deleting a virtual object based on the tracked movement.

[0312] In one embodiment of the present disclosure, the at least one processor can recognize at least one of a color and a size of at least a portion of the real object from the acquired image. The at least one processor can determine at least one of a color and a size of the virtual object based on at least one of the color and the size of at least a portion of the real object. The at least one processor can generate the virtual object having at least one of the determined color and size.

[0313] In one embodiment of the present disclosure, the at least one processor can recognize a target object pointed by the real object from the acquired image. The at least one processor can determine at least one of a color, a shape, and a shape of the virtual object based on at least one of a color, a shape, and a shape of the target object. The at least one processor can generate the virtual object having at least one of the determined color, shape, and shape.

[0314] In one embodiment of the present disclosure, the at least one processor may classify the virtual object using an artificial intelligence model that inputs a virtual image corresponding to the virtual object. The at least one processor may determine a color mapped to the classification result as the color of the virtual object. The at least one processor may generate the virtual object having the determined color.

[0315] In one embodiment of the present disclosure, the pre-registered function can be registered by recognizing the real object from the image, registering the recognized real object with the input tool, and mapping an interaction between the registered real object and the user's hand with a pre-defined function.

[0316] In one embodiment of the present disclosure, the at least one processor may measure the distance between the user's hand and the real-world object. If the distance does not exceed a predefined threshold, the at least one processor may determine to use the recognized real-world object as an input tool. If the distance exceeds a predefined threshold, the at least one processor may determine not to use the recognized real-world object as an input tool.

[0317] In one embodiment of the present disclosure, the at least one processor may perform a function of determining a performance value of the virtual object based on at least one of a color, a material, a shape of the recognized real object, a shape of the user's hand, and a position of the user's hand.

[0318] In one embodiment of the present disclosure, the at least one processor can detect the movement of the recognized real object. The at least one processor can determine whether the detected movement is a predetermined movement. The at least one processor can perform a function of generating a predetermined virtual object based on determining that the detected movement is the predetermined movement.

[0319] In one embodiment of the present disclosure, the at least one processor may activate an authentication mode requiring personal authentication input based on determining that the movement of the recognized real-world object corresponds to the predetermined movement. The at least one processor may obtain the personal authentication input. The at least one processor may generate the predetermined virtual object in response to the personal authentication input. The at least one processor may display the predetermined virtual object on a display.

[0320] In one embodiment of the present disclosure, the at least one processor can activate at least one predetermined sensor.

[0321] In one embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a camera that captures an image of a real object in a real space. The electronic device may include a memory that stores at least one instruction. The electronic device may include at least one processor that executes the at least one instruction stored in the memory. The at least one processor may recognize the real object from the acquired image. The at least one processor may register the recognized real object as an input tool. The at least one processor may map an interaction between the registered real object and a user's hand to a predefined function.

[0322] In one embodiment of the present disclosure, a method for utilizing a real-world object as an input tool of an electronic device may be provided. The method may include a step of capturing an image of a real-world object and a user's hand in a real-world space. The method may include a step of recognizing the real-world object interacting with the user's hand from the acquired image. The method may include a step of identifying a pre-registered function for using the recognized real-world object as an input tool. The method may include a step of performing the identified pre-registered function.

[0323] In one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing a method of using a real object as an input tool of an electronic device on a computer may be provided.

[0324] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0325] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. A camera (210) that captures images by taking pictures of real objects and the user's hand in a real space; A memory (260) storing at least one instruction; and At least one processor (250) for executing the at least one instruction stored in the memory; wherein the at least one processor (250) comprises: Recognize the real object interacting with the user's hand from the acquired image, Identifying a registered function that uses the above recognized real object as an input tool, An electronic device (200) performing the above-identified registered function.

2. In paragraph 1, At least one processor (250) of the above: Detecting the real object from the acquired image, Divide the above detected real object into multiple parts, Detecting at least one area among the above multiple areas occluded by the user's hand, An electronic device (200) that identifies the registered function based on at least one of the occluded portions.

3. In either of paragraphs 1 and 2, At least one processor (250) of the above: Detecting the real object from the acquired image, Divide the above detected real object into multiple parts, Based on the above segmentation results, the location where the user's hand grasps the real object is determined, An electronic device (200) that identifies the registered function based on the determined location.

4. In either of paragraphs 2 and 3, At least one processor (250) of the above: Determine the gripping direction indicating which part of the above multiple parts the user's hand is gripping the real object toward, An electronic device (200) that identifies the registered function based on the determined phage direction.

5. In any one of paragraphs 1 to 4, At least one processor (250) of the above: Detecting the real object from the acquired image, Divide the above detected real object into multiple parts, Recognizes a motion in which the user's hand touches at least one of the plurality of parts, An electronic device (200) that identifies the registered function based on the recognized motion.

6. In any one of paragraphs 1 to 5, At least one processor (250) of the above: Tracking the movement of the real object based on the acquired image, An electronic device (200) that performs a function of creating or deleting a virtual object based on the tracked movement.

7. In paragraph 6, At least one processor (250) of the above: Recognizing at least one of the color and size of at least a part of the real object from the acquired image, determining at least one of a color and a size of the virtual object based on at least one of a color and a size of at least a portion of the real object; An electronic device (200) that generates the virtual object having at least one of the determined colors and sizes.

8. In either of paragraphs 6 and 7, At least one processor (250) of the above: Recognize the target object pointed to by the real object from the acquired image, determining at least one of the color, shape, and shape of the virtual object based on at least one of the color, shape, and shape of the target object; An electronic device (200) that generates the virtual object having at least one of the determined color, shape, and shape.

9. In any one of paragraphs 6 to 8, At least one processor (250) of the above: Classify the virtual object using an artificial intelligence model that inputs a virtual image corresponding to the virtual object, The color mapped to the above classification result is determined as the color of the virtual object, An electronic device (200) that generates the virtual object having the determined color.

10. In any one of paragraphs 1 to 9, An electronic device in which the above-mentioned pre-registered function is registered by recognizing the real object from the image, registering the recognized real object with the input tool, and mapping the interaction between the registered real object and the user's hand with a pre-defined function.

11. In a method of using a real object as an input tool for an electronic device, Step (S310) of acquiring an image by photographing a real object and a user's hand in a real space; A step (S320) of recognizing the real object interacting with the user's hand from the acquired image; Step (S330) of identifying a registered function that uses the above recognized real object as an input tool; and A method comprising a step (S340) of performing the above-mentioned identified registered function.

12. In paragraph 11, The step (S320) of recognizing the real object interacting with the user's hand from the acquired image is: A step of detecting the real object from the acquired image; A step of dividing the above-detected real object into multiple parts; and Comprising a step of detecting at least one part among the plurality of parts occluded by the user's hand, The step (S330) of identifying a registered function that uses the above-mentioned recognized reality object as an input tool is: A method comprising the step of identifying the pre-registered function based on at least one of the occluded portions.

13. In either of paragraphs 11 and 12, The step (S320) of recognizing the real object interacting with the user's hand from the acquired image is: A step of detecting the real object from the acquired image; A step of dividing the above-detected real object into multiple parts; and A step of determining a position where the user's hand grasps the real object based on the segmentation result is included. The step (S330) of identifying a registered function that uses the above-mentioned recognized reality object as an input tool is: A method comprising the step of identifying the pre-registered function based on the determined location.

14. In any one of paragraphs 12 and 13, The step (S320) of recognizing the real object interacting with the user's hand from the acquired image is: A step of determining a gripping direction indicating which part among the plurality of parts the user's hand grips the real object toward, The step (S330) of identifying a registered function that uses the above-mentioned recognized reality object as an input tool is: A method comprising the step of identifying the registered function based on the determined phage direction.

15. A computer-readable recording medium having recorded thereon a program for performing the method of any one of claims 11 to 14 on a computer.

Images