Electronic device and operating method of electronic device

Abstract

An electronic device according to exemplary embodiments disclosed in the present document can include a display, a communication circuit, at least one camera, a memory, and a processor operatively connected to the display, the communication circuit, the at least one camera, and the memory. According to exemplary embodiments, the memory can include instructions that, when executed, cause the processor to provide an augmented reality (AR) environment or a virtual reality (VR) environment through the display, connect to one or more external electronic devices through the communication circuit, display the one or more external electronic devices through the display, designate a first external electronic device among the one or more displayed external electronic devices based on an input interface switching event, and control an operation of the electronic device in the AR environment or the VR environment by using the designated first external electronic device.

Description

Technical Field

[0001] The various embodiments disclosed in this disclosure relate to electronic devices and methods of operating such electronic devices. Background Technology

[0002] Augmented reality (AR) is a technology used to overlay and display virtual information on real video displayed on the screen of an electronic device. Virtual reality (VR) is a technology used to display virtual information and / or pre-captured preview images on the screen of an electronic device.

[0003] With the development of wearable devices such as head-mounted displays (HMDs), electronic devices can identify the user's external environment and provide the user with an augmented reality environment that reflects the identified external environment. Summary of the Invention

[0004] Technical issues

[0005] Electronic devices providing augmented reality and / or virtual reality environments can receive input via gestures, but may lack elements for a more intuitive input experience, such as haptic feedback. Additionally, the electronic device may struggle to discern a user's input intent solely from the gesture in space. Furthermore, when the electronic device uses a camera to recognize the gesture, it may be unable to recognize the gesture within the camera's blind spot.

[0006] The various example embodiments disclosed in this disclosure are intended to provide an electronic device that, when an augmented reality environment and / or a virtual reality environment is activated, intuitively provides at least a portion of an input method / technology using the gesture of the electronic device providing the augmented reality environment and / or the virtual reality environment to an external electronic device.

[0007] The various example embodiments disclosed herein are intended to provide an electronic device that, when an augmented reality environment and / or a virtual reality environment is activated, provides a control method related to the operation of one of a plurality of external electronic devices to another external electronic device.

[0008] Technical solution

[0009] An electronic device according to an example embodiment may include: a display; a communication circuit; at least one camera; a memory; and a processor operatively connected to the display, the communication circuit, the at least one camera, and the memory. According to the example embodiment, the memory stores instructions that, when executed, are configured to cause the at least one processor to: provide an augmented reality environment or a virtual reality environment via the display; connect the electronic device and at least one external electronic device via the communication circuit; display the at least one external electronic device via the display; designate a first external electronic device among the at least one displayed external electronic devices based on an input interface switching event; and control the operation of the electronic device in the augmented reality environment or the virtual reality environment using the designated first external electronic device.

[0010] An electronic device according to an example embodiment may include: a display; a communication circuit; at least one camera; a memory; and at least one processor operatively connected to the display, the communication circuit, the at least one camera, and the memory. According to the example embodiment, the memory stores instructions that, when executed, are configured to cause the at least one processor to: provide an augmented reality environment or a virtual reality environment via the display; connect the electronic device to an external electronic device via the communication circuit; display the external electronic device via the display; activate a first input mode for controlling the electronic device using user gestures while providing the augmented reality environment or the virtual reality environment; and activate a second input mode for controlling the electronic device using the gestures and the external electronic device based on receiving an input interface switching request provided by the user, while the first input mode is activated.

[0011] An operation method of an electronic device according to an embodiment may include: providing operation of an augmented reality environment or a virtual reality environment via a display; connecting the electronic device and at least one external electronic device via a communication circuit; displaying operation of the at least one external electronic device via the display; specifying operation of a first external electronic device among the at least one displayed external electronic devices based on an input interface switching event; and controlling operation of the electronic device in the augmented reality environment or the virtual reality environment using the specified first external electronic device.

[0012] Beneficial effects

[0013] According to specific example embodiments, an electronic device may be provided that uses the gesture of an electronic device providing an augmented reality environment and / or a virtual reality environment to an external electronic device to intuitively provide at least a portion of an input method.

[0014] According to certain example embodiments, an electronic device may be provided that, when an augmented reality environment and / or a virtual reality environment is activated, provides a control method related to the operation of one of a plurality of external electronic devices to another external electronic device.

[0015] In addition, various effects that can be directly or indirectly identified through this disclosure may be provided. Attached Figure Description

[0016] The description of the accompanying drawings may use the same or similar reference numerals for the same or similar elements. Furthermore, the above and other aspects, features, and advantages of certain embodiments will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0017] Figure 1 This is a block diagram illustrating an electronic device 101 in a network environment 100 according to various example embodiments.

[0018] Figure 2 This is a block diagram of an electronic device according to various example embodiments.

[0019] Figure 3 These are block diagrams of electronic devices and external electronic devices according to various example embodiments.

[0020] Figure 4 This is a block diagram of an electronic device according to various example embodiments.

[0021] Figure 5 It is a view used to illustrate the operation of an electronic device according to various example embodiments.

[0022] Figure 6 It is a view used to illustrate the operation of an electronic device according to various example embodiments.

[0023] Figure 7 It is a view used to illustrate the operation of an electronic device according to various example embodiments.

[0024] Figure 8 It is a view used to illustrate the operation of an electronic device according to various example embodiments.

[0025] Figure 9 It is a view used to illustrate the operation of an electronic device according to various example embodiments.

[0026] Figure 10This is a flowchart of an operation method of an electronic device according to various example embodiments.

[0027] Figure 11 This is a flowchart of an operation method of an electronic device according to various example embodiments.

[0028] Figure 12 This is a flowchart of an operation method of an electronic device according to various example embodiments.

[0029] Regarding the description of the accompanying drawings, the same or similar reference numerals may be used for the same or similar parts.

[0030] Any of the embodiments described herein may be used in conjunction with any other embodiments described herein. Detailed Implementation

[0031] Figure 1 This is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. (Refer to...) Figure 1 In network environment 100, electronic device 101 can communicate with electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or with at least one of electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, memory 130, input module 150, sound output module 155, display module 160, audio module 170, sensor module 176, interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, user identification module (SIM) 196, or antenna module 197. In various embodiments, at least one of the above components (e.g., connection terminal 178) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some of the components described above (e.g., sensor module 176, camera module 180, or antenna module 197) may be implemented as a single component (e.g., display module 160).

[0032] Processor 120 may run software (e.g., program 140) to control at least one other component (e.g., hardware or software component) coupled to electronic device 101, and may perform various data processing or calculations. According to an example embodiment, as at least part of the data processing or calculation, processor 120 may store commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the commands or data stored in volatile memory 132, and store the resulting data in non-volatile memory 134. According to an embodiment, processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor central processor, or communication processor (CP)) that is operationally independent of or combined with the main processor 121. For example, when electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 can be adapted to consume less power than the main processor 121, or adapted to be dedicated to a specific function. The auxiliary processor 123 can be implemented separately from the main processor 121, or implemented as part of the main processor 121.

[0033] When the main processor 121 is inactive (e.g., in sleep) state, the auxiliary processor 123 (rather than the main processor 121) can control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190), or when the main processor 121 is active (e.g., running an application), the auxiliary processor 123 can, together with the main processor 121, control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190). According to embodiments, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) can be implemented as part of another component (e.g., camera module 180 or communication module 190) functionally associated with the auxiliary processor 123. According to embodiments, the auxiliary processor 123 (e.g., a neural processing unit) can include hardware architectures dedicated to artificial intelligence model processing. Artificial intelligence models can be generated through machine learning. For example, such learning can be performed by electronic device 101 where artificial intelligence is performed or via a separate server (e.g., server 108). The learning algorithm can include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model can include multiple layers of artificial neural networks. The artificial neural network can be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network, or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model can include software structures in addition to hardware structures.

[0034] Memory 130 may store various data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various data may include, for example, software (e.g., program 140) and input or output data for commands associated with it. Memory 130 may include volatile memory 132 or non-volatile memory 134.

[0035] The program 140 can be stored as software in the memory 130, and the program 140 may include, for example, an operating system (OS) 142, middleware 144, or application 146.

[0036] Input module 150 can receive commands or data from outside electronic device 101 (e.g., a user) that will be used by another component of electronic device 101 (e.g., processor 120). Input module 150 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).

[0037] The audio output module 155 can output audio signals to the outside of the electronic device 101. The audio output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. According to embodiments, the receiver can be implemented separately from the speaker, or as part of the speaker.

[0038] Display module 160 can visually provide information to the outside of electronic device 101 (e.g., to a user). Display module 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a respective one of the display, holographic device, and projector. According to an embodiment, display module 160 may include a touch sensor adapted to detect touch or a pressure sensor adapted to measure the intensity of the force caused by touch.

[0039] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or via headphones of an external electronic device (e.g., electronic device 102) that is directly (e.g., wired) coupled to the electronic device 101 or wirelessly coupled to it.

[0040] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a posture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0041] Interface 177 may support one or more specific protocols used to enable direct (e.g., wired) or wireless coupling between electronic device 101 and external electronic device (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.

[0042] Connection end 178 may include a connector, through which electronic device 101 can be physically connected to an external electronic device (e.g., electronic device 102). According to embodiments, connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] The haptic module 179 can convert electrical signals into mechanical stimuli (e.g., vibration or motion) or electrical stimuli that can be recognized by a user through his touch or kinesthesia. According to embodiments, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0044] Camera module 180 can capture still or moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0045] The power management module 188 can manage the power supply to the electronic device 101. According to an embodiment, the power management module 188 can be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0046] Battery 189 can power at least one component of electronic device 101. According to embodiments, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable rechargeable battery, or a fuel cell.

[0047] Communication module 190 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. Communication module 190 may include one or more communication processors capable of operating independently of processor 120 (e.g., application processor (AP)) and supporting direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 198 (e.g., a short-range communication network, such as Bluetooth). TM The wireless communication module 192 can communicate with external electronic devices via a Wi-Fi Direct or Infrared Data Association (IrDA) network or a second network 199 (e.g., a long-range communication network, such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components (e.g., multiple chips) that are separate from each other. The wireless communication module 192 can identify and verify the electronic device 101 in the communication network (such as a first network 198 or a second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0048] Wireless communication module 192 can support 5G networks following 4G networks and next-generation communication technologies (such as new radio (NR) access technologies). NR access technologies can support enhanced mobile broadband (eMBB), massive machine-type communications (mMTC), or ultra-reliable low-latency communications (URLLC). Wireless communication module 192 can support high-frequency bands (e.g., millimeter-wave bands) to achieve, for example, high data transmission rates. Wireless communication module 192 can support various technologies used to ensure performance in high-frequency bands, such as, for example, beamforming, massive MIMO, full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. Wireless communication module 192 can support various requirements specified in electronic device 101, external electronic devices (e.g., electronic device 104), or network systems (e.g., second network 199). According to an embodiment, the wireless communication module 192 may support peak data rates (e.g., 20 Gbps or greater) for implementing eMBB, lost coverage (e.g., 164 dB or less) for implementing mMTC, or U-plane latency (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip) for implementing URLLC.

[0049] Antenna module 197 can transmit or receive signals or power to or from the outside of electronic device 101 (e.g., external electronic device). According to an embodiment, antenna module 197 may include an antenna comprising a radiating element formed of a conductive material or conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, antenna module 197 may include multiple antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as a first network 198 or a second network 199) can be selected from the multiple antennas by, for example, communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, additional components besides the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally incorporated into antenna module 197.

[0050] According to various embodiments, antenna module 197 can form a millimeter-wave antenna module. According to embodiments, the millimeter-wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and multiple antennas (e.g., an array antenna), wherein the RFIC is disposed on or adjacent to a first surface (e.g., a bottom surface) of the printed circuit board and is capable of supporting a specified high-frequency band (e.g., a millimeter-wave band), and the multiple antennas are disposed on or adjacent to a second surface (e.g., a top or side surface) of the printed circuit board and are capable of transmitting or receiving signals in the specified high-frequency band.

[0051] At least some of the aforementioned components can be coupled to each other and communicate signals (e.g., commands or data) between them via an inter-peripheral communication scheme (e.g., bus, general purpose input / output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)).

[0052] According to an embodiment, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 coupled to a second network 199. Each of electronic device 102 or electronic device 104 can be a device of the same type as electronic device 101, or a device of a different type. According to an embodiment, all or some operations that would be performed on electronic device 101 can be performed on one or more of external electronic devices 102, external electronic devices 104, or server 108. For example, if electronic device 101 is required to automatically perform a function or service, or is required to perform a function or service in response to a request from a user or another device, electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service, instead of running the function or service, or electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service in addition to running the function or service. The one or more external electronic devices receiving the request may perform at least a portion of the requested function or service, or perform additional functions or services related to the request, and transmit the result of the execution to electronic device 101. Electronic device 101 may provide the result as at least a partial response to the request, with or without further processing of the result. For this purpose, technologies such as cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing may be used. Electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and / or neural networks. According to embodiments, external electronic device 104 or server 108 may be included in a second network 199. Electronic device 101 may be applied to intelligent services based on 5G communication technology or IoT-related technologies (e.g., smart homes, smart cities, smart cars, or healthcare).

[0053] Figure 2 This is a block diagram of an electronic device according to various example embodiments.

[0054] Reference Figure 2 According to the embodiment, the electronic device 201 (e.g., Figure 1 The electronic device 101 may include: at least one processor 220 including processing circuitry (e.g., Figure 1 The processor 120 in the middle), and the communication module 290 including the communication circuit (e.g., Figure 1 The communication module 190 and the power management module 288 (e.g., the circuit) are included. Figure 1The power management module 188 in the middle), and the sensor module 276 including at least one sensor or sensing circuit (e.g., Figure 1 The sensor module 176 in the sensor module 280 includes a camera module 280 (e.g., a camera module 176 in the sensor module 176 in the sensor module 280 in the camera ... Figure 1 The camera module 180 and memory 230 (e.g., in the camera module 180) are mentioned. Figure 1 The memory 130 in the memory module 130 and / or the display module 260 including the display (e.g., Figure 1 (Display module 160 in the middle). According to an embodiment, electronic device 201 can be connected to second power management module 237 via connection terminal 235 (e.g., USB TYPE-C).

[0055] According to an example embodiment, electronic device 201 may be a wearable device that can be worn on a user's body part. For example, electronic device 201 may be worn on the user's head. For example, the display module 260 of the electronic device may include a head-mounted display (HMD). Hereinafter, it may be assumed that electronic device 201, which will be described later, is such that it is worn on the user's head and, when worn, positions the head-mounted display in front of the user's eyes. The direction in which the user looks when wearing electronic device 201 may be referred to as a first direction, and the direction opposite to the first direction and toward the user may be referred to as a second direction.

[0056] According to an example embodiment, processor 220 can run programs stored in memory 230 (e.g., Figure 1 The processor 220 (program 140) controls at least one other component (e.g., a hardware component or a software component) and performs various data processing or operations. According to an example embodiment, the processor 220 can provide an augmented reality environment (and / or augmented reality services) to a user. The processor 220 can output at least one virtual object via the display module 260, such that the at least one virtual object is seen by being added to a real space corresponding to the user's field of view of the wearable electronics 201. According to an example embodiment, the processor 220 can provide a virtual reality environment (and / or virtual reality services) to a user. The processor 220 can display virtual images or virtual videos via the display module 260.

[0057] According to an example embodiment, content displayed in a real space via display module 260 that includes at least one virtual object can be referred to as augmented reality content. For example, display module 260 may display content generated by electronic device 201 or transmitted via communication module 290 from another electronic device (e.g., Figure 1 Electronic device 102 and electronic device 104 or Figure 1Augmented reality content received by server 108 in the virtual reality environment. According to an example embodiment, the content displayed by display module 260 in a virtual reality environment can be referred to as virtual reality content. For example, processor 220 can receive augmented reality content from another electronic device (e.g., Figure 1 Electronic device 102 and electronic device 104 or Figure 1 The server 108 in the middle receives virtual reality content.

[0058] According to an example embodiment, the display module 260 of the electronic device 201 may include at least one display panel (e.g., a first display panel and / or a second display panel). The display panel may be formed of or include a transparent device, allowing a user to perceive the real space through the display module 260. The display module 260 may display at least one virtual object on at least a portion of the display panel, allowing a user of the wearing electronic device 201 to see the virtual object added to the real space. For example, the user's field of view may include the angle or range at which the user can perceive the object. According to an example embodiment, the display module 260 of the virtual reality environment may display a preview video of footage taken in the real space.

[0059] According to an example embodiment, the sensor module 276 of the electronic device 201 may include a proximity sensor 221, an illuminance sensor 222, and / or a gyroscope sensor 223. According to an example embodiment, the proximity sensor 221 can sense objects adjacent to the electronic device 201. The illuminance sensor 222 can measure the level of brightness around the electronic device 201. According to an example embodiment, the gyroscope sensor 223 can sense the attitude and / or position of the electronic device 201. For example, the gyroscope sensor 223 can sense whether the electronic device 201 is properly worn on a user's head. As another example, the gyroscope sensor 223 can sense movement of the electronic device 201 or the user wearing the electronic device 201.

[0060] According to an example embodiment, electronic device 201 can communicate with another electronic device (e.g., wireless communication circuit) via communication module 290 (e.g., communication module 290). Figure 1 Wireless communication between electronic devices 102 and 104. For example, electronic device 201 can perform wireless communication with a portable electronic device (e.g., a smartphone), and electronic device 201 can be controlled at least partially by another external electronic device (e.g., a portable electronic device). For example, electronic device 201 can perform at least one function under the control of another external electronic device.

[0061] According to an example embodiment, the camera module 280 of the electronic device 201 may include at least one of a gesture camera 211, an eye-tracking camera 213, a depth camera 215, and / or an RGB camera 217. According to an example embodiment, the gesture camera 211 can sense user movement. For example, at least one gesture camera 211 may be disposed in the electronic device 201 and can sense movement of the user's hand within a set distance. The gesture camera 211 may include a simultaneous localization and mapping (SLAM) camera for identifying information (e.g., position and / or orientation) related to the surrounding space of the electronic device 201. A gesture recognition area of ​​the gesture camera 211 may be set based on the available shooting range of the gesture camera 211. According to an example embodiment, the eye-tracking camera 213 can track the movement of the user's left and / or right eyes. According to an example embodiment, the processor 220 may use the eye-tracking camera 213 to identify the gaze direction of the left eye and / or the gaze direction of the right eye. According to an example embodiment, the depth camera 215 can measure the distance to an object positioned in front of the electronic device 201. Depth camera 215 may include a time-of-flight (TOF) camera and / or a depth camera. According to an example embodiment, electronic device 201 may use eye-tracking camera 213 to identify one of the objects positioned within the user's gaze range (e.g., field of view) and measure the distance to the corresponding object by calculating the depth using either the depth camera or the TOF camera. According to an example embodiment, RGB (red, green, blue) camera 217 may sense information related to the object's color and information about the distance to the object. According to an example embodiment, the pose camera 211, eye-tracking camera 213, depth camera 215, and / or RGB camera 217 included in camera module 280 may be independently included in electronic device 201, or some of those cameras may be implemented as an integrated camera. For example, depth camera 215 and RGB camera 217 may be implemented as a single integrated camera. According to an example embodiment, camera module 280 may be configured to perform shooting in a first direction at a position corresponding to the user's eyes when the user is wearing electronic device 201. According to an example embodiment, camera module 280 may capture video in the direction the user is looking.

[0062] According to an example embodiment, camera module 280 can identify external objects located within the user's gaze range. For example, the external object identified by camera module 280 can be a body part (such as the user's hand) or any object.

[0063] According to an example embodiment, camera module 280 can identify the position of an external object. For example, camera module 280 can identify the relative position of the object with respect to an electronic device. As another example, camera module 280 can identify the position of another external object based on one external object identified by camera module 280.

[0064] According to an example embodiment, camera module 280 can identify the position of a user's finger. For example, camera module 280 can identify the finger position based on the tip of the finger (e.g., the tip of the nail) of the finger making the pointing operation. According to an example embodiment, processor 220 can store the identified position of the user's finger in memory 230. For example, processor 220 can display virtual content (e.g., a dot or finger icon) on the tip of the user's finger (e.g., the tip of the nail) identified on augmented reality content to easily identify the user.

[0065] According to an example embodiment, power management modules 288 and 237 can manage the power supply to electronic device 201. Power management modules 288 and 237 may include a first power management module 288 directly connected to processor 220 to supply power, and / or a second power management module 237 supplied with power via connection terminal 235 (e.g., TYPE-C). The second power management module 237 may be functionally connected to battery 239 to manage a larger amount of power. According to an example embodiment, electronic device 201 can use the power supplied from power management modules 288 and 237 to drive at least one camera included in camera module 280. Electronic device 201 can manage the consumed power by adjusting display information provided through display module 260 based on information acquired using at least one camera. Figure 2 The number and location of at least one camera (e.g., gesture camera 211, eye-tracking camera 213, depth camera 215, and / or RGB camera 217) included in the illustrated electronic device 201 are not limited. For example, the number and location of at least one camera may vary based on the shape (e.g., form or size) of the electronic device 201.

[0066] According to an example embodiment, processor 220 may provide information about the selected object to the user via display module 260 in response to a user's selection input for a virtual object or a real object sensed by camera module 280. The operations performed by processor 220 of electronic device 201 according to an embodiment will be described with reference to the accompanying drawings, which will be described later.

[0067] Figure 3 These are block diagrams of electronic devices and external electronic devices according to various example embodiments.

[0068] Any of the embodiments described herein may be used in conjunction with any other embodiments described herein.

[0069] Reference Figure 3 Electronic device 310 (e.g., Figure 1 Electronic device 101 and / or Figure 2The electronic device 201 in the middle may include: a display 311 (e.g., Figure 1 The display module 160 and / or Figure 2 The display module 260 and camera 312 (e.g., Figure 1 The camera module 180 and / or Figure 2 The camera module 280 and memory 313 (e.g., in the camera module 280) are mentioned. Figure 1 The memory 130 and / or Figure 2 The memory 230 and communication circuit 314 (e.g., in the memory 230) are used in the communication circuit 314. Figure 1 The communication module 190 and / or Figure 2 The communication module 290 and / or the processor 315 including processing circuitry (e.g., Figure 1 The processor 120 and / or Figure 2 (Processor 220 in the processor). In the following, embodiments of the electronic device 310 that provides augmented reality and / or virtual reality environments will be described.

[0070] According to an example embodiment, the electronic device 310 may be the wearable device described above, including a display 311 configured to be worn on (or coupled to) a user's body part (e.g., head).

[0071] According to an example embodiment, display 311 may include a transparent device in at least a portion thereof, allowing a user to perceive the real space outside electronic device 310 through display 311. According to an example embodiment, display 311 may provide an augmented reality (AR) environment to a user. For example, display 311 may include a head-mounted display (HMD) and may provide an AR environment to a user wearing display 311 on a body part (e.g., the head). For example, when a user wears electronic device 310 on their head (or couples electronic device 310 to their head), display 311 may provide the user with an AR environment including at least one virtual object additionally displayed in real space. For example, when a user wears electronic device 310 on their head (or couples electronic device 310 to their head), display 311 may display at least one virtual object on at least a portion of display 311, allowing the user to visually recognize that the at least one virtual object has been added to the real space perceived by display 311. According to various embodiments, display 311 may display preview video captured in real space within a virtual reality environment. Even in this case, embodiments described later may be applied.

[0072] According to an example embodiment, camera 312 may include a first camera that recognizes the movement of a user's hand, a second camera that acquires an image corresponding to the user's gaze direction or the direction in which the user's head is facing, and / or a third camera that recognizes the movement of the user's eyes. For example, the first camera can recognize the user's gesture. For example, the second camera can recognize the user's gaze direction or the direction in which the user's head (or neck) is facing, and acquire an external image corresponding to the recognized direction. For example, the third camera can recognize (or sense) the movement of the user's pupils. As another example, the first or second camera can recognize the user's gesture and the user's gaze direction or the direction in which the user's head (or neck) is facing, and recognize (or sense) the external movement corresponding to the recognized direction.

[0073] According to an example embodiment, memory 313 may store at least one program, application, data, or instructions executed by processor 315. According to an example embodiment, memory 313 may include... Figure 1 At least a portion of the memory 130 shown. According to an example embodiment, memory 313 may include information or instructions for enabling at least some operations of the electronic device 310, which will be described later, to be performed. According to an example embodiment, memory 313 may include instructions related to multiple applications run by processor 315.

[0074] According to an example embodiment, the communication circuit 314 can support communication between the electronic device 310 and an external electronic device 320. For example, the communication circuit 314 enables the electronic device 310 to communicate with the external electronic device 320. For example, the communication circuit 314 can connect the electronic device 310 and the external electronic device 320 to each other in a wireless communication scheme (e.g., Bluetooth, WiFi, and 5G communication scheme). For example, the communication circuit 314 can provide (or send) control signals (e.g., electrical signals and / or electronic signals) to the external electronic device 320 for controlling the operation of the electronic device 310. For example, the communication circuit 314 can send and receive control signals (e.g., electrical signals and / or electronic signals) for controlling the operation of the electronic device 310 together with the external electronic device 320 in a wireless communication scheme (e.g., Bluetooth, WiFi, and 5G communication scheme). In various embodiments, the communication circuit 314 can connect the electronic device 310 to multiple external electronic devices.

[0075] According to an example embodiment, at least one processor 315 can transfer (or delegate or provide) control of the electronic device 310 to an external electronic device 320 in an augmented reality environment. According to an example embodiment, the processor 315 can recognize input interface switching events. For example, the processor 315 can recognize an input interface switching event when a user's gesture (e.g., a gesture and / or gaze) points at the external electronic device 320. For example, the processor 315 can recognize an input interface switching event when the user's gesture points at the external electronic device 320 for a specified time or when a virtual click operation is performed based on the user's gesture pointing at the external electronic device 320. For example, the processor 315 can recognize an input interface switching event when the user's gaze points at the external electronic device 320 for a specified time.

[0076] According to an example embodiment, processor 315 can select (or designate) external electronic device 320 as an input device of electronic device 310, at least based on an identified input interface switching event. Processor 315 can receive user input through the selected external electronic device 320 and receive control signals corresponding to the received user input from the external electronic device 320. Processor 315 can perform operations (or tasks) of the electronic device corresponding to the user input (e.g., screen control operations or data processing operations) based on the control signals.

[0077] According to an example embodiment, the processor 315 may use the external electronic device 320 to perform or control at least some operations of the electronic device 310 in an augmented reality environment and / or a virtual reality environment based on the hardware components included in the external electronic device 320 or the processing of the external electronic device 320.

[0078] According to an example embodiment, external electronic device 320 can receive raw data from a user and provide the received raw data to electronic device 310. Processor 315 can use the received raw data to perform various processing operations. According to an embodiment, processor 315 can use the sensor module 321 of external electronic device 320 to receive user input from the user and receive electrical signals corresponding to the received user input from external electronic device 320. Processor 315 can perform operations (or tasks) corresponding to the user input based on the received electrical signals. For example, when external electronic device 320 includes a touchpad (or touchscreen) and / or physical buttons, the user can provide user input such as scrolling, clicking, and / or dragging to the touchpad and / or physical buttons of external electronic device 320. External electronic device 320 can provide input signals corresponding to the user input to electronic device 310. Processor 315 of electronic device 310 can identify the user input received through external electronic device 320 based on the received input signals and perform the corresponding operation. For example, the processor 315 can scroll, click, and / or drag the screen displayed on the display 311 based on input signals.

[0079] According to an example embodiment, processor 315 may use processor 322 of external electronic device 320 to perform some processing operations on user input received in external electronic device 320. According to an example embodiment, processor 315 may perform a first processing operation (e.g., a control operation for the user interface of electronic device 310) and a second processing operation (e.g., relatively advanced data processing operations including rendering, encoding, decoding and / or transcoding) based on user input received in external electronic device 320.

[0080] According to an example embodiment, when the external electronic device 320 is capable of performing relatively advanced data processing operations, the processor 315 can control the operation of the electronic device 310 through the external electronic device 320. For example, the external electronic device 320 may include a smartphone.

[0081] According to an example embodiment, processor 315 may use an external server (not shown) (e.g., a cloud server and mobile edge computing (MEC)) to perform operations of electronic device 310. According to an example embodiment, when electronic device 310 supports streaming services, cloud services, and / or wearable platforms, processor 315 may enable the use of an external server to perform some operations of electronic device 310.

[0082] According to the example embodiment, the processor 315 can use the external electronic device 320 to perform some operations of the electronic device 310 in the above scheme based on the hardware components included in the connected external electronic device 320 or the processing methods supported by the connected external electronic device 320.

[0083] In the following description, the operations performed by the electronic device 310, as described with reference to the accompanying drawings, may be referred to as being performed by the processor 315. According to the example embodiment, the input signals input via the external electronic device 320 and the aforementioned screen control operations (scrolling, clicking, and / or dragging) or data processing operations (e.g., data input) of the electronic device 310 are merely examples for the sake of convenience, and this disclosure is not limited thereto.

[0084] According to an example embodiment, processor 315 can identify an input interface switching event in which two or more of a plurality of external electronic devices (not shown) displayed in the user's field of view in an augmented reality environment are determined (or selected) as input devices. For example, processor 315 can identify the input interface switching event based on the coincidence of the positions of the plurality of external electronic devices with the user's gaze direction (and / or the position of electronic device 310). According to one embodiment, processor 315 can also determine whether the user's gaze has lingered on at least one of the plurality of external electronic devices for a set time or longer. For example, processor 315 can identify the input interface switching event based on the fact that the user's gaze has lingered on at least one of the plurality of external electronic devices for a set time or longer. According to an example embodiment, processor 315 can enable a first external electronic device among the plurality of external electronic devices to receive input signals for controlling at least some operations of a second external electronic device based on the identified input interface switching event. According to an example embodiment, in an augmented reality environment created and / or provided to a user by the processor 315, when the user provides a specified input to the first external electronic device, the second external electronic device can perform an operation corresponding to the specified input.

[0085] Reference Figure 3 External electronic device 320 (e.g., Figure 1 The electronic device 102 or electronic device 104 may include a sensor module 321, a processor 322, a communication circuit 323, and / or a feedback module 324 including circuitry. According to an example embodiment, the external electronic device 320 may include at least one of a smartphone, smartwatch, smart ring, and / or Bluetooth headset. Multiple such external electronic devices 320 may be provided.

[0086] According to an example embodiment, sensor module 321 may include a touch sensor and / or a 6-axis sensor (e.g., a 6-axis accelerometer / gyroscope sensor). For example, sensor module 321 may use a touch sensor to sense a user's touch. For example, sensor module 321 may use a 6-axis sensor to sense the tilt angle and / or movement speed of external electronics 320.

[0087] According to an example embodiment, processor 322 can receive at least a portion of permissions for controlling electronic device 310 from electronic device 310. For example, processor 322 can use sensor module 321 to receive user input such as scrolling, clicking, and / or dragging, and can send the received user input to electronic device 310. For example, processor 322 can provide electrical and / or electronic signals corresponding to the received user input to electronic device 310 via communication circuit 323. For example, processor 315 of electronic device 310 can scroll, click, and / or drag the screen of display 311 in response to the received electrical and / or electronic signals. For example, processor 322 can receive data to be input to electronic device 310 from the user, and can provide the received data to electronic device 310 via communication circuit 323. For example, processor 315 of electronic device 310 can output the received data to display 311. For example, processor 315 of electronic device 310 can use the received data to perform various data processing operations. According to various embodiments, processor 322 can receive user input and process at least a portion of the data corresponding to the received user input to perform an operation (or task) corresponding to the user input on electronic device 310.

[0088] According to the example embodiment, the communication circuit 323 can support wireless communication schemes (e.g., Bluetooth, WiFi, and / or 5G communication schemes). For example, the communication circuit 323 can send electrical and / or electronic signals to the electronic device 310, causing the electronic device 310 to perform an operation (or task) corresponding to the user input (e.g., screen control or data processing).

[0089] According to an example embodiment, the feedback module 324 can provide feedback to the user. For example, the feedback module 324 may include an acoustic module, a tactile module, and / or a light-emitting module. According to an example embodiment, the feedback module 324 can provide the user with mechanical and / or electrical stimulation to notify the electronic device 310 that an operation corresponding to the user input has been performed. For example, the processor 322 can provide the electronic device 310 with electrical and / or electronic signals for scrolling, clicking, and / or dragging the screen of the display 311. In this case, the feedback module 324 can generate sound, vibration, and / or light so that the user recognizes that input for scrolling, clicking, and / or dragging the electronic device 310 has been input.

[0090] According to an example embodiment, electronic device 310 can be at least partially controlled by external electronic device 320. According to an example embodiment, electronic device 310 can use camera 312 to recognize a user's posture (e.g., gestures and / or gaze) and activate a first input mode that receives input for controlling the operation of electronic device 310 based at least on the recognized posture. According to an example embodiment, electronic device 310 can activate a second input mode that receives input for controlling the operation of electronic device 310 using external electronic device 320 based at least on an input interface switching event. According to an example embodiment, electronic device 310 can receive input signals that allow the operation of electronic device 310 to be controlled using external electronic device 320, thereby reducing user fatigue and allowing the user to perform input more intuitively.

[0091] According to an example embodiment, electronic device 310 can receive input signals that allow the operation of a second external electronic device to be controlled using a first external electronic device. For example, electronic device 310 can identify the first and second external electronic devices located within the user's field of view (FOV). According to an example embodiment, based on the user's gaze duration and / or gaze direction, electronic device 310 can receive input signals from the first external electronic device that allow control of at least some operations of the second external electronic device. According to an example embodiment, electronic device 310 can intuitively control the operation of the second external electronic device based on user interaction, thereby providing an improved user experience.

[0092] Figure 4 This is a block diagram of an electronic device according to various example embodiments.

[0093] According to an embodiment, electronic device 400 (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic devices 201 and / or Figure 3 The electronic device 310 in the middle may include: a display 410 (e.g., Figure 1Display module 160 in Figure 2 The display module 260 and / or Figure 3 The display 311 and camera 420 (e.g., in the display 311) are shown in the display 420. Figure 1 Camera module 180 in Figure 2 The camera module 280 and / or Figure 3 Camera 312), memory 430 (e.g., Figure 1 The memory 130 in Figure 2 The memory 230 and / or Figure 3 The memory 313 in the memory and the communication circuit 440 (e.g., Figure 1 Communication module 190 in Figure 2 The communication module 290 and / or Figure 3 The communication circuit 314) and / or the processor 450 including processing circuitry (e.g., Figure 1 The processor 120 in Figure 2 The processor 220 and / or Figure 3 The processor 315 in the middle). Because Figure 4 The components shown are Figure 3 The components of the electronic device 310 shown are identical or substantially identical, so redundant descriptions will be omitted. In the following, embodiments relating to the electronic device 310 providing augmented reality and / or virtual reality environments will be described.

[0094] According to an example embodiment, camera 420 may include multiple cameras. According to an example embodiment, a first camera among the multiple cameras can recognize gestures by tracking the movement of a user's hand. According to an example embodiment, a second camera among the multiple cameras can recognize the external space corresponding to the user's gaze direction or the direction the user's head is facing. For example, the second camera can acquire an external image corresponding to the user's gaze direction or the direction the user's head is facing. According to an example embodiment, a third camera among the multiple cameras can recognize the movement of the user's pupils. For example, processor 450 can acquire information related to the time the user's gaze lingers on an object based on the pupil movement of the user recognized by the third camera. As another example, the first camera or the second camera can perform substantially the same operation. As another example, one of the first camera and the second camera may be omitted.

[0095] According to an example embodiment, processor 450 may activate a first input mode in an augmented reality environment that uses the user's gesture as an input method for electronic device 400. For example, the user's gesture may include the user's gesture and / or the user's gaze. For example, processor 450 may use camera 420 in the augmented reality environment to recognize the user's gesture and create and / or display a user interface (UI) representing movement of the user's gesture in the augmented reality environment. For example, the user interface may include an indicator line extending from the tip of the user's finger displayed on display 410. In various embodiments, the user interface may include a circular cursor displayed on display 410 indicating the position of the tip of the user's finger. According to an example embodiment, processor 450 may move the user interface based on the movement of the user's gesture in the augmented reality environment.

[0096] According to an example embodiment, the processor 450 can recognize an input interface switching event that activates one of the second and third input modes. For example, the second input mode may correspond to the use of an external electronic device (e.g., Figure 1 Electronic device 102, electronic device 104 and / or Figure 3 An external electronic device 320 is used as an input method for electronic device 400. For example, a third input mode may correspond to a scheme using a mixture of the user's gesture and the external electronic device as an input method for electronic device 400. According to an example embodiment, processor 450 may determine one of the second and third input modes based on the hardware components and / or processing included in the external electronic device.

[0097] According to an example embodiment, in an augmented reality environment, processor 450 can recognize input interface switching events, including a first gesture by the user positioning the user interface at the location where the external electronic device is positioned and / or a second gesture virtually clicking the external electronic device. According to an example embodiment, processor 450 can activate a second input mode for using the external electronic device as an input method for electronic device 400 based on the recognized input interface switching events.

[0098] According to an example embodiment, processor 450 may determine a profile of an external electronic device (or hardware components of the external electronic device) and activate a second input mode based on the determined profile of the external electronic device. According to an example embodiment, when the external electronic device is a smartphone, processor 450 may activate a second input mode that receives input signals for controlling the movement of a user interface via the smartphone. For example, the external electronic device may provide the user with one of vibration, sound, and / or light as feedback via the smartphone after sending an input signal for controlling the operation of electronic device 400. In various embodiments, processor 450 may control the external electronic device to process user input received from the user based on the hardware components of the external electronic device.

[0099] According to example embodiments, processor 450 can determine a profile of an external electronic device (and / or hardware components of the external electronic device) and activate a third input mode based on the determined profile. For example, when the external electronic device is a smartwatch or smart ring, processor 450 can activate a third input mode that controls the movement of the user interface by recognizing the user's gestures while the external electronic device is being worn on a part of the user's body, and provides the user with one of vibration, sound, and / or light as feedback via the smartwatch or smart ring. For example, when the external electronic device is a Bluetooth headset, processor 450 can activate a third input mode that recognizes the user's gestures and receives signals for controlling the user interface, and provides the user with sound as feedback via the Bluetooth headset. According to various embodiments, processor 450 may use an external server (not shown) to control the operation of electronic device 400.

[0100] According to various embodiments, processor 450 can identify a first external electronic device and a second external electronic device positioned in the user's field of view, and receive input signals that allow the operation of the second external electronic device to be controlled using the first external electronic device. For example, processor 450 can identify that the first and second external electronic devices are positioned in an area corresponding to the user's gaze direction (or the direction the user's head is facing) for a set time. For example, processor 450 can identify the user's gaze direction to identify that electronic device 400, the first external electronic device, and the second external electronic device are positioned on a virtual straight line for a set time. According to an example embodiment, when the first and second external electronic devices are positioned in an area corresponding to the user's gaze direction for a set time, processor 450 can receive at least a portion of the input signals for controlling the operation of the second external electronic device via the first external electronic device. For example, when the first external electronic device includes a display, processor 450 can display a control UI for controlling the operation of the second external electronic device on the display of the first external electronic device. For example, when user input is received via a control UI displayed on a first external electronic device's screen, processor 450 can enable a second external electronic device to perform an operation (or task) corresponding to the received user input. Hereinafter, operations described as being performed by electronic device 400, with reference to the accompanying drawings which will be described later, may be referred to as being performed by processor 450.

[0101] Figure 5 It is a view used to illustrate the operation of an electronic device according to various example embodiments.

[0102] Reference Figure 5 User 501 can use electronic device 500 (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 and / or Figure 4 The electronic device 400 is worn on the head. Hereinafter, embodiments relating to situations where the electronic device 500 provides an augmented reality environment and / or a virtual reality environment will be described.

[0103] According to an example embodiment, when worn on the head of user 501, electronic device 500 can use a camera (e.g., Figure 1 Camera module 180 in Figure 2 Camera module 280 in Figure 3 Camera 312 and / or Figure 4 The camera 420 in the device is used to recognize the gesture 510 of the user 501. According to an example embodiment, the electronic device 500 can activate a first input mode that uses the recognized gesture 510 as input.

[0104] According to an example embodiment, the electronic device 500 can create a user interface 515 in an augmented reality environment and / or a virtual reality environment, enabling the user 501 to visually recognize the movement of the recognized user 501's gesture 510. For example, the user interface 515 may include an indicator line extending from the tip of the user 501's finger and / or a circle indicating the endpoint of the indicator line. According to an example embodiment, the electronic device 500 can move the user interface 515 based on the movement of the user 501's gesture 510.

[0105] Reference Figure 5 The electronic device 500 can recognize input interface switching events. For example, the electronic device 500 can recognize when the user 501 positions the user interface 515 on the external electronic device 520 (e.g., Figure 1 Electronic device 102, electronic device 104 and / or Figure 3 The external electronic device 520 is displayed at a point in the augmented reality and / or virtual reality environment by a first gesture 517. As another example, the electronic device 500 can recognize a second gesture 518 of the user 501 virtually touching the external electronic device 520 displayed in the augmented reality and / or virtual reality environment using a user interface 515. According to an example embodiment, the electronic device 500 can specify (or select) the external electronic device 520 and use it as an input method based on the recognition of an input interface switching event including the first gesture 517 and / or the second gesture 518. According to an example embodiment, the electronic device 500 can activate a second input mode using the external electronic device 520 as an input method of the electronic device 500, at least based on the recognition of an input interface switching event.

[0106] Reference Figure 5According to an embodiment, the electronic device 500 can control a user interface 515 such that the user interface 515 moves in response to movement of the external electronic device 520. For example, in a second input mode, the electronic device 500 can perform various operations based on the movement of the user interface 515 along the external electronic device 520. According to an example embodiment, the display 525 of the external electronic device 520 can display a screen for controlling the electronic device 500. For example, the screen may include at least one object. For example, when user input is received via the display 525 (or touchscreen) included in the external electronic device 520, the electronic device 500 can perform an operation (and / or task) corresponding to the received user input in an augmented reality environment and / or virtual reality environment. For example, user input corresponding to a click on a screen displayed on the display of the electronic device 500 can be received on the display 525 of the external electronic device 520. The electronic device 500 can control the screen displayed on its display in the augmented reality environment and / or virtual reality environment based on the user input received via the external electronic device 520. According to an example embodiment, the external electronic device 520 may further include a feedback module that provides feedback to the user 501. For example, the electronic device 500 may use the external electronic device 520 to provide feedback to the user 501 based on the execution of an operation (e.g., a click) corresponding to user input via the external electronic device 520, including at least one of vibration, sound, and / or light. As an example, because the user 501 must raise their hand so that the user 501's hand is included in the field of view (FOV) for the gesture to be recognized by the camera, the user 501's physical fatigue may increase. According to an example embodiment, the user 501 of the electronic device 500 can use the external electronic device 520 to control the operation of the electronic device 500, thereby reducing physical fatigue.

[0107] Figure 6 It is a view used to illustrate the operation of an electronic device according to various example embodiments.

[0108] Reference Figure 6 User 601 can use electronic device 600 (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 in Figure 4 Electronic devices 400 and / or Figure 5 The electronic device 500 is worn on the head. Hereinafter, embodiments relating to situations where the electronic device 600 provides an augmented reality environment and / or a virtual reality environment will be described.

[0109] According to an example embodiment, when worn on the head of user 601, electronic device 600 can use a camera (e.g., Figure 1 Camera module 180 in Figure 2 Camera module 280 in Figure 3 Camera 312 and / or Figure 4 The camera 420 in the device is used to recognize the gesture 610 of the user 601. According to an example embodiment, the electronic device 600 can activate a first input mode using gesture 610 as an input method of the electronic device 600.

[0110] According to an example embodiment, electronic device 600 can create a user interface 615 that moves along with a gesture 610 in an augmented reality environment and / or virtual reality environment, allowing user 601 to visually recognize the movement of the gesture 610. For example, the user interface 615 can be in the form of a cursor. According to an example embodiment, electronic device 600 can move the user interface 615 based on the movement of user 601's gesture 610.

[0111] Reference Figure 6 The electronic device 600 can recognize input interface switching events. For example, the electronic device 600 can recognize when the user 601 positions the user interface 615 on the external electronic device 620 (e.g., Figure 1 Electronic device 102 or electronic device 104 and / or Figure 3 The external electronic device 620 (320) is displayed at a point in the augmented reality environment and / or virtual reality environment via a first gesture 617. As another example, the electronic device 600 can recognize a second gesture 618 where the user 601 virtually touches the external electronic device 620 using the user interface 615. According to an example embodiment, the electronic device 600 can designate the external electronic device 620 based on an input interface switching event including the first gesture 617 and / or the second gesture 618. According to an example embodiment, the electronic device 600 can activate a second input mode using the designated external electronic device 620 as an input method for the electronic device 600.

[0112] Reference Figure 6Electronic device 600 can control the movement of user interface 615 in response to movement of external electronic device 620. For example, in a second input mode, electronic device 600 can perform operations based on operations of external electronic device 620. According to an example embodiment, display 625 of external electronic device 620 can display a screen for controlling electronic device 600. For example, the screen may include at least one object. For example, user 601 can provide user input to touchscreen 625 (or touchpad) included in external electronic device 620. Electronic device 600 can receive user input from external electronic device 620 and can perform operations (or tasks) corresponding to the received user input. For example, user 601 can input user input to external electronic device 620 by scrolling the touchscreen 625 of external electronic device 620 (e.g., scroll input). Electronic device 600 can scroll the display screen displayed on the display of electronic device 600 in augmented reality and / or virtual reality environments based on the scroll input received by user 601 from the touchscreen 625 of external electronic device 620. In various embodiments, when external electronic device 620 receives data input via touchscreen 625, electronic device 600 can use the data input via touchscreen 625 to perform data input operations in augmented reality and / or virtual reality environments. According to example embodiments, external electronic device 620 may also include a feedback module that provides feedback to user 601. For example, when performing an operation corresponding to user input (e.g., scrolling) via external electronic device 620, electronic device 600 can use external electronic device 620 to provide user 601 with feedback including at least one of vibration, sound, and / or light.

[0113] Figure 7 It is a view used to illustrate the operation of an electronic device according to various example embodiments.

[0114] Reference Figure 7 User 701 can use electronic device 700 (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 in Figure 4 Electronic devices 400 in Figure 5 Electronic devices 500 and / or Figure 6 The electronic device 600 is worn on the head. Hereinafter, embodiments relating to situations where the electronic device 700 provides an augmented reality environment and / or a virtual reality environment will be described.

[0115] According to an example embodiment, the electronic device 700 can use a camera (e.g., Figure 1 Camera module 180 in Figure 2Camera module 280 in Figure 3 Camera 312 or Figure 4 The electronic device 700 uses a camera 420 to recognize the user 701's gesture 710. According to one embodiment, the electronic device 700 can activate a first input mode using the user 701's gesture 710 as an input method. The electronic device 700 can create a user interface 715 in an augmented reality environment and / or a virtual reality environment, allowing the user 701 to visually recognize the movement of the user 701's gesture 710. According to an example embodiment, the electronic device 700 can control the movement of the user interface 715 based on the movement of the user 701's gesture 710.

[0116] Reference Figure 7 The electronic device 700 can identify an area (or space) 750 corresponding to the user 701's field of view (FOV) when worn on the head of the user 701. According to an example embodiment, the electronic device 700 can acquire images corresponding to the user 701's gaze direction and / or the direction the user 701's head is facing using a camera to identify the external electronic device 720 included in the area 750 corresponding to the field of view (FOV). According to an example embodiment, the electronic device 700 can use a camera to identify the movement of the user 701's pupil. According to an example embodiment, the electronic device 700 can use a camera to acquire information about the time the user 701's gaze 730, corresponding to the direction the user 701's pupil is facing, remains (or is maintained) on the external electronic device 720. According to an example embodiment, the electronic device 700 can determine whether the time the user 701's gaze 730 remains on the external electronic device 720 is equal to or greater than a set time. For example, when the user 701's gaze 730 lingers on the external electronic device 720 for a time equal to or greater than a set time, the electronic device 700 can recognize that an input interface switching event has occurred.

[0117] According to an example embodiment, electronic device 700 can determine, based on an input interface switching event, that user 701's gaze 730 has lingered on the profile of external electronic device 720 and / or the hardware components included in external electronic device 720 for a set time or longer, and determine the input scheme of electronic device 700 based on the determined profile and / or hardware components. For example, when external electronic device 720 is identified as a device capable of receiving user input from user 701 as a result of determining the profile and / or hardware components of external electronic device 720, electronic device 700 can determine a second input mode as its input scheme. According to an example embodiment, when external electronic device 720 corresponds to a smartphone, electronic device 700 can activate a second input mode that uses external electronic device 720 as its input method. For example, the second input mode may correspond to an input scheme that does not necessarily use user 701's gestures 710 as its input method, but instead uses external electronic device 720 as its input method.

[0118] Reference Figure 7 The electronic device 700 can control the movement of the user interface 715 in response to the movement of the external electronic device 720. For example, when the external electronic device 720 includes a smartphone, the electronic device 700 can perform operations (e.g., scrolling, clicking, and / or dragging) based on user input received through the external electronic device 720. According to an example embodiment, when the external electronic device 720 includes a feedback module (e.g., ...), Figure 3 When the feedback module 324 is in use, the electronic device 700 can provide feedback (e.g., sound, vibration and / or light) to the user 701 through the external electronic device 720.

[0119] Figure 8 It is a view used to illustrate the operation of an electronic device according to various example embodiments.

[0120] Reference Figure 8 User 801 can use electronic device 800 (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 in Figure 4 Electronic devices 400 in Figure 5 Electronic devices 500 Figure 6 Electronic devices 600 and / or Figure 7 The electronic device 700 is worn on the head. Hereinafter, embodiments relating to situations where the electronic device 800 provides an augmented reality environment and / or a virtual reality environment will be described.

[0121] According to an example embodiment, the electronic device 800 can use a camera (e.g., Figure 1 Camera module 180 in Figure 2 Camera module 280 in Figure 3 Camera 312 or Figure 4 The electronic device 800 uses a camera 420 to recognize the gesture 810 of the user 801. According to an example embodiment, the electronic device 800 can activate a first input mode using the gesture 810 of the user 801 as an input method. The electronic device 800 can create a user interface 815 in an augmented reality environment and / or a virtual reality environment, allowing the user 801 to visually recognize the movement of the gesture 810. According to an example embodiment, the electronic device 800 can control the movement of the user interface 815 based on the movement of the gesture 810 of the user 801.

[0122] Reference Figure 8 The electronic device 800 can identify an area (or space) 850 corresponding to the user 801's field of view (FOV). According to an example embodiment, the electronic device 800 can use a camera to acquire an image corresponding to the user 801's gaze direction or the direction the user 801's head is facing, to identify the external electronic device 820 included in the area 850 corresponding to the field of view (FOV). According to an example embodiment, the electronic device 800 can use a camera to identify the movement of the user 801's pupils. For example, the electronic device 800 can use a camera to acquire information about the time the user 801's gaze 830 remains (or is held) on the external electronic device 820. For example, when the user 801's gaze 830 remains on the external electronic device 820 for a time equal to or greater than a set time, the electronic device 800 can identify that an input interface switching event has occurred.

[0123] According to an example embodiment, electronic device 800 can determine, based on an input interface switching event, that user 801's gaze 830 has remained on the profile of external electronic device 820 for a set time or longer, and / or the hardware components included in external electronic device 820, and determine the input scheme of electronic device 800 based on the determined profile and / or hardware components. For example, when external electronic device 820 is identified as a device restricted from receiving user input from user 801, as a result of determining the profile and / or hardware components of external electronic device 820, electronic device 800 can determine a third input mode as its input scheme. Restrictions may include, for example, the presence or absence of a display, the size of the display, the presence or absence of buttons, or the wearing location. For example, when external electronic device 820 corresponds to one of a smartwatch, a smart ring, and a Bluetooth headset, electronic device 800 can activate a third input mode that uses user 801's gesture 810 and external electronic device 820 as input methods for electronic device 800. For example, the third input mode may correspond to an input scheme that uses a combination of user 801's gesture 810 and external electronic device 820 as the input method of electronic device 800.

[0124] Reference Figure 8 The electronic device 800 can recognize the user 801's gestures 810 while the external electronic device 820 is worn on a part of the user 801's body. The electronic device 800 can control the movement of the user interface 815 based on the recognized gestures 810, and can control the feedback (e.g., sound, vibration, and / or light) provided to the user 801 through a feedback module 821 included in the external electronic device 820 for operations (e.g., scrolling, clicking, and / or dragging) of the electronic device 800. For example, when the external electronic device 820 is a smartwatch including a haptic module, the electronic device 800 can receive various inputs based on the user 801's gestures 810, perform operations corresponding to the inputs, and provide feedback (e.g., vibration) to the user 801 corresponding to the inputs or operations.

[0125] Figure 9 It is a view used to illustrate the operation of an electronic device according to various example embodiments.

[0126] Reference Figure 9 User 901 can use electronic device 900 (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 in Figure 4 Electronic devices 400 in Figure 5 Electronic devices 500 Figure 6 Electronic device 600 in Figure 7 Electronic devices 700 and / or Figure 8 The electronic device 800 is worn on the head. Hereinafter, embodiments relating to situations where the electronic device 900 provides an augmented reality environment and / or a virtual reality environment will be described.

[0127] According to an example embodiment, electronic device 900 can be connected to a plurality of external electronic devices 920 and 940 via at least one communication circuit. According to an example embodiment, electronic device 900 can determine its relative position or alignment with the plurality of external electronic devices 920 and 940. Based on the determined relative position or alignment, electronic device 900 can use a first external electronic device 920 to control at least some operations of a second external electronic device 940.

[0128] According to an example embodiment, the electronic device 900 can use a camera (e.g., Figure 1 Camera module 180 in Figure 2 Camera module 280 in Figure 3 Camera 312 and / or Figure 4 The electronic device 900 uses a camera 420 to recognize the user's gestures 910. According to an example embodiment, the electronic device 900 can display a user interface 915 in an augmented reality environment and / or a virtual reality environment, so that the user 901 can visually recognize the movement of the user's gestures 910.

[0129] Reference Figure 9 The electronic device 900 can identify a region (or space) 950 corresponding to the field of view (FOV) of the user 901. According to an example embodiment, the electronic device 900 can acquire an image corresponding to the direction of the user 901's gaze or the direction in which the user 901's head is facing via a camera to identify multiple external electronic devices 920, 940, and 960 included in the region 950 corresponding to the field of view (FOV).

[0130] According to an example embodiment, electronic device 900 can use a camera to detect the movement of the pupil of user 901. For example, electronic device 900 can use a camera to identify a first external electronic device 920 and a second external electronic device 940 positioned (or aligned) in an area corresponding to the gaze 930 of user 901. In various embodiments, electronic device 900 can identify the angle of the gaze 930 of user 901 and / or the position of the first external electronic device 920 and the position of the second external electronic device 940 to determine whether the first external electronic device 920 and the second external electronic device 940 are positioned (or aligned) in the area corresponding to the gaze 930 of user 901.

[0131] According to an example embodiment, when the first external electronic device 920 and the second external electronic device 940 are positioned (or aligned) in the area corresponding to the gaze 930 of the user 901 and the user 901's gaze 930 stops on the first external electronic device 920 for a set time, the electronic device 900 can recognize that an input interface switching event has occurred. In various embodiments, when the first external electronic device 920 and the second external electronic device 940 are positioned (or aligned) in the area corresponding to the gaze 930 of the user 901 and the user interface 915 remains on the first external electronic device 920 for a set time, the electronic device 900 can recognize that an input interface switching event has occurred.

[0132] For example, when the user 901's gaze 930 stops on the first external electronic device 920 for a set time, the electronic device 900 can select the first external electronic device 920 as the input method to control the second external electronic device 940 located in the area corresponding to the user 901's gaze 930.

[0133] Reference Figure 9 Electronic device 900 can use (or select) a first external electronic device 920 as an input method for a second external electronic device 940. For example, the first external electronic device 920 may include a smartphone, while the second external electronic device 940 may include an Internet of Things (IoT) device and an electronic device (e.g., a smart TV). According to an example embodiment, electronic device 900 can display a control interface 945 for controlling the operation of the second external electronic device 940 on the first external electronic device 920 based on the selection of the first external electronic device 920. For example, the control interface 945 may include at least one object related to the operation of the second external electronic device 940. For example, electronic device 900 can display a control interface 945 for changing the channel or volume of the external electronic device 940, including a smart TV, on the first external electronic device 920. A user 901 of electronic device 900 can provide user input to the first external electronic device 920 through the control interface 945. Electronic device 900 can control the operation of the second external electronic device 940 based on the user input received from the first external electronic device 920.

[0134] In various embodiments, the second external electronic device 940 may be a virtual object displayed in an augmented reality environment and / or a virtual reality environment, rather than a real object. Even when the second external electronic device 940 is a virtual object, the first external electronic device 920 may be selected as the input method for controlling the second external electronic device 940 in the same or similar manner as in the embodiments described above.

[0135] Electronic devices according to exemplary embodiments of the present disclosure (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 in Figure 4 Electronic devices 400 in Figure 5 Electronic devices 500 Figure 6 Electronic device 600 in Figure 7 Electronic device 700 in Figure 8 Electronic devices 800 and / or Figure 9 The electronic device 900 in the middle may include: a display (e.g., Figure 1 Display module 160 in Figure 2 The display module 260 in Figure 3 The display 311 and / or Figure 4 The display 410 in the middle); communication circuit (e.g., Figure 1 Communication module 190 in Figure 2 Communication module 290 in Figure 3 Communication circuit 314 and / or Figure 4 The communication circuit 440 in the middle); at least one camera (e.g., Figure 1 Camera module 180 in Figure 2 Camera module 280 in Figure 3 Camera 312 and / or Figure 4 Camera 420 in the middle); memory (e.g., Figure 1 The memory 130 in Figure 2 The memory 230 in Figure 3 The memory 313 and / or Figure 4 The memory 430 in the memory; and at least one processor (e.g., Figure 1 The processor 120 in Figure 2 The processor 220 in Figure 3 The processor 315 and / or Figure 4 The at least one processor (450) is operatively connected to the display, the communication circuitry, the at least one camera, and the memory. According to an example embodiment, the memory may store instructions that, when executed, are configured to cause the at least one processor to: provide an augmented reality (AR) environment or a virtual reality (VR) environment via the display; connect to the electronic device and at least one external electronic device via the communication circuitry; display the at least one external electronic device via the display; designate a first external electronic device among the at least one displayed external electronic devices based on an input interface switching event; and control the operation of the electronic device in the augmented reality environment or the virtual reality environment using the designated first external electronic device.

[0136] According to an example embodiment, the at least one camera may include a first camera for recognizing a user's gestures, and the instructions may cause the processor to create a user interface in the augmented reality environment or the virtual reality environment, the user interface moving based on the movement of the gestures.

[0137] According to an example embodiment, the instructions can cause the processor to specify the first external electronic device based on the input interface switching event, the input interface switching event including: a first gesture that positions the created user interface at a point where the first external electronic device is displayed; and a second gesture that virtually touches the first external electronic device using the user interface.

[0138] According to an example embodiment, the instructions may cause the processor to move the user interface based on the movement of the first external electronic device in the augmented reality environment or the virtual reality environment.

[0139] According to an example embodiment, the instructions may cause the processor to perform a scrolling operation or a data input operation in the augmented reality environment or the virtual reality environment based on input provided by the user through the touchscreen when the first external electronic device includes a touchscreen.

[0140] According to an example embodiment, the instructions may cause the processor to: when the first external electronic device includes at least one of a tactile module, an acoustic module, and a light-emitting module, use at least one of the tactile module, the acoustic module, and the light-emitting module to provide feedback related to the operation of the electronic device, the feedback including at least one of vibration, sound, and light.

[0141] According to an example embodiment, the at least one camera may include a second camera for identifying the user's gaze direction, and the instructions may cause the processor to specify the first external electronic device based on an input interface switching event in which the first external electronic device is positioned in an area corresponding to the user's gaze direction identified by the second camera for a set time.

[0142] According to an example embodiment, the instructions can cause the processor to create a user interface in the augmented reality environment or the virtual reality environment, the user interface moving based on the movement of the first external electronic device.

[0143] According to an example embodiment, the at least one camera may include a second camera for identifying the facing direction of the user's head, and the instruction may cause the processor to: when the first and second external electronic devices included in the at least one external electronic device are positioned in an area corresponding to the user's gaze direction or the facing direction of the user's head identified by the second camera for a set time, assign the authority to control the operation of the second external electronic device to the first external electronic device.

[0144] According to an example embodiment, the instructions may enable the processor to: enable the first external electronic device to display a control interface for controlling the operation of the second external electronic device, and when the first external electronic device includes at least one of a haptic module, an acoustic module, and a light-emitting module, use at least one of the haptic module, the acoustic module, and the light-emitting module to provide feedback related to the operation of the second external electronic device, the feedback including at least one of vibration, sound, and light.

[0145] Electronic devices according to embodiments of the present disclosure (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 in Figure 4 Electronic devices 400 in Figure 5 Electronic devices 500 Figure 6 Electronic device 600 in Figure 7 Electronic device 700 in Figure 8 Electronic device 800 or Figure 9 The electronic device 900 includes: a display (e.g., Figure 1 Display module 160 in Figure 2 The display module 260 in Figure 3 The monitor 311 or Figure 4 The display 410 in the middle); communication circuit (e.g., Figure 1 Communication module 190 in Figure 2 Communication module 290 in Figure 3 Communication circuit 314 or Figure 4 The communication circuit 440 in the middle); at least one camera (e.g., Figure 1 Camera module 180 in Figure 2 Camera module 280 in Figure 3 Camera 312 or Figure 4 Camera 420 in the middle); memory (e.g., Figure 1 The memory 130 in Figure 2 The memory 230 in Figure 3 The memory 313 or Figure 4The memory 430 in the memory; and the processor (e.g., Figure 1 The processor 120 in Figure 2 The processor 220 in Figure 3 The processor 315 or Figure 4 The processor (450) is operatively connected to the display, the communication circuitry, the at least one camera, and the memory. According to an example embodiment, the memory stores instructions that, when executed, cause the processor to: provide an augmented reality (AR) environment or a virtual reality (VR) environment via the display; connect to an external electronic device via the communication circuitry; display the external electronic device via the display; activate a first input mode for controlling the electronic device using user gestures while providing the AR or VR environment; and activate a second input mode for controlling the electronic device using the gestures and the external electronic device based on a received input interface switching request from the user, while the first input mode is activated.

[0146] According to an example embodiment, the instructions may cause the processor to: display a user interface in the augmented reality environment or the virtual reality environment indicating whether the first input mode is activated; and activate the second input mode based on the user's gesture of positioning the user interface on the external electronic device.

[0147] According to an example embodiment, the instructions may cause the processor to: use the at least one camera to identify the user's gaze direction or the direction the user's head is facing; and activate the second input mode by identifying that the external electronic device is positioned in an area corresponding to the gaze direction or the direction the head is facing for a set time.

[0148] According to an example embodiment, the instructions can cause the processor to: display a control interface supporting the second input mode on a display included in the external electronic device in response to activation of the second input mode; and perform a task corresponding to the user input received through the control interface.

[0149] According to an example embodiment, the task may include at least one of the following: scrolling the screen displayed on the display in response to user input in the augmented reality environment or the virtual reality environment, and inputting data.

[0150] Figure 10 This is a flowchart of an operation method of an electronic device according to various embodiments.

[0151] According to an example embodiment, in operation 1010, the electronic device (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 in Figure 4 Electronic devices 400 in Figure 5 Electronic devices 500 Figure 6 Electronic device 600 in Figure 7 Electronic device 700 in Figure 8 Electronic device 800 or Figure 9 The electronic device 900 in the middle can be displayed (e.g., Figure 1 Display module 160 in Figure 2 The display module 260 in Figure 3 The monitor 311 or Figure 4 The display (410) is used to provide an augmented reality (AR) environment or a virtual reality (VR) environment. For example, the display may include a display that is at least partially transparent.

[0152] According to an example embodiment, in operation 1020, the electronic device can communicate via a communication circuit (e.g., Figure 1 Communication module 190 in Figure 2 Communication module 290 in Figure 3 Communication circuit 314 or Figure 4 The communication circuit 440 in the middle is connected to at least one external electronic device (e.g., Figure 3 External electronic device 320 or Figures 5 to 9 External electronic devices 520, 620, 720, 820, 920, and 940 (as described above). For example, the electronic devices can be connected to at least one external electronic device via wireless communication schemes (e.g., Bluetooth, WiFi, and 5G communication schemes).

[0153] According to an example embodiment, in operation 1030, the electronic device can display at least one external electronic device on a display. For example, the electronic device can display at least one external electronic device added to an augmented reality environment or virtual reality environment on the display. As another example, when at least a portion of the electronic device's display is transparent, the user of the electronic device can observe the external electronic device through the transparent portion of the display. In various embodiments, the electronic device can provide a virtual reality environment by displaying a preview video taken in real space on the display.

[0154] According to an example embodiment, in operation 1040, the electronic device can specify a first external electronic device among at least one external electronic device based on an input interface switching event.

[0155] In an example embodiment, the electronic device can recognize an input interface switching event, which includes a first gesture by a user positioning the user interface at a location where the first external electronic device is positioned in an augmented reality or virtual reality environment, and / or a second gesture virtually clicking the first external electronic device. The electronic device can specify (or select) the first external electronic device among at least one external electronic device based on the recognized input interface switching event.

[0156] In an example embodiment, the electronic device can identify an input interface switching event in an augmented reality or virtual reality environment based on a user's gaze lingering (or remaining) on ​​a first external electronic device among at least one external electronic device for a set time. When the user's gaze lingers (or remains) on the first external electronic device for the set time, the electronic device can specify (or select) the first external electronic device among at least one external electronic device.

[0157] According to an example embodiment, in operation 1050, the operation of an electronic device can be controlled in an augmented reality or virtual reality environment using a designated first external electronic device. For example, the electronic device can control its operation based on user input received from the first external electronic device. For instance, when user input is received from a user on a touchscreen (or touchpad) included in the external electronic device, the electronic device can perform an operation (or task) corresponding to the received user input. For example, when a user provides scrolling, clicking, and / or dragging input to the touchscreen included in the external electronic device, the electronic device can perform scrolling, clicking, and / or dragging operations on the display of the electronic device in the augmented reality or virtual reality environment.

[0158] Figure 11 This is a flowchart of an operation method of an electronic device according to various embodiments.

[0159] According to an example embodiment, in operation 1110, the electronic device (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 in Figure 4 Electronic devices 400 in Figure 5 Electronic devices 500 Figure 6 Electronic device 600 in Figure 7 Electronic device 700 in Figure 8 Electronic device 800 or Figure 9 The electronic device 900 can activate a first input mode in augmented reality and / or virtual reality environments, using the user's gestures as an input scheme for the electronic device. For example, the user's gestures may include the user's gestures and / or the user's gaze.

[0160] According to an example embodiment, in operation 1120, the electronic device can recognize that the user interface is located on an external electronic device in an augmented reality environment and / or virtual reality environment via a display (e.g., Figure 1 Display module 160 in Figure 2 The display module 260 in Figure 3 The monitor 311 or Figure 4 The display (410) is shown or viewed at the point. According to an example embodiment, when the user interface is positioned for a specified time, the electronic device can specify (or select) an external electronic device. For example, the user interface can be an object that allows the user to visually recognize the gesture along a movement path in a first input mode. In an example embodiment, after the user interface is positioned at the point where the external electronic device is displayed, the electronic device can further perform the operation of virtually clicking the external electronic device using the user interface.

[0161] According to an example embodiment, in operation 1130, the electronic device can identify the profile of an external electronic device and / or the hardware components included in the external electronic device. According to an example embodiment, the electronic device can determine the profile of a specified (or selected) external electronic device and / or the hardware components of the external electronic device, and perform operation 1140 when the profile of the external electronic device corresponds to a first profile. For example, the electronic device can perform operation 1140 when the external electronic device is capable of performing relatively advanced data processing operations. According to an example embodiment, the electronic device can determine the profile of a specified (or selected) external electronic device, and perform operation 1150 when the profile of the external electronic device corresponds to a second profile. For example, the electronic device can perform operation 1150 when the external electronic device is capable of performing relatively low-level data processing operations.

[0162] According to an example embodiment, in operation 1140, the electronic device may activate a second input mode. According to an example embodiment, the electronic device may activate the second input mode when the external electronic device is capable of performing relatively advanced data processing operations (e.g., rendering, encoding, decoding, and / or transcoding). For example, the electronic device may activate the second input mode when the external electronic device corresponds to a smartphone. According to an example embodiment, the second input mode may correspond to an input mode for controlling the operation of the electronic device using a specified (or selected) external electronic device. For example, the electronic device may perform various operations (e.g., screen control operations or data processing operations) in an augmented reality environment and / or virtual reality environment based on user input received on the specified (or selected) external electronic device.

[0163] According to an example embodiment, in operation 1150, the electronic device may activate a third input mode. According to an example embodiment, the electronic device may activate the third input mode when the external electronic device is capable of performing relatively low-level data processing operations (e.g., sensing user input and transmitting corresponding input signals). For example, the electronic device may activate the third input mode when the external electronic device corresponds to a smartwatch, smart ring, and / or Bluetooth headset. According to an example embodiment, the third input mode may correspond to an input mode used to control the operation of the electronic device using the user's gestures as an input method, and using a designated (or selected) external electronic device to provide feedback (e.g., vibration, sound, and / or light) to the user regarding the operation of the electronic device.

[0164] Figure 12 This is a flowchart of an operation method of an electronic device according to various embodiments.

[0165] According to an example embodiment, in operation 1211, electronic device 1200 (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 in Figure 4 Electronic devices 400 in Figure 5 Electronic devices 500 Figure 6 Electronic device 600 in Figure 7 Electronic device 700 in Figure 8 Electronic device 800 or Figure 9 The electronic device 900 can be connected (or paired) with the first external electronic device 1203 and the second external electronic device 1205. For example, the first external electronic device 1203 may include a smartphone.

[0166] According to an example embodiment, in operation 1212, the first external electronic device 1203 may be connected (or paired) with the second external electronic device 1205. For example, the second external electronic device 1205 may include an IoT electronic device (e.g., a smart TV). In various embodiments, operation 1212 may be omitted. In this case, the first external electronic device 1203 and the second external electronic device 1205 may not be directly connected to each other, but may be indirectly connected to each other through electronic device 1200.

[0167] According to an example embodiment, in operation 1221, electronic device 1200 can identify (or recognize) the location of the first external electronic device 1203 in an augmented reality environment and / or virtual reality environment. For example, electronic device 1200 can use a depth camera (e.g., Figure 2The depth camera 215 in the augmented reality environment and / or virtual reality environment identifies the point where the first external electronic device 1203 is located.

[0168] According to an example embodiment, in operation 1222, electronic device 1200 can identify (or recognize) the location of a second external electronic device 1205 in an augmented reality environment and / or virtual reality environment. For example, electronic device 1200 can use a depth camera (e.g., Figure 2 The depth camera 215 in the augmented reality environment and / or virtual reality environment identifies the point where the second external electronic device 1205 is located.

[0169] According to an example embodiment, in operation 1230, electronic device 1200 can determine whether the positions of electronic device 1200, the first external electronic device 1203, and the second external electronic device 1205 are aligned (or the alignment state of the positions of electronic device 1200, the first external electronic device 1203, and the second external electronic device 1205) in an augmented reality environment and / or a virtual reality environment. According to an example embodiment, when the first external electronic device 1203 and the second external electronic device 1205 are positioned in an area corresponding to the user's gaze direction, electronic device 1200 can determine that the positions of electronic device 1200, the first external electronic device 1203, and the second external electronic device 1205 are aligned, and perform operation 1241. In various embodiments, electronic device 1200 can further determine whether the user's gaze lingers on the first external electronic device 1203 for a time equal to or greater than a set time. For example, when the positions of electronic device 1200, the first external electronic device 1203, and the second external electronic device 1205 are aligned and the user's gaze lingers on the first external electronic device 1203 for a time equal to or greater than a set time, electronic device 1200 can perform operation 1241.

[0170] According to the example embodiment, when the positions of electronic device 1200, the first external electronic device 1203, and the second external electronic device 1205 are not aligned in the augmented reality environment and / or virtual reality environment, electronic device 1200 can perform operations 1221 and 1222 again.

[0171] According to an example embodiment, in operation 1241, electronic device 1200 can select a first external electronic device 1203 as an input method for controlling a second external electronic device 1205 in an augmented reality environment and / or virtual reality environment.

[0172] According to an example embodiment, in operation 1245, the first external electronic device 1203 can display a control interface configured to control the operation of the second external electronic device 1205 on its display. For example, when the second external electronic device 1205 is a smart TV, the first external electronic device 1203 can display a control interface configured to control the on / off state, channel number, and / or volume level of the second external electronic device 1205 on its display.

[0173] According to an example embodiment, in operation 1247, in an augmented reality environment and / or a virtual reality environment, electronic device 1200 can control a second external electronic device 1205. For example, when a user wears electronic device 1200 on their head, the user of electronic device 1200 can input user input (e.g., volume increase) into a control interface displayed on a first external electronic device 1203, associated with the second external electronic device 1205. The first external electronic device 1203 can provide electronic device 1200 with a first control signal corresponding to the user input received through the control interface. Electronic device 1200 can control the operation of the second external electronic device 1205 based on the received first control signal. For example, when a user wearing electronic device 1200 inputs a user input instructing the second external electronic device 1205 to increase the volume into the control interface displayed on the first external electronic device 1203, the first external electronic device 1203 can send a first control signal corresponding to the input user input to electronic device 1200. Electronic device 1200 can provide a second control signal to second external electronic device 1205 based on the received first control signal, for setting the volume level of second external electronic device 1205 to a larger value.

[0174] According to an example embodiment, electronic device 1200 can intuitively control the operation of multiple external electronic devices 1203 and 1205 in an augmented reality environment and / or virtual reality environment. For example, electronic device 1200 can select one of the multiple external electronic devices 1203 and 1205 as an input method for controlling the other based on at least one of whether the multiple external electronic devices 1203 and 1205 are aligned in the augmented reality environment and / or virtual reality environment and / or the time when the user's gaze stops.

[0175] Electronic devices according to embodiments of the present disclosure (e.g., Figure 1 Electronic device 101 in Figure 2 Electronic device 201 in Figure 3 Electronic device 310 in Figure 4 Electronic devices 400 in Figure 5Electronic devices 500 Figure 6 Electronic device 600 in Figure 7 Electronic device 700 in Figure 8 Electronic device 800 or Figure 9 The operation method of the electronic device 900 includes: via a display (e.g., Figure 1 Display module 160 in Figure 2 The display module 260 in Figure 3 The monitor 311 or Figure 4 The display 410 in the middle) provides operation of the augmented reality (AR) environment or virtual reality (VR) environment; through communication circuits (e.g., Figure 1 Communication module 190 in Figure 2 Communication module 290 in Figure 3 Communication circuit 314 or Figure 4 The communication circuit 440 in the middle is connected to at least one external electronic device (e.g., Figure 3 External electronic device 320 or Figures 5 to 9 The operation of external electronic devices 520, 620, 720, 820, 920 and 940 in the augmented reality environment; displaying the operation of the at least one external electronic device through the display; specifying the operation of a first external electronic device among the at least one external electronic device displayed based on an input interface switching event; and controlling the operation of the electronic device in the augmented reality environment or the virtual reality environment using the specified first external electronic device.

[0176] According to an example embodiment, the method may further include: creating a user interface in the augmented reality environment or the virtual reality environment, the user interface moving based on the movement of a user's gestures recognized using a camera included in the electronic device, and specifying the operation of the first external electronic device may include: specifying the operation of the first external electronic device based on the input interface switching event, the input interface switching event including a first gesture positioning the created user interface at a point where the first external electronic device is displayed, and a second gesture virtually touching the first external electronic device using the user interface.

[0177] According to an example embodiment, the method may further include: when the first external electronic device includes at least one of a tactile module, an acoustic module, and a light-emitting module, using at least one of the tactile module, the acoustic module, and the light-emitting module to provide feedback related to the operation of the electronic device, the feedback including at least one of vibration, sound, and light.

[0178] According to an example embodiment, specifying the operation of the first external electronic device may include: specifying the operation of the first external electronic device based on an input interface switching event in which the first external electronic device is positioned in an area corresponding to the gaze direction of a user identified by a camera included in the electronic device for a set time.

[0179] According to an example embodiment, the method may further include: using a camera included in the electronic device to identify the user's gaze direction or the direction the user's head is facing; and when the first external electronic device and the second external electronic device included in the at least one external electronic device are positioned in an area corresponding to the user's gaze direction or the direction the user's head is facing for a set time, assigning control of the operation of the second external electronic device to the operation of the first external electronic device.

[0180] The electronic device according to various embodiments can be one of a variety of types of electronic devices. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. According to embodiments of this disclosure, the electronic device is not limited to those described above.

[0181] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to the specific embodiments, but rather to include various changes, equivalents, or substitutions to the respective embodiments. In the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the singular form of a noun corresponding to an item may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include any one or all possible combinations of the items enumerated together with the corresponding phrase among the plurality of phrases. As used herein, terms such as “first” and “second” or “first” and “second” may be used to simply distinguish the corresponding component from another component and do not limit the component in other respects (e.g., importance or order). It will be understood that, whether or not the terms “operably” or “communically” are used, if an element (e.g., a first element) is referred to as “coupled to another element (e.g., a second element),” “coupled to another element (e.g., a second element),” “connected to another element (e.g., a second element),” or “connected to another element (e.g., a second element)”, it means that the element can be directly (e.g., wiredly) coupled to the other element, wirelessly connected to the other element, or coupled to the other element via a third element.

[0182] As used in connection with various embodiments of this disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "part," or "circuit"). A module may be a single integrated component adapted to perform one or more functions, or the smallest unit or part of such a single integrated component. For example, according to embodiments, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0183] The various embodiments set forth herein can be implemented as software (e.g., program 140) comprising one or more instructions readable by a machine (e.g., electronic device 101) stored in a storage medium (e.g., internal memory 136 or external memory 138). For example, under the control of a processor (e.g., processor 120) of the machine (e.g., electronic device 101), the processor can invoke and execute at least one of the one or more instructions stored in the storage medium, with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. Machine-readable storage media may be provided in the form of non-transitory storage media. The term "non-transitory" means only that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between data being stored semi-permanently in the storage medium and data being temporarily stored in the storage medium.

[0184] According to embodiments, methods according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TM The computer program product may be distributed online (e.g., downloaded or uploaded), or may be distributed directly between two user devices (e.g., smartphones) (e.g., downloaded or uploaded). If it is distributed online, at least a portion of the computer program product may be temporarily generated, or at least a portion of the computer program product may be temporarily stored in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a forwarding server).

[0185] According to various embodiments, each of the above-described components (e.g., a module or program) may include a single entity or multiple entities, and some of the multiple entities may be separately disposed in different components. According to various embodiments, one or more of the above-described components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding component of the multiple components performed one or more functions before integration. According to various embodiments, the operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be run in a different order or omitted, or one or more other operations may be added.

Claims

1. An electronic device, the electronic device comprising: A display configured to display an augmented reality environment or a virtual reality environment; A communication circuit configured to communicate with at least one external electronic device; At least one camera; Memory; as well as At least one processor, operatively connected to the display, the communication circuitry, the at least one camera, and the memory. The memory stores instructions that, when executed by the at least one processor, cause the electronic device to: Control is performed to connect the electronic device and the at least one external electronic device via the communication circuit; Control is performed to display the at least one external electronic device in at least one of the augmented reality environment or virtual reality environment via the display; During the first input mode, the at least one of the augmented reality environment or virtual reality environment is controlled based on the user's posture in the at least one augmented reality environment or virtual reality environment detected by the at least one camera; An input event is detected based on the position of the user's orientation and the position displayed on the first external electronic device. The first external electronic device is specified among at least one external electronic device shown; and Based on the detection of the input event, a second input mode is activated, wherein: in the second input mode, the user interface of the at least one external electronic device in the augmented reality environment or the virtual reality environment is controlled based on the user's posture detected by the at least one camera using a designated first external electronic device.

2. A wearable device, the wearable device comprising: A display that is positioned in front of the user's eyes when the wearable device is worn on the user's head; A communication circuit configured to communicate with one or more external electronic devices in a real-world space environment; Multiple cameras, including a first camera and a second camera, wherein the first camera is configured to detect pose and the second camera is configured to acquire an image corresponding to the direction in which the user's head is facing; At least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the wearable device to: Control the display to show a virtual reality environment and virtual objects, including images obtained via the second camera that correspond to the direction in which the user's head is facing. In the first input mode, The virtual reality environment displayed on the display is controlled by a first gesture, wherein the first gesture is detected based on an image obtained through the first camera; Based on the second camera, a first external electronic device is identified in the real space environment that corresponds to the direction in which the user's head is facing. While the virtual reality environment including the identified first external electronic device is displayed on the display, a second gesture for selecting the displayed first external electronic device is received; And activate the second input mode based on a second gesture for selecting the first external electronic device displayed; as well as In the second input mode, The virtual reality environment displayed on the monitor is controlled based on input to the first external electronic device and a third gesture detected based on an image obtained through the first camera.

3. The wearable device according to claim 2, wherein, When the instruction is executed by the at least one processor, the wearable device: The virtual reality environment displayed on the monitor is controlled by inputting to a physical button on the first external electronic device for providing data input.

4. The wearable device according to claim 2, wherein, When the instruction is executed by the at least one processor, the wearable device: The virtual reality environment displayed on the screen is controlled based on touch input to the touch-sensitive input device of the first external electronic device. The touch input includes at least one of scroll input (scrolling the screen displayed on the display), drag input (dragging the screen displayed on the display), or click input (tapping the screen displayed on the display).

5. The wearable device according to claim 2, wherein, When the instruction is executed by the at least one processor, the wearable device: In response to receiving the second gesture, the communication circuit is used to connect to the first external electronic device in the real space environment.

6. The wearable device according to claim 5, wherein, When the instruction is executed by the at least one processor, the wearable device: Using the communication circuit, a signal corresponding to the input to the first external electronic device is received from the first external electronic device; as well as The virtual reality environment displayed on the monitor is controlled based on the received signals.

7. The wearable device according to claim 2, wherein, The wearable device includes a head-mounted display.

8. The wearable device according to claim 2, wherein, The wearable device also includes a third camera configured to identify the user's gaze direction, and Wherein, when the instruction is executed by the at least one processor, the wearable device: When the wearable device is worn on a user's head, it detects the user's gaze; and A user interface indicating the user's gaze direction is displayed at the point where the first external electronic device is displayed.

9. The wearable device according to claim 8, wherein, When the instruction is executed by the at least one processor, the wearable device: The user interface is controlled to move in response to the movement of the first external electronic device.

10. A method for a wearable device, the wearable device including a display and a plurality of cameras, the display being positioned in front of the user's eyes when the wearable device is worn on the user's head, the plurality of cameras including a first camera and a second camera, the first camera being configured to detect pose, and the second camera being configured to acquire an image corresponding to an orientation facing the user's head, the method comprising: Control the display to show a virtual reality environment and virtual objects, including images corresponding to the direction in which the user's head is facing, obtained via the second camera; In the first input mode, The virtual reality environment displayed on the display is controlled by a first gesture, wherein the first gesture is detected based on an image obtained through the first camera; Based on the second camera, a first external electronic device is identified in the real space environment that corresponds to the direction in which the user's head is facing. While the virtual reality environment including the identified first external electronic device is displayed on the display, a second gesture for selecting the displayed first external electronic device is received; The second input mode is activated based on a second gesture used to select the first external electronic device displayed; as well as In the second input mode, The virtual reality environment displayed on the monitor is controlled based on input to the first external electronic device and a third gesture detected based on an image obtained through the first camera.

11. The method according to claim 10, further comprising: The virtual reality environment displayed on the monitor is controlled by inputting to a physical button on the first external electronic device for providing data input.

12. The method of claim 10, further comprising: The virtual reality environment displayed on the screen is controlled based on touch input to the touch-sensitive input device of the first external electronic device, and The touch input includes at least one of scroll input (scrolling the screen displayed on the display), drag input (dragging the screen displayed on the display), or click input (tapping the screen displayed on the display).

13. The method according to claim 10, further comprising: In response to receiving the second gesture, the wearable device is connected to the first external electronic device in the real space environment.

14. The method of claim 13, further comprising: Receive a signal from the first external electronic device corresponding to the input to the first external electronic device; as well as The virtual reality environment displayed on the monitor is controlled based on the received signals.

15. The method of claim 10, further comprising: When the wearable device is worn on the user's head, it can detect the user's gaze. as well as A user interface indicating the user's gaze direction is displayed at the point where the first external electronic device is displayed.

Images