Wearable electronic device that identifies user input in external electronic device and control method thereof

The wearable electronic device uses camera-based analysis of external keyboard code images to identify key inputs, addressing integration challenges in augmented and virtual reality environments by enabling secure and efficient user interaction without direct communication with the external device.

WO2024248480A1PCT designated stage expired Publication Date: 2024-12-05SAMSUNG ELECTRONICS CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/007329
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-29
Filing Date
2024-05-29
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Wearable electronic devices face challenges in seamlessly integrating with external electronic devices for user input recognition without direct communication, especially in augmented and virtual reality environments, where traditional keyboard inputs are complex and require efficient image processing for key identification.

Method used

A wearable electronic device equipped with a camera and processor that captures and analyzes code images from external keyboards, identifies key information based on color indicators, and performs corresponding operations, enabling keyboard control without direct communication, even when the external device is not communicatively connected.

Benefits of technology

Enables secure and efficient user input recognition in augmented and virtual reality scenarios by analyzing color indicators on external keyboards, improving security and reducing resource consumption through targeted image processing, allowing seamless interaction without the need for explicit communication with the external device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024007329_05122024_PF_FP_ABST
    Figure KR2024007329_05122024_PF_FP_ABST
Patent Text Reader

Abstract

This wearable electronic device may comprise: a camera; a display; and at least one processor electrically connected to the camera and the display, wherein the at least one processor can: acquire a code image corresponding to a keyboard displayed on an external electronic device by means of an image acquired by means of the camera; on the basis of the code image, obtain key information corresponding to each of a plurality of keys included in the keyboard; when a key is selected from the keyboard displayed on the external electronic device, identify key information corresponding to the selected key by means of an image acquired by means of the camera; and perform the operation corresponding to the selected key on the basis of the identified key information.
Need to check novelty before this filing date? Find Prior Art

Description

Wearable electronic device for verifying user input from external electronic device and method for controlling same

[0001] Embodiments of the present disclosure relate to a wearable electronic device for confirming user input from an external electronic device and a method for controlling the same.

[0002] The variety of services and additional features offered through electronic devices, such as smartphones, is steadily increasing. To enhance the utility of these devices and satisfy the diverse needs of users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices to offer a variety of features and differentiate themselves from competitors. Consequently, the various functions offered through electronic devices are also becoming increasingly sophisticated.

[0003] The variety of services and additional features offered through wearable electronic devices, such as augmented reality glasses (AR glasses), virtual reality glasses (VR glasses), and head-mounted displays (HMDs), is steadily increasing. To enhance the utility of these devices and satisfy the needs of diverse users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices that offer diverse features and differentiate themselves from competitors. Consequently, the various functions offered through wearable electronic devices are also becoming increasingly sophisticated.

[0004] AR glasses or VR glasses can provide users with a realistic experience by displaying virtual images while worn on their head. AR glasses or VR glasses can replace smartphones in various fields, such as gaming, entertainment, education, and social networking services (SNS). Users can access smartphone content and / or content similar to reality through the AR glasses or VR glasses worn on their head.

[0005] According to one embodiment, a wearable electronic device may include a camera, a display, and at least one processor electrically connected to the camera and the display.

[0006] According to one embodiment, the at least one processor may obtain a code image corresponding to a keyboard displayed on the external electronic device through an image obtained through the camera.

[0007] According to one embodiment, the at least one processor may obtain key information corresponding to each of a plurality of keys included in the keyboard based on the code image.

[0008] According to one embodiment, the at least one processor can identify key information corresponding to a selected key when a key is selected through a keyboard displayed on the external electronic device through an image acquired through the camera.

[0009] According to one embodiment, the at least one processor may perform an operation corresponding to the selected key based on the identified key information.

[0010] According to one embodiment, a method for controlling a wearable electronic device may include an operation of obtaining a code image corresponding to a keyboard displayed on an external electronic device through an image obtained through a camera of the electronic device.

[0011] According to one embodiment, a method for controlling a wearable electronic device may include an operation of obtaining key information corresponding to each of a plurality of keys included in the keyboard based on the code image.

[0012] According to one embodiment, a method for controlling a wearable electronic device may include an operation of identifying key information corresponding to a selected key when a key is selected through a keyboard displayed on the external electronic device through an image acquired through the camera.

[0013] According to one embodiment, a method for controlling a wearable electronic device may include performing an operation corresponding to the selected key based on the identified key information.

[0014] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may include commands for causing a wearable electronic device to obtain a code image corresponding to a keyboard displayed on an external electronic device through an image obtained through the camera.

[0015] According to one embodiment, the one or more programs may include commands for the wearable electronic device to obtain key information corresponding to each of a plurality of keys included in the keyboard based on the code image.

[0016] According to one embodiment, the one or more programs may include commands for identifying key information corresponding to a selected key when a key is selected through a keyboard displayed on the external electronic device through an image acquired through the camera by the wearable electronic device.

[0017] According to one embodiment, the one or more programs may include commands for the wearable electronic device to perform an action corresponding to the selected key based on the identified key information.

[0018] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0019] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0020] FIG. 3A is a drawing showing the front side of a wearable electronic device according to one embodiment.

[0021] FIG. 3b is a drawing showing the back of a wearable electronic device according to one embodiment.

[0022] FIG. 4 is another perspective view of a wearable electronic device according to one embodiment.

[0023] FIG. 5 is a drawing schematically illustrating an operation of an electronic device according to one embodiment of the present invention to confirm a user input from an external electronic device.

[0024] FIG. 6 is a flowchart illustrating an operation of confirming user input from an external electronic device in an electronic device according to one embodiment.

[0025] FIG. 7 is a drawing for explaining a keyboard screen displayed on an external electronic device according to one embodiment.

[0026] FIG. 8 is a drawing for explaining a user input operation through a keyboard screen displayed on an external electronic device according to one embodiment.

[0027] FIG. 9A is a diagram illustrating a color indicator display operation based on user input through a keyboard screen displayed on an external electronic device according to one embodiment.

[0028] FIG. 9b is a diagram illustrating a color indicator display operation based on user input through a keyboard screen displayed on an external electronic device according to one embodiment.

[0029] FIG. 10 is a drawing for explaining a keyboard screen of an external electronic device displayed on a virtual reality screen of an electronic device according to one embodiment.

[0030] FIG. 11 is a flowchart illustrating an operation of confirming user input of an electronic device and an external electronic device according to one embodiment.

[0031] FIG. 12 is a drawing for explaining a keyboard screen of an external electronic device that is enlarged and displayed on a virtual reality screen of an electronic device according to one embodiment.

[0032] FIG. 13 is a drawing for explaining an operation of displaying a virtual keyboard when an external electronic device disappears from the screen of the electronic device due to rotation of the electronic device according to one embodiment.

[0033] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0034] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0035] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0036] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0037] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0038] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0039] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0040] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0041] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0042] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0043] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0044] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0045] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0046] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0047] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0048] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0049] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0050] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0051] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0052] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0053] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0054] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0055] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0056] Referring to FIG. 2, a wearable electronic device (200) according to one embodiment may include at least one of a light output module (211), a display member (201), and a camera module (250).

[0057] According to one embodiment, the light output module (211) may include a light source capable of outputting an image and a lens for guiding the image to the display member (201). According to one embodiment, the light output module (211) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0058] According to one embodiment, the display member (201) may include an optical waveguide (e.g., a waveguide). According to one embodiment, an output image of the optical output module (211) incident on one end of the optical waveguide may be propagated inside the optical waveguide and provided to a user. According to one embodiment, the optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide the output image of the optical output module (211) to the user's eyes by using at least one diffractive element or reflective element.

[0059] According to one embodiment, the camera module (250) can capture still images and / or moving images. According to one embodiment, the camera module (250) can be positioned within the lens frame and can be positioned around the display member (201).

[0060] According to one embodiment, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. According to one embodiment, the first camera module (251) can periodically or aperiodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0061] According to one embodiment, the second camera module (253) can capture an external image.

[0062] According to one embodiment, the third camera module (255) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. According to one embodiment, the third camera module (255) can be used for 3 degrees of freedom (3DoF), 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. According to one embodiment, the second camera module (253) can also be used for hand detection and tracking, and user gesture recognition. According to one embodiment, at least one of the first camera module (251) to the third camera module (255) can be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module can include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0063] FIG. 3A is a drawing showing the front side of a wearable electronic device according to one embodiment.

[0064] FIG. 3b is a drawing showing the back of a wearable electronic device according to one embodiment.

[0065] Referring to FIGS. 3A and 3B, in one embodiment, camera modules (311, 312, 313, 314, 315, 316) and / or depth sensors (317) for obtaining information related to the surrounding environment of the wearable electronic device (300) may be arranged on the first surface (310) of the housing.

[0066] In one embodiment, the camera modules (311, 312) can acquire images related to the environment surrounding the wearable electronic device.

[0067] In one embodiment, the camera modules (313, 314, 315, 316) can acquire images while the wearable electronic device is worn by a user. The camera modules (313, 314, 315, 316) can be used for hand detection, tracking, and recognition of user gestures (e.g., hand movements). The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. In one embodiment, the camera modules (311, 312) can also be used for hand detection and tracking, and user gestures.

[0068] In one embodiment, a depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (217), camera modules (213, 214, 215, 216) may determine the distance to an object.

[0069] According to one embodiment, a camera module (325, 326) for facial recognition and / or a display (321) (and / or a lens) may be arranged on the second side (320) of the housing.

[0070] In one embodiment, a face recognition camera module (325, 326) adjacent to the display may be used to recognize a user's face, or may recognize and / or track both eyes of the user.

[0071] In one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). In one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among the plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIGS. 3A and 3B , the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2 .

[0072] As described above, according to one embodiment, the wearable electronic device (300) may have a form factor for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured on a body part of the user. The wearable electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0073] FIG. 4 is another perspective view of an electronic device according to one embodiment.

[0074] Referring to FIG. 4, the electronic device (400) may be a head-mounted device (HMD) capable of providing images in front of the user's eyes. The configuration of the electronic device (400) of FIG. 4 may be entirely or partially identical to the configuration of the electronic device (200) of FIG. 2.

[0075] According to one embodiment, the electronic device (400) may include a housing (410, 420, 430) that may form the exterior of the electronic device (400) and provide a space in which components of the electronic device (400) may be placed.

[0076] According to one embodiment, the electronic device (400) may include a first housing (410) capable of surrounding at least a portion of a user's head. According to one embodiment, the first housing (410) may include a first side (400a) facing the exterior of the electronic device (400) (e.g., in the +Z direction).

[0077] According to one embodiment, the first housing (410) can surround at least a portion of the internal space (I). For example, the first housing (410) can include a second side (400b) facing the internal space (I) of the electronic device (400) and a third side (400c) opposite the second side (400b). According to one embodiment, the first housing (410) can be combined with the third housing (430) to form a closed curve shape surrounding the internal space (I).

[0078] According to one embodiment, the first housing (410) can accommodate at least some of the components of the electronic device (400). For example, an optical output module and a circuit board can be placed within the first housing (410).

[0079] According to one embodiment, the electronic device (400) may include one display member (440) corresponding to the left and right eyes. The display member (440) may be placed in the first housing (410). The configuration of the display member (440) of FIG. 4 may be all or part of the same as the configuration of the display member (201) of FIG. 2.

[0080] According to one embodiment, the electronic device (400) may include a second housing (420) that can be mounted on a user's face. According to one embodiment, the second housing (420) may include a fourth surface (400d) that can at least partially face the user's face. According to one embodiment, the fourth surface (400d) may be a surface facing the internal space (I) of the electronic device (400) (e.g., -Z direction). According to one embodiment, the second housing (420) may be coupled with the first housing (410).

[0081] According to one embodiment, the electronic device (400) may include a third housing (430) that can be mounted on the back of the user's head. According to one embodiment, the third housing (430) may be coupled to the first housing (410). According to one embodiment, the third housing (430) may accommodate at least some of the components of the electronic device (400). For example, a battery (e.g., battery (189) of FIG. 1) may be placed within the third housing (430).

[0082] FIG. 5 is a drawing schematically illustrating an operation of an electronic device according to one embodiment of the present invention to confirm a user input from an external electronic device.

[0083] Referring to FIG. 5, the electronic device (101) (e.g., the electronic device (101) of FIG. 1, the processor (120) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3a, the wearable electronic device (300) of FIG. 3b, or the electronic device (400) of FIG. 4) may be a wearable electronic device (e.g., a head mounted display (HMD) device or smart glasses) mounted on the body of a user (10).

[0084] According to one embodiment, the electronic device (101) may obtain input from a user (10) through a keyboard (510) displayed on an external electronic device (104) (e.g., the electronic device (104) of FIG. 1) through a camera (180) (e.g., the camera module (180) of FIG. 1). According to one embodiment, the external electronic device (104) may be in a state of being communicatively unconnected to the electronic device (101).

[0085] In one embodiment, the external electronic device (104) may display the keyboard (510) when a user input (e.g., a ten-finger touch) is received to display the keyboard (510).

[0086] According to one embodiment, the external electronic device (104) may display a color indicator (520, 521) corresponding to a key selected by the user (10) among a plurality of keys included in the keyboard (510).

[0087] According to one embodiment, the electronic device (101) can capture a color indicator (520, 521) displayed on an external electronic device (104) through a camera (180) and identify a key selected by a user (10) based on the color information of the color indicator.

[0088] According to one embodiment, when the electronic device (101) displays a virtual reality (VR) screen, which is an image of a virtual space rather than an image of a real space acquired through a camera, on the display, the electronic device (101) may acquire an image corresponding to an external electronic device (104) included in the image acquired through the camera, and display the image corresponding to the external electronic device (104) on the virtual reality screen. For example, the electronic device (101) may crop an image corresponding to the external electronic device (104) included in the image of a real space acquired through the camera, and display the cropped image on the virtual reality screen. As a result, the electronic device (101) may replace only the area mapped to the boundary of the external electronic device (104) with the image of a real space, thereby converting the area of ​​the external electronic device (104) into a see-through screen.

[0089] According to one embodiment, the electronic device (101) can use the external electronic device (104) as a keyboard without scanning and pairing operations for communication connection with the external electronic device (104). For example, in an XR environment, the hassle of having to enter and set up a communication connection screen, a password input screen, and / or a keyboard use setting screen on the electronic device (101) and control the power of the keyboard in order to use the external electronic device (104) as a keyboard can be reduced. In addition, the electronic device (101) can use the external electronic device (104) as a keyboard without displaying a virtual keyboard for password input on the password input screen and receiving a user input through the virtual keyboard.

[0090] Referring to FIGS. 6 to 13 below, the operation of using an external electronic device as a keyboard will be described in more detail.

[0091] FIG. 6 is a flowchart illustrating an operation of confirming user input from an external electronic device in an electronic device according to one embodiment.

[0092] Referring to FIG. 6, in operation 610, an electronic device (e.g., the electronic device (101) of FIG. 1, the processor (120) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3A, the wearable electronic device (300) of FIG. 3B, or the electronic device (400) of FIG. 4) may acquire a code image corresponding to a keyboard displayed on an external electronic device (e.g., the electronic device (104) of FIG. 1) through an image acquired through a camera (e.g., the camera module (180) of FIG. 1). According to one embodiment, the code image may include at least one of a QR code or a barcode.

[0093] According to one embodiment, the electronic device may further include a communication module (e.g., the communication module (190) of FIG. 1). According to one embodiment, the communication module may be in a non-communicative state with an external electronic device.

[0094] According to one embodiment, an electronic device may display an augmented reality (AR) screen via a display (e.g., the display module (160) of FIG. 1). According to one embodiment, the AR screen may display an image of a real space acquired through a camera on the display. According to one embodiment, the AR screen may further display virtual objects along with the image of the real space.

[0095] In one embodiment, when displaying an augmented reality screen, the electronic device may display an external electronic device included in an image of a real space on the display. For example, the external electronic device may display a keyboard and a code image corresponding to the keyboard based on a user input for displaying the keyboard. In one embodiment, the electronic device may display the keyboard and the code image corresponding to the keyboard on the display, and acquire the code image displayed on the external electronic device.

[0096] In one embodiment, the electronic device may display a virtual reality (VR) screen via a display. In one embodiment, the VR screen may display an image of a virtual space on the display, rather than an image of a real space acquired via a camera.

[0097] According to one embodiment, while displaying a virtual reality screen, the electronic device may acquire an image corresponding to an external electronic device included in an image acquired through a camera, and display the image corresponding to the external electronic device on the virtual reality screen. For example, the electronic device may crop an image corresponding to an external electronic device included in an image of a real space acquired through a camera, and display the cropped image on the virtual reality screen.

[0098] In one embodiment, the electronic device may display a cropped image on a virtual reality screen based on the location of an external electronic device included in an image of a real space.

[0099] According to one embodiment, the electronic device may enlarge a cropped image and display it on a virtual reality screen. The operation of enlarging a cropped image and displaying it on a virtual reality screen according to one embodiment will be described in more detail below with reference to FIG. 12.

[0100] In one embodiment, the external electronic device may display a keyboard and a code image corresponding to the keyboard based on a user input for displaying the keyboard. In one embodiment, the electronic device may display the keyboard and the code image corresponding to the keyboard on the external electronic device on the display, and may obtain the code image displayed on the external electronic device.

[0101] In this way, the electronic device can convert the area of ​​the external electronic device into a see-through screen by replacing only the area mapped to the boundary of the external electronic device with an image of real space.

[0102] According to one embodiment, the operation of replacing the area where the external electronic device is placed with an image of a real space while displaying a virtual reality screen will be described in more detail with reference to FIG. 10 below.

[0103] In one embodiment, the external electronic device may be a foldable device. In one embodiment, the electronic device may detect an unfolding operation of the external electronic device, which is a foldable device, through a camera. In one embodiment, the unfolding operation may include an operation of unfolding the external electronic device from a folded state.

[0104] In one embodiment, the external electronic device may be a rollable device. In one embodiment, the electronic device may detect, through a camera, an unfolding motion (or an unfolding motion) of the display of the external electronic device, which is a rollable device. In one embodiment, the unfolding motion (or the unfolding motion) may include an unfolding motion of the display of the external electronic device from a rolled state (or a folded state).

[0105] According to one embodiment, the electronic device may acquire an image corresponding to the external electronic device included in an image acquired through a camera based on detecting an unfolding action of the external electronic device, and display the image corresponding to the external electronic device on a virtual reality screen.

[0106] In one embodiment, when the external electronic device is in the form of a non-folding bar, the electronic device may acquire an image corresponding to the external electronic device included in an image acquired through a camera, and display the image corresponding to the external electronic device on a virtual reality screen, when the external electronic device displays a keyboard and a code image corresponding to the keyboard.

[0107] According to one embodiment, in operation 620, the electronic device can obtain key information corresponding to each of a plurality of keys included in the keyboard based on a code image.

[0108] According to one embodiment, the electronic device may analyze a code image to obtain information (or key information) about a plurality of keys included in a keyboard, which are included in the code image. For example, the electronic device may obtain information about each of a plurality of keys included in the keyboard, which are included in the code image (e.g., key code values), color information corresponding to each of the plurality of keys, information about positions where the color information corresponding to the plurality of keys is displayed, and / or position information about the plurality of keys on the keyboard. According to one embodiment, the electronic device may map information about each of a plurality of keys included in the keyboard, color information corresponding to each of the plurality of keys, information about positions where the color information corresponding to the plurality of keys is displayed, and / or position information about the plurality of keys on the keyboard, by key, and store the information in a memory (e.g., memory (130) of FIG. 1).

[0109] Although the present disclosure has described a code image as including color information for distinguishing multiple keys, this is not limited to this. For example, instead of, or in addition to, color information for distinguishing multiple keys, the code image may include pattern information, text information, or image information corresponding to the multiple keys.

[0110] According to one embodiment, in operation 630, the electronic device can identify key information corresponding to a selected key when a key is selected through a keyboard displayed on an external electronic device through an image acquired through a camera.

[0111] According to one embodiment, the electronic device can identify the color of a color indicator displayed on an external electronic device through an image acquired through a camera. According to one embodiment, the color indicator may correspond to a key selected through a keyboard displayed on the external electronic device. According to one embodiment, the color indicator may be changed to correspond to the order of the keys selected through the external electronic device with a color corresponding to each key selected. According to one embodiment, the color indicator may be displayed at the location of the selected key, or may be displayed at a location unrelated to the selected key. According to one embodiment, the color indicator may be displayed at the same location, or may be displayed at a different location each time it is displayed.

[0112] According to one embodiment, the arrangement of a plurality of keys included in a keyboard may be changed in response to a key selection operation. For example, the arrangement of a plurality of keys included in the keyboard may be changed in response to a set number of key selection operations being received (e.g., one or more times). According to one embodiment, at least one of the arrangement in which the plurality of keys may be arranged or the positions in which the plurality of keys are arranged may be changed. In this way, even if the arrangement of the keys included in the keyboard is changed, since the colors corresponding to the keys remain the same, the electronic device can identify the color of the color indicator corresponding to the selected key displayed on the external electronic device.

[0113] According to one embodiment, the electronic device can identify selected key information based on a plurality of color information pieces included in the key information and the color of the color indicator. For example, the electronic device can identify color information corresponding to the color of the color indicator among the plurality of color information pieces included in the key information. According to one embodiment, the electronic device can identify a key code value corresponding to the identified color information.

[0114] According to one embodiment, the electronic device can identify the color of the color indicator and the order of the colors. According to one embodiment, the electronic device can identify the key corresponding to the color of the color indicator based on color information stored in the memory, and the order of the colors of the color indicator. For example, when the 'y' key and the 'v' key are sequentially selected among a plurality of keys, the external electronic device can sequentially display an indicator of a color corresponding to the 'y' key (e.g., yellow) and an indicator of a color corresponding to the 'v' key (e.g., blue). According to one embodiment, the electronic device can identify the colors of the indicator as yellow and blue in that order, and can identify that the 'v' key is selected after the 'y' key is selected based on the order of the colors.

[0115] According to one embodiment, the electronic device may further obtain positional information of a plurality of keys included in the keyboard based on the code image, and identify a selected key by further considering the position of the color indicator.

[0116] According to one embodiment, the electronic device can determine selection of two or more keys included in a keyboard based on a code image, and can identify selection of two or more keys based on a color of a color indicator and / or a position of the color indicator.

[0117] According to one embodiment, the operation of identifying a selected key based on a color indicator is described in more detail with reference to FIGS. 9A and 9B below.

[0118] In one embodiment, the electronic device may identify a selected key based on the color of a color indicator displayed at a location independent of the location of the selected key on the external electronic device. For example, the color indicator may be displayed at the same location regardless of the location of the selected key on the external electronic device. For example, the location of the color indicator displayed on the external electronic device may change randomly each time it is displayed on the external electronic device.

[0119] In one embodiment, the electronic device may identify the location of a highlight displayed on an external electronic device through an image acquired through a camera. In one embodiment, the highlight may be displayed at the location of a key selected through a keyboard displayed on the external electronic device. For example, the highlight may be displayed on at least a portion of the border of the selected key, or on at least a portion of the key area.

[0120] According to one embodiment, the electronic device can identify a key code value corresponding to the location of the identified highlight.

[0121] In one embodiment, when a code image includes pattern information, text information, or image information corresponding to a plurality of keys, instead of or together with color information for distinguishing a plurality of keys, the electronic device may identify the pattern information, text information, or image information displayed on an external electronic device through an image acquired through a camera. In one embodiment, the electronic device may identify a key code value corresponding to the identified pattern information, text information, or image information.

[0122] In one embodiment, in operation 640, the electronic device may perform an operation corresponding to a selected key based on the identified key information.

[0123] According to one embodiment, the electronic device may perform an operation corresponding to the identified key code value based on the color information and / or position information of the color indicator. For example, the electronic device may display text corresponding to the identified key code value on the display. According to one embodiment, the electronic device may perform a function corresponding to the identified key code value, such as copy, paste, shift, delete, and backspace.

[0124] In one embodiment, the electronic device may display a virtual keyboard on the display after acquiring information through a code image, and if an external electronic device is not included in the image acquired through the camera based on the rotation of the electronic device, the electronic device may display the virtual keyboard. An embodiment of displaying the virtual keyboard when an external electronic device is not captured due to the rotation of the electronic device will be described in more detail below with reference to FIG. 13.

[0125] FIG. 7 is a drawing for explaining a keyboard screen displayed on an external electronic device according to one embodiment.

[0126] Referring to FIG. 7, an external electronic device (104) (e.g., the electronic device (104) of FIG. 1) may display a keyboard (710) when a user's (10) input for displaying a keyboard is received. For example, the external electronic device (104) may display a keyboard (710) when a user input in which ten fingers of the user (10) are touched is received. For example, the external electronic device (104) may display a keyboard (710) when an input (e.g., a gesture or voice recognition) for executing the keyboard (710) is received.

[0127] According to one embodiment, the external electronic device (104) may further display a code image (711) corresponding to the keyboard (710) together with the keyboard (710). For example, the code image (711) may include at least one of a QR code or a barcode including information related to the keyboard (710). According to one embodiment, the code image (711) may include at least one of information on a plurality of keys included in the keyboard (710), color information corresponding to each of the plurality of keys, information on a location where the color information corresponding to the plurality of keys is displayed, or location information of the plurality of keys.

[0128] According to one embodiment, an electronic device (e.g., an electronic device (101) of FIG. 1) can capture a code image (711) through a camera included in the electronic device (e.g., a camera module (180) of FIG. 1) and analyze the captured code image (711) to obtain information about a keyboard (710) displayed on an external electronic device (104).

[0129] According to one embodiment, the electronic device may store information for code image analysis in a memory (e.g., memory (130) of FIG. 1), and obtain information about a keyboard (710) displayed on an external electronic device (104) based on a code image (711) acquired through a camera and the information for code image analysis stored in the memory.

[0130] According to one embodiment, the electronic device may transmit a code image (711) acquired through a camera to a server (e.g., server (108) of FIG. 1) and obtain information about a keyboard (710) displayed on an external electronic device (104) from the server.

[0131] According to one embodiment, when at least one key among a plurality of keys included in the keyboard (710) is selected by the user (10), the external electronic device (104) may display a color indicator of a color corresponding to at least one selected key. According to one embodiment, when a plurality of keys are sequentially selected, the external electronic device (104) may display a color indicator of a color corresponding to the plurality of keys in a manner corresponding to the order of the plurality of keys. According to one embodiment, when a plurality of keys are selected simultaneously, the external electronic device (104) may simultaneously display a plurality of color indicators, each including a color corresponding to the plurality of keys, in different areas.

[0132] According to one embodiment, the electronic device can obtain color information of a color indicator through a camera and identify a selected key by comparing it with stored color information.

[0133] FIG. 8 is a drawing for explaining a user input operation through a keyboard screen displayed on an external electronic device according to one embodiment.

[0134] Referring to FIG. 8, in operation 810, an external electronic device (e.g., the electronic device (104) of FIG. 1) may display a keyboard and a code image on the display based on the reception of an input for displaying a keyboard. For example, when an input for displaying a keyboard is received, such as touching ten fingers, receiving a voice command for displaying a keyboard, or selecting an icon for displaying a keyboard, the external electronic device may display a keyboard and a code image corresponding to the keyboard on the display. According to one embodiment, the external electronic device may display information about each key included in the keyboard, or may only display the layout of the keys without information about the keys.

[0135] In one embodiment, when the external electronic device is a foldable device, the external electronic device may display a keyboard and a code image corresponding to the keyboard on the display when an action of unfolding the display is detected and / or an input for displaying a keyboard is received.

[0136] In one embodiment, when the external electronic device is a rollable device, the external electronic device may display a keyboard and a code image corresponding to the keyboard on the display when an input for detecting a motion to unfold the display and / or displaying a keyboard is received.

[0137] In one embodiment, when the external electronic device is a foldable device or a rollable device, the external electronic device may detect a motion of unfolding the display and, if an input for displaying a keyboard (e.g., ten-finger touch input, gesture input, or voice input) is received within a specified time, display a keyboard and a code image corresponding to the keyboard on the display.

[0138] According to one embodiment, the code image may include at least one of a QR code or a barcode containing information related to the keyboard. According to one embodiment, the code image may include at least one of information regarding a plurality of keys included in the keyboard, color information corresponding to each of the plurality of keys, information regarding a location where the color information corresponding to the plurality of keys is displayed, or position information of the plurality of keys.

[0139] In one embodiment, in operation 820, the external electronic device may receive an input for selecting at least one of a plurality of keys included in the displayed keyboard. For example, the external electronic device may receive an input for selecting at least one of a plurality of keys included in the keyboard through a user's touch. In one embodiment, the external electronic device may receive an input for selecting at least one of the plurality of keys included in the keyboard through an input device connected to the external electronic device, such as a stylus pen or a mouse, in addition to the user's touch.

[0140] According to one embodiment, in operation 830, the external electronic device may display color information corresponding to each of the selected at least one key.

[0141] According to one embodiment, the external electronic device may display a color indicator of a color corresponding to at least one selected key among a plurality of keys included in a keyboard when at least one key is selected. According to one embodiment, the external electronic device may display a color indicator of a color corresponding to the plurality of keys in a manner corresponding to the order of the plurality of keys when the plurality of keys are sequentially selected. According to one embodiment, the external electronic device may display a color indicator corresponding to the plurality of selected keys when a plurality of keys are selected simultaneously or may simultaneously display a plurality of color indicators, each including a color corresponding to the plurality of selected keys, in different areas.

[0142] According to one embodiment, the external electronic device may display a color indicator at the location of the selected key. According to one embodiment, the external electronic device may also display the color indicator at a location unrelated to the location of the selected key. For example, the color indicator may be displayed in a specific area on the external electronic device unrelated to the location of the selected key. For example, the location at which the color indicator is displayed on the external electronic device may randomly change each time it is displayed on the external electronic device.

[0143] FIG. 9A is a diagram illustrating a color indicator display operation based on user input through a keyboard screen displayed on an external electronic device according to one embodiment.

[0144] Referring to FIG. 9A, when an external electronic device (104) (e.g., the electronic device (104) of FIG. 1) receives a user input for displaying a keyboard, the external electronic device (104) may display a keyboard (910) and a code image corresponding to the keyboard (910). For example, the code image may include at least one of a QR code or a barcode including information related to the keyboard (910). According to an embodiment, the code image may include at least one of information on a plurality of keys included in the keyboard (910), color information corresponding to each of the plurality of keys, information on a location where color information corresponding to the plurality of keys is displayed, or location information of the plurality of keys.

[0145] In one embodiment, the external electronic device (104) may display only the keyboard (910) and the code image and then, after a set period of time, delete the code image.

[0146] According to one embodiment, when the external electronic device (104) receives a user input for selecting at least one of a plurality of images included in the keyboard (910), the external electronic device (104) may display a color indicator (920, 921) corresponding to the selected key. For example, when the external electronic device (104) receives a user input for selecting the 'y' key and the 'v' key among the plurality of keys, the external electronic device (104) may display an indicator (920) of a color (e.g., yellow) corresponding to the 'y' key and an indicator (921) of a color (e.g., blue) corresponding to the 'v' key.

[0147] According to one embodiment, when the 'y' key and the 'v' key are sequentially selected among a plurality of keys, the external electronic device (104) may sequentially display an indicator (920) of a color corresponding to the 'y' key (e.g., yellow) and an indicator (921) of a color corresponding to the 'v' key (e.g., blue).

[0148] In one embodiment, when the 'y' key and the 'v' key are selected simultaneously among a plurality of keys, the external electronic device (104) may simultaneously display an indicator (920) of a color corresponding to the 'y' key (e.g., yellow) and an indicator (921) of a color corresponding to the 'v' key (e.g., blue).

[0149] According to one embodiment, the external electronic device (104) may display a color indicator (920) corresponding to 'y' at the position of the selected 'y' key, and may display a color indicator (921) corresponding to 'v' at the position of the selected 'v' key.

[0150] According to one embodiment, the external electronic device (104) may display the color indicator (920, 921) at a location independent of the locations of the selected 'y' key and 'v' key. For example, the external electronic device (104) may display the color indicator (920, 921) at a specific area of ​​the keyboard (910). For example, the external electronic device (104) may display the color indicator (920, 921) at a different area (e.g., at a random location) of the keyboard (910) each time it displays the color indicator (920, 921).

[0151] In this way, as the external electronic device displays a color indicator corresponding to the selected key, the electronic device can identify the keyboard input by recognizing the color of the color indicator through the camera even if it is not communicatively connected to the external electronic device.

[0152] Additionally, if the color indicator is displayed in a location unrelated to the selected key, it may be difficult to visually determine which key is selected, which may improve security when entering personal information or passwords.

[0153] Additionally, if the color indicator is displayed in a location unrelated to the selected key, the problem of the color indicator being covered by the user's finger can be reduced.

[0154] In one embodiment, although FIG. 9A illustrates displaying a keyboard (910) including key layout information, as illustrated in FIG. 9B, the external electronic device may also display a keyboard that does not include key layout information.

[0155] In one embodiment, the external electronic device may display a color indicator for a set period of time. For example, if the display period of the color indicator is set to be short, the external electronic device may end the display of the color indicator before the next key selection input is received.

[0156] In one embodiment, the external electronic device may maintain the display of the color indicator corresponding to a previously selected key even after the next input for selecting the key is received, or may display the color differently in relation to the order in which the keys were selected, if the display time of the color indicator is set to be long.

[0157] In one embodiment, when multiple color indicators are displayed on an external electronic device, the color indicator corresponding to a later selected key may be a color indicating that it was selected later than a key corresponding to another color indicator displayed together. For example, when only the 'y' key is selected, a color indicator of 'light yellow' may be displayed, and when the 'y' key is selected after another key is selected, and two color indicators are displayed together, a color indicator of 'dark yellow' may be displayed indicating that 'y' is the later selected key. In one embodiment, information regarding colors that are displayed differently in relation to the order in which the keys are selected may be included in the code image.

[0158] In this way, when the time for which the color indicator is displayed is set long, the selected key can be identified through color identification even when the image processing speed of the electronic device is slow.

[0159] FIG. 9b is a diagram illustrating a color indicator display operation based on user input through a keyboard screen displayed on an external electronic device according to one embodiment.

[0160] Referring to FIG. 9B, an external electronic device (e.g., electronic device (104) of FIG. 1) may display a keyboard (930) that does not include keyboard layout information of keys. In one embodiment, the external electronic device may display a keyboard (930) that only includes the layout of keys without keyboard layout information of keys. In one embodiment, even if the external electronic device does not display keyboard layout information of keys, the electronic device may provide keyboard layout information of keys based on layout information of keys obtained through a code image displayed on the external electronic device.

[0161] According to one embodiment, the external electronic device may display a color indicator (940, 941) corresponding to at least one selected key among a plurality of keys included in the keyboard (930).

[0162] According to one embodiment, the external electronic device may display a color indicator (940, 941) corresponding to the selected key at the location of at least one selected key.

[0163] According to one embodiment, the external electronic device may display the color indicator (940, 941) at a location independent of the location of the selected key. For example, the external electronic device may display the color indicator (940, 941) at a specific area of ​​the keyboard (930), or may display the color indicator (940, 941) at a different area of ​​the keyboard (930) each time it displays the color indicator (940, 941) (e.g., at a random location).

[0164] In this way, as the external electronic device displays a color indicator corresponding to the selected key, the electronic device can identify the keyboard input by recognizing the color of the color indicator through the camera even if it is not communicatively connected to the external electronic device.

[0165] Additionally, if the color indicator is displayed in a location unrelated to the selected key, it may be difficult to visually determine which key is selected, which may improve security when entering personal information or passwords.

[0166] Additionally, by not displaying the keyboard information of the keys, security can be improved when entering personal information or passwords.

[0167] FIG. 10 is a drawing for explaining a keyboard screen of an external electronic device displayed on a virtual reality screen of an electronic device according to one embodiment.

[0168] Referring to FIG. 10, an electronic device (e.g., an electronic device (101) of FIG. 1, a processor (120) of FIG. 1, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may display a virtual reality (VR) screen (1010) through a display (e.g., a display module (160) of FIG. 1). According to one embodiment, the virtual reality screen (1010) may be an image of a virtual space displayed on the display, rather than an image of a real space acquired through a camera (e.g., a camera module (180) of FIG. 1).

[0169] According to one embodiment, the electronic device may, while displaying a virtual reality screen, acquire an image (1020) corresponding to an external electronic device included in an image acquired through a camera, and display the image (1020) corresponding to the external electronic device on the virtual reality screen (1010). For example, the electronic device may crop an image (1020) corresponding to an external electronic device included in an image of a real space acquired through a camera, and display the cropped image (1020) on the virtual reality screen (1010).

[0170] According to one embodiment, when the electronic device recognizes that the image acquired through the camera includes a keyboard screen or recognizes that the image includes a code image, the electronic device may crop an image (1020) corresponding to an external electronic device included in the image of the real space acquired through the camera, and display the cropped image (1020) on the virtual reality screen (1010).

[0171] According to one embodiment, the electronic device may display a cropped image (1020) on a virtual reality screen (1010) based on the location of an external electronic device included in an image of a real space.

[0172] According to one embodiment, the electronic device may enlarge a cropped image and display it on a virtual reality screen. The operation of enlarging a cropped image and displaying it on a virtual reality screen according to one embodiment will be described in more detail below with reference to FIG. 12.

[0173] According to one embodiment, the electronic device can display a keyboard and an external electronic device displaying a code image corresponding to the keyboard on the display, and obtain the code image displayed on the external electronic device.

[0174] In this way, the electronic device can convert the area of ​​the external electronic device into a see-through screen by replacing only the area mapped to the boundary of the external electronic device with an image of real space.

[0175] In one embodiment, the external electronic device may be a foldable device. In one embodiment, the electronic device may detect an unfolding operation of the external electronic device, which is a foldable device, through a camera. In one embodiment, the unfolding operation may include an operation of unfolding the external electronic device from a folded state.

[0176] In one embodiment, the external electronic device may be a rollable device. In one embodiment, the electronic device may detect an unfolding motion (or unfolding motion) of the external electronic device, which is a rollable device, through a camera. In one embodiment, the unfolding motion may include an action of unfolding the display of the external electronic device from a rolled state.

[0177] According to one embodiment, the electronic device may acquire an image corresponding to the external electronic device included in an image acquired through a camera based on detecting an unfolding or unfolding motion of the external electronic device, and display the image corresponding to the external electronic device on a virtual reality screen. For example, when the electronic device detects an unfolding motion of the external electronic device, the electronic device may display an image of the external electronic device showing only a black screen on the virtual reality screen. According to one embodiment, when an input for keyboard input is received on the unfolded external electronic device, and a keyboard and a code image corresponding to the keyboard are displayed on the external electronic device, the electronic device may display an image of the external electronic device showing the keyboard and the code image on the virtual reality screen.

[0178] According to one embodiment, the electronic device can capture a code image displayed on an external electronic device, analyze the captured code image, and obtain information about a keyboard displayed on the external electronic device (e.g., key information, color information corresponding to a key, and / or position information of a key).

[0179] FIG. 11 is a flowchart illustrating an operation for confirming user input of an electronic device and an external electronic device according to an embodiment. For example, the electronic device (101) may be in a state of being communicatively unconnected with an external electronic device (104).

[0180] Referring to FIG. 11, an electronic device (101) (e.g., the electronic device (101) of FIG. 1, the processor (120) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3A, the wearable electronic device (300) of FIG. 3B, or the electronic device (400) of FIG. 4) may execute an XR mode in operation 1101. For example, the electronic device (101) may execute an XR mode when it detects that a user is wearing the electronic device (101).

[0181] In one embodiment, in operation 1102, an external electronic device (104) (e.g., the external electronic device (104) of FIG. 1 ) may receive an input for displaying a keyboard. For example, the external electronic device (104) may receive an input for touching a display of the external electronic device (104) with ten fingers, receiving a voice command, or selecting an icon for displaying the keyboard.

[0182] According to one embodiment, in operation 1103, the external electronic device (104) may display a keyboard and a code image. For example, the code image may include information on a plurality of keys included in the keyboard, color information corresponding to each of the plurality of keys, location information indicating color information corresponding to each of the plurality of keys, and / or location information corresponding to each of the plurality of keys.

[0183] According to one embodiment, in operation 1104, the electronic device (101) may display an area of ​​an external electronic device. For example, when the electronic device (101) displays an augmented reality screen that displays an image of a real space acquired through a camera, the electronic device (101) may display an external electronic device included in the image of the real space. According to one embodiment, when the electronic device (101) displays a virtual reality screen that displays an image of a virtual space rather than a real space, the electronic device (101) may crop an image of the external electronic device from the image of the real space acquired through the camera, and display the cropped image of the external electronic device on the virtual reality screen.

[0184] According to one embodiment, in operation 1105, the electronic device (101) may obtain keyboard information by capturing a code image displayed on an external electronic device (104). For example, the electronic device (101) may capture a code image displayed on the external electronic device (104) through a camera, analyze the captured code image, and obtain information about the keyboard included in the code image (e.g., information about a plurality of keys included in the keyboard, color information corresponding to each of the plurality of keys, and / or position information corresponding to each of the plurality of keys).

[0185] In one embodiment, in operation 1106, the external electronic device (104) may receive input on a keyboard. For example, the external electronic device (104) may receive a touch input from a user selecting at least one of a plurality of keys included in the keyboard, a touch input from a stylus pen, and / or an input from an input device connected to the external electronic device (104), such as a mouse.

[0186] In one embodiment, in operation 1107, the external electronic device (104) may display a color corresponding to the input. For example, the external electronic device (104) may display a color indicator of a color corresponding to each of the selected at least one key.

[0187] According to one embodiment, in operation 1108, the electronic device (101) can identify the selected key through color imaging. For example, the electronic device (101) can capture an image of a color indicator displayed on an external electronic device (104) through a camera and identify the color of the color indicator. According to one embodiment, the electronic device (101) can identify key information corresponding to the color of the color indicator using color information acquired through the code image. According to one embodiment, the electronic device can identify key information by taking into account not only the color of the color indicator, but also the location of the color indicator.

[0188] According to one embodiment, in operation 1109, the electronic device (101) may perform an operation corresponding to the selected key. For example, the electronic device (101) may identify a key selected from an external electronic device (104) based on the verified key information, and may perform a text input corresponding to the selected key or a function such as copy, paste, shift, delete, or backspace corresponding to the selected key.

[0189] In this way, even if the electronic device and the external electronic device are not communicatively connected, the electronic device can identify the keyboard input entered from the external electronic device through the color information.

[0190] FIG. 12 is a drawing for explaining a keyboard screen of an external electronic device that is enlarged and displayed on a virtual reality screen of an electronic device according to one embodiment.

[0191] Referring to FIG. 12, an electronic device (e.g., an electronic device (101) of FIG. 1, a processor (120) of FIG. 1, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may acquire an image (1211) of an external electronic device (e.g., an electronic device (104) of FIG. 1) from an image (1210) of a real space acquired through a camera (e.g., a camera module (180) of FIG. 1). According to one embodiment, the image (1211) of the external electronic device may include a keyboard displayed on the external electronic device, and may further include a hand of a user (10) performing input on the keyboard.

[0192] According to one embodiment, the electronic device may enlarge an image (1211) of an acquired external electronic device and display the enlarged image (1221) on a virtual reality screen (1220) being displayed on a display (e.g., display module (160) of FIG. 1).

[0193] In this way, by enlarging the image of an external electronic device including a keyboard and displaying it on a virtual reality screen, the user can more easily use the keyboard displayed on the external electronic device.

[0194] According to one embodiment, when an electronic device identifies an area of ​​an image (1211) of an external electronic device in an image (1210) of a real space, the electronic device may set the area of ​​the image (1211) of the external electronic device as a region of interest (ROI). According to one embodiment, the electronic device may acquire an image by exposing and reading out only the ROI area, rather than acquiring an image by exposing and reading out the entire area that can be acquired through the camera.

[0195] In this way, resource consumption can be reduced by continuously acquiring and / or image processing only a portion of the image to identify keyboard input.

[0196] FIG. 13 is a drawing for explaining an operation of displaying a virtual keyboard when an external electronic device disappears from the screen of the electronic device due to rotation of the electronic device according to one embodiment.

[0197] Referring to FIG. 13, an electronic device (e.g., an electronic device (101) of FIG. 1, a processor (120) of FIG. 1, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may acquire an image (1311) of an external electronic device (e.g., an electronic device (104) of FIG. 1) from an image (1310) of a real space acquired through a camera (e.g., a camera module (180) of FIG. 1). According to one embodiment, the image (1311) of the external electronic device may include a keyboard displayed on the external electronic device, and may further include a hand of a user (10) performing input on the keyboard.

[0198] According to one embodiment, the electronic device may enlarge an image (1311) of an acquired external electronic device and display the enlarged image (1321) on a virtual reality screen (1320) being displayed on a display (e.g., display module (160) of FIG. 1).

[0199] In one embodiment, when the head of a user wearing an electronic device rotates and the external electronic device moves out of the field of view of the camera of the electronic device, the image (1330) of the real space acquired through the camera may not include the external electronic device, but only the user's hand (10).

[0200] In one embodiment, when an image (1330) of a real space acquired through a camera does not include an external electronic device, the electronic device may change a virtual reality screen (1320) that displayed an enlarged image (1321) of an external electronic device in a real space in a certain area to a full virtual reality screen (1340).

[0201] In one embodiment, the electronic device may display a virtual keyboard (1341) and a virtual hand image (1342) corresponding to the user's hand (10) by replacing the enlarged image (1321) of an external electronic device in real space.

[0202] In this way, even if the external electronic device is moved out of the field of view of the camera of the electronic device due to rotation of the electronic device, keyboard input can be performed through the virtual keyboard.

[0203] According to one embodiment, a wearable electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3A, the wearable electronic device (300) of FIG. 3B, or the electronic device (400) of FIG. 4) may include a camera (e.g., the camera module (180) of FIG. 1), a display (e.g., the display module (160) of FIG. 1), and at least one processor (e.g., the processor (120) of FIG. 1) electrically connected to the camera and the display.

[0204] According to one embodiment, the at least one processor may obtain a code image (e.g., a code image (711) of FIG. 7) corresponding to a keyboard (e.g., a keyboard (510) of FIG. 5, a keyboard (710) of FIG. 7, a keyboard (910) of FIG. 9A, or a keyboard (930) of FIG. 9B) displayed on the external electronic device (e.g., an electronic device (104) of FIG. 1 or an external electronic device (104) of FIG. 5) through an image obtained through the camera.

[0205] According to one embodiment, the at least one processor may obtain key information corresponding to each of a plurality of keys included in the keyboard based on the code image.

[0206] According to one embodiment, the at least one processor can identify a color of key information corresponding to a selected key when a key is selected through a keyboard displayed on the external electronic device through an image acquired through the camera.

[0207] According to one embodiment, the at least one processor may perform an operation corresponding to the selected key based on the identified key information.

[0208] According to one embodiment, the key information may include a plurality of key code values ​​and a plurality of color information, each corresponding to the plurality of keys.

[0209] According to one embodiment, the at least one processor can identify a color indicator displayed on the external electronic device (e.g., color indicators (520, 521) of FIG. 5, color indicators (920, 921) of FIG. 9A, or color indicators (940, 941) of FIG. 9B) through an image acquired through the camera.

[0210] According to one embodiment, the at least one processor can identify color information corresponding to the identified color among the plurality of color information.

[0211] According to one embodiment, the at least one processor can identify a key code value corresponding to the identified color information.

[0212] According to one embodiment, the at least one processor may perform an operation corresponding to the identified key code value.

[0213] In one embodiment, the color indicator may correspond to the selected key.

[0214] According to one embodiment, the at least one processor may acquire an image corresponding to the external electronic device included in an image acquired through the camera while displaying a virtual reality screen through the display.

[0215] According to one embodiment, the at least one processor may display an image corresponding to the external electronic device (e.g., an image corresponding to the external electronic device (1020) of FIG. 10) on the virtual reality screen (e.g., a virtual reality screen (1010) of FIG. 10).

[0216] In one embodiment, the at least one processor may magnify an image corresponding to the external electronic device.

[0217] In one embodiment, the at least one processor may display the enlarged image on the virtual reality screen.

[0218] According to one embodiment, the external electronic device may be a foldable device or a rollable device.

[0219] According to one embodiment, the at least one processor may acquire an image corresponding to the external electronic device included in an image acquired through the camera based on detecting an unfolding action of the external electronic device.

[0220] According to one embodiment, the at least one processor may display an image corresponding to the external electronic device on the virtual reality screen.

[0221] According to one embodiment, the at least one processor may display text corresponding to the selected key on the display based on the identified key information.

[0222] In one embodiment, the at least one processor may display a virtual keyboard on the display if the external electronic device is not included in the image acquired through the camera according to the rotation of the electronic device after acquiring information through the code image.

[0223] According to one embodiment, the at least one processor may further obtain position information of the plurality of keys included in the keyboard based on the code image.

[0224] According to one embodiment, the at least one processor may identify the key information corresponding to the selected key by further considering the position of the color indicator.

[0225] In one embodiment, the at least one processor may identify key information corresponding to the selected key based on a color of the color indicator displayed at a location unrelated to the location of the selected key on the external electronic device.

[0226] In one embodiment, the color indicator may be such that the location at which it is displayed on the external electronic device changes randomly each time it is displayed on the external electronic device.

[0227] According to one embodiment, the electronic device may further include a communication module (e.g., the communication module (190) of FIG. 1).

[0228] In one embodiment, the communication module may be in a non-communicative state with the external electronic device.

[0229] According to one embodiment, a method for controlling a wearable electronic device may include an operation of obtaining a code image corresponding to a keyboard displayed on an external electronic device through an image obtained through a camera of the electronic device.

[0230] According to one embodiment, a method for controlling a wearable electronic device may include an operation of obtaining key information corresponding to each of a plurality of keys included in the keyboard based on the code image.

[0231] According to one embodiment, a method for controlling a wearable electronic device may include an operation of identifying key information corresponding to a selected key when a key is selected through a keyboard displayed on the external electronic device through an image acquired through the camera.

[0232] According to one embodiment, a method for controlling a wearable electronic device may include performing an operation corresponding to the selected key based on the identified key information.

[0233] According to one embodiment, the key information may include a plurality of key code values ​​and a plurality of color information, each corresponding to the plurality of keys.

[0234] According to one embodiment, the operation of identifying the key information may identify the color of a color indicator displayed on the external electronic device through an image acquired through the camera.

[0235] According to one embodiment, the operation of identifying the key information may identify color information corresponding to the identified color among the plurality of color information.

[0236] According to one embodiment, the operation of identifying the key information may identify a key code value corresponding to the identified color information.

[0237] According to one embodiment, the operation of performing an operation corresponding to the selected key may perform an operation corresponding to the identified key code value.

[0238] In one embodiment, the color indicator may correspond to the selected key.

[0239] According to one embodiment, a method for controlling a wearable electronic device may further include an operation of acquiring an image corresponding to the external electronic device included in an image acquired through the camera while displaying a virtual reality screen through the display.

[0240] According to one embodiment, a method for controlling a wearable electronic device may further include an action of displaying an image corresponding to the external electronic device on the virtual reality screen.

[0241] According to one embodiment, a method for controlling a wearable electronic device may further include an operation of enlarging an image corresponding to the external electronic device.

[0242] According to one embodiment, the method for controlling a wearable electronic device may further include an action of displaying the enlarged image on the virtual reality screen.

[0243] According to one embodiment, the external electronic device may be a foldable device or a rollable device.

[0244] According to one embodiment, a method for controlling a wearable electronic device may further include an operation of acquiring an image corresponding to the external electronic device included in an image acquired through the camera based on detecting an unfolding operation of the external electronic device.

[0245] In one embodiment, the act of displaying the enlarged image on the virtual reality screen may display an image corresponding to the external electronic device on the virtual reality screen.

[0246] According to one embodiment, a control method of a wearable electronic device may further include an operation of displaying a text corresponding to the selected key based on the identified key information.

[0247] According to one embodiment, a method for controlling a wearable electronic device may further include an action of displaying a virtual keyboard on the display when the external electronic device is not included in an image acquired through the camera according to rotation of the electronic device after acquiring information through the code image.

[0248] According to one embodiment, a method for controlling a wearable electronic device may further include an operation of obtaining position information of the plurality of keys included in the keyboard based on the code image.

[0249] According to one embodiment, the operation of identifying key information corresponding to the selected key may identify key information corresponding to the selected key by further considering the position of the color indicator.

[0250] According to one embodiment, the operation of identifying key information corresponding to the selected key may identify key information corresponding to the selected key based on the color of the color indicator displayed at a location unrelated to the location of the selected key on the external electronic device.

[0251] In one embodiment, the color indicator may be such that the location at which it is displayed on the external electronic device changes randomly each time it is displayed on the external electronic device.

[0252] According to one embodiment, the communication module of the electronic device may be in a state of not being connected to the external electronic device.

[0253] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may include commands for causing a wearable electronic device to obtain a code image corresponding to a keyboard displayed on an external electronic device through an image obtained through the camera.

[0254] According to one embodiment, the one or more programs may include commands for the wearable electronic device to obtain key information corresponding to each of a plurality of keys included in the keyboard based on the code image.

[0255] According to one embodiment, the one or more programs may include commands for identifying key information corresponding to a selected key when a key is selected through a keyboard displayed on the external electronic device through an image acquired through the camera by the wearable electronic device.

[0256] According to one embodiment, the one or more programs may include commands for the wearable electronic device to perform an action corresponding to the selected key based on the identified key information.

[0257] According to one embodiment, the key information may include a plurality of key code values ​​and a plurality of color information, each corresponding to the plurality of keys.

[0258] According to one embodiment, the one or more programs may include commands for the wearable electronic device to identify a color of a color indicator displayed on the external electronic device through an image acquired through the camera.

[0259] According to one embodiment, the one or more programs may include instructions for the wearable electronic device to identify color information corresponding to the identified color among the plurality of color information.

[0260] According to one embodiment, the one or more programs may include instructions for the wearable electronic device to identify a key code value corresponding to the identified color information.

[0261] According to one embodiment, the one or more programs may include instructions that cause the wearable electronic device to perform an action corresponding to the identified key code value.

[0262] In one embodiment, the one or more programs may be configured such that the wearable electronic device has a color indicator corresponding to the selected key.

[0263] According to one embodiment, the one or more programs may include commands for obtaining an image corresponding to the external electronic device included in an image obtained through the camera while the wearable electronic device is displaying a virtual reality screen through the display.

[0264] According to one embodiment, the one or more programs may include commands for the wearable electronic device to display an image corresponding to the external electronic device on the virtual reality screen.

[0265] In one embodiment, the one or more programs may include commands for the wearable electronic device to enlarge an image corresponding to the external electronic device.

[0266] In one embodiment, the one or more programs may include commands for the wearable electronic device to display the enlarged image on the virtual reality screen.

[0267] According to one embodiment, the external electronic device may be a foldable device or a rollable device.

[0268] According to one embodiment, the one or more programs may include commands for causing the wearable electronic device to acquire an image corresponding to the external electronic device included in an image acquired through the camera based on detection of an unfolding action of the external electronic device.

[0269] According to one embodiment, the one or more programs may include commands for the wearable electronic device to display an image corresponding to the external electronic device on the virtual reality screen.

[0270] According to one embodiment, the one or more programs may include instructions for the wearable electronic device to display text corresponding to the selected key on the display based on the identified key information.

[0271] In one embodiment, the one or more programs may include commands for causing the wearable electronic device to display a virtual keyboard on the display when the external electronic device is not included in the image acquired through the camera according to the rotation of the electronic device after acquiring information through the code image.

[0272] According to one embodiment, the one or more programs may include instructions for the wearable electronic device to further obtain positional information of the plurality of keys included in the keyboard based on the code image.

[0273] According to one embodiment, the one or more programs may include instructions for the wearable electronic device to identify key information corresponding to the selected key by further considering the location of the color indicator.

[0274] According to one embodiment, the one or more programs may include instructions for the wearable electronic device to identify key information corresponding to the selected key based on a color of the color indicator displayed at a location unrelated to the location of the selected key on the external electronic device.

[0275] In one embodiment, the color indicator may be such that the location at which it is displayed on the external electronic device changes randomly each time it is displayed on the external electronic device.

[0276] According to one embodiment, the electronic device may further include a communication module.

[0277] In one embodiment, the communication module may be in a non-communicative state with the external electronic device.

[0278] Electronic devices according to the embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0279] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0280] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0281] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0282] According to one embodiment, the method according to one embodiment disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0283] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable electronic device (101, 200, 300, 400), Camera (180); display (160); and At least one processor (120) electrically connected to the camera and the display, wherein the at least one processor comprises: Through the image acquired through the above camera, a code image (711) corresponding to the keyboard (510, 710, 910, 930) displayed on the external electronic device (104) is acquired, Based on the above code image, key information corresponding to each of the multiple keys included in the keyboard is obtained, Through the image acquired through the above camera, when a key is selected through the keyboard displayed on the external electronic device, key information corresponding to the selected key is identified, A wearable electronic device configured to perform an action corresponding to the selected key based on the identified key information.

2. In paragraph 1, The above key information is, A wearable electronic device including a plurality of key code values ​​and a plurality of color information, each corresponding to a plurality of keys.

3. In paragraph 2, At least one processor of the above, Through the image acquired through the above camera, the color of the color indicator (520, 521, 920, 921, 940, 941) displayed on the external electronic device is identified, Identifying color information corresponding to the identified color among the plurality of color information, Identify the key code value corresponding to the above identified color information, Perform an action corresponding to the above identified key code value, A wearable electronic device, wherein the color indicator corresponds to the selected key.

4. In any one of paragraphs 1 to 3, At least one processor of the above, In a state where a virtual reality screen (1010) is displayed through the above display, an image (1020) corresponding to the external electronic device included in the image acquired through the above camera is acquired, A wearable electronic device that displays an image corresponding to the external electronic device on the virtual reality screen.

5. In paragraph 4 At least one processor of the above, Enlarge the image corresponding to the above external electronic device, A wearable electronic device that displays the enlarged image on the virtual reality screen.

6. In paragraph 4 or 5, The above external electronic device comprises a foldable device or a rollable device, At least one processor of the above, Based on detecting the unfolding motion of the external electronic device, an image corresponding to the external electronic device included in the image acquired through the camera is acquired, A wearable electronic device that displays an image corresponding to the external electronic device on the virtual reality screen.

7. In any one of paragraphs 1 to 6, At least one processor of the above, A wearable electronic device that displays text corresponding to the selected key on the display based on the identified key information.

8. In any one of paragraphs 3 to 7, At least one processor of the above, Based on the above code image, further obtaining location information of the plurality of keys included in the keyboard, An electronic device that identifies key information corresponding to the selected key by further considering the position of the color indicator.

9. In any one of paragraphs 3 to 8, At least one processor of the above, A wearable electronic device that identifies key information corresponding to the selected key based on the color of the color indicator displayed at a location unrelated to the location of the selected key on the external electronic device.

10. In paragraph 9, The above color indicators are, A wearable electronic device, wherein the location displayed on the external electronic device changes randomly whenever the device is displayed on the external electronic device.

11. In any one of paragraphs 1 to 10, Further comprising a communication module (190); A wearable electronic device in which the above communication module is not connected to the external electronic device.

12. A method for controlling a wearable electronic device, An action of obtaining a code image corresponding to a keyboard displayed on an external electronic device through an image obtained through a camera of the electronic device; An operation of obtaining key information corresponding to each of a plurality of keys included in the keyboard based on the above code image; An operation of identifying key information corresponding to a selected key when a key is selected through a keyboard displayed on the external electronic device through an image acquired through the camera; and A method for controlling a wearable electronic device, comprising: performing an operation corresponding to the selected key based on the identified key information.

13. In paragraph 12, An operation of acquiring an image corresponding to the external electronic device included in an image acquired through the camera while displaying a virtual reality screen through the display; and A method for controlling a wearable electronic device, further comprising: an operation of displaying an image corresponding to the external electronic device on the virtual reality screen.

14. In paragraph 12 or 13, A method for controlling a wearable electronic device, further comprising: an action of displaying a text corresponding to the selected key based on the identified key information.

15. In any one of paragraphs 12 to 14, A method for controlling a wearable electronic device in which the communication module of the electronic device is not connected to the external electronic device.

Citation Information

Patent Citations

  • method of processing virtual keypad

    KR101246605B1

  • Head mounted display and the method for controlling the same

    KR1020140064384A

  • Touchscreen image mapping for augmented reality devices

    US10325101B2

  • Virtual input devices for pressure sensitive surfaces

    US20210089161A1

  • Numeric keypad encryption for augmented reality devices

    US9697384B2