Electronic device and virtual image providing method for electronic device
The integration of a processor in wearable devices to adjust virtual object characteristics based on foldable device angle information addresses the accuracy and smoothness issues in user control, enhancing the virtual environment experience.
Patent Information
- Application Number
- PCT/KR2024/007007
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-07-12
- Filing Date
- 2024-05-23
- Publication Date
- 2024-12-05
AI Technical Summary
Current wearable devices providing virtual environments, such as VR, AR, and MR, face challenges in accurately detecting user movements for controlling virtual objects due to low motion detection accuracy and limited control operations, leading to smooth control issues.
An electronic device equipped with a display, camera, and communication circuit, where a processor recognizes folding angle information from a foldable device and adjusts virtual image characteristics, such as size, position, and angle, based on this information to enhance user control over virtual objects.
Improves the accuracy and smoothness of user control over virtual objects by dynamically adjusting their visual characteristics in response to the folding angle of an adjacent foldable device, providing a more immersive and interactive virtual environment.
Smart Images

Figure KR2024007007_05122024_PF_FP_ABST
Abstract
Description
Electronic devices and methods for providing virtual images of electronic devices
[0001] The present disclosure relates to electronic devices, and for example, to a method for providing a virtual image in a wearable electronic device that can visually provide a virtual environment when worn by a user.
[0002] Various technologies are being developed to provide users with virtual environments, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR). These virtual environments can be provided by wearable devices, such as head-mounted devices or eyewear-type devices (e.g., AR glasses). The virtual images provided by wearable devices can include various virtual objects.
[0003] A foldable device may include a structure that allows the entire device to be folded, with a flexible display. The angle between the facing displays of the foldable device can be changed according to the user's manipulation, and the foldable device may be positioned in, for example, a fully folded state, an intermediate state, or a fully unfolded state.
[0004] When providing a virtual image containing various virtual objects on a wearable device, the user may desire to change the position, size, angle, etc. of the virtual objects. For example, the wearable device can detect the movement of a part of the user's body (e.g., a finger) and control the virtual object in response to the movement. However, this type of motion detection can be inaccurate, and the control actions that can be mapped to the user's movements may be limited, making it difficult to control the virtual object according to the user's actual intention.
[0005] An electronic device according to various embodiments of the present disclosure (or specification, invention) may include a display for outputting a virtual image, a camera for acquiring a real image including an external subject, a communication circuit, and a processor operatively connected to the display, the camera, and the communication circuit.
[0006] According to one embodiment, the processor may be configured to recognize the foldable device based on at least one of a real image including an external foldable device acquired from the camera or a signal received from the foldable device via the communication circuit.
[0007] According to one embodiment, the processor may be configured to receive folding angle information of the foldable device from the foldable device through the communication circuit, determine visual characteristics of at least one virtual object constituting the virtual image based on the received folding angle information, and render the virtual image including the at least one virtual object in which the determined visual characteristics are reflected, and output the virtual image through the display.
[0008] A method for providing a virtual image of an electronic device according to various embodiments of the present document may include an operation of recognizing a foldable device based on at least one of a real image including an external foldable device acquired from a camera or a signal received from the foldable device, an operation of receiving folding angle information of the foldable device from the foldable device, an operation of determining a visual characteristic of at least one virtual object constituting the virtual image based on the received folding angle information, and an operation of rendering and outputting the virtual image including the at least one virtual object to which the determined visual characteristic is reflected.
[0009] According to various embodiments of the present document, an electronic device and a method for providing a virtual image of the electronic device can be provided, which can render a virtual image and adjust visual characteristics of a virtual object included in the virtual image based on a position and / or a folding angle of an adjacent foldable device.
[0010] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.
[0011] FIG. 2 is a perspective view of a wearable device according to various embodiments.
[0012] FIG. 3 illustrates a wearable device and a foldable device according to various embodiments.
[0013] FIG. 4 illustrates a virtual office space provided by a wearable device according to various embodiments.
[0014] Figure 5 is a block diagram of a wearable device according to various embodiments.
[0015] FIG. 6 is a flowchart of a method for rendering a virtual image of a wearable device according to one embodiment.
[0016] FIG. 7 is a flowchart of a method for a wearable device to configure a virtual image using information of a foldable device according to one embodiment.
[0017] FIG. 8 is a flowchart of a method for providing a virtual image of a wearable device and a foldable device according to one embodiment.
[0018] FIG. 9 illustrates an example of a wearable device according to one embodiment controlling the angle of a virtual image according to the folding angle of the foldable device.
[0019] FIG. 10 illustrates an example of a wearable device according to one embodiment controlling the size of a virtual image according to a folding angle of a foldable device.
[0020] FIG. 11 illustrates an example of a wearable device according to one embodiment controlling the distance of a virtual image according to a folding angle of a foldable device.
[0021] FIG. 12 illustrates an example of a wearable device according to one embodiment configuring position coordinates of a virtual object according to a folding angle of a foldable device.
[0022] FIG. 13 is a flowchart of a method for determining a virtual object to be controlled by a wearable device according to one embodiment.
[0023] FIG. 14 illustrates an example of a wearable device according to one embodiment controlling a virtual object according to a folding angle of a foldable device.
[0024] FIG. 15 illustrates an example of a wearable device according to one embodiment controlling a virtual object according to a folding angle of a foldable device.
[0025] FIG. 16 illustrates an example of a wearable device according to one embodiment controlling a virtual object according to a folding angle of a foldable device.
[0026] FIG. 17 illustrates an example of a wearable device according to one embodiment controlling a grid including virtual objects according to a folding angle of a foldable device.
[0027] FIG. 18 illustrates an example of a wearable device according to one embodiment configuring a grid including virtual objects according to the orientation of the foldable device.
[0028] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.
[0029] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.
[0030] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).
[0031] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.
[0032] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.
[0033] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).
[0034] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).
[0035] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).
[0036] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.
[0037] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.
[0038] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).
[0039] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.
[0040] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
[0041] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).
[0042] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.
[0043] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.
[0044] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).
[0045] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.
[0046] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).
[0047] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.
[0048] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).
[0049] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.
[0050] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).
[0051] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.
[0052] FIG. 2 is a perspective view of a wearable device according to various embodiments.
[0053] Although FIG. 2 describes the structure of a wearable device in the form of glasses (e.g., AR glasses, smart glasses), the wearable device according to various embodiments of the present disclosure may be implemented as another type of wearable device that can be worn by a user and provide a virtual environment such as VR (virtual reality), AR (augmented reality), MR (mixed reality), or XR (extended reality) when worn by the user. The wearable device (200) may further include at least some of the configurations and / or functions of the wearable device (101) of FIG. 1. Hereinafter, the wearable device may also be referred to as an electronic device or a wearable electronic device.
[0054] Referring to FIG. 2, a wearable device (200) according to various embodiments may include a bridge (bridge, 201), a first rim (rim, 210), a second rim (220), a first end piece (end piece, 230), a second end piece (240), a first temple (temple, 250), and / or a second temple (260).
[0055] According to one embodiment, the bridge (201) can connect the first limb (210) and the second limb (220). The bridge (201) can be positioned over the user's nose when the user wears the wearable device (200). The bridge (201) can separate the first limb (210) and the second limb (220) based on the user's nose.
[0056] According to various embodiments, the bridge (201) may include a camera module (203), a first line-of-sight tracking camera (205), a second line-of-sight tracking camera (207), and / or an audio module (209).
[0057] According to various embodiments, the camera module (203) (e.g., the camera module (180) of FIG. 1) can capture a front view (e.g., in the -y-axis direction) of the user and acquire image data. The camera module (203) can capture an image corresponding to the user's field of view (FOV) or measure a distance to a subject. The camera module (203) can include an RGB camera, a high resolution (HR) camera, and / or a photo video (PV) camera. The camera module (203) can include a color camera having an auto focus (AF) function and an optical image stabilization (OIS) function to acquire high-quality images.
[0058] According to various embodiments, the first gaze tracking camera (205) and the second gaze tracking camera (207) can determine the gaze of the user. The first gaze tracking camera (205) and the second gaze tracking camera (207) can capture the pupil of the user's eye in a direction substantially opposite to the capturing direction of the camera module (203) (e.g., y direction). For example, the first gaze tracking camera (205) can capture at least a portion of the user's left eye, and the second gaze tracking camera (207) can capture at least a portion of the user's right eye. The first gaze tracking camera (205) and the second gaze tracking camera (207) can detect the user's pupil (e.g., left eye and right eye) and track the gaze direction. The tracked gaze direction can be utilized to move the center of a virtual image including a virtual object in response to the gaze direction. The first gaze tracking camera (205) and / or the second gaze tracking camera (207) can track the user's gaze using, for example, at least one of an EOG sensor (electro-oculography or electrooculogram), a coil system, a dual Purkinje system, bright pupil systems, or dark pupil systems.
[0059] According to various embodiments, the audio module (209) (e.g., the audio module (170) of FIG. 1) may be positioned between the first gaze tracking camera (205) and the second gaze tracking camera (207). The audio module (209) may convert a user's voice and / or external sound into an electrical signal, or convert an electrical signal into sound. The audio module (209) may include a microphone.
[0060] According to one embodiment, the first limb (210) and the second limb (220) may form a frame of the wearable device (200) (e.g., a frame of AR glasses). The first limb (210) may be disposed in a first direction (e.g., an x-axis direction) of the bridge (201). The first limb (210) may be disposed at a position corresponding to the left eye of the user. The second limb (220) may be disposed in a second direction (e.g., a -x-axis direction) of the bridge (201) which is substantially opposite to the first direction (e.g., an x-axis direction). The second limb (220) may be disposed at a position corresponding to the right eye of the user. The first limb (210) and the second limb (220) may be formed of a metal material and / or a non-conductive material (e.g., a polymer).
[0061] According to various embodiments, the first rim (210) may surround and support at least a portion of the first glass (215) disposed on the inner surface. The first glass (215) may be positioned in front of the user's left eye. The second rim (220) may surround and support at least a portion of the second glass (225) disposed on the inner surface. The second glass (225) may be positioned in front of the user's right eye. The user of the wearable device (200) may view a foreground (e.g., a real image or real information) of an external object through the first glass (215) and the second glass (225). The wearable device (200) may implement augmented reality by superimposing a virtual image on real information including an external object and displaying it.
[0062] According to various embodiments, the first glass (215) and the second glass (225) may include projection-type transparent displays. The first glass (215) and the second glass (225) may each form a reflective surface as a transparent plate (or transparent screen), and an image generated in the wearable device (200) may be reflected (e.g., total internal reflection) through the reflective surface and may be incident on the user's left and / or right eyes.
[0063] According to various embodiments, the first glass (215) may include a waveguide (or optical waveguide) that transmits light generated from a light source (not shown) of the wearable device (200) to the user's left eye. For example, the waveguide may be formed of glass, plastic, or a polymer material, and may include a nano-pattern (e.g., a grating structure or a mesh structure having a polygonal or curved shape) formed on the inside or surface of the first glass (215). The waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or a reflective element (e.g., a reflective mirror). The waveguide may guide display light emitted from the light source to the user's eye by using at least one diffractive element or reflective element included in the waveguide. In various embodiments, the diffractive element may include an input / output optical member, and the reflective element may include a total internal reflection (TIR). For example, light emitted from a light source may be guided along an optical path to a waveguide through an input optical element (e.g., an in coupler), and light traveling inside the waveguide may be guided toward a user's eye through an output optical element (e.g., an out coupler).
[0064] The second glass (225) can be implemented in substantially the same manner as the first glass (215). The optical path formed through the wave guides of the first glass (215) and the second glass (225) will be described in more detail with reference to FIG. 3.
[0065] According to various embodiments, the first glass (215) and the second glass (225) may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), a light emitting diode (LED) on silicon (LEDoS), an organic light emitting diode (OLED), an organic light emitting diode on silicon (OLEDoS), or a micro light emitting diode (micro LED). Although not shown, when the first glass (215) and the second glass (225) are formed of one of a liquid crystal display, a digital mirror display, or a silicon liquid crystal display, the wearable device (200) may include a light source that irradiates light to the screen output areas of the first glass (215) and the second glass (225). In another embodiment, if the first glass (215) and the second glass (225) are capable of generating light on their own, for example, if they are made of either organic light emitting diodes or micro LEDs, the wearable device (200) can provide a relatively good quality virtual image to the user even without including a separate light source.
[0066] According to various embodiments, the first limb (210) may include a first microphone (211), a first recognition camera (213), a first light-emitting device (217), and / or a first display module (219). The second limb (220) may include a second microphone (221), a second recognition camera (223), a second light-emitting device (227), and / or a second display module (229).
[0067] In various embodiments, the first light-emitting device (217) and the first display module (219) may be included in the first end piece (230), and the second light-emitting device (227) and the second display module (229) may be included in the second end piece (240).
[0068] According to various embodiments, the first microphone (211) and / or the second microphone (221) can receive the voice and / or external sound of the user of the wearable device (200) and convert it into an electrical signal.
[0069] According to various embodiments, the first recognition camera (213) and / or the second recognition camera (223) can recognize the surrounding space of the wearable device (200). The first recognition camera (213) and / or the second recognition camera (223) can detect the user's gesture within a certain distance (e.g., a certain space) of the wearable device (200). The first recognition camera (213) and / or the second recognition camera (223) can include a GS (global shutter) camera capable of reducing the RS (rolling shutter) phenomenon in order to detect and track the user's rapid hand movements and / or fine movements of the fingers. The wearable device (200) can detect the eye corresponding to the dominant eye and / or the auxiliary eye among the user's left and / or right eyes using the first gaze tracking camera (205), the second gaze tracking camera (207), the first recognition camera (213), and / or the second recognition camera (223). For example, the wearable device (200) can detect the eye corresponding to the dominant eye and / or the auxiliary eye based on the user's gaze direction toward an external object or a virtual object.
[0070] According to various embodiments, the first light emitting device (217) and / or the second light emitting device (227) may emit light to increase the accuracy of the camera module (203), the first gaze tracking camera (205), the second gaze tracking camera (207), the first recognition camera (213) and / or the second recognition camera (223). The first light emitting device (217) and / or the second light emitting device (227) may be used as an auxiliary means to increase the accuracy when capturing the user's pupil using the first gaze tracking camera (205) and / or the second gaze tracking camera (207). The first light emitting device (217) and / or the second light emitting device (227) may be used as an auxiliary means when the user's gesture is captured using the first recognition camera (213) and / or the second recognition camera (223), and when it is not easy to detect an object to be captured (e.g., a subject) due to a dark environment or mixing and reflected light from multiple light sources. The first light emitting device (217) and / or the second light emitting device (227) may include, for example, an LED, an IR LED, or a xenon lamp.
[0071] According to various embodiments, the first display module (219) and / or the second display module (229) can emit light and transmit it to the user's left eye and / or right eye using the first glass (215) and / or the second glass (225). The first glass (215) and / or the second glass (225) can display various image information using the light emitted through the first display module (219) and / or the second display module (229). The wearable device (200) can display a foreground for an external object and an image emitted through the first display module (219) and / or the second display module (229) by overlapping them through the first glass (215) and / or the second glass (225).
[0072] In one embodiment, the first end piece (230) may be coupled to a portion (e.g., in the x-axis direction) of the first limb (210). The second end piece (240) may be coupled to a portion (e.g., in the -x-axis direction) of the second limb (220). In various embodiments, the first light-emitting device (217) and the first display module (219) may be included in the first end piece (230). The second light-emitting device (227) and the second display module (229) may be included in the second end piece (240).
[0073] According to various embodiments, the first end piece (230) can connect the first rim (210) and the first temple (250). The second end piece (240) can connect the second rim (220) and the second temple (260).
[0074] In one embodiment, the first temple (250) can be operatively connected to the first end piece (230) using a first hinge portion (255). The first hinge portion (255) can be configured to be rotatable so that the first temple (250) can be folded or unfolded with respect to the first limb (210). The first temple (250) can extend, for example, along the left side of the user's head. At least a portion of a distal portion (e.g., in the y-axis direction) of the first temple (250) can be configured to be bent so that it is supported on, for example, at least a portion of the user's left ear when the user wears the wearable device (200). The second temple (260) can be operatively connected to the second end piece (240) using a second hinge portion (265). The second hinge portion (265) may be configured to be rotatable so that the second temple (260) can be folded or unfolded with respect to the second rim (220). The second temple (260) may extend, for example, along the right side of the user's head. At least a portion of a distal portion (e.g., in the y-axis direction) of the second temple (260) may be configured to be bent so that it is supported on, for example, at least a portion of the user's right ear when the user wears the wearable device (200).
[0075] According to various embodiments, the first temple (250) may include a first printed circuit board (251), a first audio output module (253) (e.g., audio output module (155) of FIG. 1) and / or a first battery (257) (e.g., battery (189) of FIG. 1). The second temple (260) may include a second printed circuit board (261), a second audio output module (263) (e.g., audio output module (155) of FIG. 1) and / or a second battery (267) (e.g., battery (189) of FIG. 1).
[0076] According to various embodiments, various electronic components (e.g., at least some of the components included in the electronic device (101) of FIG. 1), such as the processor (120), memory (130), interface (177), and / or wireless communication module (192) disclosed in FIG. 1, may be disposed on the first printed circuit board (251) and / or the second printed circuit board (261). The processor may include, for example, one or more of a central processing unit, an application processor, a graphic processing unit, an image signal processor, a sensor hub processor, or a communication processor. The first printed circuit board (251) and / or the second printed circuit board (261) may include, for example, a printed circuit board (PCB), a flexible PCB (FPCB), or a rigid-flexible PCB (RFPCB). In some embodiments, the first printed circuit board (251) and / or the second printed circuit board (261) may include a primary PCB, a secondary PCB positioned to partially overlap the primary PCB, and / or an interposer substrate between the primary PCB and the secondary PCB. The first printed circuit board (251) and / or the second printed circuit board (261) may be electrically connected to other components (e.g., a camera module (203), a first line-of-sight tracking camera (205), a second line-of-sight tracking camera (207), an audio module (209), a first microphone (211), a first recognition camera (213), a first light-emitting device (217), a first display module (219), a second microphone (221), a second recognition camera (223), a second light-emitting device (227), a second display module (229), a first audio output module (253) and / or a second audio output module (263)) using electrical paths such as FPCBs and / or cables. For example, the FPCBs and / or cables may be disposed on at least a portion of the first limb (210), the bridge (201) and / or the second limb (220).In some embodiments, the wearable device (200) may include only one of the first printed circuit board (251) or the second printed circuit board (261).
[0077] According to various embodiments, the first audio output module (253) and / or the second audio output module (263) may transmit audio signals to the user's left and / or right ears. The first audio output module (253) and / or the second audio output module (263) may include, for example, a piezo speaker (e.g., a bone conduction speaker) that transmits audio signals without a speaker hole. In some embodiments, the wearable device (200) may include only one of the first audio output module (253) or the second audio output module (263).
[0078] According to various embodiments, the first battery (257) and / or the second battery (267) may supply power to the first printed circuit board (251) and / or the second printed circuit board (261) using a power management module (e.g., the power management module (188) of FIG. 1). The first battery (257) and / or the second battery (267) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell. In some embodiments, the wearable device (200) may include only one of the first battery (257) or the second battery (267).
[0079] According to various embodiments, the wearable device (200) may include a sensor module (e.g., the sensor module (176) of FIG. 1). The sensor module may generate an electrical signal or data value corresponding to an internal operating state of the wearable device (200) or an external environmental state. The sensor module may further include, for example, at least one of a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a color sensor, an infrared (IR) sensor, a biometric sensor (e.g., an HRM sensor), a temperature sensor, a humidity sensor, or an illuminance sensor. In some embodiments, the sensor module may recognize a user's biometric information using various biometric sensors (or biometric recognition sensors), such as an e-nose sensor, an electromyography sensor, an electroencephalogram sensor, an ECG sensor, or an iris sensor.
[0080] FIG. 3 illustrates a wearable device and a foldable device according to various embodiments.
[0081] According to one embodiment, the wearable device (300) (e.g., the wearable device (200) of FIG. 2) may be a device that can be worn by a user on the body, for example, a device that can be worn on a part of the head. The wearable device (300) may be, but is not limited to, a head-mounted device or a glasses-type device.
[0082] According to one embodiment, the wearable device (300) may generate a virtual image constituting a virtual environment and provide it to the user. For example, the virtual environment provided by the wearable device (300) may include, but is not limited to, any one of virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR). The wearable device (300) may store data (e.g., virtual objects) for implementing the virtual image in memory, receive it from an external server via a network, and / or obtain it from another electronic device of the user (e.g., a foldable device (400)) via short-range wireless communication.
[0083] There is no set type of virtual environment that can be provided through a wearable device (300), but it may include, for example, a virtual office space (desk work space) as described later with reference to FIG. 4.
[0084] According to one embodiment, the foldable device (400) may be a device having a wireless communication function and capable of executing various applications, such as a smart phone or a tablet PC. The foldable device (400) may be folded based on a folding axis. For example, a flexible display of the foldable device (400) may include a first region and a second region that are divided based on the folding axis, and in an unfolded state, both the first region and the second region may be exposed to the outside, and in a folded state, the first region and the second region may be folded in a form facing each other. The first region of the flexible display may be disposed on a first housing, the second region may be disposed on a second housing, and the first housing and the second housing may be rotatably coupled through a hinge structure.
[0085] The foldable device (400) can be folded or unfolded by a user's manipulation, so that the folding angle, which is the angle between the first region and the second region of the flexible display, can be changed. The foldable device (400) can include an angle sensor for detecting such folding angle. The foldable device (400) can be fixed to maintain the angle in at least some angle range among an intermediate state between a folded state and an unfolded state (e.g., a state in which the angle between the first region and the second region is between 0 degrees and 180 degrees), such as a flex mode.
[0086] Although FIG. 3 illustrates a foldable device (400) with a foldable display, the foldable device (400) of this document is not limited thereto. For example, a device including two physically separated displays on an area defined by a folding axis may be used. Alternatively, a device that does not include a display may also be the foldable device (400) of this document.
[0087] According to one embodiment, the wearable device (300) and the foldable device (400) can be wirelessly connected to each other via short-range wireless communication (e.g., Bluetooth, WiFi) to transmit and receive various data. For example, the foldable device (400) can transmit folding angle information (e.g., the angle between the first and second regions of the flexible display) to the wearable device (300) via short-range wireless communication.
[0088] According to one embodiment, when providing a virtual image that implements a virtual environment, the wearable device (300) may determine the visual characteristics of at least one virtual object included in the virtual image based on the position and / or folding angle information of the foldable device (400). For example, the wearable device (300) may change the size, distance, and / or angle of a specific virtual object according to a change in the folding angle of the foldable device (400).
[0089] Hereinafter, various embodiments in which a wearable device (300) renders a virtual image and adjusts visual characteristics of a virtual object included in the virtual image based on the position and / or folding angle of an adjacent foldable device (400) will be described.
[0090] FIG. 4 illustrates a virtual office space provided by a wearable device according to various embodiments.
[0091] According to one embodiment, a wearable device (e.g., the wearable device (300) of FIG. 3) may generate a virtual image constituting a virtual environment and provide it to a user. For example, the virtual environment provided by the wearable device may include, but is not limited to, any one of virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR).
[0092] FIG. 4 illustrates a virtual office space (desk workspace) (500), which is an example of various virtual environments that can be provided through a wearable device. Referring to FIG. 4, the virtual office space (500) may include a UX screen (520), a virtual desk (530), a task board (540), and virtual objects (511, 512, 513). Here, the UX screen (520) may include a screen that displays work content, such as a TV or monitor in a real environment. The virtual desk (530) may provide a user with the feeling of sitting at a desk and working. The task board (540) may be configured to allow the user to record work-related information.
[0093] According to one embodiment, the virtual image may include at least one virtual object (511, 512, 513). The wearable device may generate the virtual object (511, 512, 513) and determine its location based on user input (e.g., change in gaze direction, change in hand position and / or shape, voice input). For example, the virtual object (511, 512, 513) may include an image having the shape of a real object, a widget (e.g., weather, calendar, clock), an avatar, etc., but there is no limitation on the type. The user may arrange the virtual object (511, 512, 513) in the entire space in the virtual environment, place it on a desk (530), or attach it to a task board (540).
[0094] According to one embodiment, when providing a virtual image that implements a virtual environment, the wearable device may determine the visual characteristics of at least one virtual object (511, 512, 513) included in the virtual image based on the position and / or folding angle information of the foldable device (e.g., the foldable device (400) of FIG. 3). As illustrated in FIG. 4, when the virtual image includes a plurality of virtual objects, the wearable device may select at least some of the plurality of virtual objects (511, 512, 513) according to a user input, and determine the visual characteristics of at least one selected virtual object (511, 512, 513) based on the position and / or folding angle information of the foldable device.
[0095] Figure 5 is a block diagram of a wearable device according to various embodiments.
[0096] According to various embodiments, the wearable device (300) may include a camera module (320), a display module (310), a communication module (330), a processor (350), and a memory (340), and may implement various embodiments of the present document even if some of the illustrated components are omitted or replaced. At least some of the illustrated components may be operatively, electrically, and / or functionally connected to each other. The wearable device (300) may include at least some of the components and / or functions of the electronic device (101) of FIG. 1 and / or the wearable device (200) of FIG. 2.
[0097] According to one embodiment, the wearable device (300) may be a glasses-type device (200) having the structure of FIG. 2, but is not limited thereto. For example, the wearable device (300) may be another type of device that can be worn by a user, such as a head-mounted device.
[0098] According to one embodiment, the display module (310) can output light that constitutes a virtual image. The display module (310) is an emissive or projector-type display, and can be composed of, but is not limited to, liquid crystal on silicon (LCoS), organic light emitting diode (OLED) (or uOLED), light emitting diode (LED) (or uLED), or LED on silicon (LEDoS). The light output from the display module (310) is transmitted through an optical structure, so that a virtual image can be recognized in the direction of a user's gaze.
[0099] According to one embodiment, the camera module (320) may include a front camera that captures the front of the user to obtain image data of the actual environment, and a gaze tracking camera that tracks the gaze position of the user. The front camera may capture an image corresponding to the user's field of view (FOW). The camera module (320) may include, but is not limited to, an RGB camera, a high resolution (HR) camera, and / or a photo video (PV) camera. The gaze tracking camera may capture the user's pupils from a direction opposite to the shooting direction of the front camera. The processor (350) may determine the user's gaze direction from the image of the pupils captured by the gaze tracking camera.
[0100] According to one embodiment, the communication module (330) may include various hardware and / or software configurations for communicating with an external device via a wireless communication network. The communication module (330) may include a cellular communication module that supports cellular wireless communication (e.g., 4G, 5G cellular communication) and a short-range wireless communication module that supports short-range wireless communication (e.g., Wi-Fi, Bluetooth). For example, the wearable device (300) may use the cellular wireless communication module or the short-range wireless communication module to communicate with another electronic device (e.g., the electronic device (102) or the electronic device (104) of FIG. 1) and / or a server on a network (e.g., the server (108) of FIG. 1). According to one embodiment, the wearable device (300) may receive a signal transmitted from an adjacent foldable device via the communication module (330).
[0101] According to one embodiment, the memory (340) may temporarily or permanently store various data, including volatile memory and non-volatile memory. The memory (340) may include at least some of the configuration and / or functions of the memory (130) of FIG. 1 and may store the program (140) of FIG. 1. The memory (340) may store various instructions that may be executed by the processor (350). Such instructions may include control commands such as arithmetic and logical operations, data movement, and input / output that may be recognized by the processor (350).
[0102] According to one embodiment, the processor (350) may be configured to perform calculations or data processing related to control and / or communication of each component of the wearable device (300), and may be configured with one or more processors. The processor (350) may include at least some of the configurations and / or functions of the processor (120) of FIG. 1. Although the calculation and data processing functions that the processor (350) may implement on the wearable device (300) are not limited, this document will describe in detail various embodiments for generating and outputting a virtual image that implements a virtual environment, and controlling the visual characteristics of a virtual object included in the virtual image according to the folding angle of the foldable device. The operations of the processor (350) described below may be performed by loading instructions stored in the memory (340).
[0103] In this document, the description that the processor (350) (or the wearable device (300)) can perform a certain operation (or function, work, task) may be interpreted to mean substantially the same as that an instruction (or command, computer program) that causes the wearable device (300) (or the processor (350)) to perform the corresponding operation is stored in the memory (340) (e.g., non-volatile memory, storage). In addition, the description that the processor (350) can perform a certain operation may be interpreted to mean substantially the same as that at least one processor (350) can perform the corresponding operation.
[0104] According to one embodiment, the wearable device (300) can output a virtual image constituting a virtual environment. For example, the virtual environment provided by the wearable device (300) may include, but is not limited to, any one of virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended reality (XR). The wearable device (300) may store data (e.g., virtual objects) for implementing the virtual image in the memory (340), receive it from an external server via a network, and / or obtain it from another electronic device (e.g., a foldable device) of the user via short-range wireless communication. There is no set type of virtual environment that can be provided through the wearable device (300), but may include, for example, a virtual office space (desk work space) (500) described with reference to FIG. 4.
[0105] According to one embodiment, the processor (350) can generate a virtual image and output it through the display module (310). The display module (310) outputs light that constitutes the virtual image, and the light output through the optical structure can be perceived by the user's eyes through a reflection and / or diffraction process.
[0106] According to one embodiment, the virtual image may include at least one virtual object (e.g., virtual object (511, 512, 513) of FIG. 4). For example, the virtual object may include an image having the shape of a real object, a widget (e.g., weather, calendar, clock), an avatar, etc., but the type thereof is not limited.
[0107] In one embodiment, the processor (350) can recognize an adjacent foldable device. For example, the processor (350) can recognize the foldable device based on a short-range wireless communication signal transmitted from the foldable device and received via the communication module (330). Alternatively, the processor (350) can recognize the foldable device through object recognition in an image acquired via the camera module (320).
[0108] According to one embodiment, the processor (350) can check the status information of the recognized foldable device. Here, the status information of the foldable device can check the location information and folding angle information of the foldable device. According to one embodiment, the processor (350) can recognize the current location of the foldable device and anchor at least one virtual object based on the recognized location. For example, the processor (350) can anchor at least one virtual object so that the virtual object is recognized adjacent to the foldable device based on the user's line of sight.
[0109] Additionally, the processor (350) can determine visual characteristics (e.g., size, distance, angle) of at least one virtual object based on folding angle information received from the foldable device.
[0110] According to one embodiment, the processor (350) may render a virtual image including a virtual object. For example, the processor (350) may render a virtual image so that it can be recognized by a user's gaze by reflecting the visual characteristics of the virtual object determined based on status information of the foldable device.
[0111] According to one embodiment, the processor (350) may receive folding angle information from the foldable device through the communication module (330) while outputting a virtual image through the display module (310). For example, the foldable device may transmit folding angle information to the wearable device (300) in real time (or at a set period), or, when the folding angle information is transmitted upon initial connection and the folding angle changes to a value greater than a reference value, the changed folding angle information may be transmitted to the wearable device (300).
[0112] According to one embodiment, the processor (350) may determine a visual characteristic of at least one virtual object constituting a virtual image based on the received folding angle information of the foldable device. For example, the visual characteristic may include at least one of the size, position, or angle of the virtual object.
[0113] According to one embodiment, the processor (350) can change the angle of the virtual object in response to a change in the folding angle of the foldable device. For example, the processor (350) can change the angle of the virtual object based on the user's gaze direction and the vertical direction in response to a change in the folding angle. This embodiment will be described in more detail with reference to FIG. 9.
[0114] According to one embodiment, the processor (350) may determine the size of a virtual object based on the folding angle of the foldable device. For example, when the foldable device is in a fully folded state, the size of the virtual object may be determined as a first size, and when the folding angle increases according to a user's operation, the size of the virtual object may be changed to a second size greater than the first size. The processor (350) may increase or decrease the size of the virtual object in real time as the folding angle of the foldable device changes according to a user's operation. This embodiment will be described in more detail with reference to FIG. 10.
[0115] According to one embodiment, the processor (350) may determine the distance of a virtual object based on the folding angle of the foldable device. When the distance of the virtual object is determined, the processor (350) may render the virtual object so that the virtual object is perceived as being located at the determined distance from the user's line of sight. This embodiment will be described in more detail with reference to FIG. 11.
[0116] According to one embodiment, the processor (350) may set a coordinate system that defines the position of a virtual object within a virtual space. For example, the coordinate system may use a rectangular coordinate system or a spherical coordinate system, and may include variables (x, y, z) that constitute a rectangular coordinate system and a folding angle of the foldable device, or variables (r, φ, θ) that constitute a spherical coordinate system. The processor (350) may use a rectangular coordinate system or a spherical coordinate system depending on the range of the folding angle of the foldable device. In this document, the wearable device (300) is described as determining the position coordinates of a virtual object using a rectangular coordinate system (e.g., (a) of FIG. 12) and a spherical coordinate system (e.g., (b) of FIG. 12), but the wearable device (300) may also use another coordinate system, such as a cylindrical coordinate system.
[0117] According to one embodiment, the processor (350) can consistently arrange various virtual objects according to a set coordinate system (e.g., rectangular coordinate system, spherical coordinate system). For example, the wearable device (300) can align virtual objects at set intervals in a vertical direction based on the position of the recognized foldable device according to a rectangular coordinate system, or arrange them in the shape of a sphere surrounding the user according to a spherical coordinate system.
[0118] According to one embodiment, the processor (350) may move the position of the virtual object on the set coordinate system when the folding angle of the foldable device is changed. For example, when the folding angle of the foldable device is changed in a state set to a rectangular coordinate system, the processor (350) may move the virtual object in the x, y, or z direction, and when the folding angle of the foldable device is changed in a state set to a spherical coordinate system, the processor (350) may move the virtual object in the r, φ, or θ direction.
[0119] According to one embodiment, when the folding angle of the foldable device is within a first angle range, the processor (350) may determine the position of the virtual object in response to the folding angle on a rectangular coordinate system. Here, the first angle range may be 0 to 90 degrees, but is not limited thereto. When the folding angle changes within the first angle range, the processor (350) may move the virtual object in a predetermined linear direction (e.g., z direction) within the virtual image in response to the change in the folding angle. Accordingly, when the folding angle is increased in a fully folded state, it may be recognized that the virtual object is moved in a direction vertically upward with respect to the user's gaze direction. Alternatively, the processor (350) may determine the position of the virtual object using a rectangular coordinate system when the foldable device is in a flex mode and is being used at a predetermined angle.
[0120] According to one embodiment, when the folding angle of the foldable device is within a second angle range, the processor (350) may determine the position of the virtual object in response to the folding angle on a spherical coordinate system. Here, the second angle range may be, but is not limited to, 90 degrees to 180 degrees. When the folding angle changes within the second angle range, the processor (350) may move the virtual object in a predetermined curved direction within the virtual image in response to the change in the folding angle. Accordingly, when the folding angle of the foldable device increases in a fully unfolded state in flex mode, the processor (350) may move away from the foldable device along a circle formed around the foldable device. When the processor (350) provides a list of virtual objects by using a spherical coordinate system, the processor (350) may configure a UI field that matches the shape of the foldable device. This embodiment will be described in more detail with reference to FIG. 12.
[0121] According to one embodiment, the processor (350) may determine the number of virtual objects based on the folding angle of the foldable device. For example, if the virtual image includes a music player application or widget, the virtual image may provide a list of each track as a virtual object. If the folding angle of the foldable device is in a first angle range (e.g., 0 to 90 degrees), the processor (350) may arrange each track in the x, y, and z directions in a rectangular coordinate system, and may increase the number of provided tracks when the folding angle changes.
[0122] According to one embodiment, the processor (350) may determine the amount of information contained in a virtual object based on the folding angle of the foldable device. For example, when the folding angle of the foldable device increases to a second angle range (e.g., 90 degrees to 180 degrees), the processor (350) may place each track in a spherical coordinate system. In this case, for a virtual object (e.g., a track in a music list) placed closer to the user (or the foldable device), the title may be displayed in a large size, and additional information such as song lyrics, artist information, and image information may be displayed.
[0123] According to one embodiment, a virtual image may include one virtual object or multiple virtual objects. There is no limit to the number of virtual objects included in a virtual image.
[0124] According to one embodiment, if a virtual image includes a single virtual object, the processor (350) can automatically anchor the virtual object. For example, the wearable device (300) can specify the location of the virtual object so that the virtual object is placed adjacent to a specific object in the real environment. In this case, even if the user changes the direction of their gaze or moves to a different location, the wearable device (300) can render the virtual image so that the virtual object is displayed adjacent to the specific object in the real environment.
[0125] According to one embodiment, when a virtual image includes a plurality of virtual objects, the processor (350) may provide a user interface that allows selection of an individual object to be controlled according to a folding angle of the foldable device among the plurality of virtual objects. For example, the user interface may include an interface that allows selection of at least one virtual object of the virtual image to be controlled in response to an operation of the foldable device. The user interface may be provided in the form of a cue indicating a specific virtual object, or may be provided as a list of a plurality of virtual objects.
[0126] According to one embodiment, the processor (350) can anchor at least one virtual object selected via the user interface. In this case, when the folding angle of the foldable device changes, the visual characteristics of at least one virtual object selected via the user interface can change in response to the change in folding angle. Conversely, the visual characteristics of at least one non-selected virtual object can be maintained regardless of the change in folding angle. This embodiment will be described in more detail with reference to FIGS. 13 to 16.
[0127] According to one embodiment, the virtual image includes a plurality of virtual objects, and the plurality of virtual objects may be arranged on a single grid. For example, the processor (350) may form the grid to be parallel to a first region (or a first housing including the first region) of a flexible display of the foldable device. The processor (350) may anchor at least one virtual object on the grid. Each virtual object may be arranged at a position parallel to each other on the grid. According to one embodiment, the processor (350) may form the grid as a single plane or curved surface according to a set coordinate system. For example, when arranging virtual objects using a rectangular coordinate system, the grid may be formed as an xz plane having a specific y value based on the position of the foldable device (or the position of the user), and the grid may be moved in the y direction when the folding angle of the foldable device is changed. Alternatively, when arranging virtual objects using a spherical coordinate system, the grid can be formed as a spherical curved surface having a specific r value based on the position of the foldable device (or the user's position), and when the folding angle of the foldable device changes, the grid can be moved in the direction in which r increases. In this way, by arranging multiple virtual objects on the grid, the arrangement form is regular, thereby providing a visual effect and ease of operation for the user.
[0128] This embodiment will be described in more detail with reference to FIGS. 17 and 18.
[0129] Instructions for performing the operations of the wearable device (300) (or processor (350)) described above may be stored on a computer-readable recording medium. The recording medium may be tangible and non-transitory. The recording medium may store one or more computer programs including the instructions.
[0130] FIG. 6 is a flowchart of a method for rendering a virtual image of a wearable device according to one embodiment.
[0131] The illustrated method can be performed by a wearable device (e.g., the wearable device (300) of FIG. 5), and can be performed by, for example, a processor of the wearable device (e.g., the processor (350) of FIG. 5).
[0132] According to one embodiment, in operation 610, the wearable device can determine whether the wearable device is worn by the user. For example, the wearable device can determine that the wearable device is worn by the user when contact and / or proximity to the user's body is recognized through a contact sensor and / or a proximity sensor, when movement is detected by the user while the wearable device is worn through a motion sensor, and / or when the user's eyes are recognized through a camera module.
[0133] According to one embodiment, in operation 620, the wearable device can determine whether an adjacent foldable device (e.g., the foldable device (400) of FIG. 3) is recognized. For example, the wearable device can support short-range wireless communication (e.g., Bluetooth, WiFi) using a communication module (e.g., the communication module (330) of FIG. 5) and can recognize the foldable device based on a short-range wireless communication signal transmitted from the foldable device. Alternatively, the wearable device can obtain image information of the front corresponding to the user's gaze direction (or field of view) through a camera module (e.g., the camera module (320) of FIG. 5) and recognize a foldable device positioned adjacently through object recognition of the obtained image.
[0134] According to one embodiment, in operation 630, the wearable device may identify at least one virtual object constituting a virtual image. For example, the wearable device may determine a virtual environment (e.g., a virtual workspace of FIG. 4) to be provided when the wearable device is worn based on a default setting or a user's selection, and may identify at least one virtual object to be provided in a virtual image that implements the virtual environment. The virtual object may include an image having the shape of a real object, a widget (e.g., weather, calendar, clock), an avatar, etc., but there is no set type. The virtual object to be provided through the virtual image may be predetermined based on a default setting or a user's selection, and a new virtual object may be added and / or a previously provided virtual object may be removed based on a user's selection while the virtual image is displayed.
[0135] According to one embodiment, in operation 640, the wearable device may link status information of the foldable device with a virtual object. Here, the status information may include location information of the foldable device and / or folding angle information of the foldable device. For example, the wearable device may recognize the current location of the foldable device and determine a location to render a virtual object based on the recognized location. Furthermore, the wearable device may determine visual characteristics (e.g., size, distance, angle) of at least one virtual object based on folding angle information received from the foldable device.
[0136] The detailed operation of the operation (640) of connecting the status information of the foldable device and the virtual object will be described in more detail with reference to FIG. 7.
[0137] According to one embodiment, in operation 650, the wearable device may render a virtual image including a virtual object. The wearable device may render the virtual image so that it can be recognized by the user's gaze, reflecting the visual characteristics of the virtual object determined based on the status information of the foldable device.
[0138] FIG. 7 is a flowchart of a method for a wearable device to configure a virtual image using information of a foldable device according to one embodiment.
[0139] The illustrated method can be performed by a wearable device (e.g., the wearable device (300) of FIG. 5), and can be performed, for example, by a processor of the wearable device (e.g., the processor (350) of FIG. 5). FIG. 7 may include detailed operations of an operation (640) of connecting state information of the foldable device of FIG. 6 and a virtual object.
[0140] According to one embodiment, in operation 710, the wearable device may receive folding angle information of the foldable device from the foldable device (e.g., the foldable device (400) of FIG. 3). For example, the foldable device may include a flexible display including a first region and a second region that are distinguished based on a folding axis, and may include an angle sensor that senses an angle between the first region and the second region and / or an angle between a first housing including the first region and a second housing including the second region. The foldable device may transmit folding angle information sensed using the angle sensor to the wearable device via short-range wireless communication. The foldable device may transmit the folding angle information to the wearable device in real time (or at a set interval), or, if the folding angle changes to a reference value or more after transmitting the folding angle information at the time of initial connection, the changed folding angle information may be transmitted to the wearable device.
[0141] According to one embodiment, in operation 720, the wearable device may collect location information of the foldable device. For example, the wearable device may determine the location of the foldable device through object recognition on a signal received from the foldable device via short-range wireless communication and / or an image acquired via a camera module (e.g., the camera module (320) of FIG. 5 ).
[0142] According to one embodiment, in operation 730, the wearable device may determine information about the foldable device to be applied to the virtual object. For example, the wearable device may determine visual characteristics, such as the location at which the virtual object is to be placed on the virtual image, the size of the virtual object, and the angle, based on folding angle information and / or position information received from the foldable device.
[0143] According to one embodiment, in operation 740, the wearable device may configure a layout including various virtual objects within a virtual space. According to one embodiment, the wearable device may configure a layout to be used for arranging virtual objects according to the type of the virtual image or the virtual object included in the virtual image. For example, if the virtual object is a 2D object, the wearable device may arrange each virtual object on the x, y, and z coordinates in a rectangular coordinate system, and move the virtual object in a direction in which any one of the x, y, and z values increases / decreases according to a change in the folding angle of the foldable device. According to one embodiment, if the virtual image including the virtual object provides a 3D environment in a panoramic format, the wearable device may arrange each virtual object based on a cylindrical coordinate system, and if the virtual image provides a VR environment, the wearable device may arrange each virtual object based on a spherical coordinate system. According to one embodiment, the wearable device may use a rectangular coordinate system or a spherical coordinate system, and may match the folding angle of the foldable device with variables (x, y, z) constituting the rectangular coordinate system or variables (r, φ, θ) constituting the spherical coordinate system. When the folding angle of the foldable device is changed, at least one of the coordinate (x, y, z) values of the virtual object may be changed when the rectangular coordinate system is set, and at least one of the coordinate (r, φ, θ) values of the virtual object may be changed when the spherical coordinate system is set.
[0144] According to one embodiment, in operation 750, the wearable device may generate at least one virtual object to be placed within a virtual space. The virtual object to be provided within the virtual image may be predetermined, and the wearable device may determine visual characteristics of the predetermined virtual object based on folding angle information and / or position information of the foldable device.
[0145] According to one embodiment, in operation 760, the wearable device may render a virtual object. The wearable device may render the virtual object so that it is recognizable to the user's gaze based on visual characteristics determined based on folding angle information and / or position information of the foldable device.
[0146] FIG. 8 is a flowchart of a method for providing a virtual image of a wearable device and a foldable device according to one embodiment.
[0147] According to one embodiment, in operation 810, a wearable device (e.g., wearable device (300) of FIG. 5) may request status information of the foldable device from the foldable device. For example, the wearable device may be connected to the foldable device via short-range wireless communication (e.g., Bluetooth, WiFi) and request status information from the foldable device. The status information may include folding angle information of the foldable device.
[0148] According to one embodiment, in operation 815, the foldable device may transmit status information of the foldable device to the wearable device in response to a request from the wearable device. The foldable device may include an angle sensor and transmit folding angle information determined through the angle sensor to the wearable device.
[0149] According to one embodiment, in operation 820, the wearable device may define a reference position of the foldable device using a camera module (e.g., the camera module (320) of FIG. 3). For example, the wearable device may recognize the foldable device in a real image of the front acquired using the camera module (or front camera), and define the recognized position of the foldable device as a reference position for anchoring virtual objects included in the virtual image.
[0150] According to one embodiment, in operation 825, the wearable device may generate a virtual space based on the reference position and state information of the foldable device. For example, the wearable device may anchor at least one virtual object constituting a virtual image adjacent to the recognized foldable device.
[0151] According to one embodiment, in operation 830, the wearable device may generate and render each virtual object based on status information of the foldable device. For example, the wearable device may determine visual characteristics (e.g., angle, distance, location) of the virtual object based on folding angle information of the foldable device.
[0152] According to one embodiment, in operation 835, the wearable device may merge at least one virtual object generated within a virtual image. For example, the virtual image may be rendered by composing a virtual image by merging at least one generated virtual object.
[0153] According to one embodiment, in operation 840, the foldable device can determine the occurrence of a change event of a folding angle according to a user's operation.
[0154] According to one embodiment, in operation 845, the foldable device may transmit status information of the foldable device to the wearable device in response to the occurrence of the event. The foldable device may include an angle sensor that detects a folding angle, and when a change in the folding angle is detected through the angle sensor, information about the changed folding angle may be transmitted to the wearable device via short-range wireless communication.
[0155] According to one embodiment, in operation 850, the wearable device may generate and render a virtual object based on the received status information of the foldable device. For example, the wearable device may change the visual characteristics (e.g., angle, distance, position) of the virtual object according to the folding angle, and render a virtual image including the virtual object with the changed visual characteristics.
[0156] According to one embodiment, in operation 855, the wearable device may merge at least one virtual object generated within the virtual image.
[0157] FIG. 9 illustrates an example of a wearable device according to one embodiment controlling the angle of a virtual image according to the folding angle of the foldable device.
[0158] In FIG. 9, the foldable device (400) is a foldable device (400) existing in a real environment, and the virtual object is an image output by a display module (e.g., a display module (310) of FIG. 5) of a wearable device (e.g., a wearable device (300) of FIG. 5) and can be recognized as being adjacent to (or overlapping with) the foldable device (400) in the real environment in the user's field of vision.
[0159] According to one embodiment, a wearable device (e.g., a processor (350) of FIG. 5) may determine an angle at which at least one virtual object among virtual objects included in a virtual image is displayed based on a folding angle of the foldable device (400). Here, the folding angle may be an angle between a first region (411) and a second region (412) of the flexible display, and the foldable device (400) may sense the folding angle using an angle sensor and transmit the sensed folding angle to the wearable device.
[0160] According to one embodiment, the wearable device can maintain the angle of the virtual object without changing it when the folding angle of the foldable device (400) is within a first angle range (e.g., 0 to 90 degrees), and can maintain the angle of the virtual object as if the foldable device (400) is in a fully folded state or is being held in the user's hand. The wearable device can change the angle of the virtual object in response to a change in the folding angle when the folding angle is within a second angle range (e.g., 90 to 180 degrees). Here, examples of the first angle range and the second angle range are not limited thereto.
[0161] Referring to FIG. 9, when the folding angle of the foldable device (400) is 0 degrees, i.e., in a fully folded state, the wearable device can arrange the virtual object (910) so that it is perpendicular to the user's gaze direction. When the folding angle of the foldable device (400) increases according to the user's operation, the angle of the virtual object (910) can be maintained until the folding angle reaches 90 degrees, which is within the first angle range. When the folding angle of the foldable device (400) further increases to A2, which is within the second angle range, the wearable device can change the angle of the virtual object (920) so that it appears to be tilted by A2, which is the same as the folding angle, based on the user's gaze direction. When the folding angle of the foldable device (400) further increases to A3, which is close to 180 degrees, the wearable device can change the angle of the virtual object (930) so that it appears to be tilted by A3.
[0162] Accordingly, the user can manipulate the folding angle of the foldable device (400) to place a virtual object at a desired angle within the virtual image.
[0163] FIG. 10 illustrates an example of a wearable device according to one embodiment controlling the size of a virtual image according to a folding angle of a foldable device.
[0164] In FIG. 10, the foldable device (400) is a foldable device (400) existing in a real environment, and the virtual object is an image output by the display module of the wearable device, and can be recognized as being adjacent to (or overlapping with) the foldable device (400) in the real environment in the user's field of vision.
[0165] According to one embodiment, a wearable device (e.g., the processor (350) of FIG. 5) may determine the size of at least one virtual object among virtual objects included in a virtual image based on the folding angle of the foldable device (400). The wearable device may increase or decrease the size of the virtual object in real time as the folding angle of the foldable device (400) changes by a user's operation.
[0166] Referring to FIG. 10, when the foldable device (400) is in a fully folded state, the wearable device can determine the size of the virtual object as a first size and render a virtual object (1010) of the first size. When the folding angle of the foldable device (400) increases to A2 by a user's operation, the wearable device can gradually increase the size of the virtual object (1020) from the first size to a second size in response to the increase in the folding angle. When the folding angle of the foldable device (400) increases to A3 by a user's operation and enters a fully unfolded state, the wearable device can gradually increase the size of the virtual object (1030) to a third size that is larger than the second size.
[0167] FIG. 11 illustrates an example of a wearable device according to one embodiment controlling the distance of a virtual image according to a folding angle of a foldable device.
[0168] In FIG. 11, the foldable device (400) is a foldable device (400) existing in a real environment, and the virtual object is an image output by the display module of the wearable device (300) (e.g., the display module (310) of FIG. 5) and can be recognized as being adjacent to (or overlapping with) the foldable device (400) in the real environment in the user's field of vision.
[0169] According to one embodiment, the wearable device (300) (e.g., the processor (350) of FIG. 5) may determine the distance of at least one virtual object among virtual objects included in a virtual image based on the folding angle of the foldable device (400). When the distance of the virtual object is determined, the wearable device (300) may render the virtual object so that the virtual object can be recognized as being located at the determined distance from the user's line of sight.
[0170] Referring to FIG. 11, the wearable device (300) can position a virtual object (1110) at a predetermined position when the foldable device (400) is in a fully folded state. For example, the position of the virtual object can be determined as a position adjacent to the upper end of the flexible display of the foldable device (400). When the folding angle of the foldable device (400) increases from A2 to A3 to A4 to A5 by a user's manipulation, the wearable device (300) can position the virtual object (1120 - 1130 - 1140 - 1150) away from the user in response to the increase in the folding angle. The wearable device (300) can render the virtual object so that the user can recognize that the virtual object is positioned at a distance determined according to the folding angle.
[0171] FIG. 12 illustrates an example of a wearable device according to one embodiment configuring position coordinates of a virtual object according to a folding angle of a foldable device.
[0172] According to one embodiment, a wearable device (e.g., the wearable device (300) of FIG. 5) may set a coordinate system that defines the position of a virtual object within a virtual space. For example, the coordinate system may use a rectangular coordinate system or a spherical coordinate system, and may include a folding angle of the foldable device (400) and variables (x, y, z) that constitute a rectangular coordinate system or variables (r, φ, θ) that constitute a spherical coordinate system. The processor may use a rectangular coordinate system or a spherical coordinate system depending on the range of the folding angle of the foldable device (400).
[0173] According to one embodiment, the wearable device can determine the position of a virtual object in response to the folding angle of the foldable device (400) in a rectangular coordinate system when the folding angle is within a first angle range. Here, the first angle range may be from 0 degrees to 90 degrees, but is not limited thereto.
[0174] Referring to (a) of FIG. 12, when the foldable device (400) is in a fully folded state, the wearable device may generate a virtual object (1210) and display it adjacent to the foldable device (400). For example, the wearable device may display the virtual object in the z direction of the foldable device (400). When the folding angle increases to A2 within the first angle range according to the user's operation, the wearable device may move the position of the virtual object (1220) in the z direction and increase the size of the virtual object in response to the folding angle. When the folding angle increases to A3 according to the user's operation, the wearable device may move the position of the virtual object (1230) in the z direction and increase the size of the virtual object in response to the folding angle.
[0175] According to one embodiment, the wearable device can determine the position of a virtual object in response to the folding angle of the foldable device (400) in a spherical coordinate system when the folding angle is within a second angle range. Here, the second angle range may be, but is not limited to, 90 degrees to 180 degrees. When the folding angle changes within the second angle range, the processor can move the virtual object in a predetermined curved direction within the virtual image in response to the change in the folding angle.
[0176] Referring to (b) of FIG. 12, when the folding angle is A4, which is within the second angle range, the wearable device can arrange the virtual object along a circle centered on the folding axis of the foldable device (400). When the folding angle increases to A5 according to the user's operation, the wearable device can move the position of the virtual object further along the circle and increase the size of the virtual object in response to the folding angle.
[0177] This document describes that a wearable device determines the position coordinates of a virtual object using a rectangular coordinate system (e.g., (a) of FIG. 12) and a spherical coordinate system (e.g., (b) of FIG. 12), but the wearable device may also use other coordinate systems, such as a cylindrical coordinate system.
[0178] FIG. 13 is a flowchart of a method for determining a virtual object to be controlled by a wearable device according to one embodiment.
[0179] The illustrated method can be performed by a wearable device (e.g., the wearable device (300) of FIG. 5), and can be performed by, for example, a processor of the wearable device (e.g., the processor (350) of FIG. 5).
[0180] According to one embodiment, in operation 1310, the wearable device may determine whether the wearable device is worn by the user.
[0181] According to one embodiment, when the wearable device is confirmed to be in a wearing state, the wearable device may recognize an adjacent foldable device and attempt to connect with the foldable device. If the wearable device is a device that provides AR (Augmented Reality) (1315-AR), in operation 1320, the wearable device may recognize the foldable device through object recognition of an image acquired by a camera. Alternatively, if the wearable device is a device that provides AR (Virtual Reality) (1315-VR), in operation 1325, the wearable device may recognize the foldable device based on a short-range wireless communication signal transmitted from the foldable device.
[0182] According to one embodiment, in operation 1330, the wearable device can determine whether a virtual object provided through a virtual image exists. For example, the wearable device can determine a virtual environment (e.g., a virtual workspace of FIG. 4) to be provided when the wearable device is worn based on a default setting or a user's selection, and can identify at least one virtual object to be provided in a virtual image that implements the virtual environment. The virtual image may include only one virtual object, or may include two or more virtual objects.
[0183] According to one embodiment, in operation 1335, the wearable device can check whether the number of virtual objects (N) in the virtual image is 1. If the number of virtual objects in the virtual image is 1, in operation 1340, the wearable device can automatically anchor the virtual object. For example, the wearable device can specify the location of the virtual object so that the virtual object is placed adjacent to a specific object in the real environment. In this case, even if the user changes the direction of his / her gaze or moves to a different location, the wearable device can render the virtual image so that the virtual object is displayed adjacent to the specific object in the real environment. According to one embodiment, the wearable device can provide the user with information indicating that the virtual object has been automatically anchored in the form of a dialog.
[0184] According to one embodiment, when the number of virtual objects in the virtual image is two or more, in operation 1345, the wearable device can determine whether all of the plurality of virtual objects are controllable. Here, whether or not they are controllable may mean whether or not their visual characteristics are to be controlled in response to an operation of the foldable device. The wearable device can determine whether or not to control all of the plurality of virtual objects in response to an operation of the foldable device based on properties of the plurality of virtual objects and / or a user's selection.
[0185] In one embodiment, when a plurality of virtual objects are all controlled, in operation 1350, the wearable device may anchor the plurality of virtual objects. In this case, when the folding angle of the foldable device changes, the visual characteristics (e.g., angle, size, distance) of all of the plurality of virtual objects may change in response to the change in the folding angle.
[0186] According to one embodiment, if the plurality of virtual objects are not all controllable, in operation 1355, the wearable device may provide a user interface for selecting individual objects to be controlled. For example, the user interface may include an interface for selecting at least one of the plurality of virtual objects of the virtual image to be controlled in response to manipulation of the foldable device. An example of the user interface will be described in more detail with reference to FIGS. 15 and 16 .
[0187] According to one embodiment, at operation 1360, the wearable device may anchor multiple virtual objects. In this case, when the folding angle of the foldable device changes, the visual characteristics of at least one virtual object selected through the user interface may change in response to the change in folding angle. Conversely, the visual characteristics of at least one unselected virtual object may be maintained regardless of the change in folding angle.
[0188] FIG. 14 illustrates an example of a wearable device according to one embodiment controlling a virtual object according to a folding angle of a foldable device.
[0189] In FIG. 14, the foldable device (400) is a foldable device (400) existing in an actual environment, and a wearable device (e.g., the wearable device (300) of FIG. 5) can render a virtual image so that a virtual environment is placed around the foldable device (400) in the user's line of sight. FIG. 14 illustrates a user interface that can determine visual characteristics to be controlled depending on a folding angle when the virtual image includes one virtual object.
[0190] According to one embodiment, a wearable device can identify a virtual object to be provided through a virtual image. Referring to (a) of FIG. 14, the virtual image may include a UX screen (1411), a virtual desk (1412), and a single virtual object (1415). In FIG. 14, the virtual object (1415) is illustrated as an example of a flower pot image, but is not limited thereto.
[0191] According to one embodiment, the wearable device may recognize the location of the foldable device (400) and anchor a virtual object around the foldable device (400) based on object recognition of an image acquired from a camera and / or a short-range wireless communication signal transmitted from the foldable device (400). Referring to (b) of FIG. 14, a virtual object (1415) may be anchored adjacent to the foldable device (400).
[0192] According to one embodiment, the wearable device may receive folding angle information of the foldable device (400) from the foldable device (400) and change visual characteristics of the virtual object (1415) according to the folding angle information. Here, the visual characteristics may include at least one of distance, height, position, or size.
[0193] According to one embodiment, the wearable device may provide a user interface (1450) that can determine a visual characteristic to be controlled based on the folding angle of the foldable device (400). Referring to FIG. 14 (c), the user interface (1450) may include a menu that can select at least one of a distance, a height, a position, and a size that can be controlled based on the folding angle. The user may select any one of the menus on the user interface (1450) based on a change in gaze position, a hand gesture, and / or a voice input.
[0194] In one embodiment, the wearable device may change a selected visual characteristic of a virtual object when the folding angle changes, if any one of the selected visual characteristics is selected through a menu of the user interface (1450). For example, if the user selects a size through the user interface (1450), the size of the virtual object (1416) may increase in response to an increase in the folding angle of the foldable device (400), as illustrated in (d) of FIG. 14 .
[0195] FIG. 15 illustrates an example of a wearable device according to one embodiment controlling a virtual object according to a folding angle of a foldable device.
[0196] In Fig. 15, the foldable device (400) is a foldable device (400) existing in an actual environment, and the wearable device can render a virtual image so that a virtual environment is placed around the foldable device (400) in the user's line of sight. Fig. 15 illustrates an interface for selecting at least one virtual object whose visual characteristics will be changed according to a folding angle when the virtual image includes multiple virtual objects.
[0197] According to one embodiment, the wearable device may provide a user interface for selecting at least one virtual object whose visual characteristics will be changed according to the folding angle of the foldable device (400), when the virtual image includes multiple virtual objects. For example, the user interface for selecting at least one virtual object may include a selectable cue indicating a specific virtual object, or may include a list including multiple virtual objects.
[0198] Referring to (a) of FIG. 15, a virtual image may include three virtual objects (1511, 1512, 1513). The user interface may provide cues (1521, 1522) indicating specific virtual objects. For example, a cue (1521) indicating a third virtual object (1513) may be displayed, and when the user's gaze direction changes, a hand gesture, a voice input, or a predetermined time elapses, the virtual object indicated by the cue may change to a first virtual object (1511) or a second virtual object (1512). When the user inputs a predetermined input while the cue (1522) is provided to the first virtual object (1511), the wearable device may select the corresponding virtual object. The wearable device may anchor the selected virtual object and provide the user with information that the corresponding virtual object has been selected through a dialog.
[0199] Referring to (b) of FIG. 15, the user interface may be provided in the form of a list (1530) containing multiple virtual objects. The user may select at least one virtual object from the list (1530) by changing the direction of the gaze, making a hand gesture, or using a voice input. The wearable device may anchor the selected virtual object.
[0200] The wearable device may thus allow the user to select at least one virtual object through the user interface, thereby removing a specific virtual object from the field of view and / or reconfiguring the layout of the virtual objects.
[0201] FIG. 16 illustrates an example of a wearable device according to one embodiment controlling a virtual object according to a folding angle of a foldable device.
[0202] In Fig. 16, the foldable device (400) is a foldable device (400) existing in an actual environment, and the wearable device can render a virtual image so that a virtual environment is placed around the foldable device (400) in the user's line of sight. Fig. 16 illustrates an interface that allows selection of at least one virtual object whose visual characteristics will be changed according to a folding angle when the virtual image includes multiple virtual objects.
[0203] According to one embodiment, a wearable device can identify a virtual object to be provided through a virtual image. Referring to (a) of FIG. 16, the virtual image may include a UX screen (1611), a virtual desk (1612), and multiple virtual objects (1621, 1622, 1623, 1624). In FIG. 16, the multiple virtual objects (1621, 1622, 1623, 1624) include, but are not limited to, a news widget, a weather widget, a clock widget, and a message widget, respectively.
[0204] According to one embodiment, the wearable device may verify that there are multiple virtual objects (1621, 1622, 1623, 1624) included in a virtual image, and provide a user interface that allows selection of at least one virtual object among the multiple virtual objects (1621, 1622, 1623, 1624) for controlling visual characteristics according to a folding angle of the foldable device (400).
[0205] Referring to (b) of FIG. 16, a cue indicating a clock widget (1623) may be displayed. Thereafter, based on a user input (e.g., a change in gaze direction, a hand gesture, or a voice input), the cue may change to indicate a weather widget (1622), as shown in (c) of FIG. 16. According to one embodiment, the wearable device may determine the order of virtual objects to be indicated by the cue based on the order of closest distance to the user or the order of the user's usage log.
[0206] Referring to (d) of FIG. 16, when a weather widget (1622) is selected, the wearable device can anchor the selected virtual object adjacent to the foldable device (400).
[0207] According to one embodiment, the wearable device may change the visual characteristics of at least one virtual object anchored according to a user's selection among a plurality of virtual objects in response to a change in the folding angle of the foldable device (400). Referring to (e) of FIG. 16, when the folding angle of the foldable device (400) increases, the wearable device may increase the size of the selected weather widget (1632) and / or move its position upward. In this case, other virtual objects (1621, 1623, 1624) that are not selected may be displayed as is without changing their visual characteristics.
[0208] FIG. 17 illustrates an example of a wearable device according to one embodiment controlling a grid including virtual objects according to a folding angle of a foldable device.
[0209] According to one embodiment, the virtual image includes a plurality of virtual objects, and the plurality of virtual objects can be arranged on a grid.
[0210] Referring to (a) of FIG. 17, a grid (1710) constituting one plane may be formed to correspond to a folding angle of the foldable device (400). For example, the grid (1710) may be formed to be parallel to a first area (or a first housing including the first area) of a flexible display of the foldable device (400). The wearable device may anchor at least one virtual object (1711, 1712) on the grid (1710). Each virtual object (1711, 1712) may be arranged at a position parallel to each other on the grid (1710). The grid (1710) illustrated in FIG. 17 may not be displayed on a virtual image, and only at least one virtual object (1711, 1712) included in the grid (1710) may be displayed.
[0211] According to one embodiment, the wearable device can change the visual characteristics of a plurality of virtual objects arranged on a grid according to a change in the folding angle of the foldable device (400). Referring to FIG. 17 (b), when the folding angle of the foldable device (400) increases according to a user's operation, the angle of the grid (1720) can also change according to the folding angle. Accordingly, the angles of each virtual object (1721, 1722) arranged on the grid (1720) can also change according to the folding angle.
[0212] In one embodiment, the foldable device (400) serves as a reference point for the work board, allowing the user to set an interaction area. The angle of the grid can be adjusted based on the first area of the flexible display of the foldable device (400), and the user can form the grid and adjust its angle while using the foldable device (400) in flex mode.
[0213] FIG. 18 illustrates an example of a wearable device according to one embodiment configuring a grid including virtual objects according to the orientation of the foldable device.
[0214] According to one embodiment, the wearable device can determine the arrangement direction of the grid including at least my object based on the mounting direction of the foldable device (400). For example, the foldable device (400) can be mounted on the floor with one side of the housing in a state where the flexible display is only partially folded, such as in flex mode, and in this case, the direction in which the flexible display faces can be mounted in various directions based on the user's viewing direction.
[0215] Referring to (a) of FIG. 18, a user may position the flexible display of the foldable device (400) in a direction from which the user is viewing. In this case, a grid (1710) including at least one object (1711, 1712) may be positioned in front from the direction of the user's gaze.
[0216] Referring to (b) of FIG. 18, a user may position the flexible display of the foldable device (400) in a vertical direction relative to the direction in which the user is looking. In this case, a grid (1720) including at least one object (1721, 1722) may be positioned to the side relative to the direction in which the user is looking.
[0217] Referring to (a) of FIG. 18, a user may place one side of the housing of the foldable device (400) sideways rather than on the floor. In this case, the wearable device may place both the grid (1710) on the front and the grid (1720) on the side in the direction of the user's gaze.
[0218] An electronic device according to various embodiments of the present document may include a display for outputting a virtual image, a camera for acquiring a real image including an external subject, a communication circuit, a memory, and a processor operatively connected to the display, the camera, the communication circuit, and the memory.
[0219] According to one embodiment, the memory may store instructions that are executable by at least one processor and, when executed, cause the electronic device to recognize the foldable device based on at least one of a real image including the external foldable device acquired by the camera or a signal received from the foldable device via the communication circuit.
[0220] According to one embodiment, the memory may store instructions that cause the electronic device to receive folding angle information of the foldable device from the foldable device through the communication circuit, determine visual characteristics of at least one virtual object constituting the virtual image based on the received folding angle information, and render the virtual image including the at least one virtual object in which the determined visual characteristics are reflected, and output the virtual image through the display.
[0221] According to one embodiment, the memory may store instructions that cause the electronic device to determine a location of the at least one virtual object within the virtual image based on location information of the foldable device.
[0222] According to one embodiment, the memory may store instructions that cause the electronic device to change a visual characteristic of the at least one virtual object based on changed folding angle information when changed folding angle information is received from the foldable device.
[0223] According to one embodiment, the visual characteristic may include at least one of a size, a position, or an angle of the virtual object.
[0224] According to one embodiment, the memory may store instructions that cause the electronic device to move the virtual object in a predetermined straight direction within the virtual image in response to a change in the folding angle when the folding angle of the foldable device is changed within a first angle range, and to move the virtual object in a curved direction within the virtual image in response to a change in the folding angle when the folding angle of the foldable device is changed within a second angle range greater than the first angle range.
[0225] According to one embodiment, the memory may store instructions that cause the electronic device to determine a position of the virtual object in correspondence with the folding angle on a rectangular coordinate system when the folding angle of the foldable device is within the first angle range, and to determine a position of the virtual object in correspondence with the folding angle on a spherical coordinate system when the folding angle of the foldable device is within the second angle range.
[0226] According to one embodiment, the memory may store instructions that cause the electronic device to change an angle of the virtual object relative to a user's gaze direction and a vertical direction in response to a change in a folding angle of the foldable device.
[0227] According to one embodiment, the memory may store instructions that cause the electronic device to identify a location of the foldable device on a real image acquired from the camera and to form the virtual image such that the virtual object is placed adjacent to the identified location of the foldable device.
[0228] According to one embodiment, the memory may store instructions that cause the electronic device to determine, when the virtual image includes a plurality of virtual objects, at least one of the plurality of virtual objects whose visual characteristics are to be changed according to a folding angle of the foldable device.
[0229] According to one embodiment, the memory may store instructions that cause the electronic device to provide a UI through the display for selecting at least one of the plurality of virtual objects whose visual characteristics will be changed according to a folding angle of the foldable device.
[0230] According to one embodiment, the foldable device includes a flexible display including a first region and a second region that are distinguished based on a folding axis, and the folding angle information may include an angle between the first region and the second region.
[0231] In one embodiment, the virtual image may implement augmented reality or virtual reality.
[0232] In one embodiment, the virtual image may include a virtual work space, and the virtual object may include a widget.
[0233] A method for providing a virtual image of an electronic device according to various embodiments of the present document may include an operation of recognizing a foldable device based on at least one of a real image including an external foldable device acquired from a camera or a signal received from the foldable device, an operation of receiving folding angle information of the foldable device from the foldable device, an operation of determining a visual characteristic of at least one virtual object constituting the virtual image based on the received folding angle information, and an operation of rendering and outputting the virtual image including the at least one virtual object to which the determined visual characteristic is reflected.
[0234] According to one embodiment, the method may further include an operation of determining a position of the at least one virtual object within the virtual image based on position information of the foldable device.
[0235] According to one embodiment, the method may further include an operation of changing a visual characteristic of the at least one virtual object based on changed folding angle information when changed folding angle information is received from the foldable device.
[0236] According to one embodiment, the visual characteristic may include at least one of a size, a position, or an angle of the virtual object.
[0237] According to one embodiment, the operation of determining a visual characteristic of the at least one virtual object may include an operation of moving the virtual object in a predetermined straight direction within the virtual image in response to a change in the folding angle when the folding angle of the foldable device is changed within a first angle range, and an operation of moving the virtual object in a curved direction within the virtual image in response to a change in the folding angle when the folding angle of the foldable device is changed within a second angle range greater than the first angle range.
[0238] According to one embodiment, the operation of determining a visual characteristic of the at least one virtual object may include an operation of changing an angle of the virtual object relative to a direction perpendicular to a user's gaze direction in response to a change in a folding angle of the foldable device.
[0239] According to one embodiment, the method may further include, when the virtual image includes a plurality of virtual objects, an operation of determining at least one of the plurality of virtual objects to change a visual characteristic according to a folding angle of the foldable device.
[0240] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.
[0241] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.
[0242] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).
[0243] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.
[0244] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.
[0245] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.
Claims
1. In electronic devices, A display that outputs virtual images; A camera that acquires real-world images including external subjects; communication circuit; memory; and At least one processor operatively connected to the display, the camera, the communication circuit, and the memory; The above memory is executable by at least one processor, and when executed, causes the electronic device to: Recognizing the foldable device based on at least one of a real image including an external foldable device acquired from the camera or a signal received from the foldable device through the communication circuit, Through the above communication circuit, folding angle information of the foldable device is received from the foldable device, Based on the received folding angle information, determining a visual characteristic of at least one virtual object constituting the virtual image, and An electronic device storing instructions for rendering the virtual image including at least one virtual object reflecting the determined visual characteristics and outputting the virtual image through the display.
2. In paragraph 1, The above memory, the electronic device, An electronic device storing instructions for determining a position of at least one virtual object within the virtual image based on position information of the foldable device.
3. In paragraph 1 or 2, The above memory, the electronic device, An electronic device storing instructions for changing a visual characteristic of at least one virtual object based on changed folding angle information when changed folding angle information is received from the foldable device.
4. In any one of paragraphs 1 to 3, The above visual characteristics are, An electronic device comprising at least one of the size, position or angle of the virtual object.
5. In any one of paragraphs 1 to 4, The above memory, the electronic device, When the folding angle of the above foldable device is changed within a first angle range, the virtual object is moved in a predetermined straight line direction within the virtual image in response to the change in the folding angle, and An electronic device storing instructions for moving the virtual object in a curved direction within the virtual image in response to a change in the folding angle when the folding angle of the foldable device is changed in a second angle range greater than the first angle range.
6. In any one of paragraphs 1 to 5, The above memory, the electronic device, An electronic device storing instructions for changing the angle of the virtual object based on a user's gaze direction and a vertical direction in response to a change in the folding angle of the foldable device.
7. In any one of paragraphs 1 to 6, The above memory, the electronic device, An electronic device storing instructions for identifying a location of the foldable device on a real image acquired from the camera and forming the virtual image so that the virtual object is placed adjacent to the identified location of the foldable device.
8. In any one of paragraphs 1 to 7, The above memory, the electronic device, An electronic device storing instructions for determining at least one of the plurality of virtual objects to change a visual characteristic according to a folding angle of the foldable device when the virtual image includes a plurality of virtual objects.
9. In any one of paragraphs 1 to 8, The above foldable device includes a flexible display including a first region and a second region separated based on a folding axis, An electronic device wherein the folding angle information includes an angle between the first region and the second region.
10. In any one of paragraphs 1 to 9, An electronic device wherein the virtual image includes a virtual work space, and the virtual object includes a widget.
11. In a method for providing a virtual image of an electronic device, An operation of recognizing a foldable device based on at least one of a real image including an external foldable device acquired from a camera or a signal received from the foldable device; An operation of receiving folding angle information of the foldable device from the foldable device; An operation of determining a visual characteristic of at least one virtual object constituting the virtual image based on the received folding angle information; and A method comprising an action of rendering and outputting the virtual image including the at least one virtual object in which the determined visual characteristics are reflected.
12. In paragraph 11, A method further comprising an action of determining a position of at least one virtual object within the virtual image based on position information of the foldable device.
13. In paragraph 11 or 12, A method further comprising an operation of changing a visual characteristic of the at least one virtual object based on the changed folding angle information when changed folding angle information is received from the foldable device.
14. In any one of paragraphs 11 to 13, The above visual characteristics are, A method comprising at least one of a size, position or angle of the virtual object.
15. In any one of paragraphs 11 to 14, The operation of determining a visual characteristic of at least one virtual object is: When the folding angle of the foldable device is changed within a first angle range, an operation of moving the virtual object in a predetermined straight line direction within the virtual image in response to the change in the folding angle; and A method comprising an action of moving the virtual object in a curved direction within the virtual image in response to a change in the folding angle when the folding angle of the foldable device is changed in a second angle range greater than the first angle range.
Citation Information
Patent Citations
Method and Apparatus for Processing Virtual World
KR101835413B1
The apparatus and method for display system displaying augmented reality image
KR1020150096947A
Gas injection system and apparatus for atomic layer deposition includinf the system
KR1020210108042A
Salivary gland and lymph gland complex stimulation device
KR102769974B1
KR20230054227A
