Method and apparatus for determining operating command of controller

Abstract

According to the electronic device of an embodiment, when it is determined that an abnormality has occurred in a target controller among a plurality of controllers registered in the electronic device, graphical function visibility corresponding to the target controller is displayed based on an abnormal state of the target controller, a user gesture with respect to the displayed graphical function visibility is detected using a camera, and the graphical function visibility is displayed based on the detected user gesture. And determining an operation command for the target controller indicated by the detected user gesture.

Description

Technical Field

[0001] The following description relates to a method for determining the operating commands of a controller. Background Technology

[0002] Recently, virtual reality, augmented reality, and mixed reality technologies that utilize computer graphics have been under development. In this context, virtual reality technology refers to the technology of using computers to construct virtual spaces that do not exist in the real world and making users feel that the virtual space is real, while augmented reality or mixed reality technology refers to the technology of expressing information by overlaying computer-generated information onto the real world, that is, the technology of allowing real-time interaction between users and the system by combining the real world and the virtual world.

[0003] Among these technologies, augmented reality and mixed reality are being integrated with and used in various technological fields, such as broadcasting, medical technology, and gaming. A representative example of augmented reality's application in broadcasting is the natural changing of weather maps in front of a weather forecaster on television, or the insertion of advertising images that are not actually present in a stadium onto the broadcast screen as if they were truly there during a sports broadcast.

[0004] Representative services that provide users with augmented or mixed reality include the metaverse. The metaverse is a compound word meaning both the virtual or abstract "meta" and the real world "universe," referring to a three-dimensional virtual world. The metaverse is a more advanced concept than the traditional term "virtual reality environment" and provides augmented reality environments in which virtual worlds, such as networks and the internet, are integrated into the real world. Summary of the Invention

[0005] Technical solution

[0006] According to an embodiment, an electronic device includes: a camera for capturing user gestures; a memory for storing computer-executable instructions; and a processor for accessing the memory and executing the instructions, wherein the instructions, when executed, cause the processor to: when it is determined that a target controller among a plurality of controllers registered with the electronic device is abnormal, display a graphical affordance corresponding to the target controller based on the abnormal state of the target controller; detect a user gesture for the displayed graphical affordance using the camera; and determine an operation command for the target controller indicated by the detected user gesture.

[0007] According to an embodiment, a method performed by an electronic device includes: when it is determined that a target controller among a plurality of controllers registered with the electronic device is abnormal, displaying a graphical function visibility corresponding to the target controller based on the abnormal state of the target controller; detecting a user gesture in response to the displayed graphical function visibility using a camera; and determining an operation command for the target controller indicated by the detected user gesture. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating exemplary configurations of electronic devices according to various embodiments.

[0009] Figure 2 Examples of optical spectroscopy (OST) devices according to various embodiments are shown.

[0010] Figure 3 Examples of optical systems for eye-tracking (ET) cameras, transparent components, and displays according to various embodiments are shown.

[0011] Figure 4a and Figure 4b This is a diagram illustrating examples of front and rear views of an electronic device according to various embodiments.

[0012] Figure 5 Examples of the construction of virtual spaces according to various embodiments and of inputs from and outputs to users within the virtual space are shown.

[0013] Figure 6 This is a flowchart schematically illustrating the process of determining operational commands for the controller according to various embodiments.

[0014] Figure 7 This illustrates a scenario where a user uses multiple controllers according to various embodiments.

[0015] Figure 8 This is a diagram illustrating a situation where an electronic device, according to various embodiments, detects an operational error of a button included in the controller.

[0016] Figure 9a and Figure 9b This is a diagram illustrating the process by which an electronic device displays the visibility of a graphical function corresponding to the controller when it detects an operational error of a button included in the controller, according to various embodiments.

[0017] Figure 10 This is a diagram illustrating the process by which an electronic device, according to various embodiments, displays a graphical function with an increased size for the controller.

[0018] Figure 11This is a diagram illustrating the process of tracking a user's hand when an electronic device detects an operational error of a tracking sensor included in a controller, according to various embodiments.

[0019] Figure 12 This is a diagram illustrating the process by which the electronic device displays the visibility of a graphical function corresponding to the controller when the connection between the controller and the electronic device is terminated or when the controller is separated from the user's hand, according to various embodiments.

[0020] Figure 13 This is a diagram illustrating the process by which an electronic device, according to various embodiments, changes the visibility of a graphical function displayed corresponding to a controller. Detailed Implementation

[0021] In the following description, embodiments will be described in detail with reference to the accompanying drawings. When describing embodiments with reference to the accompanying drawings, the same reference numerals refer to the same elements, and repeated descriptions related to them will be omitted.

[0022] Figure 1 This is a block diagram illustrating exemplary configurations of electronic devices according to various embodiments.

[0023] Figure 1 This is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. Reference Figure 1 In network environment 100, electronic device 101 can communicate with electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or with at least one of electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, memory 130, input module 150, sound output module 155, display module 160, audio module 170, sensor module 176, interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, user identification module (SIM) 196, or antenna module 197. In some embodiments, at least one component (e.g., connection terminal 178) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some of the components (e.g., sensor module 176, camera module 180, or antenna module 197) may be integrated into a single component (e.g., display module 160).

[0024] Processor 120 can execute, for example, software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of electronic device 101 connected to processor 120, and can perform various data processing or calculations. According to embodiments, as at least part of data processing or calculation, processor 120 can store commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the commands or data stored in volatile memory 132, and store the resulting data in non-volatile memory 134. According to embodiments, processor 120 may include a main processor 121 (e.g., central processing unit (CPU) or application processor (AP)) or an auxiliary processor 123 (e.g., graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor central processor, or communication processor (CP)) operating independently of or in conjunction with main processor 121. For example, when electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or to be adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or may be implemented as part of the main processor 121.

[0025] When the main processor 121 is inactive (e.g., in sleep) state, the auxiliary processor 123 (rather than the main processor 121) can control at least some of the functions or states associated with at least one of the components of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190), or when the main processor 121 is active (e.g., running an application), the auxiliary processor 123 can work with the main processor 121 to control at least some of the functions or states associated with at least one of the components of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190). According to embodiments, the auxiliary processor 123 (e.g., ISP or CP) can be implemented as part of another component (e.g., camera module 180 or communication module 190) functionally associated with the auxiliary processor 123. According to embodiments, the auxiliary processor 123 (e.g., NPU) can include hardware architecture specifically for artificial intelligence model processing. Artificial intelligence models can be generated through machine learning. Machine learning can be performed by, for example, an electronic device 101 that performs AI, or via a separate server (e.g., server 108). Learning algorithms can include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model can include multiple layers of artificial neural networks. Artificial neural networks can include, but are not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), deep Q-networks, or combinations of two or more of these. The artificial intelligence model may additionally or alternatively include software structures in addition to hardware structures.

[0026] Memory 130 may store various data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various data may include, for example, software (e.g., program 140) and input or output data for commands associated with it. Memory 130 may include volatile memory 132 or non-volatile memory 134.

[0027] Program 140 may be stored as software in memory 130 and may include, for example, an operating system (OS) 142, middleware 144, or application 146.

[0028] Input module 150 can receive commands or data from outside electronic device 101 (e.g., a user) to be used by another component of electronic device 101 (e.g., processor 120). Input module 150 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).

[0029] The audio output module 155 can output audio signals to the outside of the electronic device 101. The audio output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or playing recordings. The receiver can be used to receive incoming calls. According to embodiments, the receiver can be implemented separately from the speaker or as part of the speaker.

[0030] Display module 160 (e.g., a display) can visually provide information to the outside of electronic device 101 (e.g., a user). Display module 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a respective one of the display, holographic device, and projector. According to an embodiment, display module 160 may include a touch sensor adapted to detect touch or a pressure sensor adapted to measure the intensity of the force caused by touch.

[0031] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or an external electronic device (e.g., electronic device 102, such as a speaker or headphones) directly or wirelessly connected to the electronic device 101.

[0032] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and generate electrical signals or data values ​​corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a posture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0033] Interface 177 may support one or more specified protocols for direct (e.g., wired) or wireless coupling between electronic device 101 and external electronic device (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, or an audio interface.

[0034] Connection terminal 178 may include a connector via which electronic device 101 can be physically connected to an external electronic device (e.g., electronic device 102). According to an embodiment, connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] The haptic module 179 can convert electrical signals into mechanical stimuli (e.g., vibration or motion) or electrical stimuli that can be recognized by a user through their touch or kinesthesia. According to embodiments, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0036] Camera module 180 can capture still images and moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an ISP, or a flash.

[0037] The power management module 188 can manage the power supply to the electronic device 101. According to an embodiment, the power management module 188 can be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0038] Battery 189 can power at least one component of electronic device 101. According to embodiments, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable rechargeable battery, or a fuel cell.

[0039] Communication module 190 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. Communication module 190 may include one or more CPs that can operate independently of processor 120 (e.g., AP) and support direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 198 (e.g., a short-range communication network, such as Bluetooth). TM The wireless communication module 192 can communicate with external electronic devices 104 via a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a fifth-generation (5G) network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components separate from each other (e.g., multiple chips). The wireless communication module 192 can use user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the SIM 196 to identify and authenticate electronic devices 101 within a communication network (such as a first network 198 or a second network 199).

[0040] Wireless communication module 192 can support 5G networks beyond fourth-generation (4G) networks, as well as next-generation communication technologies such as New Radio (NR) access technology. NR access technology can support enhanced mobile broadband (eMBB), massive machine-type communication (mMTC), or ultra-reliable and low-latency communication (URLLC). Wireless communication module 192 can support high-frequency bands (e.g., millimeter-wave bands) to achieve, for example, high data transmission rates. Wireless communication module 192 can support various technologies used to ensure performance in high-frequency bands, such as beamforming, massive MIMO, full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive MIMO. Wireless communication module 192 can support various requirements specified in electronic device 101, external electronic device (e.g., electronic device 104), or network system (e.g., second network 199). According to an embodiment, the wireless communication module 192 may support peak data rates (e.g., 20 Gbps or higher) for implementing eMBB, lost coverage (e.g., 164 dB or lower) for implementing mMTC, or user plane (U plane) latency (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for round trip) for implementing URLLC.

[0041] Antenna module 197 can transmit or receive signals or power to or from the outside of electronic device 101 (e.g., external electronic device). According to an embodiment, antenna module 197 may include an antenna comprising a radiating element comprising conductive material or conductive patterns formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, antenna module 197 may include multiple antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as a first network 198 or a second network 199) can be selected from the multiple antennas by, for example, communication module 190. Signals or power can be transmitted or received between communication module 190 and external electronic device via at least one selected antenna. According to an embodiment, another component besides the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally incorporated into antenna module 197.

[0042] According to various embodiments, antenna module 197 can form a millimeter-wave antenna module. According to embodiments, the millimeter-wave antenna module may include: a PCB; an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the PCB and capable of supporting a specified high-frequency band (e.g., millimeter-wave band); and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top or side surface) of the PCB and capable of transmitting or receiving signals in a specified high-frequency band.

[0043] At least some of the aforementioned components may be coupled to each other and transmit signals (e.g., commands or data) between them via a peripheral communication scheme (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)).

[0044] According to an embodiment, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 coupled to second network 199.

[0045] Each of the external electronic devices 102 and 104 and the server 108 may be a device of the same or different type as electronic device 101. According to embodiments, all or some of the operations performed by electronic device 101 may be performed by one or more external electronic devices (e.g., external electronic devices 102 and 104 and server 108). For example, if electronic device 101 needs to automatically perform a function or service, or in response to a request from a user or another device, electronic device 101 may request one or more external electronic devices to perform at least a portion of that function or service, rather than performing that function or service, or other than performing that function or service. Upon receiving the request, one or more external electronic devices may perform at least a portion of the requested function or service, or perform additional functions or services related to the request, and may send the results of the execution to electronic device 101. Electronic device 101 may provide the results, with or without further processing, as at least part of a response to the request. In this disclosure, the following example will be described primarily: Electronic device 101 is an augmented reality (AR) device (e.g., Figure 2 Electronic equipment 201 Figure 3 Electronic devices 301 or Figure 4b The electronic device 401, as well as external electronic devices 102 and 104 and server 108, send the results of executing the virtual space and the additional functions or services associated with the virtual space to the electronic device 101.

[0046] Server 108 may include processor 181, communication module 182, and memory 183. Processor 181, communication module 182, and memory 183 may be configured similarly to processor 120, communication module 190, and memory 130 of electronic device 101. For example, processor 181 can provide interaction between a virtual space and a user within the virtual space by executing instructions stored in memory 183. Processor 181 can generate at least one of visual, auditory, or tactile information about the virtual space and objects within it. For example, as visual information, processor 181 can generate rendering data (e.g., visual rendering data) obtained by rendering the appearance (e.g., shape, size, color, or texture) of the virtual space and the appearance (e.g., shape, size, color, or texture) of objects located in the virtual space. Additionally, processor 181 can generate rendering data that renders interactions or changes (e.g., changes in object appearance, the appearance of sound, or the appearance of touch) between objects (e.g., physical objects, virtual objects, or avatar objects) in virtual space based on one or more user inputs to objects (e.g., physical objects, virtual objects, or avatar objects). Communication module 182 can establish communication with a user's first electronic device (e.g., electronic device 101) and another user's second electronic device (e.g., electronic device 102). Communication module 182 can send at least one of the aforementioned visual, tactile, or auditory information to the first and second electronic devices. For example, communication module 182 can send rendering data.

[0047] For example, after rendering content data executed by the application, server 108 can send the content data to electronic device 101, and electronic device 101 receiving the data can output the content data to display module 160. If electronic device 101 detects user movement via an inertial measurement unit (IMU) sensor or the like, processor 120 of electronic device 101 can correct the rendered data received from external electronic device 102 based on the movement information and output the data to display module 160. Alternatively, the processor can send movement information to server 108 to request rendering, causing screen data to be updated accordingly. However, embodiments are not limited to this, and rendering can be performed by various types of external electronic devices (e.g., 102 and 104), such as smartphones or housing devices for storing electronic device 101 and charging electronic device 101. Rendered data corresponding to virtual space generated by external electronic devices 102 and 104 can be provided to electronic device 101. In another example, electronic device 101 can receive virtual space information (e.g., defining the vertex coordinates, texture, and color of the virtual space) and object information (e.g., defining the vertex coordinates, texture, and color of the object's appearance) from server 108, and perform rendering based on the received data.

[0048] Figure 2 Examples of optical spectroscopy (OST) devices according to various embodiments are shown.

[0049] Electronic device 201 may include a display (e.g., Figure 1 The electronic device 201, which has a transparent display and provides images through the transparent display, may be referred to as an OST device. (The device includes at least one of a display module 160), a vision sensor, light sources 230a and 230b, an optical element, or a substrate.)

[0050] For example, the display may include a liquid crystal display (LCD), a digital mirror device (DMD) or liquid crystal on silicon (LCoS), an organic light-emitting diode (OLED), or a micro light-emitting diode (micro LED).

[0051] In one embodiment, when the display is one of LCD, DMD, or LCoS, the electronic device 201 may include light sources 230a and 230b that emit light to the screen output areas of the display (e.g., screen display portions 215a and 215b). In another embodiment, when the display is capable of generating its own light, for example, when the display is an OLED or micro-LED, the electronic device 201 can provide the user with a virtual image of relatively high quality even without separate light sources 230a and 230b. In one embodiment, when the display is implemented as an OLED or micro-LED, the light source 230a or 230b may be optional, thus reducing the weight of the electronic device 201.

[0052] refer to Figure 2 The electronic device 201 may include a display, a first transparent member 225a, and / or a second transparent member 225b. A user can use the electronic device 201 while wearing it on their face. The first transparent member 225a and / or the second transparent member 225b may be formed of glass, plastic, or polymer, and may be transparent or translucent. According to an embodiment, the first transparent member 225a may be positioned facing the user's right eye, and the second transparent member 225b may be positioned facing the user's left eye. The display may include a first display 205 configured to output a first image (e.g., a right image) corresponding to the first transparent member 225a, and a second display 210 configured to output a second image (e.g., a left image) corresponding to the second transparent member 225b. In an embodiment, when each of the displays is transparent, the displays and transparent members may be positioned facing the user's eyes to configure screen display portions 215a and 215b.

[0053] In an embodiment, light emitted from display 205 or 210 can be guided by a waveguide through input optics 220a or 220b. Light moving into the display waveguide can be guided through output optics (e.g., ...). Figure 3 The output optical component 340 is guided toward the user's eye. The screen display portions 215a and 215b can be determined based on the light emitted toward the user's eye.

[0054] For example, light emitted from displays 205 and 210 can be reflected from the grating regions of the waveguides formed in input optical components 220a and 220b and screen display portions 215a and 215b, and can be transmitted to the user's eyes.

[0055] Optical elements may include at least one of a lens or an optical waveguide.

[0056] The lens can adjust the focus so that the output to the display screen is visible to the user's eyes. The lens can include at least one of, for example, a Fresnel lens, a pancake lens, or a multi-channel lens.

[0057] Optical waveguides can transmit the light rays of an image generated by a display to the user's eyes. For example, the image rays can represent light rays emitted by light sources 230a or 230b and passing through the screen output area of ​​the display. Optical waveguides can be formed of glass, plastic, or polymer. Optical waveguides may include nanopatterns formed on a portion of the inner surface or a portion of the outer surface, such as polygonal or curved grating structures. See below for reference. Figure 3 An exemplary structure for describing a waveguide.

[0058] A vision sensor may include at least one of a camera or a depth sensor.

[0059] The first camera 265a or 265b can be a recognition camera, and can be a camera used for 3-DOF or 6DoF head tracking, hand detection, hand tracking, and spatial recognition. The first cameras 265a and 265b can primarily consist of global shutter (GS) cameras. Since head tracking and spatial recognition require stereo cameras, the first cameras 265a and 265b can include two or more GS cameras. GS cameras can have superior performance compared to rolling shutter (RS) cameras in detecting and tracking fine movements, such as rapid movements of the hand or fingers. For example, GS cameras can have low image blur. The first cameras 265a and 265b can capture image data for simultaneous localization and mapping (SLAM) functions via 6DoF depth capture and spatial recognition. Additionally, user pose recognition functions can be performed based on the image data captured by the first cameras 265a and 265b.

[0060] The second cameras 270a and 270b, serving as eye-tracking (ET) cameras, can be used to capture image data for detecting and tracking the user's pupils. (See below for reference.) Figure 3 Describe the second cameras 270a and 270b.

[0061] The third camera 245 can be a camera used for image capture. The third camera 245 may include a high-resolution (HR) camera to capture HR images or photo-video (PV) images. The third camera 245 may include a color camera with features for obtaining high-quality images, such as autofocus (AF) and optical image stabilization (OIS). The third camera 245 can be a GS camera or an RS camera.

[0062] Fourth camera (for example, below) Figure 4b The facial recognition cameras (425 and 426) are facial recognition cameras, and the facial tracking (FT) cameras can be used to detect and track a user's facial expressions.

[0063] A depth sensor (not shown) can be a sensor configured to sense information such as time-of-flight (TOF) for determining the distance to an object. TOF is a technique that uses signals (e.g., near-infrared, ultrasonic, laser, etc.) to measure the distance to an object. A TOF-based depth sensor can send a signal from a transmitter, measure the signal in a receiver, and measure the TOF of the signal.

[0064] Light source 230a or 230b (e.g., an illumination module) may include elements (e.g., LEDs) that emit light of various wavelengths. The illumination module can be attached to various locations depending on the purpose. For example, a first illumination module (e.g., an LED element) attached to the periphery of the frame of an AR glasses device may emit light to assist gaze detection when eye movement is tracked by an ET camera. The first illumination module may include, for example, an IR LED of an infrared wavelength. For example, a second illumination module (e.g., an LED element) may be attached to a camera mounted on the periphery of a bridge connecting the frames to each other or on the periphery of hinges 240a or 240b connecting the frames to the temples. When the camera captures an image, the second illumination module may emit light to supplement ambient light. The second illumination module may emit light when objects are not easily detected in dark environments.

[0065] The substrate 235a or 235b (e.g., PCB) can support the aforementioned components.

[0066] A PCB can be mounted on the temples of the eyeglasses. A flexible PCB (FPCB) can transmit electrical signals to each module (e.g., a camera, display, audio module, and sensor module) and another PCB. In an embodiment, at least one PCB may include a first substrate, a second substrate, and an interposer layer disposed between the first and second substrates. For example, the PCB may be mounted on the central portion of the kit. Electrical signals can be transmitted via the FPCB to each module and another PCB.

[0067] Other components may include at least one of, for example, multiple microphones (e.g., first microphone 250a, second microphone 250b and third microphone 250c), multiple speakers (e.g., first speaker 255a and second speaker 255b), battery 260, antenna or sensor (e.g., accelerometer, gyroscope, touch sensor, etc.).

[0068] Figure 3 Examples of optical systems for ET cameras, transparent components, and displays according to various embodiments are shown.

[0069] Figure 3 This is a diagram illustrating the operation of an ET camera included in an electronic device according to an embodiment. Figure 3 The following process is illustrated: the ET camera 310 of the electronic device 301 according to the embodiment (e.g., Figure 2 The first ET camera 270a and the second ET camera 270b) use a display 320 (e.g., Figure 2 The light (e.g., infrared light) output by the first display 205 and the second display 210 tracks the user's eye 309 (e.g., the user's gaze).

[0070] Second camera (e.g., Figure 2 The second cameras 270a and 270b can be an ET camera 310, which collects information for locating the center of a virtual image projected onto the electronic device 301 based on the direction of the wearer's pupil gaze. The second camera may also include a GS camera to detect the pupil and track rapid pupil movement. ET cameras can be mounted for both the right and left eyes, and ET cameras with the same camera performance and specifications can be used. The ET camera 310 may include an eye gaze tracking sensor 315. The ET sensor 315 may be included inside the ET camera 310. Infrared light output from the display 320 can be transmitted to the user's eye 309 as reflected infrared light 303 via a semi-reflective mirror. The ET sensor 315 can detect the transmitted infrared light 305 generated when the reflected infrared light 303 is reflected from the user's eye 309. The ET camera 310 can track the user's eye 309, i.e., the user's gaze, based on the detection results of the ET sensor 315.

[0071] Display 320 may include multiple visible light pixels and multiple infrared pixels. Visible light pixels may include R pixels, G pixels, and B pixels. Visible light pixels can output visible light corresponding to a virtual object image. Infrared pixels can output infrared light. Display 320 may include, for example, micro-LEDs or OLEDs.

[0072] Show waveguide 350 and ET waveguide 360 ​​can be included in transparent member 370 (e.g., Figure 2 In the first transparent member 225a and the second transparent member 225b). The transparent member 370 can be formed, for example, a glass plate, a plastic plate or a polymer, and can be formed transparently or translucently. The transparent member 370 can be positioned facing the user's eyes. In this case, the distance between the transparent member 370 and the user's eyes 309 can be referred to as the "eye relief" 380.

[0073] The transparent component 370 may include a waveguide 350 and an ET waveguide 360. The transparent component 370 may include an input optical component 330 and an output optical component 340. Furthermore, the transparent component 370 may include an ET beam splitter 375 that splits the input light into several waveguides.

[0074] According to an embodiment, light incident on one end of the display waveguide 350 can propagate within the display waveguide 350 via a nanopattern and can be provided to a user. Alternatively, the display waveguide 350, formed by a free-form prism, can provide the incident light as image light to the user via a reflector. The display waveguide 350 may include at least one of a diffractive element (e.g., a diffractive optical element (DOE) or a holographic optical element (HOE)) or a reflective element (e.g., a reflector). The display waveguide 350 can use at least one of the diffractive or reflective elements included in the display waveguide 350 to guide display light (e.g., image light) emitted from a light source to the user's eye. For reference, although... Figure 3 The output optics 340 is shown to be separate from the ET waveguide 360, but the output optics 340 may be included in the ET waveguide 360.

[0075] According to an embodiment, the diffraction element may include an input optical component 330 and an output optical component 340. For example, the input optical component 330 may refer to, for example, an "input grating region." The output optical component 340 may refer to, for example, an "output grating region." The input grating region may serve as an input end for diffracting (or reflecting) light output from a light source (e.g., a micro-LED) to transmit light to transparent components (e.g., a first transparent component and a second transparent component) of the screen display portion. The output grating region may serve as an outlet for diffracting (or reflecting) light transmitted to the transparent components (e.g., the first transparent component and the second transparent component) of the waveguide to the user's eye.

[0076] According to embodiments, the reflective element may include a total internal reflection waveguide or a total internal reflection optical element for total internal reflection (TIR). For example, as one of the schemes for inducing light, a TIR may form an incident angle such that light entering through the input grating region (e.g., a virtual image) is completely reflected from a surface (e.g., a specific surface) of the waveguide to completely transmit the light to the output grating region.

[0077] In this embodiment, light emitted from display 320 can be guided by a waveguide through input optics 330. The light traveling in the waveguide can then be directed toward the user's eye via output optics 340. The portion of the screen display can be determined based on the light emitted toward the user's eye.

[0078] Figure 4a and Figure 4b This is a diagram illustrating examples of front and rear views of an electronic device according to various embodiments. Figure 4a This can be the appearance of the electronic device 401 as viewed from the first direction ①. Figure 4b This can refer to the appearance of the electronic device 401 as viewed from the second direction ②. When a user wears the electronic device 401, it can... Figure 4b The image shows the appearance as seen by the user's eyes.

[0079] Reference Figure 4a According to various embodiments, electronic device 401 (e.g., Figure 1 Electronic devices 101 Figure 2 Electronic devices 201 or Figure 3 Electronic devices 301) may provide users with services that offer extended reality (XR) experiences. For example, XR or XR services may be defined as services collectively referred to as virtual reality (VR), augmented reality (AR), and / or mixed reality (MR).

[0080] According to an embodiment, electronic device 401 may refer to a head-mounted device or head-mounted display (HMD) worn on a user's head, but may be provided in the form of at least one of glasses, goggles, a helmet, or a hat. Electronic device 401 may include types such as OST type, which is configured such that external light reaches the user's eyes through the glasses when worn, or video see-through (VST) type, which is configured such that light emitted from the display reaches the user's eyes when worn, but external light is blocked from reaching the user's eyes.

[0081] According to an embodiment, electronic device 401 can be worn on a user's head and provide the user with images related to XR services. For example, electronic device 401 can provide XR content (hereinafter also referred to as XR content images) that is output such that at least one virtual object appears to be overlaid in a display area or an area defined as the user's field of view (FoV). According to an embodiment, XR content may refer to images related to real space obtained by a camera (e.g., an image capture camera) or images or videos in which at least one virtual object is added to virtual space. According to an embodiment, electronic device 401 may be based on the functions performed by electronic device 401 and / or by external electronic devices (e.g., Figure 1 Electronic devices 102 and 104 and Figure 1 At least one or more external electronic devices in server 108 are performing functions to provide XR content.

[0082] According to an embodiment, electronic device 401 may be at least partially composed of external electronic devices (e.g., Figure 1 The device may be controlled by an electronic device 102 or 104, or may perform at least one function under the control of an external electronic device, or may perform at least one function independently.

[0083] refer to Figure 4a The vision sensor may be disposed on a first surface of the housing of the main body 410 of the electronic device 401. The vision sensor may include a camera (e.g., second functional cameras 411 and 412 and a first functional camera 415) and / or a depth sensor 417 for obtaining information relating to the surrounding environment of the electronic device 401.

[0084] In an embodiment, second functional cameras 411 and 412 can acquire images relating to the surrounding environment of the electronic device 401. A first functional camera 415 can acquire images using a wearable electronic device worn by a user. The first functional camera 415 can be used for hand detection and tracking, and for recognizing the user's posture (e.g., hand gestures). The first functional camera 415 can be used for 3DoF and 6DoF head tracking, location (space, environment) recognition, and / or motion recognition. In an embodiment, second functional cameras 411 and 412 can also be used for hand detection and tracking, and for recognizing the user's posture.

[0085] In an embodiment, depth sensor 417 may be configured to send signals and receive signals reflected from an object, and may be used to determine the distance to the object based on Time-of-Flight (TOF). Alternatively or additionally, cameras 411, 412, and 415 may replace depth sensor 417 in determining the distance to the object.

[0086] Reference Figure 4b The facial recognition cameras 425 and 426 and / or the display 421 (and / or the lens) may be disposed on the second surface 420 of the housing of the main body 410.

[0087] In an embodiment, facial recognition cameras 425 and 426 adjacent to the display may be used to identify the user’s face, or to identify and / or track the user’s eyes.

[0088] In an embodiment, a display 421 (and / or a lens) may be disposed on a second surface 420 of the electronic device 401. In an embodiment, the electronic device 401 may not include some of the plurality of cameras 415. Although in Figure 4a and Figure 4b Not shown, but electronic device 401 may also include Figure 2 At least one of the components shown.

[0089] According to an embodiment, the electronic device 401 may include: a main body 410 on which a... Figure 1 At least some of the components; display 421 (e.g., Figure 1The main body 410 includes a display module 160, which is disposed in a first direction ①; a first functional camera 415 (e.g., a recognition camera), which is disposed in a second direction ②; second functional cameras 411 and 412 (e.g., image capture cameras), which are disposed in the second direction ②; a third functional camera 428 (e.g., an ET camera), which is disposed in the first direction ①; fourth functional cameras 425 and 426 (e.g., a face recognition camera), which are disposed in the first direction ①; a depth sensor 417, which is disposed in the second direction ②; and a touch sensor 413, which is disposed in the second direction ②. Although not shown in the figures, the main body 410 may include a memory (e.g., Figure 1 The memory 130) and the processor (e.g., Figure 1 The processor 120, and may also include Figure 1 Other components shown.

[0090] According to an embodiment, the display 421 may include an LCD, a DMD, an LCoS device, an OLED, or a micro LED.

[0091] In one embodiment, when the display 421 is one of an LCD, DMD, or LCoS device, the electronic device 401 may include a light source that emits light to the screen output area of ​​the display 421. In another embodiment, when the display 421 is capable of generating light itself, for example, when the electronic device 401 is formed of one of an OLED or a micro-LED, the electronic device 401 can provide the user with XR content images of relatively high quality even without a separate light source. In one embodiment, when the display 421 is implemented as an OLED or a micro-LED, a light source may be unnecessary, which may result in a lighter electronic device 401.

[0092] According to embodiments, the display 421 may include a first transparent member 421a and / or a second transparent member 421b. A user can use the electronic device 401 while it is worn on the face. The first transparent member 421a and / or the second transparent member 421b may be formed of a glass plate, a plastic plate, or a polymer, and may be transparent or translucent. According to embodiments, the first transparent member 421a may be positioned facing the user's left eye in a third direction ③, and the second transparent member 421b may be positioned facing the user's right eye in a fourth direction ④. According to various embodiments, when the display 421 is transparent, the display 421 may be positioned facing the user's eyes to form a display area.

[0093] According to an embodiment, display 421 may include a lens comprising a transparent waveguide. The lens can be used to adjust the focus so that the screen output to display 421 (e.g., an XR content image) will be viewed by the user's eyes. For example, light emitted from the display panel can pass through the lens and be transmitted to the user via the waveguide formed within the lens. The lens may include, for example, a Fresnel lens, a pancake lens, or a multi-channel lens.

[0094] An optical waveguide (e.g., a waveguide) can be used to transmit light generated by display 421 to a user's eye. The optical waveguide can be formed of glass, plastic, or polymer, and can have nanopatterns formed on an inner surface or a portion of an outer surface, such as polygonal or curved grating structures. According to an embodiment, light incident on one end of the optical waveguide (i.e., the output image of display 421) can propagate within the optical waveguide to be provided to the user. Alternatively, an optical waveguide formed of a free-form prism can provide incident light to the user via a mirror. The optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE) and a holographic optical element (HOE)) or at least one reflective element (e.g., a mirror). The optical waveguide can guide the image output from display 421 to the user's eye using at least one diffractive or reflective element included in the optical waveguide.

[0095] According to an embodiment, the diffraction element may include an input optics component / output optics component (not shown). For example, the input optics component may refer to an input grating region, and the output optics component (not shown) may refer to an output grating region. The input grating region may serve as an input end, which diffracts (or reflects) light emitted from a light source (e.g., a micro-LED) to transmit the light to transparent components (e.g., a first transparent component 421a and a second transparent component 421b) in the display area. The output grating region may serve as an outlet, which diffracts (or reflects) light transmitted to the transparent components (e.g., the first and second transparent components) of the optical waveguide to the user's eye.

[0096] According to embodiments, the reflective element may include a TIR optics element or a TIR waveguide for TIR. For example, as a scheme for guiding light, the TIR can generate an incident angle such that light input through the input grating region (e.g., a virtual image) will be reflected substantially 100% from one surface (e.g., a specific side) of the optical waveguide, and the light will be transmitted substantially 100% to the output grating region.

[0097] In this embodiment, light emitted from display 421 can be guided to the optical path of the waveguide via input optics. Light traveling inside the waveguide can then be guided towards the user's eye via output optics. The display area can be determined based on the light emitted in the direction of the eye.

[0098] According to an embodiment, the electronic device 401 may include a plurality of cameras. For example, the cameras may include a first functional camera 415 (e.g., a recognition camera) disposed in a second direction ② of the main body 410, second functional cameras 411 and 412 (e.g., image capture cameras) disposed in the second direction ②, a third functional camera 425 (e.g., an ET camera) disposed in the first direction ①, and a fourth functional camera 425 and 426 (e.g., a face recognition camera) disposed in the first direction ①, and may also include other functional cameras (not shown).

[0099] A first functional camera 415 (e.g., a recognition camera) can be used to detect user movement or recognize user posture. The first functional camera 415 can support at least one of head tracking, hand detection and tracking, and spatial recognition. For example, the first functional camera 415 can primarily use a GS camera, which has superior performance compared to an RS camera, to detect and track fine gestures or movements of the hand and fingers, and can be configured as a stereo camera including two or more GS cameras for head tracking and spatial recognition. The first functional camera 415 can perform functions such as 6DoF spatial recognition and SLAM functions for recognizing information (e.g., position and / or orientation) associated with the surrounding space through depth imaging.

[0100] Secondary cameras 411 and 412 (e.g., image capture cameras) can be used to capture external images, generate corresponding images or videos, and send them to a processor (e.g., Figure 1 The processor 120 can display images provided by the second function cameras 411 and 412 on the display 421. The second function cameras 411 and 412 may also be referred to as HR or PV cameras, and may include HR cameras. For example, the second function cameras 411 and 412 may be color cameras equipped with functions for obtaining high-quality images, such as AF and OIS, but are not limited thereto. The second function cameras 411 and 412 may also include GS cameras or RS cameras.

[0101] A third-function camera 425 (e.g., an ET camera) can be mounted on the display 421 (or inside the main unit) such that the camera lens faces the user's eyes when the user wears the electronic device 401. The third-function camera 425 can be used to detect and track the pupil (e.g., ET). The processor can verify the gaze direction by tracking the movement of the user's left and right eyes in the images received from the third-function camera 425. By tracking the position of the pupil in the image, the processor can be configured to center the XR content image displayed on the display area according to the gaze direction of the pupil. For example, the third-function camera 425 can use a GS camera to detect the pupil and track its movement. The third-function camera 425 can be mounted for each of the left and right eyes and can have the same camera performance and specifications.

[0102] When a user wears electronic device 401, fourth functional cameras 425 and 426 (e.g., facial recognition cameras) can be used to detect and track the user's facial expressions (e.g., FT).

[0103] According to an embodiment, electronic device 401 may include an illumination unit (e.g., an LED) (not shown) as an auxiliary device for a camera. For example, a third functional camera 425 may use an illumination unit included in a display as an auxiliary device for facilitating gaze detection when tracking eye movements, to direct emitted light (e.g., an IR LED of IR wavelength) toward the user's eyes. In another example, second functional cameras 411 and 412 may also include an illumination unit (e.g., a flash) as an auxiliary device for supplementing ambient brightness when capturing external images.

[0104] According to an embodiment, depth sensor 417 (or depth camera) can be used to verify the distance to an object (e.g., a target) via, for example, Time-of-Flight (TOF). TOF is a technique that uses a signal (e.g., near-infrared, ultrasonic, or laser) to measure the distance to an object. It can transmit a signal from a transmitter, then measure the signal via a receiver, and the distance to the object can be measured based on the TOF signal.

[0105] According to an embodiment, the touch sensor 413 can be disposed on the body 410 in a second direction ②. For example, when a user wears the electronic device 401, the user's eyes can view the device in a first direction ①. The touch sensor 413 can be implemented as a single type or a left / right separated type based on the shape of the body 410, but is not limited thereto. For example, when the touch sensor 413 is implemented as... Figure 4aIn the case of the left / right separation type shown, when the user wears the electronic device 401, the first touch sensor 413a can be arranged in the third direction ③ at a position corresponding to the user's right eye, and the second touch sensor 413b can be arranged in the fourth direction ④ at a position corresponding to the user's left eye.

[0106] Touch sensor 413 can identify touch input using at least one of, for example, capacitive, resistive, infrared, or ultrasonic methods. For example, touch sensor 413 using a capacitive method can identify physical touch (or contact) input or hover (or proximity) input from an external object. According to some embodiments, electronic device 401 can use a proximity sensor (not shown) to identify proximity to an external object.

[0107] According to an embodiment, the touch sensor 413 may have a two-dimensional (2D) surface and send touch data (e.g., touch coordinates) of an external object (e.g., a user's finger) that contacts the touch sensor 413 to the processor 120. The touch sensor 413 may detect hover input from an external object (e.g., a user's finger) that approaches within a first distance from the touch sensor 413, or detect touch input that contacts the touch sensor 413.

[0108] In an embodiment, when an external object touches the touch sensor 413, the touch sensor 413 can provide the processor 120 with two-dimensional information about the contact point as "touch data". The touch data can be described as a "touch pattern". When an external object is within a first distance from the touch sensor 413 (or hovering above the proximity sensor or touch sensor), the touch sensor 413 can provide the processor 120 with hover data about the time or position of the external object hovering around the touch sensor 413. The hover data can also be described as a "hover pattern / proximity pattern".

[0109] According to an embodiment, electronic device 401 may use at least one of touch sensor 413, proximity sensor (not shown) and / or depth sensor 417 to obtain hover data to generate information about the distance, position or time point between touch sensor 413 and an external object.

[0110] According to an embodiment, the main body 410 may include a processor (e.g., Figure 1 The processor 120) and memory (e.g., Figure 1 (Memory 130).

[0111] Memory 130 may store various instructions that can be executed by a processor. Instructions may include control instructions that can be recognized by the processor, such as arithmetic and logical operations, data movement, or input / output. Memory may include volatile memory (e.g., Figure 1volatile memory 132) and non-volatile memory (e.g., Figure 1 Non-volatile memory 134) is used to temporarily or permanently store various data.

[0112] The processor may be operatively, functionally, and / or electrically connected to each of the components of the electronic device 401 to perform control and / or communication-related calculations or data processing for each of the components. Operations performed by the processor may be stored in memory and, when executed, may be performed by instructions that cause the processor to operate.

[0113] While there are no restrictions on the computational and data processing functions implemented by the processor on electronic device 401, a series of operations related to XR content service functions will be described below. The processor operations described below can be executed by executing instructions stored in memory.

[0114] According to an embodiment, the processor can generate virtual objects based on virtual information derived from image information. The processor can output virtual objects related to the XR service and background space information via display 421. For example, the processor can obtain image information by capturing images related to the real space of the field of view (FoV) corresponding to the user wearing the electronic device 401 via second functional cameras 411 and 412, or generate a virtual space for a virtual environment. For example, the processor can control display 421 to display output XR content (hereinafter referred to as the XR content screen) such that at least one virtual object appears in the area of ​​the field of view (FoV) or is defined as the user's field of view.

[0115] According to an embodiment, electronic device 401 may have a shape factor that is worn on a user's head. Electronic device 401 may also include straps and / or wearable components to be secured to a part of the user's body. Electronic device 401 may provide a VR, AR, and / or MR-based user experience when worn on a user's head.

[0116] Figure 5 Examples of the construction of virtual spaces according to various embodiments and of inputs from and outputs to users within the virtual space are shown.

[0117] Electronic devices (e.g., Figure 1 Electronic devices 101 Figure 2 Electronic equipment 201 Figure 3 Electronic devices 301 or Figure 4bThe electronic device 401 can use a sensor to obtain spatial information about the physical space in which the sensor is located. The spatial information may include the geographical location of the physical space, the size of the space, the appearance of the space, the location of the physical object 551 placed in the space, the size of the physical object 551, the appearance of the physical object 551, and lighting information. The appearance of the space and the physical object 551 may include at least one of the shape, texture, or color of the space and the physical object 551. Information about the light source, which is information about the light emitting light that functions in the physical space, may include at least one of the intensity, direction, or color of the lighting. The aforementioned sensor can collect information to provide AR. For example, referencing... Figure 2 , Figure 3 , Figure 4a and Figure 4b The AR device shown may include a camera and a depth sensor. However, the example is not limited to this, and the sensor may also include at least one of an infrared sensor, a depth sensor (e.g., a lidar sensor, a radar sensor, or a stereo camera), a gyroscope sensor, an accelerometer sensor, or a geomagnetic sensor.

[0118] Electronic device 501 can collect spatial information across multiple time frames. For example, in each time frame, electronic device 501 can collect spatial information about a portion of the scene within the sensing range (e.g., FoV) of a sensor at the location of electronic device 501 in the physical space. Electronic device 501 can analyze the spatial information of the time frames to track changes in objects over time (e.g., positional movement or state change). Electronic device 501 can synthesize the spatial information collected by multiple sensors to obtain comprehensive spatial information of the combined sensing range of multiple sensors (e.g., an image obtained by spatially stitching the scene around electronic device 501 in the physical space).

[0119] Electronic device 501 can analyze physical space as three-dimensional (3D) information using various input signals from sensors (e.g., sensing data from an RGB camera, infrared sensor, depth sensor, or stereo camera). For example, electronic device 501 can analyze at least one of the shape, size, or position of physical space and the shape, size, or position of physical object 551.

[0120] For example, electronic device 501 can use camera sensing data (e.g., captured images) to detect objects captured in a scene corresponding to the camera's FoV. Electronic device 501 can determine the label of a physical object 551 (e.g., information indicating object classification, including values ​​indicating chairs, monitors, or plants) from the camera's 2D scene image and the area (e.g., bounding box) occupied by the physical object 551 in the 2D scene. Therefore, electronic device 501 can obtain 2D scene information from the position where user 590 is viewing. Additionally, electronic device 501 can also calculate its position in physical space based on the camera's sensing data.

[0121] Electronic device 501 can use sensing data (e.g., depth data) from a depth sensor to obtain the user 590's position information and the depth information of the real space in the viewing direction. The depth information can be information indicating the distance from the depth sensor to each point and can be represented in the shape of a depth map. Electronic device 501 can analyze the distance of each pixel unit to the 3D position viewed by the user 590.

[0122] Electronic device 501 can use various sensing data to obtain information including 3D point clouds and meshes. Electronic device 501 can obtain planes, meshes, or 3D coordinate point clusters of the configuration space by analyzing the physical space. Electronic device 501 can obtain a 3D point cloud representing a physical object based on the information obtained as described above.

[0123] Electronic device 501 can obtain information including at least one of the following: 3D position coordinates, 3D shape, or 3D size (e.g., 3D bounding box) of physical objects arranged in the physical space by analyzing the physical space.

[0124] Therefore, electronic device 501 can obtain physical object information detected in 3D space and semantic segmentation information about 3D space. Physical object information may include at least one of the physical object 551's position, appearance (e.g., shape, texture, and color), or size in 3D space. Semantic segmentation information, which is obtained by semantically segmenting 3D space into subspaces, may include, for example, information indicating that 3D space is segmented into objects and background, and information indicating that the background is segmented into walls, floors, and ceilings. As described above, electronic device 501 can obtain and store 3D information (e.g., spatial information) about physical object 551 and physical space. Electronic device 501 may store the user 590's 3D position information in space along with the spatial information.

[0125] In this embodiment, the electronic device 501 can establish a virtual space 500 based on the physical location of the user 590 and / or the electronic device 501. The electronic device 501 can generate the virtual space 500 by referring to the aforementioned spatial information. The electronic device 501 can generate a virtual space 500 with the same scale as the physical space based on the spatial information, and arrange objects within the generated virtual space 500. The electronic device 501 can provide complete VR to the user 590 by outputting an image that replaces the entire physical space. The electronic device 501 can provide MR or AR by outputting an image that replaces a portion of the physical space. Although the construction of the virtual space 500 based on spatial information obtained through analysis of the physical space has been described, the electronic device 501 can also construct a virtual space 500 regardless of the physical location of the user 590. Here, the virtual space 500 can be a space corresponding to AR or VR, and can be referred to as a metaverse space.

[0126] For example, electronic device 501 can provide a virtual graphical representation that replaces at least a portion of the physical space. OST-based electronic device 501 can output a virtual graphical representation by overlaying it onto a screen area corresponding to at least a portion of the space in a screen display portion. VST-based electronic device 501 can output an image generated by replacing an image area corresponding to at least a portion of the space in a spatial image with a virtual graphical representation, the spatial image corresponding to the physical space and rendered based on spatial information. Electronic device 501 can replace at least a portion of the background in the physical space with a virtual graphical representation, but embodiments are not limited thereto. Electronic device 501 can additionally arrange virtual objects 552 in a virtual space 500 based solely on spatial information without changing the background.

[0127] Electronic device 501 can arrange and output virtual object 552 in virtual space 500. Electronic device 501 can set a manipulation area for virtual object 552 in the space occupied by virtual object 552 (e.g., volume corresponding to the appearance of virtual object 552). The manipulation area can be the area in which manipulation of virtual object 552 occurs. In addition, electronic device 501 can replace physical object 551 with virtual object 552 and output virtual object 552. Virtual object 552 corresponding to physical object 551 can have the same or similar shape as the corresponding physical object 551. However, the embodiments are not limited to this, and electronic device 501 can set the manipulation area only in the space occupied by physical object 551 or at the position corresponding to physical object 551, without outputting virtual object 552 replacing physical object 551. That is, electronic device 501 can send visual information representing physical object 551 (e.g., light reflected from physical object 551 or image obtained by capturing physical object 551) to user 590 as is without change, and set the manipulation area in the corresponding physical object 551. The manipulation area can be set to have the same shape and volume as, but is not limited to, the space occupied by the virtual object 552 or the physical object 551. The electronic device 501 can be set to a manipulation area smaller than the space occupied by the virtual object 552 or the physical object 551.

[0128] According to an embodiment, electronic device 501 can arrange virtual objects (not shown) representing user 590 (e.g., avatar objects) in virtual space 500. When the avatar object is presented in a first-person view, electronic device 501 can provide user 590 with a visual graphical representation corresponding to a portion of the avatar object (e.g., a hand, torso, or leg) via the aforementioned display (e.g., an OST display or a VST display). However, the embodiment is not limited to this, and when the avatar object is presented in a third-person view, electronic device 501 can provide user 590 with a visual graphical representation corresponding to the entire shape of the avatar object (e.g., a rear view) via the aforementioned display. Electronic device 501 can provide user 590 with an experience integrated with the avatar object.

[0129] Furthermore, electronic device 501 can provide an avatar of another user entering the same virtual space 500. Electronic device 501 can receive feedback information that is the same as or similar to the feedback information (e.g., information based on at least one of visual, auditory, or tactile sensations) provided to another electronic device 501 entering the same virtual space 500. For example, when an object is arranged in a virtual space 500 and multiple users access the virtual space 500, each electronic device 501 of multiple users 590 can receive feedback information (e.g., graphical representation, sound signal, or tactile feedback) of the same object arranged in the virtual space 500 and provide the feedback information to each user 590.

[0130] Electronic device 501 can detect input to an avatar of another electronic device (not shown) and can receive feedback information from the avatar of the other electronic device. The exchange of feedback and input in each virtual space 500 can be handled by a server (e.g., Figure 1 The server 108 performs the operation. For example, the server (e.g., a server providing metaverse space) can send input and feedback between user 590's avatar and another user 590's avatar. However, the embodiments are not limited to this, and electronic device 501 can establish direct communication with another electronic device to provide input or receive feedback based on the avatar, without going through the server.

[0131] For example, based on the detected user input that selects a manipulation area, electronic device 501 can determine that user 590 has selected a physical object 551 corresponding to the selected manipulation area. User 590's input may include at least one of gesture input using body parts (e.g., hands or eyes), input using a separate VR accessory device, or user voice input.

[0132] Pose input can be input corresponding to a pose identified by tracking the body parts 510 of the user 590, and can include, for example, input indicating or selecting an object. Pose input can include at least one of the following: a pose in which a body part (e.g., a hand) moves toward an object for a predetermined period of time or longer; a pose in which a body part (e.g., a finger, eye, or head) points at an object; or a pose in which a body part and an object are in spatial contact with each other. An eye-pointing pose can be identified based on ET (Eye Pointing). A head-pointing pose can be identified based on head tracking.

[0133] Tracking of user 590's body parts 510 can be performed primarily based on the camera of electronic device 501, but is not limited thereto. Electronic device 501 can track body parts 510 based on sensing data (e.g., image data from the camera and depth data from the depth sensor) and information collected by accessory devices described below (e.g., controller tracking and finger tracking in the controller). Finger tracking can be performed by sensing the distance or contact between a single finger and the controller based on sensors (e.g., infrared sensors) embedded in the controller.

[0134] Accessory devices for VR may include riding devices, wearable devices, controller devices 520, or other sensor-based devices. Riding devices may be devices that the user 590 rides and manipulates, and may include at least one of, for example, a treadmill-type device or a chair-type device. Wearable devices may be manipulation devices worn on at least a part of the user 590's body, and may include at least one of, for example, a full-body or half-body suit controller, a vest-type controller, a shoe-type controller, a bag-type controller, a glove-type controller (e.g., a haptic glove), or a mask-type controller. Controller devices 520 may include, for example, input devices (e.g., stick controllers or guns) manipulated by hands, feet, toes, or other body parts 510.

[0135] Electronic device 501 can track at least one of the location or movement of the accessory device by establishing direct communication with it, but the examples are not limited thereto. Electronic device 501 can also communicate with the accessory device via a base station for VR.

[0136] For example, electronic device 501 can determine that virtual object 552 has been selected based on the eye gaze tracking technology described above, which detects a gaze at virtual object 552 for a predetermined period of time or longer. In another example, electronic device 501 can identify a gesture pointing at virtual object 552 using hand tracking technology. Electronic device 501 can determine that virtual object 552 has been selected based on the direction of the tracked hand pointing at virtual object 552 for a predetermined period of time or longer, or based on the user 590's hand touching or entering the area occupied by virtual object 552 in virtual space 500.

[0137] The user's voice input (which corresponds to the user's voice received by electronic device 501) can be sensed by, for example, an input module (e.g., a microphone) of electronic device 501, or may include voice data received from an external electronic device of electronic device 501. By analyzing the user's voice input, electronic device 501 can determine that a physical object 551 or a virtual object 552 has been selected. For example, based on detecting a keyword in the user's voice input that indicates at least one of physical object 551 or virtual object 552, electronic device 501 can determine that at least one of physical object 551 or virtual object 552 corresponding to the detected keyword has been selected.

[0138] Electronic device 501 can provide feedback as described below in response to the input of user 590.

[0139] Feedback can include visual, auditory, tactile, olfactory, or gustatory feedback. Feedback can be derived from references. Figure 1 The described server 108, electronic device 101, or external electronic device 102 are presented.

[0140] Visual feedback may include the operation of outputting images through a display of electronic device 501 (e.g., a transparent display or an opaque display).

[0141] Auditory feedback may include the operation of outputting sound through the speaker of electronic device 501.

[0142] Haptic feedback can include force feedback that simulates weight, shape, texture, size, and dynamics. For example, a haptic glove can include haptic elements (e.g., electronic muscles) for simulating tactile sensations by tightening and relaxing the user's body 590. The haptic elements in the haptic glove can function as tendons. The haptic glove can provide haptic feedback to the entire hand of the user 590. Electronic device 501 can provide feedback representing the shape, size, and stiffness of an object through the haptic glove. For example, the haptic glove can generate forces that simulate the shape, size, and stiffness of an object. The exoskeleton of the haptic glove (or suit-type device) can include sensors and finger movement measurement devices that can transmit cable tension (e.g., electromagnetic force, DC-based force, or pneumatic force) to the user's fingers, thereby transmitting haptic information to the body. Hardware providing haptic feedback can include sensors, actuators, power supplies, and wireless transmission circuitry. The haptic glove can be operated by inflating and deflating inflatable air bladders on the surface of the glove.

[0143] Based on the selection of an object in virtual space 500, electronic device 501 can provide feedback to user 590. For example, electronic device 501 can output a graphical representation indicating the selected object via a display (e.g., highlighting the selected object). For example, electronic device 501 can output a sound (e.g., voice) to notify the selected object via a speaker. In another example, electronic device 501 can send electrical signals to a haptic support accessory device (e.g., a haptic glove), thereby providing user 590 with haptic movement that simulates the tactile sensation of the corresponding object.

[0144] Figure 6 This is a flowchart schematically illustrating the process of determining operational commands for the controller according to various embodiments.

[0145] An electronic device according to an embodiment (e.g., Figure 1 Electronic devices 101 Figure 2 Electronic equipment 201 Figure 3 Electronic equipment 301, Figure 4b Electronic devices 401 or Figure 5 The electronic device (501) can be worn by the user. The electronic device can be an extended reality device (XR device). XR devices can include all of the following: VR devices, AR devices, and MR devices.

[0146] Electronic devices can construct virtual spaces based on the physical location of the electronic device and / or the user (e.g., Figure 5 The electronic device can communicately connect to one or more handheld controllers (hereinafter referred to as "controllers") including buttons and sensors to interact with the virtual objects arranged in the virtual space. The controller can detect user actions and send operation commands corresponding to the detected user actions to the electronic device. The electronic device can communicate with multiple controllers individually and receive operation commands from each of the multiple controllers separately.

[0147] In an embodiment, the electronic device may display graphical functional visibilitys corresponding to multiple controllers on a display screen. Graphical functional visibilitys may represent user-interactive graphical user interface (GUI) objects displayed on the electronic device's display. Typically, the electronic device may determine and display the shape and size of the graphical functional visibilitys corresponding to the controllers as being the same as the shape and size of the controllers in real space, but the embodiments are not limited thereto. The electronic device may also determine the shape and size of the graphical functional visibilitys to be different from the shape and size of the controllers in real space.

[0148] In another embodiment, instead of displaying the graphical functionality visibility corresponding to the multiple controllers on a display screen, the electronic device can use a camera to capture (real-time) images of the multiple controllers and display the images on the display screen.

[0149] In another embodiment, the electronic device may display information about multiple controllers (e.g., battery information) on a display screen without displaying the graphical functionality visibility or images corresponding to the multiple controllers.

[0150] In this embodiment, when the electronic device determines that the controller is malfunctioning, the electronic device can display the corresponding graphical function visibility on the display screen based on the malfunctioning state of the controller. The method for displaying the corresponding graphical function visibility based on the malfunctioning state of the controller is briefly described below.

[0151] In operation 610, when the electronic device determines that a target controller among a plurality of controllers registered with the electronic device is abnormal, the electronic device may display the graphical function visibility corresponding to the target controller based on the abnormal state of the target controller.

[0152] In this embodiment, the electronic device can establish communication with and register with multiple controllers. The electronic device can individually determine whether each of the multiple controllers is abnormal.

[0153] In one embodiment, the electronic device may display graphical functionality visibility on a display screen corresponding to a target controller among a plurality of controllers that has been identified as having an anomaly.

[0154] For example, an electronic device may be displaying the existing graphical functionality visibility for a target controller. When the electronic device determines that the target controller is malfunctioning, it can change the existing graphical functionality visibility for the target controller based on the malfunctioning state of the target controller.

[0155] In another example, the electronic device may be displaying an image captured by a target controller. When the electronic device determines that the target controller has an anomaly, it can overlay the graphical functionality visibility based on the anomaly state onto the image capturing the target controller.

[0156] For example, an electronic device may not display existing graphical functionality visibility for the target controller or capture an image of the target controller. In this case, the electronic device can generate and display new graphical functionality visibility corresponding to the target controller based on the abnormal state of the target controller.

[0157] In one embodiment, for a target controller determined to be abnormal, the electronic device can use the graphical functionality visibility displayed corresponding to the target controller to determine the operation command indicated by the user's gesture for the target controller. Alternatively, the electronic device can directly receive operation commands for the remaining controllers that are not abnormal.

[0158] In operation 620, the electronic device can use a camera that captures user gestures to detect user gestures in relation to the visibility of the displayed graphical functions.

[0159] In one embodiment, when the electronic device determines that the target controller is malfunctioning, the electronic device may activate a pass-through camera capable of capturing user gestures (e.g., Figure 1 The first camera 265a or 265b or Figure 4a (Functional camera 415). The electronic device can detect user gestures in relation to the visibility of graphical functions displayed corresponding to the target controller. User gestures may include, for example, gestures corresponding to pressing a button on the controller, gestures corresponding to rotating the controller, or gestures corresponding to the touch sensor of the touch controller.

[0160] In operation 630, the electronic device can determine the operation command for the target controller indicated by the detected user gesture.

[0161] In an embodiment, the electronic device can determine an operation command for a target controller indicated by a user gesture for the visibility of the graphical function, and can receive the determined operation command for the target controller as an input signal. Furthermore, the electronic device can perform an operation based on the determined operation command for the target controller. In other words, the electronic device can receive an operation command for the target controller indicated by a user gesture for the visibility of the graphical function as an input signal, so that even when some components included in the target controller (e.g., buttons, touch sensors, or gyroscope sensors) are not operational, the user can still send controller operation commands based on non-operating components to the electronic device.

[0162] Figure 7 This illustrates a scenario where a user uses multiple controllers according to various embodiments.

[0163] In this embodiment, the electronic device can establish communication with multiple controllers 721 and 722 individually. The electronic device can register multiple controllers 721 and 722 to itself by establishing communication individually with each of them. Controller 721 can be designed to be operated with the user's left hand 701, and controller 722 can be designed to be operated with the user's right hand 702.

[0164] In this embodiment, the electronic device can generate a virtual space and arrange virtual objects within it. The electronic device can then display a VR image 720 corresponding to the virtual space on a display screen.

[0165] In an embodiment, the electronic device can individually determine whether each of the plurality of controllers 721 and 722 is malfunctioning. The electronic device can determine whether a controller (e.g., controller 721) is malfunctioning based on signals received from the controller.

[0166] In this embodiment, the electronic device can determine that the multiple controllers 721 and 722 are not in an abnormal state. In other words, the electronic device can determine that the multiple controllers 721 and 722 are operating normally. In this case, as... Figure 7 As shown, the electronic device may not display the graphical functionality visibility corresponding to the controller (e.g., controller 721 or controller 722) or the image of the controller captured by the electronic device on the display screen.

[0167] In an embodiment, the electronic device can determine that the controller 721 is malfunctioning. For example, the electronic device can detect an operational error of the button 731 included in the controller 721 based on signals received from the controller 721. When the electronic device determines that the controller 721 is malfunctioning, it can display a graphical function visibility (not shown) corresponding to the controller 721 based on the malfunction status. (See reference...) Figure 8 , Figure 9a , Figure 9b and Figure 10 A method for displaying abnormal states based on controller 721 and the corresponding graphical function visibility (not shown) of controller 721 is described in more detail.

[0168] Figure 8 This is a diagram illustrating a situation where an electronic device according to an embodiment detects an operational error of a button included in a controller.

[0169] In an embodiment, the electronic device can determine the abnormal state of the controller 721 based on signals received from the controller 721. The abnormal state of the controller 721 can manifest in various ways. For example, the types of abnormal states of the controller can include operational errors of buttons included in the controller, operational errors of touch sensors included in the controller, operational errors of tracking sensors included in the controller, or operational errors of gyroscope sensors included in the controller.

[0170] In this embodiment, when the electronic device determines that the controller 721 is malfunctioning, the electronic device can display the corresponding graphical function visibility on the display screen based on the malfunction state of the controller 721. The electronic device can display the corresponding graphical function visibility in different ways depending on the type of malfunction state of the controller 721.

[0171] refer to Figure 8 When the electronic device determines that the controller 721 is malfunctioning, it can activate a camera that captures the controller (e.g., controller 721 or controller 722) and the user's posture, and can display the controller's image 820 on the screen. If the electronic device determines that the controller 721 is malfunctioning while displaying a VR image on the screen, it can display an AR image instead of a VR image on the screen.

[0172] In addition, the electronic device can display an interface object 810 on the screen that includes information indicating that the controller 721 has an anomaly.

[0173] In this embodiment, when the electronic device receives user input requesting the use of interface object 811, which is determined to have a malfunction in controller 721, the electronic device can display the graphical functionality visibility corresponding to controller 721 on the screen. Conversely, when the electronic device receives user input requesting the termination of use of interface object 812, which is determined to have a malfunction in controller 721, the electronic device can terminate communication between controller 721 and the electronic device, and may not display the graphical functionality visibility corresponding to controller 721. The following description primarily focuses on the case where a user requests the use of controller 721, which is determined to have a malfunction.

[0174] In an embodiment, the electronic device can detect operational errors of the button 731 included in the controller 721 based on signals received from the controller 721. The button 731 included in the controller 721 can detect physical pressure from the user and generate an electrical signal indicating whether the button 731 has been pressed, which is sent to the processor of the controller 721. Operational errors of the button 731 can include various types of errors, such as a press detection error due to a defect in the button sensor included in the button 731, where the button 731 fails to detect physical pressure from the user, or a sensitivity adjustment error due to inaccurate adjustment of the button 731's sensitivity. However, the types of operational errors of the button 731 are not limited to these.

[0175] When an operational error is detected in button 731, controller 721 can notify the electronic device of the operational error in button 731. Hereinafter, a method by which controller 721 determines an operational error in button 731 is described as an example. For example, controller 721 can use a tracking sensor to track the position of a user's hand operating controller 721 (hereinafter, "user hand tracking"). Controller 721 can determine whether the user has physically pressed button 731 by user hand tracking. For example, controller 721 can determine whether the user's finger has contacted the upper surface of button 731 by performing user hand tracking, and can determine whether the user has physically pressed button 731 by determining whether the finger contacting the upper surface of button 731 has performed a flexing movement. When user hand tracking determines that the user has physically pressed button 731, but button 731 fails to detect physical pressure from the user, controller 721 can determine that an operational error has occurred in button 731. When it is determined that an operational error has occurred in button 731, controller 721 can send a notification to the electronic device that an operational error has occurred in button 731.

[0176] Figure 9a and Figure 9b This is a diagram illustrating the process by which an electronic device displays the visibility of a graphical function corresponding to the controller when it detects an operational error of a button included in the controller, according to various embodiments.

[0177] In an embodiment, the electronic device can determine that the controller 721 is malfunctioning. For example, the electronic device can detect an operational error of a button 731 included in the controller 721. The electronic device can receive a selection request to use an interface object (e.g., the controller 721 determined to be malfunctioning) of the controller 721. Figure 8 User input in the interface object 811.

[0178] refer to Figure 9a The electronic device can display the graphical functionality visibility 911 corresponding to the controller 721 on the display screen.

[0179] In an embodiment, the electronic device may overlay a graphical functional visibility 911 corresponding to the controller 721 onto an image 820 capturing the controller 721. In this case, the electronic device may generate a graphical functional visibility 911 corresponding to the controller 721 for a button 731 determined to be malfunctioning. In other words, the electronic device may generate a graphical functional visibility 911 that replaces a portion of the controller 721. When the electronic device detects an operational error of the button 731 included in the controller 721, the electronic device may display a graphical functional visibility 911 in the image 820 capturing the controller 721 that has an increased size compared to at least one of the width and height of the button 731.

[0180] In another embodiment, the electronic device may generate and display a graphical functional visibility (not shown) replacing the entire controller 721, instead of displaying the image 820 capturing the controller 721. In this case, in addition to the graphical functional visibility for accessories (e.g., button 731) determined to have an anomaly in the controller 721, the electronic device may also display graphical functional visibility for other accessories (e.g., triggers or grip buttons of the controller). In this case, the electronic device may detect an operational error of button 731 included in the controller 721. In this case, the electronic device may display graphical functional visibility of other accessories in the controller 721 that correspond to the shape of other accessories appearing in image 820, while displaying the graphical functional visibility of the button 731 with an operational error in the controller 721 by increasing at least one of the width and height of the button 731 appearing in image 820.

[0181] In an embodiment, when the electronic device receives a selection request, it uses the interface object of the controller 721 (e.g., Figure 8 When user input occurs to the interface object 811, the electronic device can provide a tutorial screen for performing user gestures to operate the button 731 in the controller 721 that has an error. The electronic device can display a graphical function visibility 911 corresponding to the controller 721, determine whether the user gesture to the graphical function visibility 911 indicates input to the button 731, and provide the user with the determination result. In this case, the electronic device can keep the camera on to recognize the user gesture.

[0182] In one embodiment, the electronic device can determine whether a user has pressed button 731 by using a camera to detect user gestures for the visibility of the graphics function 911. The camera (e.g., Figure 1 The first camera 265a or 265b or Figure 4aThe 415-inch camera can detect user posture. The camera can be used for real-world spatial recognition and user hand tracking.

[0183] In an embodiment, the electronic device can more easily determine whether a user gesture indicates input to button 731 by displaying a graphical functional visibility 911 in image 820 that has an increased size compared to at least one of the width and height of button 731. The electronic device can detect the user gesture toward the graphical functional visibility 911 and can determine whether the detected user gesture indicates input to button 731 included in controller 721. For example, when the electronic device determines that the user's left-hand 701 finger moves downward a distance greater than a threshold distance after contacting the upper surface of button 731, the electronic device can determine that the user gesture indicates input to button 731.

[0184] The electronic device can display a graphical functional visibility 911 in image 820 that is larger than the width of button 731. In this case, the electronic device can more easily determine whether the user's left hand 701 finger (e.g., index finger) covers the upper surface of button 731. The electronic device can also display a graphical functional visibility 911 in image 820 that is larger than the height of button 731. In this case, the electronic device can more easily determine whether the user's left hand 701 finger (e.g., index finger) has pressed button 731. This is because when a graphical functional visibility 911 with an increased height of button 731 is displayed on the screen, it is expected that the user will move their finger (e.g., thumb) further downward to perform a pressing operation on button 731.

[0185] When the electronic device determines that the user's gesture indicates that the button 731 has been pressed, the electronic device can determine that the user's gesture indicates input to the button 731. When the user's gesture indicates input to the button 731, the electronic device can receive the operation command for the button 731 from the controller 721 as an input signal. Furthermore, the electronic device can perform an operation according to the operation command for the button 731.

[0186] refer to Figure 9b The electronic device can, after providing a tutorial screen for executing an operation command on the button 731 in the controller 721 that has an error using the user's gestures, display the VR image 920 again on the screen to allow the user to enter the virtual space. In this case, the electronic device can display the VR image 920 and an AR image 930 including a graphical function visibility 911 corresponding to the controller 721 on the screen. For example, the AR image 930 may be an image in which the graphical function visibility 911 corresponding to the controller 721 is overlaid (e.g., Figure 9aThe image 920 is an image on the area corresponding to the controller 721 and the hand 701 of the user operating the controller 721. The electronic device can display the VR image 920 and the AR image 930 together on the screen in a picture-in-picture (PIP) manner. The electronic device can provide a virtual space to the user by displaying the VR image 920 and the AR image 930 together on the screen, while helping the user to perform gestures to generate operation commands for the button 731 that has been erroneously pressed.

[0187] Figure 10 This is a diagram illustrating the process by which an electronic device, according to various embodiments, displays a graphical function with increased visibility to the controller.

[0188] In this embodiment, the electronic device can detect operational errors of a gyroscope sensor (not shown) included in the controller 1021 based on signals received from the controller 1021. The gyroscope sensor included in the controller 1021 can detect rotational movement of a user's hand 1001 that manipulates the controller 1021. More specifically, the gyroscope sensor included in the controller 1021 can measure the rotational speed, direction, and amount of rotation of the controller 1021. The controller 1021 can determine the rotational movement of the user's hand 1001 based on the measured rotational speed, direction, and amount of rotation. The controller 1021 can send information about the rotational movement of the user's hand 1001, as measured by the gyroscope sensor, to the electronic device in real time. In this case, the electronic device can detect operational errors of the gyroscope sensor included in the controller 1021 either by itself or by receiving a signal from the controller 1021 indicating that an operational error of the gyroscope sensor has occurred. For example, when the electronic device fails to receive information from the controller 1021 about the rotational movement of the user's hand 1001, the electronic device can determine that an operational error has occurred in the gyroscope sensor included in the controller 1021.

[0189] In one embodiment, the electronic device may display a graphical visibility 1031 with an enlarged size for the controller 1021. (See reference...) Figure 10The electronic device can generate and display a graphical functional visibility 1031 that replaces the entire controller 1021. The electronic device can display images 1020 on the screen capturing controllers 1021 and 1022, as well as the hands 1001 and 1002 of the user manipulating controllers 1021 and 1022. The electronic device can remove the area corresponding to controller 1021 from the image 1020 and can overwrite the graphical functional visibility 1031 corresponding to controller 1021. The electronic device can generate a graphical functional visibility 1031 that replaces the entire controller 1021. When the electronic device detects an operational error of the gyroscope sensor included in controller 1021, the electronic device can display a graphical functional visibility 1031 with an increased size compared to controller 1021 in the image 1020 capturing controller 1021.

[0190] In an embodiment, the electronic device can detect user gestures for the graphics function visibility 1031 and determine whether the detected user gestures indicate an operation related to the rotation of the controller 1021.

[0191] In an embodiment, the electronic device can calculate the rotational speed, direction, and amount of rotation of the user's hand 1001 based on the detected user posture. The electronic device can then determine whether the user posture indicates an operation related to the rotation of the controller 1021 (e.g., a rolling operation, a flicking operation, or a swinging operation) based on the calculated rotational speed, direction, and amount of rotation. For example, when the rotational speed and amount of rotation of the user's hand 1001 are greater than or equal to a threshold speed and a threshold amount, the electronic device can determine that the user posture indicates an operation related to the rotation of the controller 1021.

[0192] In an embodiment, the electronic device can more easily determine whether a user's posture indicates an operation related to the rotation of the controller 1021 by displaying a graphical functional visibility 1031 with an increased size compared to the controller 1021 on a display screen. As described above, the electronic device can determine that the user's posture indicates an operation related to the rotation of the controller 1021 when the rotational speed and amount of the user's hand 1001 are greater than or equal to a threshold speed and a threshold amount. Because the electronic device displays a graphical functional visibility 1031 with an increased size compared to the controller 1021 on the screen, the user viewing the graphical functional visibility 1031 can increase the rotational speed and amount of the user's hand 1001 to rotate the graphical functional visibility 1031 corresponding to the controller 1021 compared to displaying a graphical functional visibility with a size equal to that of the controller 1021. In other words, when the user intends to perform an operation related to the rotation of the controller 1021, the rotational speed and amount of the user's hand 1001 can be increased, and the electronic device can more easily determine whether the user's posture indicates an operation related to the rotation of the controller 1021.

[0193] Furthermore, for user convenience, when the electronic device detects an operational error of the gyroscope sensor included in the controller 1021, the electronic device can display information as a graphical object on the display screen about the rotation speed and amount of rotation required to perform operations related to the rotation of the controller 1021.

[0194] Figure 11 This is a diagram illustrating the process of tracking a user's hand when an electronic device detects an operational error of a tracking sensor included in a controller, according to various embodiments.

[0195] In one embodiment, the electronic device can detect operational errors of the tracking sensor included in the controller 1121 based on signals received from the controller 1121. The tracking sensor included in the controller 1121 can track the position of a user's hand 1101 that manipulates the controller 1121. The controller 1121 can send the position information of the user's hand 1101, measured by the tracking sensor, to the electronic device in real time. In this case, the electronic device can detect operational errors of the tracking sensor included in the controller 1121 itself, or it can receive a signal from the controller 1121 indicating an operational error of the tracking sensor. For example, when the electronic device fails to receive the position information of the user's hand 1101 from the controller 1121, the electronic device can determine that an operational error has occurred in the tracking sensor included in the controller 1121.

[0196] In an embodiment, when the electronic device detects an operational error of the tracking sensor included in the controller 1121, the electronic device can set the hand 1101 of a user manipulating the controller 1121 as the tracking target. The electronic device can capture the hand 1101 of the user manipulating the controller 1121 using a camera, and can mark the area 1130 corresponding to the user's hand 1101 in the image 1120 of the captured hand 1101 using markers. For example, the image 1120 of the user's hand 1101 can represent images of the capture controllers 1121 and 1122, as well as the user's hand 1101 and 1102.

[0197] In this embodiment, since the controller 1121 fails to track the position of the user's hand 1101, the electronic device can continuously track the user's hand 1101 by setting the user's hand 1101 as a tracking target through image analysis based on image 1120. The electronic device can analyze the position and shape of the user's hand 1101 based on image 1120 captured by the camera. Since the electronic device cannot receive tracking information about the user's hand 1101 from the controller 1121, the electronic device can mark the area 1130 in image 1120 corresponding to the user's hand 1101 with a marker, and can display the marked area 1130 on a display screen. The electronic device can notify the user that the user's hand 1101 is being tracked by the electronic device and that an operational error has occurred in the tracking sensor included in the controller 1121 by displaying the marked area 1130 corresponding to the user's hand 1101.

[0198] Figure 12 This is a diagram illustrating the process by which the electronic device displays the visibility of a graphical function corresponding to the controller when the connection between the controller and the electronic device is terminated or when the controller is separated from the user's hand, according to various embodiments.

[0199] In an embodiment, when it is detected that the connection with controller 1221 has been terminated or that controller 1221 and the hand 1201 of the user operating controller 1221 are separated from each other, the electronic device may display a graphical functional visibility 1231 for generating operation commands for controller 1221.

[0200] For example, the connection between the electronic device and the controller 1221 can be terminated when the battery of the controller 1221 discharges or when the wireless communication between the controller 1221 and the electronic device is interrupted. Additionally, the controller 1221 can detect contact with the user's hand 1201, and if no contact with the user's hand 1201 is detected within a predetermined period, the controller 1221 can determine that the user's hand 1201 has been separated from the controller 1221. The controller 1221 can then send information to the electronic device indicating that the user's hand 1201, which operates the controller 1221, has been separated from the controller 1221.

[0201] The shape of the graphical functional visibility 1231 used to generate operation commands for controller 1221 may be similar to or different from the shape of controller 1221. The electronic device may display on the screen an image 1220 capturing controllers 1221 and 1222, as well as the hands 1201 and 1202 of the user manipulating controllers 1221 and 1222. The electronic device may display the graphical functional visibility 1231 corresponding to the controller overlaid on the image 1220.

[0202] When the electronic device detects that the connection with the controller 1221 has been terminated or that the controller 1221 and the user's hand 1201 operating the controller 1221 have separated from each other, the electronic device can activate the camera to detect the user's posture for the graphics function visibility 1231. The electronic device can determine through the graphics function visibility 1231 whether the user's posture indicates an operation command for the controller 1221, and can execute an operation according to the determined operation command.

[0203] Figure 13 This is a diagram illustrating the process by which an electronic device, according to various embodiments, changes the visibility of a graphical function displayed in accordance with the controller to the type of object.

[0204] In an embodiment, the electronic device can determine the type of object whose graphical functionality visibility corresponds to that of the controller. Object types that can be used as the controller may include, but are not limited to, joystick types, keyboard types, and instrument types.

[0205] In an embodiment, when a user gesture corresponding to one of a plurality of object types that can be used as a controller is identified, the electronic device can determine the object type of the graphical functionality visibility displayed in virtual space 1200 corresponding to the controller. For example, the electronic device can acquire an image of the user's hand via a camera and detect the user gesture from the acquired image through image analysis. When the detected user gesture is a keyboard input gesture, the electronic device can determine the object type of the graphical functionality visibility 1231 displayed corresponding to the controller as a keyboard type.

[0206] In an embodiment, the electronic device may display an area (e.g., area 1341, 1342, 1343, or 1344) in the graphical functional visibility 1331 displayed corresponding to the controller, such that this area is distinct from the other areas. Reference Figure 13 The electronic device can display the area in the graphical functional visibility 1331 that is used to generate operation commands for the controller in a different color than other areas. For example, when the electronic device detects a user gesture (e.g., touch area 1341) on area 1341 of the graphical functional visibility 1331, the electronic device can generate an operation command for the controller corresponding to area 1341.

[0207] Furthermore, the electronic device can determine the operation command for the controller indicated by the detected user gesture and output the determined operation command as voice or text. In an embodiment, the electronic device can output the operation command for the controller as voice or text before executing the operation corresponding to the operation command for the controller indicated by the user gesture. In other words, the electronic device can send the operation command for the controller recognized by the electronic device to the user before executing the operation corresponding to the operation command for the controller indicated by the user gesture. The user can cancel the operation command for the controller before the electronic device executes the operation command corresponding to the operation command for the controller indicated by the user gesture. For example, the electronic device can determine the operation command for the controller indicated by the user gesture and display an interface object for canceling the operation corresponding to the determined operation command. The user can prevent the electronic device from executing the operation corresponding to the determined operation command by interacting with the interface object for canceling the operation.

[0208] The electronic device according to various embodiments can be one of a variety of types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to embodiments, the electronic device is not limited to those described above.

[0209] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to particular embodiments, but rather to include various changes, equivalents, or substitutions for the respective embodiments. The same reference numerals may be used for similar or related parts in conjunction with the description of the accompanying drawings. It should be understood that, unless the relevant context clearly indicates otherwise, the singular form of a noun corresponding to an item may include one or more things. As used herein, each of “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “A, B, or C” may include any one of the items listed together in a corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from other components discussed and may not limit the component in other respects (e.g., importance or order). It will be understood that, whether the terms “operably” or “communically” are used or not, if a component (e.g., a first component) is referred to as “combined with another component (e.g., a second component),” “combined to another component (e.g., a second component),” “connected to another component (e.g., a second component),” or “connected to another component (e.g., a second component)”, it means that the component can be directly (e.g., via a wire), wirelessly connected to the other component, or connected to the other component via a third component.

[0210] As used in conjunction with various embodiments of this disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "part," or "circuit"). A module may be a single integrated component adapted to perform one or more functions, or its smallest unit or part. For example, according to an embodiment, a module may be implemented in the form of an ASIC.

[0211] The various embodiments set forth herein can be implemented as software (e.g., program 140) comprising one or more instructions readable by a machine (e.g., electronic device 101) stored in a storage medium (e.g., internal memory 136 or external memory 138). For example, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke and execute at least one of the instructions stored in the storage medium. This allows the machine to perform at least one function according to the invoked instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is temporarily stored in the storage medium.

[0212] According to embodiments, methods according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TM The computer program product may be distributed online (e.g., downloaded or uploaded) or directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as the memory of a manufacturer's server, an app store's server, or a relay server.

[0213] According to embodiments, each of the above components (e.g., a module or program) may include a single entity or multiple entities, and some of the multiple entities may be separately located in different components. According to embodiments, one or more of the above components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to embodiments, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as the corresponding component performed before integration. According to various embodiments, operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be performed in a different order or omitted, or one or more other operations may be added.

[0214] The units described herein can be implemented using hardware components, software components, and / or combinations thereof. Processing devices can be implemented using one or more general-purpose or special-purpose computers, such as, for example, processors, controllers, and arithmetic logic units (ALUs), digital signal processors (DSPs), microcomputers, field-programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or any other device capable of responding to and executing instructions in a defined manner. Processing devices can run an operating system (OS) and one or more software applications running on the OS. Processing units can also access, store, manipulate, process, and generate data in response to the execution of software. For simplicity, the description of processing units is used as singular; however, those skilled in the art will understand that processing units can include multiple processing elements and various types of processing elements. For example, a processing unit can include multiple processors, or a single processor and a single controller. Furthermore, different processing configurations are possible, such as parallel processors.

[0215] Software can include computer programs, code, instructions, or combinations thereof, to independently or uniformly instruct or configure a processing device to operate as needed. Software and data can be permanently or temporarily embodied in any type of machine, component, physical or virtual device, or computer storage medium or device capable of providing instructions or data to or being interpreted by the processing device. Software can also be distributed across network-coupled computer systems, enabling it to be stored and executed in a distributed manner. Software and data can be stored on one or more non-transitory computer-readable recording media.

[0216] The methods described in the examples above can be recorded in a non-transitory computer-readable medium comprising program instructions for implementing the various operations described in the examples. The medium may also include data files, data structures, etc., alone or in combination with the program instructions. The program instructions recorded on the medium may be program instructions specifically designed and constructed for the purposes of the examples, or they may be of types known and available to those skilled in the art of computer software. Examples of non-transitory computer-readable media include magnetic media, such as hard disks, floppy disks, and magnetic tapes; optical media, such as CD-ROMs and / or digital universal disks; magneto-optical media, such as optical discs; and hardware devices specifically configured to store and execute program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, etc. Examples of program instructions include machine code generated by a compiler and files containing higher-level code that can be executed by a computer using an interpreter.

[0217] The aforementioned hardware device can be configured to function as one or more software modules to perform the operations of the above embodiments, and vice versa.

Claims

1. An electronic device (101; 201; 301； 401; 501), including: Cameras (265a; 265b; 415) that capture user poses; Memory (130) storing computer-executable instructions; and The processor (120) accesses the memory (130) and executes the instructions. When the instruction is executed, it causes the processor (120) to: When it is determined that a target controller among the multiple controllers (721; 722) registered with the electronic device (101; 201; 301; 401; 501) is abnormal, a graphical function visibility corresponding to the target controller is displayed based on the abnormal state of the target controller. The camera (265a; 265b; 415) is used to detect the user gesture in response to the displayed graphical function visibility, and an operation command for the target controller indicated by the detected user gesture is determined.

2. The electronic device (101; 201; 301; 401; 501) of claim 1, wherein, When the instruction is executed, it causes the processor (120) to: The abnormal state of the target controller is determined based on the signals received from the target controller.

3. The electronic device (101; 201; 301; 401; 501) according to any one of claims 1 to 2, wherein, When the instruction is executed, it causes the processor (120) to: When it is determined that the target controller is abnormal, the camera (265a; 265b; 415) is turned on.

4. The electronic device (101; 201; 301; 401; 501) according to any one of claims 1 to 3, wherein, When the instruction is executed, it causes the processor (120) to: When an operational error is detected in a button included in the target controller, a graphical functional visibility with an increased size compared to at least one of the width and height of the button is displayed in an image of the target controller.

5. The electronic device (101; 201; 301; 401; 501) according to any one of claims 1 to 4, wherein, When the instruction is executed, it causes the processor (120) to: When an operational error is detected in the gyroscope sensor included in the target controller, a graphical feature with an increased size compared to the size of the target controller is displayed in the image of the captured target controller.

6. The electronic device (101; 201; 301; 401; 501) according to any one of claims 1 to 5, wherein, When the instruction is executed, it causes the processor (120) to: When an operational error is detected in the tracking sensor included in the target controller, the hand of the user operating the target controller is set as the tracking target, and the area corresponding to the user's hand is displayed as a marker in the image capturing the user's hand.

7. The electronic device (101; 201; 301; 401; 501) according to any one of claims 1 to 6, wherein, When the instruction is executed, it causes the processor (120) to: Virtual reality (VR) images and augmented reality (AR) images, including the visibility of graphical functions corresponding to the target controller, are displayed on the display screen.

8. The electronic device (101; 201; 301; 401; 501) according to any one of claims 1 to 7, wherein, When the instruction is executed, it causes the processor (120) to: Determine the object type for which the graphical functionality visibility is displayed in response to the target controller.

9. The electronic device (101; 201; 301; 401; 501) according to any one of claims 1 to 8, wherein, When the instruction is executed, it causes the processor (120) to: When a user gesture is identified that corresponds to one of a plurality of object types that can be used as a controller, the object type of the graphical functionality visibility displayed in response to the target controller is determined.

10. The electronic device (101; 201; 301; 401; 501) according to any one of claims 1 to 9, wherein, When the instruction is executed, it causes the processor (120) to: The operation commands for the target controller indicated by the detected user gestures are output as voice or text.

11. A device comprising electronic equipment (101; 201; The method executed by (301; 401; 501) includes: When it is determined that a target controller among the multiple controllers (721, 722) registered with the electronic device (101; 201; 301; 401; 501) is abnormal, the graphical functional visibility corresponding to the target controller is displayed based on the abnormal state of the target controller. The user's pose is detected using a camera (265a; 265b; 415) for the visibility of the displayed graphical features; and Determine the operation command for the target controller indicated by the detected user gesture.

12. The method according to claim 11, wherein, The visibility of the graphical functions corresponding to the target controller includes: The abnormal state of the target controller is determined based on the signals received from the target controller.

13. The method according to any one of claims 11 to 12, wherein, The detection of the user's gesture includes: When it is determined that the target controller is abnormal, the camera (265a; 265b; 415) is turned on.

14. The method according to any one of claims 11 to 13, wherein, The visibility of the graphical functions corresponding to the target controller includes: When an operational error is detected in a button included in the target controller, a graphical functional visibility with an increased size compared to at least one of the width and height of the button is displayed in an image of the target controller.

15. The method according to any one of claims 11 to 14, wherein, The visibility of the graphical functions corresponding to the target controller includes: When an operational error is detected in the gyroscope sensor included in the target controller, a graphical feature with an increased size compared to the size of the target controller is displayed in the image of the captured target controller.

Images