Extended reality device and operation method thereof

Abstract

Disclosed is an operation method of an extended reality device, comprising: acquiring an image of an external user by controlling a camera; identifying, within the acquired image, an area representing a counterpart extended reality device; removing, from the image, the identified area representing the counterpart extended reality device; receiving gaze information of both eyes of the external user from the counterpart extended reality device; generating first modeling data by synthesizing an eye image corresponding to the gaze information of both eyes of the external user with an area from which the area representing the counterpart extended reality device has been removed, from among the entire area of the image; and outputting the first modeling data to a display.

Description

Extended reality device and method of operation thereof

[0001] The present disclosure relates to an extended reality device that virtually displays a virtual image on a real object, and more particularly, to an extended reality device and an operating method thereof for increasing the usability of face-to-face interaction between users.

[0002] Extended Reality is a technology that overlays virtual images onto the physical environment or real world objects of the real world.

[0003] Augmented reality devices typically allow the user to view a scene through a see-through display positioned close to the user's eyes while worn. The scene includes one or more real-world objects within the physical environment or space directly visible to the user. Augmented reality devices transmit virtual images to the user's eyes through the see-through display, allowing the user to simultaneously view both real-world objects and virtual images.

[0004] However, when both the user and the person with whom the user wishes to interact are wearing an extended reality device, there is a problem in that the usefulness of the interaction is reduced because the user and the person interact with their eyes and faces obscured by the extended reality device.

[0005] An extended reality device according to one embodiment of the present disclosure may include at least one display, at least one camera, a communication interface for performing data communication with a counterpart extended reality device, a memory for storing at least one command, and at least one processor for executing at least one command stored in the memory. The at least one processor may, by executing the at least one command, control the at least one camera to capture an image of an external user wearing the counterpart extended reality device, thereby obtaining an image of the external user. The at least one processor may, by executing the at least one command, identify an area representing the counterpart extended reality device within the obtained image. The at least one processor may, by executing the at least one command, remove an area representing the identified counterpart extended reality device from the image. The at least one processor may, by executing the at least one command, control the communication interface to receive gaze information of both eyes of the external user from the counterpart extended reality device. The at least one processor may, by executing the at least one command, synthesize an eye image corresponding to gaze information of both eyes of the external user into an area of ​​the entire image from which the area representing the counterpart extended reality device has been removed, thereby generating first modeling data. The at least one processor can output the first modeling data to the at least one display by executing the at least one command.

[0006] An extended reality device according to one embodiment of the present disclosure may include at least one gaze tracking sensor, a communication interface for performing data communication with a counterpart extended reality device, a memory for storing at least one command, and at least one processor for executing at least one command stored in the memory. The at least one processor may control the at least one gaze tracking sensor by executing the at least one command to obtain gaze information of both eyes of a user. The at least one processor may control the communication interface by executing the at least one command to transmit gaze information of both eyes of the user to the counterpart extended reality device.

[0007] A method for operating an extended reality device according to one embodiment of the present disclosure may include a step of acquiring an image of an external user wearing a relative extended reality device by photographing the external user through at least one camera. A method for operating an extended reality device according to one embodiment of the present disclosure may include a step of identifying an area representing the relative extended reality device within the acquired image. A method for operating an extended reality device according to one embodiment of the present disclosure may include a step of removing an area representing the identified relative extended reality device from the image. A method for operating an extended reality device according to one embodiment of the present disclosure may include a step of receiving gaze information of both eyes of the external user from the relative extended reality device via a communication interface. A method for operating an extended reality device according to one embodiment of the present disclosure may include a step of generating first modeling data by synthesizing an eye image corresponding to gaze information of both eyes of the external user with an area of ​​the entire area of ​​the image from which the area representing the relative extended reality device has been removed. A method for operating an extended reality device according to one embodiment of the present disclosure may include a step of outputting the first modeling data to the at least one display. A method for operating an extended reality device according to one embodiment of the present disclosure may include a step of acquiring gaze information of both eyes of the user through at least one gaze tracking sensor. A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of transmitting gaze information of both eyes of the user to a counterpart extended reality device through a communication interface.

[0008] FIG. 1 is a diagram illustrating an example of a user wearing an extended reality device according to one embodiment of the present disclosure interacting with an external user wearing a counterpart extended reality device.

[0009] FIG. 2 is a block diagram illustrating components of an extended reality device according to one embodiment of the present disclosure.

[0010] FIG. 3 is a diagram illustrating components of an extended reality device according to one embodiment of the present disclosure.

[0011] FIG. 4 is a diagram illustrating a method for an extended reality device according to one embodiment of the present disclosure to generate modeling data.

[0012] FIG. 5 is a diagram illustrating a gaze tracking sensor, which is a component of an extended reality device according to one embodiment of the present disclosure.

[0013] FIG. 6 is a diagram illustrating a gaze tracking sensor, which is a component of an extended reality device according to one embodiment of the present disclosure.

[0014] Figure 7 is a drawing showing a three-dimensional eye model for the user's gaze.

[0015] FIG. 8 is a block diagram illustrating components of an extended reality device according to one embodiment of the present disclosure.

[0016] FIG. 9 is a diagram illustrating a method for an extended reality device according to one embodiment of the present disclosure to output modeling data.

[0017] FIG. 10 is a flowchart illustrating an operation method of an extended reality device according to one embodiment of the present disclosure.

[0018] FIG. 11 is a flowchart illustrating an operation method of an extended reality device according to one embodiment of the present disclosure.

[0019] FIG. 12 is a flowchart illustrating an operation method of an extended reality device according to one embodiment of the present disclosure.

[0020] FIG. 13 is a flowchart illustrating an operation method of an extended reality device according to one embodiment of the present disclosure.

[0021] FIG. 14 is a flowchart illustrating an operation method of an extended reality device according to one embodiment of the present disclosure.

[0022] FIG. 15 is a flowchart illustrating an operation method of a first extended reality device and a second extended reality device according to one embodiment of the present disclosure.

[0023] The terms used in this specification will be briefly explained, and the present invention will be described in detail.

[0024] The terms used in this invention have been selected from widely used, current terms, taking into account the functions of the invention. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should not be defined simply as names, but rather based on their inherent meanings and the overall content of the invention.

[0025] When a part of the specification is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used throughout the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0026] Additionally, the description 'at least one of A, B, and C' means that it can be any one of 'A', 'B', 'C', 'A and B', 'A and C', 'B and C', and 'A, B, and C'.

[0027] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement the present invention. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description have been omitted to clearly explain the present invention, and similar parts have been designated with similar reference numerals throughout the specification.

[0028] In this disclosure, 'Extended Reality' means displaying a virtual image together within the physical environment space of the real world or displaying a real object and a virtual image together.

[0029] In the present disclosure, an 'extended reality device' is a device capable of expressing extended reality, and may be, for example, a head-mounted display apparatus (HMD) worn on the head or face of a user, as well as an augmented reality glasses in the shape of glasses, an augmented reality helmet, etc.

[0030] In the embodiments of this specification, the term “user” may mean a person who controls a system, function, or operation.

[0031] FIG. 1 is a diagram illustrating an example of a user wearing an extended reality device according to one embodiment of the present disclosure interacting face-to-face with an external user wearing an opposite extended reality device.

[0032] When an external user with whom a user wishes to interact is wearing an augmented reality device, the user interacts with the external user with some areas of the face (e.g., eyes and nose) covered by the augmented reality device (110).

[0033] Here, the relative extended reality device may mean an extended reality device worn by an external user who interacts with a user wearing the extended reality device according to one embodiment of the present disclosure.

[0034] External users may refer to users wearing augmented reality devices.

[0035] When a user interacts with an external user while a portion of the face is obscured by an AR device, nonverbal communication expressed in that portion of the face may not be conveyed between the user and the external user. Consequently, when a user wearing an AR device interacts with an external user wearing another AR device, the usability of the interaction is reduced.

[0036] Therefore, in order to improve the usability of interaction, there is a need to display some areas of the face that are obscured by the extended reality device.

[0037] Hereinafter, the present disclosure will describe a method or device for improving usability between a user and an external user by expressing (120) an area covered by a relative extended reality device on the face of an external user wearing a relative extended reality device.

[0038] FIG. 2 is a block diagram illustrating components of an extended reality device according to one embodiment of the present disclosure.

[0039] An extended reality device (200) may include at least one eye tracking sensor (210), a communication interface (250), a memory (240) storing at least one command, and at least one processor (230) executing at least one command stored in the memory (240).

[0040] The memory (240) can store various data, programs or applications for driving and controlling the extended reality device (200) according to one embodiment of the present disclosure. The memory (240) can include, for example, a non-volatile memory including at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory, etc.), a ROM (Read-Only Memory), and an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), and a volatile memory such as a RAM (Random Access Memory) or an SRAM (Static Random Access Memory).

[0041] The memory (240) may store instructions, data structures, and program codes that can be read by the processor (230). In the following embodiments, the processor (230) may be implemented by executing instructions or codes of a program stored in the memory (240).

[0042] Programs stored in the memory (240) can be classified into multiple modules according to their functions, and may include, for example, a gaze information acquisition module (242) and a gaze information transmission module (244).

[0043] The gaze information acquisition module (242) and the gaze information transmission module (244) stored in the memory (240) refer to a unit that processes a function or operation performed by the processor (230), and this can be implemented as software such as commands, algorithms, data structures, or program codes.

[0044] However, the commands or codes of the program stored in the memory (240) are not limited thereto. For example, the stored gaze information acquisition module (242) and gaze information transmission module (244) are configurations for one embodiment, and the configuration of the modules included in the memory (240) may be integrated, added, or omitted according to the specifications of the extended reality device (200) actually implemented. That is, two or more modules may be combined into one module, or one module may be divided into two or more modules and configured.

[0045] The communication interface (250) can perform communication with an external device or server through at least one wired or wireless communication network by the processor. According to one embodiment, the communication interface can include at least one short-range communication module that performs communication according to a communication standard such as Bluetooth, Wi-Fi, BLE (Bluetooth Low Energy), NFC (Near Field Communication) / RFID (Radio-Frequency Identification), Wi-Fi Direct, UWB (Ultra-Wide Band), or ZIGBEE, and a long-range communication module that performs communication with a server for supporting long-range communication according to a long-range communication standard. The long-range communication module can perform communication through a communication network according to a 3G, 4G, and / or 5G communication standard, or a network for Internet communication.

[0046] The gaze tracking sensor (210) is a device that tracks the gaze direction of a user's eyes. The gaze tracking sensor (210) can detect the user's gaze direction by detecting an image of a person's pupil or pupils, or by detecting the direction or amount of light reflected from the cornea by near-infrared light. The gaze tracking sensor (210) includes a left-eye gaze tracking sensor and a right-eye gaze tracking sensor, and can detect the gaze direction of the user's left eye and the gaze direction of the user's right eye, respectively. Detecting the user's gaze direction may include an operation of acquiring gaze information related to the user's gaze.

[0047] In one embodiment of the present disclosure, the gaze tracking sensor (210) may include one or more infrared irradiators, multiple infrared detectors, and a gaze tracking camera. However, the present disclosure is not limited thereto, and the gaze tracking sensor (210) may be configured to include an infrared irradiator and an infrared detector, or an infrared irradiator and a gaze tracking camera.

[0048] The gaze tracking sensor (210) can obtain information about the user's eyes, including the size of the pupil, by photographing the user's eyes. In one embodiment of the present disclosure, the gaze tracking sensor (210) can photograph the user's eyes, including the pupil and iris, and provide the obtained image to the processor (230).

[0049] The specific structure and operation of the gaze tracking sensor (210) will be described later in FIGS. 5 to 7.

[0050] The processor (230) can control the overall operation of the augmented reality device. The processor (230) can operate the gaze information acquisition module (242) and the gaze information transmission module (244) by executing one or more commands stored in the memory (240).

[0051] The processor (230) may be composed of hardware components that perform arithmetic, logic, and input / output operations and signal processing. The processor (230) may be composed of at least one of, for example, a central processing unit (CPU), a microprocessor, a graphic processing unit (GRAP), an application specific integrated circuits (ASICs), a digital signal processor (DSPs), a digital signal processing device (DSPDs), a programmable logic device (PLDs), and a field programmable gate array (FPGAs), but is not limited thereto.

[0052] In FIG. 2, only essential components for explaining the operation of the extended reality device (200) are shown, and the components included in the extended reality device (200) are not limited as shown in FIG. 2.

[0053] The gaze information acquisition module (242) may be configured with instructions or program codes related to a function and / or operation of controlling at least one gaze tracking sensor (210) to acquire gaze information of both eyes of the user.

[0054] According to one embodiment of the present disclosure, a processor (230) can control at least one gaze tracking sensor (210) by executing commands or program codes of a gaze information acquisition module (242) to acquire gaze information of both eyes of a user.

[0055] The gaze information of the user's two eyes may include, but is not limited to, at least one of the position of the user's eyes or the direction of the user's gaze.

[0056] A specific method of obtaining gaze information of both eyes of a user by controlling at least one gaze tracking sensor by executing commands or program codes stored in a memory (240) of a processor (230) according to one embodiment of the present disclosure will be described later with reference to FIGS. 5 to 7.

[0057] The gaze information transmission module (244) may be configured with instructions or program codes related to the function and / or operation of transmitting gaze information of the user's two eyes to the corresponding extended reality device by controlling the communication interface (250).

[0058] According to one embodiment of the present disclosure, a processor (230) can execute commands or program codes of a gaze information transmission module (244) to control a communication interface (250) to transmit gaze information of both eyes of a user to a relative extended reality device.

[0059] The relative extended reality device may mean an extended reality device worn by an external user who interacts with a user wearing an extended reality device (200) according to one embodiment of the present disclosure.

[0060] External users may refer to users wearing augmented reality devices.

[0061] A processor (230) according to one embodiment of the present disclosure can obtain a facial image of a user not wearing an extended reality device (200) according to one embodiment of the present disclosure by executing commands or program codes stored in a memory (240).

[0062] A user can take a picture of his / her own face through a camera on his / her electronic device connected wired or wirelessly to an extended reality device (200) according to one embodiment of the present disclosure.

[0063] Here, the electronic device of the user wirelessly connected to the extended reality device (200) according to one embodiment may mean, but is not limited to, the electronic device of the user connected to the extended reality device (200) according to one embodiment of the present disclosure based on at least one of Bluetooth or Wi-Fi direct.

[0064] A processor (230) according to one embodiment of the present disclosure can control a communication interface (250) to obtain a user's face image when the user transmits a user's face image from an electronic device of the user that is wirelessly connected to an extended reality device (200) according to one embodiment of the present disclosure.

[0065] The user's electronic device means an electronic device that supports a wired or wireless connection with an extended reality device (200) according to one embodiment of the present disclosure.

[0066] The user's electronic device may include, but is not limited to, at least one of a cell phone, a digital camera, or a tablet personal computer that supports wired or wireless connections.

[0067] The user can store an image of his / her own face captured through the user's electronic device in the memory (240).

[0068] A user may store a virtual image in memory (240). The virtual image may refer to an image including a human face. The virtual image may include, but is not limited to, at least one of a computer graphic image including a human face or an avatar including a human face. The human face may include all of the eyes, nose, mouth, ears, etc. included in the human face.

[0069] The user can store a customized virtual image based on the user's settings in the memory (240).

[0070] An extended reality device according to one embodiment of the present disclosure may further include a camera.

[0071] The camera is configured to capture images (videos) by capturing the real world surrounding the extended reality device (200). The camera can capture image frames, such as still images or videos, through the image sensor when an application requiring a shooting function is executed.

[0072] The camera may include, but is not limited to, at least one of an RGB camera, a depth camera, or a time of flight (TOF) camera.

[0073] An extended reality device (200) according to one embodiment of the present disclosure can acquire a field of view (FOV) based on at least one of a still image or image frame acquired from at least one camera.

[0074] The structure of a camera according to one embodiment of the present disclosure will be described with reference to FIG. 3.

[0075] FIG. 3 is a diagram illustrating components of an extended reality device according to one embodiment of the present disclosure.

[0076] Referring to Figure 3,

[0077] An extended reality device (200) according to one embodiment of the present disclosure may further include a camera (220).

[0078] A camera (220) included in an extended reality device (200) according to one embodiment of the present disclosure may include at least one of one or more forward cameras (222, 224) or one or more downward cameras (226, 228).

[0079] The front camera (222, 224) may refer to a camera that photographs an object located in front of the extended reality device (200) according to one embodiment of the present disclosure, based on the extended reality device (200) according to one embodiment of the present disclosure.

[0080] The downward camera (226, 228) may be positioned in a downward direction to capture an object positioned downward relative to the extended reality device (200) according to one embodiment of the present disclosure. In one embodiment of the present disclosure, the downward camera (226, 228) may capture an object positioned downward relative to the extended reality device including the user's mouth.

[0081] According to one embodiment of the present disclosure, a processor (230) can execute commands or program codes stored in a memory (240) to control at least one camera (220) facing the user's face to obtain a facial image of the user.

[0082] A processor (230) according to one embodiment of the present disclosure can acquire user's facial expression data by controlling at least one camera by executing commands or program codes stored in a memory (240).

[0083] A user's facial expression data may indicate changes in facial structure due to changes in the user's facial expression. For example, changes in facial structure may include, but are not limited to, at least one of: a change in the height of the corners of the mouth or the position of the eyebrows when the user smiles; or a change in the height of the corners of the mouth when the user cries.

[0084] Referring to FIG. 3, an extended reality device (200) according to one embodiment of the present disclosure may include at least one downward-facing camera (226, 228) to capture an object positioned downward relative to the extended reality device (200) according to one embodiment of the present disclosure.

[0085] A processor (230) according to one embodiment of the present disclosure may execute commands or program codes stored in a memory (240) to obtain facial expression data of a user expressed downward relative to an extended reality device (200) according to one embodiment of the present disclosure through at least one downward camera (226, 228).

[0086] For example, the user's facial expression data may include, but is not limited to, changes in mouth size when the user opens and smiles, changes in the height of the corners of the mouth when the user cries, and the like.

[0087] For example, the user's facial expression data may include, but is not limited to, at least one of changes in mouth size when the user speaks.

[0088] A processor (230) according to one embodiment of the present disclosure can obtain user's facial expression data by controlling at least one camera (220) by executing commands or program codes stored in a memory (240).

[0089] A processor (230) according to one embodiment of the present disclosure may execute commands or program codes stored in a memory (240) to obtain facial expression data of a user through at least one camera (220) included in an extended reality device (200) according to one embodiment of the present disclosure facing the user's face.

[0090] For example, when a user smiles, at least one of the following can be obtained: eyebrow position, eye size, change in wrinkles around the eyes, or movement between the eyebrows, but this is not limited thereto.

[0091] For example, if a user cries, at least one of the following may be obtained: eyebrow position, eye size, change in wrinkles around the eyes, or movement between the eyebrows, but is not limited thereto.

[0092] A processor (230) according to one embodiment of the present disclosure may execute commands or program codes stored in a memory (240) to store data on changes in a user's facial structure according to changes in the user's facial expression when the user wears an extended reality device (200) according to one embodiment of the present disclosure in the memory (240).

[0093] A processor (230) according to one embodiment of the present disclosure may execute commands or program codes stored in a memory (240) to output a guide message that causes a user to make various facial expressions.

[0094] In one embodiment of the present disclosure, the extended reality device (200) further includes a display (not shown), and the processor (230) executes commands or program codes stored in the memory (240) to output guidance messages through the display that cause the user to make various facial expressions, but is not limited thereto.

[0095] In one embodiment of the present disclosure, the extended reality device (200) further includes a speaker (not shown), and the processor (230) executes commands or program codes stored in the memory (240) to output guidance messages that prompt the user to make various facial expressions as voice messages through the speaker. However, the present disclosure is not limited thereto.

[0096] According to one embodiment of the present disclosure, the processor (230) executes commands or program codes stored in the memory (240), and when the user makes a facial expression according to the outputted guidance message, the facial expression data according to the user's facial expression corresponding to the guidance message can be stored in the memory (240).

[0097] For example, when the processor (230) executes instructions or program codes stored in the memory (240) and outputs "Smile" (a guidance message) on the display, the user may make a smiling expression. The processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in the memory (240) and store facial expression data related to the user's smiling expression in the memory (240).

[0098] A processor (230) according to one embodiment of the present disclosure can acquire user's facial expression data through a microphone (not shown) by executing commands or program codes stored in a memory (240).

[0099] A processor (230) according to one embodiment of the present disclosure can receive a user's voice input through a microphone by executing commands or program codes stored in a memory (240).

[0100] A processor (230) according to one embodiment of the present disclosure can convert a received user's voice input into text by executing commands or program codes stored in a memory (240).

[0101] A processor (230) according to one embodiment of the present disclosure can convert a user's voice input received based on an artificial intelligence model into text by executing commands or program codes stored in a memory (240).

[0102] Artificial intelligence technology (hereinafter referred to as “AI technology”) is a technology that performs operations through a neural network to analyze and / or classify input data and obtain the desired result.

[0103] These AI technologies can be implemented using algorithms. Here, an algorithm or set of algorithms for implementing AI technologies is called a neural network. Here, a neural network can receive input data, perform the aforementioned analysis and / or classification operations, and output result data. In order for a neural network to accurately output result data corresponding to the input data, the neural network needs to be trained. Here, "training" can mean inputting various data into the neural network and training the neural network so that it can discover or acquire methods for analyzing the input data, classifying the input data, and / or extracting features necessary for generating result data from the input data. Specifically, through the learning process, the neural network can learn from training data (e.g., multiple different images) and optimize and set the weight values ​​within the neural network. Then, the neural network with the optimized weight values ​​learns the input data on its own, thereby outputting the desired result.

[0104] Specifically, a neural network can be classified as a deep neural network if the number of hidden layers, which are internal layers that perform calculations, is multiple, that is, if the depth of the neural network that performs calculations increases. Examples of neural networks include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and deep Q-networks. In addition, a neural network can be subdivided. For example, a CNN neural network can be subdivided into a deep convolutional neural network (DCNN) or a capsnet neural network.

[0105] In the disclosed embodiments, an "AI model" may refer to a neural network comprising at least one layer that receives input data and operates to output a desired result. Furthermore, an "AI model" may refer to an algorithm or a set of multiple algorithms that perform operations through a neural network to output a desired result, a processor for executing such an algorithm or a set thereof, software for executing such an algorithm or a set thereof, or hardware for executing such an algorithm or a set thereof.

[0106] Neural networks can be trained by receiving training data. The trained neural network then receives input data as input, and the output can analyze the input data and perform operations to produce the desired output data. Operations performed by neural networks can be performed through hidden layers. These hidden layers can be composed of multiple layers.

[0107] The commands or programs stored in the memory (240) according to one embodiment of the present disclosure may include an Automatic Speech Recognition model (ASR model) that converts a voice signal into text.

[0108] In one embodiment of the present disclosure, an 'automatic speech recognition model' may mean a speech recognition model that recognizes a user's speech, and a model trained to convert a speech input received from a user into text and output it.

[0109] In one embodiment of the present disclosure, the automatic speech recognition model may be an artificial intelligence model including an acoustic model, a pronunciation dictionary, and a language model.

[0110] In one embodiment of the present disclosure, the automatic speech recognition model may be an end-to-end speech recognition model with a structure that includes an integrated neural network, rather than separately including an acoustic model, a pronunciation dictionary, and a language model. By utilizing the integrated neural network, the end-to-end automatic speech recognition model can convert a speech signal into text without first recognizing phonemes from the speech signal and then converting the phonemes into text.

[0111] In one embodiment of the present disclosure, the processor (230) can convert voice input received from a user into text using an automatic speech recognition model.

[0112] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to store text expressing a human facial expression in a look-up table (LUT) in the memory (240).

[0113] For example, the processor (230) according to one embodiment of the present disclosure may store the text "happy" in a table by labeling it as (1,1), store the text "funny" in a table by labeling it as (1,2), or store the text "sad" in a table by labeling it as (2,1). However, the present disclosure is not limited thereto.

[0114] A processor (230) according to one embodiment of the present disclosure can execute commands or program codes stored in a memory (240) to store facial expression data corresponding to text expressing facial expressions stored as a table in the memory (240).

[0115] For example, a processor (230) according to one embodiment of the present disclosure may store, in the memory, expression data related to a user's smiling expression obtained through at least one camera, when the text "funny" is labeled as (1,2) and stored in a table by executing instructions or program codes stored in the memory (240).

[0116] For example, a processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to store, in a table, the text "sad" labeled as (2,1), and may store, in the memory, facial expression data related to a sad facial expression of a user acquired through at least one camera. However, the present disclosure is not limited thereto.

[0117] A processor (230) according to one embodiment of the present disclosure may execute commands or program codes stored in a memory (240) to match text converted from a received voice input with a table stored in the memory (240).

[0118] A processor (230) according to one embodiment of the present disclosure can obtain user's facial expression data based on converted text and facial expression data stored in a table by executing commands or program codes stored in a memory (240).

[0119] For example, a processor (230) according to an embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to label the text "happy" as (1,1) and store it in a table, and if the text converted from a user's voice input includes the expression "happy," the converted text may be matched to (1,1). A processor (230) according to an embodiment of the present disclosure may obtain user facial expression data corresponding to (1,1) in the table stored in the memory (240). However, the present disclosure is not limited thereto.

[0120] A processor (230) according to one embodiment of the present disclosure may execute commands or program codes stored in a memory (240) to generate first modeling data including a user's facial expression based on the acquired user's facial image and facial expression data.

[0121] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to generate first modeling data including a user's facial expression based on rendering from a user's facial image and facial expression data.

[0122] Rendering can mean converting a two-dimensional image into a three-dimensional image.

[0123] A processor (230) according to one embodiment of the present disclosure can store a general human facial muscle model in the memory (240) by executing instructions or program codes stored in the memory (240).

[0124] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to map a facial image of a user not wearing an extended reality device (200) according to one embodiment of the present disclosure to a facial muscle model stored in the memory (240).

[0125] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to map a facial image of a user not wearing an extended reality device (200) according to one embodiment of the present disclosure to a facial muscle model stored in the memory (240) based on an artificial intelligence model.

[0126] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to map muscles that induce changes in a user's facial expression in a facial image of a user not wearing an extended reality device (200) according to one embodiment of the present disclosure to a facial muscle model stored in the memory (240).

[0127] A specific method of mapping muscles that induce a change in a user's facial expression in a facial image of a user not wearing an extended reality device (200) according to an embodiment of the present disclosure to a facial muscle model stored in the memory (240) by executing instructions or program codes stored in the memory (240) by the processor (230) according to an embodiment of the present disclosure will be described with reference to FIG. 4.

[0128] FIG. 4 is a diagram for explaining a method for generating modeling data by an extended reality device (200) according to one embodiment of the present disclosure.

[0129] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to map the orbicularis oculi (410a, 410b) that induces an eye closing or opening motion in an acquired user's face image (460) to the orbicularis oculi (410c, 410d) of a facial muscle model (470).

[0130] A processor (230) according to one embodiment of the present disclosure may execute commands or program codes stored in a memory (240) to form wrinkles between the eyebrows in an acquired user's face image (460) and map corrugator muscles (420a, 420b) that induce changes in the eyebrows to corrugator muscles (420c, 420d) of a facial muscle model (470).

[0131] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to map the zygomaticus minor (430a, 430b) or the zygomaticus major (440a, 440b), which induce changes in the corners of the mouth in an acquired user's face image (460), to the zygomaticus minor (430c, 430d) or the zygomaticus major (440c, 440d) of a facial muscle model (470), respectively.

[0132] A processor (230) according to one embodiment of the present disclosure can execute instructions or program codes stored in a memory (240) to map the depressor anguli oris (450a, 450b) that induces a change in the corners of the mouth in an acquired user's face image (460) to the depressor anguli oris (450c, 450d) of a facial muscle model (470).

[0133] However, the method of mapping a user's facial image to a facial muscle model by executing commands or program codes stored in a memory (240) by a processor (230) according to an embodiment of the present disclosure described above is only one embodiment and is not limited thereto.

[0134] Referring again to Figure 2,

[0135] A processor (230) according to one embodiment of the present disclosure may synthesize a user's face image into a facial muscle model by executing instructions or program codes stored in a memory (240) and mapping muscles that induce a change in the user's facial expression in the user's face image to a facial muscle model stored in the memory (240).

[0136] Synthesis may mean combining an image of the user's face with a facial muscle model.

[0137] The synthesis may be performed based on at least one of a method of overlapping multiple images, for example, multiple user face images and facial muscle models, to form layers and adjusting the transparency of each layer for synthesis, a method of segmenting a portion of the user's face image, for example, the eyes, nose, or mouth, and synthesizing each of them onto a facial muscle model, or a method of synthesis based on an artificial intelligence model, but is not limited thereto.

[0138] A processor (230) according to one embodiment of the present disclosure can execute commands or program codes stored in a memory (240) to obtain first modeling data based on a facial muscle model to which a facial image is mapped and the obtained user's facial expression data.

[0139] According to one embodiment of the present disclosure, a processor (230) can control the movement of at least one muscle included in a facial muscle model in which muscles of a facial image that induce a change in the facial structure are mapped, by executing instructions or program codes stored in a memory (240), when the facial structure changes according to a change in the user's facial expression.

[0140] A processor (230) according to one embodiment of the present disclosure can generate first modeling data including a user's facial expression by executing instructions or program codes stored in a memory (240) to control the movement of at least one muscle included in a facial muscle model in which muscles of a facial image are mapped to induce a change in facial structure.

[0141] A processor (230) according to one embodiment of the present disclosure can transmit first modeling data to a relative extended reality device by controlling a communication interface (250) by executing commands or program codes stored in a memory (240).

[0142] A processor (230) according to one embodiment of the present disclosure may execute commands or program codes stored in a memory (240) to generate second modeling data including gaze information of both eyes of the user based on the first modeling data and gaze information.

[0143] A processor (230) according to one embodiment of the present disclosure can determine the position of the user's eyes from the acquired binocular gaze information by executing commands or program codes stored in the memory (240).

[0144] A processor (230) according to one embodiment of the present disclosure can identify the position of the user's eyes in the first modeling data based on the determined position of the user's eyes by executing instructions or program codes stored in the memory (240).

[0145] A processor (230) according to one embodiment of the present disclosure can execute commands or program codes stored in a memory (240) to synthesize the user's binocular information including the user's gaze information at the location of the user's eyes identified in the first modeling data.

[0146] The synthesis may be performed by at least one of, but not limited to, a method of forming a layer by overlapping multiple images, for example, multiple binocular images of the user, at the location of the user's eyes identified in the first modeling data, and adjusting the transparency of each layer to synthesize, a method of segmenting a portion of the user's binocular images, for example, the pupil, the iris, or the white of the eye, and synthesizing each at the location of the user's eyes identified in the first modeling data, or a method of synthesizing based on an artificial intelligence model.

[0147] A processor (230) according to one embodiment of the present disclosure can transmit second modeling data to a relative extended reality device by controlling a communication interface (250) by executing instructions or program codes stored in a memory (240).

[0148] A processor (230) according to one embodiment of the present disclosure can execute instructions or program codes stored in a memory (240) to recognize mutual agreement on transmission and reception of at least one of the user's binocular gaze information, first modeling data, or second modeling data between the user and the relative extended reality device.

[0149] The relative extended reality device may mean an extended reality device worn by an external user who interacts with a user wearing an extended reality device (200) according to one embodiment of the present disclosure.

[0150] External users may refer to users wearing augmented reality devices.

[0151] A processor (230) according to one embodiment of the present disclosure can recognize a relative extended reality device by executing instructions or program codes stored in a memory (240) and controlling at least one forward-facing camera.

[0152] A processor (230) according to one embodiment of the present disclosure can recognize an identifier included in a relative extended reality device by executing commands or program codes stored in a memory (240).

[0153] Here, the identifier may mean an image in the form of a code containing a pattern located on the front of the relative extended reality device.

[0154] The identifier may include, but is not limited to, at least one of a QR (Quick Response) code or an Aruco Marker.

[0155] A processor (230) according to one embodiment of the present disclosure may execute instructions or program code stored in a memory (240) to assign an identifier to a relative extended reality device.

[0156] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to obtain distance information between at least one camera of an extended reality device of the present disclosure and a relative extended reality device through at least one camera.

[0157] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to assign an identifier to a relative extended reality device based on acquired distance information.

[0158] For example, when the front of the camera (220) of the extended reality device (200) according to one embodiment of the present disclosure is assumed to be 0°, if a relative extended reality device exists at a distance of about 1 m and 30° to the left, an identifier of a “first relative extended reality device” can be assigned to the relative extended reality device located at a distance of about 1 m and 30° to the left.

[0159] A processor (230) according to one embodiment of the present disclosure can recognize a plurality of relative extended reality devices by executing commands or program codes stored in a memory (240).

[0160] For example, a processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to recognize a plurality of relative augmented reality devices based on at least one of a QR code or an ARCOMaker.

[0161] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to obtain identifiers for each of a plurality of relative extended reality devices.

[0162] A processor (230) according to one embodiment of the present disclosure can execute commands or program codes stored in a memory (240) to control at least one camera (220) and obtain distance information between at least one camera (220) and a plurality of relative extended reality devices, respectively.

[0163] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to assign identifiers to each of the relative extended reality devices based on the acquired distance information.

[0164] For example, when there are multiple relative extended reality devices, and assuming that the front of the camera (220) of the extended reality device (200) according to one embodiment of the present disclosure is 0°, when a first relative extended reality device is present at a distance of 30° to the left and about 1 m, and a second extended reality device is present at a distance of 30° to the right and about 2 m, the processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in the memory (240) to assign an identifier of “relative extended reality device 1” to the first relative extended reality device located at a distance of 30° to the left and about 1 m, and an identifier of “relative extended reality device 2” to the second relative extended reality device located at a distance of 30° to the right and about 2 m, respectively.

[0165] A processor (230) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (240) to perform mutual agreement for transmission and reception of at least one of the user's binocular gaze information, first modeling data, or second modeling data between the recognized at least one relative extended reality device and the user.

[0166] Mutual agreement may mean that there is consent from a user to transmit at least one of the user's binocular gaze information, first modeling data, or second modeling data to the other extended reality device by the extended reality device (200) according to one embodiment of the present disclosure, and there is consent from an external user to receive at least one of the user's binocular gaze information, first dimensional modeling data, or second modeling data.

[0167] Mutual agreement between an extended reality device (200) and a counterpart extended reality device according to one embodiment of the present disclosure can be performed by a communication interface (250) included in the extended reality device (200) and the counterpart extended reality device according to one embodiment of the present disclosure.

[0168] According to one embodiment of the present disclosure, a processor (230) may execute commands or program codes stored in a memory (240) to output a notification message for obtaining a user's input regarding whether to transmit at least one of the user's binocular gaze information, first modeling data, or second modeling data when a relative extended reality device is recognized.

[0169] In one embodiment of the present disclosure, the extended reality device (200) further includes a display (not shown), and the processor (230) executes commands or program codes stored in the memory (240) to output a notification message through the display for obtaining a user's input regarding whether to transmit at least one of the user's binocular gaze information, first modeling data, or second modeling data.

[0170] In one embodiment of the present disclosure, the extended reality device (200) further includes a speaker (not shown), and the processor (230) executes commands or program codes stored in the memory (240) to output a notification message as a voice message through the speaker to obtain a user's input regarding whether to transmit at least one of the user's binocular gaze information, first modeling data, or second modeling data. However, the present disclosure is not limited thereto.

[0171] A processor (230) according to one embodiment of the present disclosure may obtain a user's input for an outputted notification message by executing commands or program codes stored in a memory (240).

[0172] User input may include, but is not limited to, at least one of the user's gaze direction, the user's voice, or the user's hand movements.

[0173] A processor (230) according to one embodiment of the present disclosure may execute commands or program codes stored in a memory (240) to obtain information related to whether an external user wearing a relative extended reality device consents to receiving at least one of the user's binocular gaze information, first modeling data, or second modeling data through a communication interface (250).

[0174] Accordingly, the processor (230) according to one embodiment of the present disclosure can recognize a mutual agreement on transmission and reception of at least one of the gaze information of the user's two eyes, first modeling data, or second modeling data between the relative extended reality device and the user through a communication interface.

[0175] According to one embodiment of the present disclosure, the processor (230) executes instructions or program codes stored in the memory (240), and when a mutual agreement on transmission and reception of at least one of the user's binocular gaze information, first modeling data, or second modeling data is recognized between the user and the relative extended reality device, the processor (230) can transmit at least one of the user's binocular gaze information, first modeling data, or second modeling data to the relative extended reality device through the communication interface.

[0176] According to one embodiment of the present disclosure, the processor (230) executes commands or program codes stored in the memory (240) to transmit first modeling data to the relative extended reality device, and may transmit the user's gaze information acquired through at least one gaze tracking sensor to the relative extended reality device.

[0177] Accordingly, limited resources can be efficiently used in transmitting and receiving data between an extended reality device and a counterpart extended reality device according to one embodiment of the present disclosure.

[0178] FIG. 5 is a diagram illustrating a gaze tracking sensor, which is a component of an extended reality device according to one embodiment of the present disclosure.

[0179] The gaze tracking sensor (210a) may include an infrared irradiation unit (214a) and a plurality of infrared detection units (212a to 212f). Although the number of the plurality of infrared detection units (212a to 212f) is illustrated as six in FIG. 5, this is for convenience of explanation, and the number of the plurality of infrared detection units (212a to 212f) is not limited as illustrated.

[0180] The infrared irradiation unit (214a) irradiates infrared light to the cornea where the lens of the eye (E) is located, and the plurality of infrared detection units (212a to 212f) can detect infrared light reflected from the cornea. In one embodiment of the present disclosure, the gaze tracking sensor (210a) can obtain information about the amount of infrared light detected by each of the plurality of infrared detection units (212a to 212f), and obtain information about the gaze direction of the user's eye (E) based on the obtained amount of infrared light. The gaze tracking sensor (210a) can provide the information about the obtained gaze direction to the processor (230, see FIG. 2). For example, the information about the obtained gaze direction may be gaze angle information in the horizontal and vertical directions of the left eye, and gaze angle information in the horizontal and vertical directions of the right eye.

[0181] The gaze tracking sensor (210a) of this embodiment is described as an IR scanner method using infrared illumination, but is not limited thereto. As another example, the gaze tracking sensor (210a) may include an image sensor that captures an image of a human eye, pupil (10), or iris (20). A gaze tracking sensor (210b) including an image sensor will be described in detail in FIG. 6.

[0182] FIG. 6 is a diagram illustrating a gaze tracking sensor, which is a component of an extended reality device according to one embodiment of the present disclosure.

[0183] Referring to FIG. 6, the gaze tracking sensor (210b) can track the user's gaze based on the position of reflected light (611, 612, 613, 614, 615) reflected from the user's eyes (E), thereby obtaining information about the gaze direction. The gaze tracking sensor (210b) can include a light source (214b) and a camera (216).

[0184] The light source (214b) may include an infrared light emitting diode (IR LED). In the embodiment illustrated in FIG. 6, the light source (214b) may include a plurality of light emitting diodes positioned at different locations. When photographing the user's eye (E), the light source (214b) may provide light (e.g., infrared light) to the eye (E). As light is provided to the user's eye (E), reflected light may be generated that is reflected from the user's eye (E).

[0185] The camera (216) may be composed of at least one camera. The camera (216) may be implemented as an infrared camera (IR). The augmented reality device may track the gaze of the user's eye (E) using images (601 to 605) of the user's eye (E) captured by the camera (216). For example, the gaze tracking sensor (210b) may track the user's gaze by detecting the pupil (10) and reflected light (611 to 615) in the user's eye images (601 to 605), thereby obtaining a gaze vector. The gaze tracking sensor (210b) may detect the positions of the pupil (10) and reflected light (611 to 615) in the user's eye images (601 to 605), and determine the gaze direction of the user's eye (E) based on the relationship between the position of the pupil (10) and the position of the reflected light (611 to 615).

[0186] For example, the gaze tracking sensor (210b) can detect a pupil (10) and reflected light (611) in the captured first eye image (601), and determine a gaze direction (621) of the user's eyes based on the relationship between the position of the pupil (10) and the position of the reflected light (611). In the same manner, the eye tracking sensor (210b) can detect a pupil (10) and reflected light (612, 613, 614, 615) in each of the second to fifth eye images (602, 603, 604, 605), and determine a gaze direction (622, 623, 624, 625) of the user's eyes based on the relationship between the position of the pupil (10) and the position of the reflected light (612, 613, 614, 615).

[0187] In one embodiment of the present disclosure, the gaze tracking sensor (210b) can acquire a gaze vector based on information regarding the determined gaze direction. The gaze tracking sensor (210b) can provide data regarding the vector value and direction of the acquired gaze vector to the processor (230, see FIG. 2).

[0188] In another embodiment, the gaze tracking sensor (210b) provides only coordinate values ​​regarding the position of the pupil (10) and the position of the reflected light (611 to 615) detected from multiple eye images (601 to 605) to the processor (230, see FIG. 2), and the processor (230) can calculate the gaze vector of the user's eye (E) based on the coordinate values ​​obtained from the gaze tracking sensor (210b).

[0189] In one embodiment of the present disclosure, the gaze tracking sensor (210b) can provide a plurality of eye images (601 to 605) to the processor (920). The processor (920) can detect an area corresponding to the pupil (10) and the iris (20) from the plurality of eye images (601 to 605).

[0190] Figure 7 is a drawing showing a three-dimensional eye model for the user's gaze.

[0191] The eye model can be modeled by assuming that the human eye (720) is spherical and ideally rotates along the direction of gaze. Furthermore, the eye model can be mathematically expressed as in the following mathematical equations 1 and 2.

[0192] [Mathematical Formula 1]

[0193]

[0194]

[0195] [Equation 2]

[0196]

[0197]

[0198] In mathematical expression 1, d represents the distance between the center of the user's eyes (710) and the virtual screen (730). represents the angle at which the user's eyes are rotated in the x-axis direction, based on the case where the user's eyes are looking straight at the virtual screen (730). represents the angle at which the user's eyes are rotated in the y-axis direction, based on the case where the user's eyes are looking straight at the virtual screen (730). In addition, in mathematical expression 2, r represents the radius of the sphere when the user's eyes are assumed to be spheres.

[0199] According to one embodiment, the gaze tracking sensor (210) uses the method described in FIGS. 5 to 7 to measure the degree of rotation of the user's eye (e.g., the left eye) (e.g., and ) can be measured, and the extended reality device (200) can measure the degree of rotation of the user's eyes (for example, and ) can be used to calculate the two-dimensional coordinates of the user's eye gaze direction on a virtual screen.

[0200] FIG. 8 is a block diagram illustrating components of an extended reality device according to one embodiment of the present disclosure.

[0201] An extended reality device (800) may include at least one display (810), at least one camera (830), a communication interface (840), a memory (850) storing at least one command, and at least one processor (820) executing at least one command stored in the memory (850).

[0202] The display (810) may be configured as a physical device including at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode (OLED), a flexible display, a 3D display, and an electrophoretic display, but is not limited thereto.

[0203] The memory (850) can store various data, programs or applications for driving and controlling the extended reality device according to one embodiment of the present disclosure. The memory (240) can include, for example, a non-volatile memory including at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory, etc.), a ROM (Read-Only Memory), and an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), and a volatile memory such as a RAM (Random Access Memory) or an SRAM (Static Random Access Memory).

[0204] The memory (850) may store instructions, data structures, and program codes that can be read by the processor (820). In the following embodiments, the processor (820) may be implemented by executing instructions or codes of a program stored in the memory (850).

[0205] The programs stored in the memory (850) can be classified into a plurality of modules according to their functions, and may include, for example, a relative extended reality device area identification module (852), a modeling data generation module (854), and a modeling data output module (856).

[0206] The relative extended reality device area identification module (852), modeling data generation module (854), and modeling data output module (856) stored in the memory (850) refer to a unit that processes functions or operations performed by the processor (820), and this can be implemented as software such as commands, algorithms, data structures, or program codes.

[0207] However, the instructions or codes of the program stored in the memory (850) are not limited thereto. For example, the stored relative extended reality device area identification module (852), modeling data generation module (854), and modeling data output module (856) are configurations for one embodiment, and the configuration of the modules included in the memory (250) may be integrated, added, or omitted according to the specifications of the extended reality device (800) actually implemented. That is, two or more modules may be combined into one module, or one module may be divided into two or more modules and configured.

[0208] The camera (830) is configured to capture images (videos) of the real world surrounding the extended reality device (800). When an application requiring a shooting function is executed, the camera (830) can capture image frames, such as still images or videos, through an image sensor. The camera (830) may include, for example, at least one of an RGB camera, a depth camera, or a time of flight (TOF) camera, but is not limited thereto.

[0209] An extended reality device (800) according to one embodiment of the present disclosure can acquire a field of view (FOV) based on at least one still image or image frame acquired from at least one camera (830).

[0210] The processor (820) can control the overall operation of the augmented reality device. The processor (820) can operate the relative augmented reality device area identification module (852), the modeling data generation module (854), and the modeling data output module (856) by executing one or more commands stored in the memory (850).

[0211] The processor (820) may be configured with hardware components that perform arithmetic, logic, and input / output operations and signal processing. The processor (820) may be configured with at least one of, for example, a central processing unit (CPU), a microprocessor, a graphic processing unit (GRAP), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs), but is not limited thereto.

[0212] In FIG. 8, only essential components for explaining the operation of the extended reality device (800) are shown, and the components included in the extended reality device (800) are not limited as shown in FIG. 8.

[0213] The relative extended reality device area identification module (852) may be configured with instructions or program codes related to a function and / or operation of acquiring an image of an external user wearing a relative extended reality device by controlling at least one camera, and acquiring location information about an area where an image representing the relative extended reality device is located within the image of the external user.

[0214] According to one embodiment of the present disclosure, a processor (820) can acquire an image of an external user wearing a relative extended reality device by controlling at least one camera (830) by executing instructions or program codes of a relative extended reality device area identification module (852).

[0215] According to one embodiment of the present disclosure, a processor (820) can acquire an image of an external user by controlling at least one camera (830) to capture an image of an external user wearing a relative extended reality device by executing instructions or program codes of a relative extended reality device area identification module (852).

[0216] The relative extended reality device may mean an extended reality device worn by an external user who interacts with a user wearing an extended reality device (800) according to one embodiment of the present disclosure.

[0217] External users may refer to users wearing augmented reality devices.

[0218] A processor (820) according to one embodiment of the present disclosure can execute instructions or program codes of a relative extended reality device region identification module (852) to identify a region representing a relative extended reality device within an image of an external user.

[0219] According to one embodiment of the present disclosure, a processor (820) may execute instructions or program codes of a relative extended reality device area identification module (852) to obtain device information corresponding to the relative extended reality device from device information previously stored in a memory (850).

[0220] According to one embodiment of the present disclosure, a processor (820) may execute instructions or program codes of a relative extended reality device area identification module (852) to control at least one camera from device information stored in a memory (850) to obtain device information corresponding to a relative extended reality device included in an image of an external user.

[0221] Device information stored in memory (850) can be changed by the user.

[0222] Device information may mean at least one of model information of the extended reality device, size information of the extended reality device, appearance information of the extended reality device, or color information of the extended reality device, but is not limited thereto.

[0223] According to one embodiment of the present disclosure, the processor (820) may execute instructions or program codes of the relative extended reality device area identification module (852) to acquire at least one of model information, size information, appearance information, or color information of the relative extended reality device based on device information corresponding to the acquired relative extended reality device. However, the present disclosure is not limited thereto.

[0224] According to one embodiment of the present disclosure, a processor (820) can execute instructions or program codes of a relative extended reality device area identification module (852) to identify an area indicated by a relative extended reality device in an image of an external user based on device information corresponding to the acquired relative extended reality device.

[0225] A processor (820) according to one embodiment of the present disclosure may execute instructions or program codes of a relative extended reality device area identification module (852) to obtain location information about an area where an image representing a relative extended reality device is located within an image of an external user.

[0226] The location information may include location information of the relative extended reality device relative to the face of an external user wearing the relative extended reality device.

[0227] According to one embodiment of the present disclosure, a processor (820) can detect the appearance of a relative extended reality device by executing instructions or program codes of a relative extended reality device area identification module (852) to control at least one camera (830).

[0228] The appearance of the relative extended reality device may refer to the edge of the relative extended reality device.

[0229] For example, a processor (820) according to one embodiment of the present disclosure may execute instructions or program codes of a relative extended reality device area identification module (852) to detect an appearance of a relative extended reality device based on a difference in color between a face color of an external user and a color of the relative extended reality device through at least one camera.

[0230] For example, a processor (820) according to one embodiment of the present disclosure may execute instructions or program codes of a relative augmented reality device area identification module (852) to detect the appearance of a relative augmented reality device based on segmentation. Segmentation may mean clustering pixels of an image into a predetermined number of groups. However, the above-described example is only one embodiment and is not limited thereto.

[0231] According to one embodiment of the present disclosure, a processor (820) may execute instructions or program codes of a relative extended reality device area identification module (852) to obtain location information about an area where an image representing a relative extended reality device is located within an image of an external user based on an appearance of a detected relative extended reality device.

[0232] The modeling data generation module (854) is configured with instructions or program codes related to functions and / or operations for removing an area representing an identified relative extended reality device from an acquired image of an external user.

[0233] A processor (820) according to one embodiment of the present disclosure may execute instructions or program codes of a modeling data generation module (854) to remove an area representing an identified relative extended reality device from an acquired image of an external user.

[0234] According to one embodiment of the present disclosure, the processor (820) may execute instructions or program codes of the modeling data generation module (854) to remove an image area representing a relative extended reality device from an image of an external user based on location information about an area where an image representing a relative extended reality device is located.

[0235] A processor (820) according to one embodiment of the present disclosure can detect the appearance of a relative extended reality device through at least one camera by executing commands or program codes of a modeling data generation module (854).

[0236] Here, the appearance of the relative extended reality device may mean the edge of the relative extended reality device.

[0237] A processor (820) according to one embodiment of the present disclosure may execute instructions or program codes of a modeling data generation module (854) to remove an image area representing a relative augmented reality device from an image of an external user based on an appearance of a detected relative augmented reality device.

[0238] The modeling data generation module (854) may be configured with instructions or program codes related to the function and / or operation of receiving gaze information of an external user's two eyes from a relative extended reality device by controlling the communication interface (840).

[0239] According to one embodiment of the present disclosure, a processor (820) can receive information about the two eyes of an external user from a relative extended reality device by executing commands or program codes of a modeling data generation module (854) to control a communication interface (840).

[0240] The modeling data generation module (854) may further include instructions or program codes related to a function and / or operation of generating first modeling data by synthesizing an eye image corresponding to the gaze information of both eyes of an external user into an area from which an area representing a relative extended reality device has been removed from the entire area of ​​the acquired image.

[0241] According to one embodiment of the present disclosure, the processor (820) may execute commands or program codes of the modeling data generation module (854) to generate first modeling data by synthesizing an eye image corresponding to the gaze information of both eyes of an external user in an area from which an area representing a relative extended reality device has been removed among the entire area of ​​the acquired image.

[0242] The gaze information may include, but is not limited to, at least one of the position of the user's eyes or the direction of the user's gaze.

[0243] The synthesis may be performed based on at least one of a method of forming a layer by superimposing eye images corresponding to the gaze information of both eyes of an external user on an area from which an area representing a relative extended reality device has been removed from the entire area of ​​the acquired image, or a method of synthesis based on an artificial intelligence model, but is not limited thereto.

[0244] The modeling data output module (856) may further include instructions or program codes related to the function and / or operation of outputting the generated first modeling data to at least one display (810).

[0245] A processor (820) according to one embodiment of the present disclosure may execute commands or program codes of a modeling data output module (856) to output first modeling data to at least one display (810).

[0246] A processor (820) according to one embodiment of the present disclosure can execute commands or program codes stored in a memory (850) to render both eyes of an eye image in a direction corresponding to a gaze direction of an external user obtained from gaze information of both eyes of the external user.

[0247] Rendering can mean converting a two-dimensional image into a three-dimensional image.

[0248] A processor (820) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (850) to receive second modeling data from a relative extended reality device.

[0249] Here, the second modeling data may mean modeling data including the facial expression of an external user.

[0250] According to one embodiment of the present disclosure, a processor (820) may execute instructions or program codes stored in a memory (850) to receive second modeling data from a relative extended reality device through a communication interface (840).

[0251] By executing instructions or program codes stored in the processor (820) memory (850) according to one embodiment of the present disclosure, the second modeling data can be synthesized in the entire area of ​​the acquired external user's image, in an area where an area representing a relative extended reality device has been removed, to generate third modeling data.

[0252] Synthesis may mean combining second modeling data with image areas from which the relative extended reality device has been removed.

[0253] The synthesis may be performed by at least one of a method of forming layers by overlapping a plurality of images, for example, a plurality of second modeling data and an area from which the relative extended reality device has been removed, and adjusting the transparency of each layer for synthesis, a method of segmenting a portion of a user's face image included in the second modeling data, for example, an eye, a nose, or a mouth, and synthesizing each of them at a location from which the relative extended reality device has been removed, or a method of synthesis based on an artificial intelligence model, but is not limited thereto.

[0254] According to one embodiment of the present disclosure, a processor (820) may execute commands or program codes stored in a memory (850) to generate third modeling data in which a relative extended reality device is replaced with second modeling information within an image of an external user based on the acquired location information and second modeling data.

[0255] The modeling data output module (856) may further include instructions or program codes related to the function and / or operation of outputting the third modeling data to at least one display.

[0256] A processor (820) according to one embodiment of the present disclosure can execute commands or program codes of a modeling data output module (856) to output generated third modeling data to at least one display.

[0257] The modeling data output module (856) may further include instructions or program codes related to functions and / or operations for determining the output range of the first modeling data or the third modeling data.

[0258] A processor (820) according to one embodiment of the present disclosure may execute commands or program codes of a modeling data output module (856) to determine an output range of the first modeling data or the third modeling data.

[0259] A method of determining an output range of first modeling data or third modeling data by executing commands or program codes of a modeling data output module (856) according to one embodiment of the present disclosure by a processor (820) will be described with reference to FIG. 9.

[0260] FIG. 9 is a diagram illustrating a method for an extended reality device according to one embodiment of the present disclosure to output modeling data.

[0261] FIG. 9 is a drawing showing a user looking straight ahead while wearing an extended reality device (800) according to one embodiment.

[0262] A processor (820) according to one embodiment of the present disclosure can obtain gaze information of a user's two eyes through at least one gaze tracking sensor (not shown) by executing commands or program codes of a modeling data output module (856).

[0263] The gaze information may include, but is not limited to, at least one of the position of the user's eyes or the direction of the user's gaze.

[0264] A gaze tracking sensor is a device that tracks the gaze direction of a user's eyes. The gaze tracking sensor can detect the user's gaze direction by detecting an image of the human eye or pupil, or by detecting the direction or amount of light reflected from the cornea, such as near-infrared light. Such gaze tracking sensors include a left-eye gaze tracking sensor and a right-eye gaze tracking sensor, and can detect the gaze direction of the user's left eye and the gaze direction of the user's right eye, respectively. Detecting the user's gaze direction may include an operation of acquiring gaze information related to the user's gaze.

[0265] In one embodiment of the present disclosure, the gaze tracking sensor may include one or more infrared irradiators, multiple infrared detectors, and an gaze tracking camera. However, the present disclosure is not limited thereto, and the gaze tracking sensor may be configured to include an infrared irradiator and an infrared detector, or an infrared irradiator and an gaze tracking camera. The gaze tracking sensor may obtain information about the user's eyes, including the size of the pupil, by photographing the user's eyes.

[0266] In one embodiment of the present disclosure, the gaze tracking sensor may capture an image of the user's eye, including the pupil and iris, and provide the acquired image to the processor (820).

[0267] A specific method for obtaining gaze information of a user's two eyes through at least one gaze tracking sensor by an extended reality device according to an embodiment of the present disclosure has been described above with reference to FIGS. 5 to 7, and therefore, a redundant description thereof will be omitted.

[0268] A processor (820) according to one embodiment of the present disclosure may execute commands or program codes of a modeling data output module (856) to determine an output range of the first modeling data or the third modeling data based on the acquired line of sight information.

[0269] The output range may refer to the extent to which the first modeling data or the second modeling data is output.

[0270] For example, a processor (820) according to one embodiment of the present disclosure may execute commands or program codes of a modeling data output module (856) to determine an output range to output only the outline of a face included in the first modeling data or the third modeling data based on gaze information.

[0271] For example, a processor (820) according to one embodiment of the present disclosure may execute commands or program codes of a modeling data output module (856) to determine an output range to output the first modeling data or the second modeling data with a lower resolution based on the line of sight information.

[0272] For example, a processor (820) according to one embodiment of the present disclosure may execute commands or program codes of a modeling data output module (856) to determine an output range to output the facial outline, eyes, and nose among data included in the first modeling data or the third modeling data based on gaze information. However, the present disclosure is not limited thereto.

[0273] According to one embodiment of the present disclosure, a processor (820) may execute commands or program codes of a modeling data output module (856) to set a first virtual line (910) in which a user's gaze direction is set to 0°, a second virtual line (920) located to the left of the first virtual line (910) that touches the end point of the first virtual line (910), and a third virtual line (930) located to the right of the first virtual line (910) that touches the end point of the first virtual line (910).

[0274] At least one of the angle formed by the first virtual line (910) and the second virtual line (920) or the angle formed by the first virtual line (910) and the third virtual line (930) (940, 950) can be changed based on a user's input.

[0275] If no user input exists, the angle (940) formed by the first virtual line (910) and the second virtual line (920) and the angle (950) formed by the first virtual line (910) and the third virtual line (930) can be set to approximately 15° each as default values.

[0276] According to one embodiment of the present disclosure, the processor (820) may execute commands or program codes of the modeling data output module (856) to determine an output range to output all of the first modeling data or the third modeling data when a relative extended reality device exists between the second virtual line and the third virtual line based on the line of sight information (960).

[0277] According to one embodiment of the present disclosure, the processor (820) may execute commands or program codes of the modeling data output module (856) to determine an output range to output only a portion of the first modeling data or the third modeling data when a relative extended reality device exists outside the second virtual line and the third virtual line based on the line of sight information.

[0278] For example, when the relative extended reality device is located to the right of the third virtual line, the processor (820) according to one embodiment of the present disclosure may execute commands or program codes of the modeling data output module (856) to determine an output range to output the facial outline among the information included in the first modeling data or the third modeling data at a low resolution (980).

[0279] For example, when the relative augmented reality device is located to the left of the second virtual line, the processor (820) according to one embodiment of the present disclosure may execute commands or program codes of the modeling data output module (856) to determine an output range to output only the outline of the face among the information included in the first modeling data or the third modeling data (970). However, the present invention is not limited thereto.

[0280] A processor (820) according to one embodiment of the present disclosure may execute commands or program codes of a modeling data output module (856) to output first modeling data or third modeling data to at least one display according to a determined output range.

[0281] Referring again to Figure 8,

[0282] A processor (820) according to one embodiment of the present disclosure can execute instructions or program codes stored in a memory (850) to identify whether there is mutual agreement on transmission and reception of at least one of gaze information or second modeling data between the relative extended reality device and the external user.

[0283] The second modeling data may include modeling data including an external user's facial expression and an external user's gaze information, or modeling data including an external user's facial expression.

[0284] A processor (820) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (850) to recognize an extended reality device based on at least one forward-facing camera (830).

[0285] According to one embodiment of the present disclosure, the processor (820) may execute instructions or program codes stored in the memory (850) to recognize an identifier included in a relative augmented reality device. Here, the identifier may refer to an image in the form of a code including a pattern located on the front of the relative augmented reality device. The identifier may include, for example, at least one of a QR (Quick Response) code or an Aruco Marker, but is not limited thereto.

[0286] A processor (820) according to one embodiment of the present disclosure may execute instructions or program code stored in a memory (850) to assign an identifier to a relative extended reality device.

[0287] A processor (820) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (850) to obtain distance information between at least one camera (830) and a relative extended reality device through at least one camera (830).

[0288] A processor (820) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (850) to assign an identifier to a relative extended reality device based on acquired distance information.

[0289] For example, when the front of the camera (830) of the extended reality device (800) according to one embodiment of the present disclosure is assumed to be 0°, if the relative extended reality device is located 30° to the left and at a distance of about 1 m, the processor (820) according to one embodiment of the present disclosure may execute instructions or program codes stored in the memory (850) to assign an identifier of a “first relative extended reality device” to the relative extended reality device located 30° to the left and at a distance of about 1 m.

[0290] A processor (820) according to one embodiment of the present disclosure can recognize a plurality of relative extended reality devices by executing instructions or program codes stored in a memory (850).

[0291] A processor (820) according to one embodiment of the present disclosure can execute instructions or program codes stored in a memory (850) to recognize a plurality of relative augmented reality devices based on at least one of a QR code or an arcomarker.

[0292] A processor (820) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (850) to assign identifiers to each of a plurality of relative extended reality devices.

[0293] A processor (820) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (850) to obtain distance information between at least one camera (830) and a plurality of relative extended reality devices through at least one camera (830).

[0294] A processor (820) according to one embodiment of the present disclosure may execute instructions or program codes stored in a memory (850) to assign identifiers to each of the relative extended reality devices based on the acquired distance information.

[0295] For example, when there are multiple relative extended reality devices, and assuming that the front of the camera (830) of the extended reality device (800) according to one embodiment of the present disclosure is 0°, when a first relative extended reality device is present at a distance of 30° to the left and about 1 m, and a second extended reality device is present at a distance of 30° to the right and about 2 m, the processor (820) according to one embodiment of the present disclosure may execute instructions or program codes stored in the memory (850) to assign an identifier of “relative extended reality device 1” to the first relative extended reality device located at a distance of 30° to the left and about 1 m, and an identifier of “relative extended reality device 2” to the second relative extended reality device located at a distance of 30° to the right and about 2 m, respectively.

[0296] Mutual consent means that there is consent from the user to receive at least one of the external user's gaze information or second modeling data through the external extended reality device, and there is consent from the external user to transmit at least one of the external user's gaze information or second modeling data.

[0297] Mutual agreement between an extended reality device (800) and a counterpart extended reality device according to one embodiment of the present disclosure can be performed by a communication interface (840) included in the extended reality device (800) and the counterpart extended reality device according to one embodiment of the present disclosure.

[0298] The communication interface (840) can communicate with an external device or server through at least one wired or wireless communication network. According to one embodiment, the communication interface (840) can include at least one short-range communication module that performs communication according to a communication standard such as Bluetooth, Wi-Fi, BLE (Bluetooth Low Energy), NFC (Near Field Communication) / RFID (Radio-Frequency Identification), Wi-Fi Direct, UWB (Ultra-Wide Band), or ZIGBEE, and a long-range communication module that performs communication with a server for supporting long-range communication according to a long-range communication standard. The long-range communication module can communicate through a communication network according to a 3G, 4G, and / or 5G communication standard, or a network for Internet communication.

[0299] According to one embodiment of the present disclosure, a processor (820) may execute commands or program codes stored in a memory (850) to output a notification message for obtaining a user's input regarding whether to receive at least one of the external user's binocular gaze information or second modeling data when a relative extended reality device is recognized.

[0300] In one embodiment of the present disclosure, the processor (820) may execute instructions or program codes stored in the memory (850) to output a notification message through a display for obtaining a user's input regarding whether to receive at least one of the external user's binocular gaze information or second modeling data, but is not limited thereto.

[0301] In one embodiment of the present disclosure, the extended reality device (800) further includes a speaker (not shown), and the processor (820) executes commands or program codes stored in the memory (850) to output a notification message as a voice message through the speaker to obtain a user's input regarding whether to receive at least one of the external user's binocular gaze information or second modeling data. However, the present disclosure is not limited thereto.

[0302] A processor (820) according to one embodiment of the present disclosure may obtain a user's input for an outputted notification message by executing instructions or program codes stored in a memory (850).

[0303] User input may include, but is not limited to, at least one of the user's gaze direction, the user's voice, or the user's hand movements.

[0304] A processor (820) according to one embodiment of the present disclosure may obtain, through a communication interface (840), information on whether an external user wearing a relative extended reality device consents to transmitting at least one of the external user's binocular gaze information or second modeling data by executing instructions or program codes stored in a memory (850).

[0305] Accordingly, the processor (820) according to one embodiment of the present disclosure can execute instructions or program codes stored in the memory (850) to identify mutual agreement on transmission and reception of at least one of the gaze information of the two eyes of the external user or the second modeling data between the relative extended reality device and the external user through the communication interface (840).

[0306] The modeling data output module (856) may further include instructions or program codes related to a function and / or operation of outputting an image of an external user wearing a relative extended reality device or a virtual image set by the user to the display (810) when a mutual agreement on transmission and reception of gaze information of both eyes of the relative extended reality device and the external user and a mutual agreement on transmission and reception of second modeling data are not identified.

[0307] According to one embodiment of the present disclosure, the processor (820) may execute commands or program codes of the modeling data output module (856) to output an image of an external user wearing a relative extended reality device or a virtual image set by the user to the display (810) when a mutual agreement on transmission and reception of gaze information of both eyes of the relative extended reality device and the external user and a mutual agreement on transmission and reception of second modeling data are not identified.

[0308] If no mutual agreement is identified, this may mean that there is non-consent from at least one of the user or the external user regarding the transmission and reception of gaze information of the external user's eyes and the second modeling data.

[0309] A user may store a virtual image in a memory (850) according to one embodiment of the present disclosure.

[0310] The virtual image set by the user may mean an image stored in the memory (850) according to one embodiment of the present disclosure.

[0311] A virtual image may refer to an image that includes a human face. The virtual image may include, but is not limited to, at least one of a computer graphic image that includes a human face or an avatar that includes a human face. A human face may include all of the eyes, nose, mouth, ears, and other elements of a human face.

[0312] A user may store a customized virtual image based on the user's settings in a memory included in an extended reality device according to one embodiment of the present disclosure.

[0313] FIG. 10 is a flowchart illustrating an operation method of an extended reality device according to one embodiment of the present disclosure.

[0314] In step 1010, the extended reality device can obtain gaze information of the user's two eyes through at least one gaze tracking sensor.

[0315] The gaze information of the user's two eyes may include, but is not limited to, at least one of the position of the user's eyes or the direction of the user's gaze.

[0316] The method for obtaining gaze information of a user's two eyes through at least one gaze tracking sensor in an extended reality device according to one embodiment of the present disclosure has been described above with reference to FIGS. 5 to 7, and therefore, a redundant description thereof will be omitted.

[0317] At step 1020, the augmented reality device can transmit gaze information of the user's two eyes to the counterpart augmented reality device.

[0318] An extended reality device according to one embodiment of the present disclosure can transmit gaze information of a user's two eyes to a counterpart extended reality device through a communication interface.

[0319] The relative extended reality device may mean an extended reality device worn by an external user who interacts with a user wearing the extended reality device according to one embodiment of the present disclosure.

[0320] External users may refer to users wearing augmented reality devices.

[0321] FIG. 11 is a flowchart illustrating an operation method of an extended reality device according to one embodiment of the present disclosure.

[0322] Step 1020 of FIG. 11 corresponds to step 1020 of FIG. 10.

[0323] At step 1110, the augmented reality device can acquire a facial image of a user not wearing the augmented reality device.

[0324] A user can take a picture of his or her own face through a camera on an electronic device of the user connected wired or wirelessly to an extended reality device according to one embodiment of the present disclosure.

[0325] Here, the electronic device of the user wirelessly connected to the extended reality device according to one embodiment may mean, but is not limited to, the electronic device of the user connected to the extended reality device according to one embodiment of the present disclosure based on at least one of Bluetooth or Wi-Fi Direct.

[0326] The user's electronic device refers to an electronic device that supports a wired or wireless connection with an extended reality device according to one embodiment of the present disclosure. The user's electronic device may include, but is not limited to, at least one of a cell phone, a digital camera, or a tablet personal computer that supports a wired or wireless connection.

[0327] A user may store an image of his or her own face captured through the user's electronic device in a memory included in an extended reality device according to one embodiment of the present disclosure.

[0328] A user may store a virtual image in a memory included in an extended reality device according to one embodiment of the present disclosure. The virtual image may refer to an image including a human face.

[0329] The virtual image may include, but is not limited to, at least one of a computer graphic image including a human face or an avatar including a human face. The human face may include all of the eyes, nose, mouth, ears, etc. included in the human face.

[0330] A user may store a customized virtual image based on the user's settings in a memory included in an extended reality device according to one embodiment of the present disclosure.

[0331] An extended reality device according to one embodiment of the present disclosure can obtain a face image of a user through at least one camera included in the extended reality device according to one embodiment of the present disclosure facing the face of the user.

[0332] At step 1120, the extended reality device can acquire the user's facial expression data through at least one camera.

[0333] A camera is configured to capture images (videos) by capturing the real world surrounding an extended reality device according to one embodiment of the present disclosure. The camera can capture image frames, such as still images or video, through an image sensor when an application requiring a shooting function is executed. The camera may include, but is not limited to, at least one of an RGB camera, a depth camera, or a time-of-flight (TOF) camera, for example.

[0334] An extended reality device (200) according to one embodiment of the present disclosure can acquire a field of view (FOV) based on at least one of a still image or image frame acquired from at least one camera.

[0335] A user's facial expression data may indicate changes in facial structure due to changes in the user's facial expression.

[0336] Changes to facial structure may include, but are not limited to, changes in the height of the corners of the mouth or the position of the eyebrows when the user makes a smiling expression.

[0337] Changes in facial structure may include, but are not limited to, at least one of a change in the height of the corners of the mouth, for example, when the user makes a crying expression.

[0338] Referring to FIG. 3, an extended reality device according to one embodiment of the present disclosure may include at least one downward-facing camera to capture an object positioned downward relative to the extended reality device according to one embodiment of the present disclosure.

[0339] An extended reality device according to one embodiment of the present disclosure can obtain facial expression data of a user expressed downward relative to the extended reality device according to one embodiment of the present disclosure through at least one downward camera.

[0340] For example, the user's facial expression data may include, but is not limited to, changes in mouth size when the user opens and smiles, changes in the height of the corners of the mouth when the user cries, and the like.

[0341] For example, the user's facial expression data may include, but is not limited to, at least one of changes in mouth size when the user speaks.

[0342] An extended reality device according to one embodiment of the present disclosure can obtain facial expression data of a user through at least one camera included in the extended reality device according to one embodiment of the present disclosure facing the user's face.

[0343] For example, the augmented reality device may capture at least one of the following when a user smiles: the position of the eyebrows, the size of the eyes, changes in wrinkles around the eyes, or movement between the eyebrows.

[0344] For example, the augmented reality device may be able to detect at least one of, but not limited to, the position of the eyebrows, the size of the eyes, changes in wrinkles around the eyes, or movement between the eyebrows when the user cries.

[0345] An extended reality device according to one embodiment of the present disclosure can store, in memory, data on changes in the user's facial structure according to changes in the user's facial expression when the user wears the extended reality device according to one embodiment of the present disclosure.

[0346] An extended reality device according to one embodiment of the present disclosure can output a guide message that prompts a user to make various facial expressions.

[0347] An extended reality device according to one embodiment of the present disclosure may output a guidance message through a display unit that prompts a user to make various facial expressions, but is not limited thereto.

[0348] An extended reality device according to one embodiment of the present disclosure may output a guidance message as a voice message through a speaker, prompting the user to make various facial expressions. However, the present invention is not limited thereto.

[0349] An extended reality device according to one embodiment of the present disclosure can store, in memory, expression data according to the user's expression corresponding to the expression message when the user makes an expression according to the outputted guidance message.

[0350] For example, if an augmented reality device according to one embodiment of the present disclosure displays "Smile" (a guidance message) on its display, the user may make a smiling expression. The augmented reality device according to one embodiment of the present disclosure may store facial expression data related to the user's smiling expression in its memory.

[0351] An extended reality device according to one embodiment of the present disclosure can obtain user facial expression data through a microphone included in the extended reality device according to one embodiment of the present disclosure.

[0352] An extended reality device according to one embodiment of the present disclosure can receive a user's voice input through a microphone.

[0353] An extended reality device according to one embodiment of the present disclosure can convert a received user's voice input into text.

[0354] An extended reality device according to one embodiment of the present disclosure can convert a user's voice input received based on an artificial intelligence model into text.

[0355] Artificial intelligence technology (hereinafter referred to as “AI technology”) is a technology that performs operations through a neural network to analyze and / or classify input data and obtain the desired result.

[0356] These AI technologies can be implemented using algorithms. Here, an algorithm or set of algorithms for implementing AI technologies is called a neural network. Here, a neural network can receive input data, perform the aforementioned analysis and / or classification operations, and output result data. In order for a neural network to accurately output result data corresponding to the input data, the neural network needs to be trained. Here, "training" can mean inputting various data into the neural network and training the neural network so that it can discover or acquire methods for analyzing the input data, classifying the input data, and / or extracting features necessary for generating result data from the input data. Specifically, through the learning process, the neural network can learn from training data (e.g., multiple different images) and optimize and set the weight values ​​within the neural network. Then, the neural network with the optimized weight values ​​learns the input data on its own, thereby outputting the desired result.

[0357] Specifically, a neural network can be classified as a deep neural network if the number of hidden layers, which are internal layers that perform calculations, is multiple, that is, if the depth of the neural network that performs calculations increases. Examples of neural networks include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and deep Q-networks. In addition, a neural network can be subdivided. For example, a CNN neural network can be subdivided into a deep convolutional neural network (DCNN) or a capsnet neural network.

[0358] In the disclosed embodiments, an "AI model" may refer to a neural network comprising at least one layer that receives input data and operates to output a desired result. Furthermore, an "AI model" may refer to an algorithm or a set of multiple algorithms that perform operations through a neural network to output a desired result, a processor for executing such an algorithm or a set thereof, software for executing such an algorithm or a set thereof, or hardware for executing such an algorithm or a set thereof.

[0359] Neural networks can be trained by receiving training data. The trained neural network then receives input data as input, and the output can analyze the input data and perform operations to produce the desired output data. Operations performed by neural networks can be performed through hidden layers. These hidden layers can be composed of multiple layers.

[0360] An extended reality device according to one embodiment of the present disclosure can convert a user's voice input received into text based on an Automatic Speech Recognition model (ASR model).

[0361] An automatic speech recognition model is a speech recognition model that recognizes a user's speech, and may refer to a model trained to convert speech input received from a user into text and output it. In one embodiment of the present disclosure, the automatic speech recognition model may be an artificial intelligence model including an acoustic model, a pronunciation dictionary, and a language model. In one embodiment of the present disclosure, the automatic speech recognition model may be an end-to-end speech recognition model having a structure including an integrated neural network without separately including an acoustic model, a pronunciation dictionary, and a language model. By utilizing the integrated neural network, the end-to-end automatic speech recognition model can convert a speech signal into text without a process of converting the phonemes into text after recognizing phonemes from the speech signal.

[0362] An extended reality device according to one embodiment of the present disclosure can store text expressing a person's facial expression in memory as a look-up table (LUT).

[0363] For example, an extended reality device according to one embodiment of the present disclosure may store text "happy" in a table by labeling it as (1,1), store text "funny" in a table by labeling it as (1,2), or store text "sad" in a table by labeling it as (2,1). However, the present disclosure is not limited thereto.

[0364] An extended reality device according to one embodiment of the present disclosure can store facial expression data corresponding to text expressing facial expressions stored as a table in memory.

[0365] For example, an extended reality device according to one embodiment of the present disclosure may store, in memory, expression data related to a user's smiling expression obtained through at least one camera, if the text "funny" is labeled as (1,2) and stored in a table.

[0366] For example, an extended reality device according to one embodiment of the present disclosure may store, in memory, expression data related to a user's sad expression acquired through at least one camera, when the text "sad" is labeled as (2,1) and stored in a table. However, the present disclosure is not limited thereto.

[0367] An extended reality device according to one embodiment of the present disclosure can match text converted from received voice input with a table stored in memory.

[0368] An extended reality device according to one embodiment of the present disclosure can obtain user expression data based on converted text and expression data stored in a table.

[0369] For example, if an extended reality device according to an embodiment of the present disclosure stores the text "I'm happy" in a table by labeling it as (1,1), and if the text converted from a user's voice input includes the expression "I'm happy," the extended reality device according to an embodiment of the present disclosure can match the converted text to (1,1). The extended reality device according to an embodiment of the present disclosure can obtain the user's facial expression data corresponding to (1,1) in the table stored in the memory. However, the present invention is not limited thereto.

[0370] In step 1130, the extended reality device can generate first modeling data including the user's facial expression based on the acquired user's facial image and facial expression data.

[0371] An extended reality device according to one embodiment of the present disclosure can generate first modeling data including a user's facial expression based on rendering from a user's facial image and facial expression data.

[0372] Rendering can mean converting a two-dimensional image into a three-dimensional image.

[0373] An extended reality device according to one embodiment of the present disclosure can store a general facial muscle model of a human being in memory.

[0374] An extended reality device according to one embodiment of the present disclosure can map a facial image of a user not wearing the extended reality device according to one embodiment of the present disclosure to a facial muscle model stored in memory.

[0375] An extended reality device according to one embodiment of the present disclosure can map a facial image of a user not wearing the extended reality device according to one embodiment of the present disclosure to a facial muscle model stored in a memory based on an artificial intelligence model.

[0376] An extended reality device according to one embodiment of the present disclosure can map muscles that induce changes in a user's facial expression in a facial image of a user not wearing the extended reality device according to one embodiment of the present disclosure to a facial muscle model stored in a memory.

[0377] For example, an extended reality device according to one embodiment of the present disclosure may map the corrugator muscle, which forms a frown line between the eyebrows and induces a change in the eyebrows, to the corrugator muscle of a facial muscle model in an acquired user's face image.

[0378] For example, an extended reality device according to one embodiment of the present disclosure may map the orbicularis oculi muscle, which induces an eye closing or opening motion in an acquired user's face image, to the orbicularis oculi muscle of a facial muscle model.

[0379] For example, an extended reality device according to one embodiment of the present disclosure may map the zygomaticus minor or zygomaticus major muscle, which induces a change in the corner of the mouth in an acquired user's face image, to the zygomaticus minor or zygomaticus major muscle of a facial muscle model, respectively.

[0380] For example, an extended reality device according to one embodiment of the present disclosure can map the depressor anguli oris muscle, which induces a change in the corners of the mouth in an acquired user's face image, to the depressor anguli oris muscle of a facial muscle model.

[0381] However, the method of mapping a user's facial image to a facial muscle model by an extended reality device according to an embodiment of the present disclosure described above is only one embodiment and is not limited thereto.

[0382] An extended reality device according to one embodiment of the present disclosure can synthesize a user's face image onto a facial muscle model based on mapping muscles that induce changes in the user's facial expression in the user's face image onto a facial muscle model stored in memory.

[0383] Synthesis may mean combining an image of the user's face with a facial muscle model.

[0384] The synthesis may be performed based on at least one of a method of overlapping multiple images, for example, multiple user face images and facial muscle models, to form layers and adjusting the transparency of each layer for synthesis, a method of segmenting a portion of the user's face image, for example, the eyes, nose, or mouth, and synthesizing each of them onto a facial muscle model, or a method of synthesis based on an artificial intelligence model, but is not limited thereto.

[0385] An extended reality device according to one embodiment of the present disclosure can obtain first modeling data based on a facial muscle model to which a facial image is mapped and obtained user expression data.

[0386] An extended reality device according to one embodiment of the present disclosure can control the movement of at least one muscle included in a facial muscle model in which muscles of a facial image that induce a change in the facial structure are mapped when the facial structure changes according to a change in the user's facial expression.

[0387] An extended reality device according to one embodiment of the present disclosure can generate first modeling data including a user's facial expression by controlling the movement of at least one muscle included in a facial muscle model in which muscles of a facial image are mapped to induce a change in facial structure.

[0388] At step 1140, the augmented reality device can transmit the first modeling data to the counterpart augmented reality device.

[0389] An extended reality device according to one embodiment of the present disclosure can transmit first modeling data to a counterpart extended reality device via a communication interface.

[0390] An extended reality device according to one embodiment of the present disclosure can generate second modeling data including gaze information of both eyes of a user based on first modeling data and gaze information.

[0391] Referring to step 1130, an extended reality device according to an embodiment of the present disclosure can generate first modeling data including a user's facial expression by controlling the movement of at least one muscle included in a facial muscle model in which muscles of a facial image are mapped to induce a change in facial structure.

[0392] An extended reality device according to one embodiment of the present disclosure can determine the position of a user's eyes from acquired binocular gaze information.

[0393] An extended reality device according to one embodiment of the present disclosure can identify the position of the user's eyes in the first modeling data based on the determined position of the user's eyes.

[0394] An extended reality device according to one embodiment of the present disclosure can synthesize a user's binocular information including the user's gaze information into the location of the user's eyes identified in the first modeling data.

[0395] The synthesis may be performed by at least one of a method of overlapping multiple images, for example, multiple binocular images of the user, at the location of the user's eyes identified in the first modeling data to form a layer and adjusting the transparency of each layer to synthesize, a method of segmenting a portion of the user's binocular images, for example, the pupil, the iris, or the white of the eye, and synthesizing each at the location of the user's eyes identified in the first modeling data, or a method of synthesizing based on an artificial intelligence model, but is not limited thereto.

[0396] An extended reality device according to one embodiment of the present disclosure can transmit second modeling data to a counterpart extended reality device.

[0397] An extended reality device according to one embodiment of the present disclosure can transmit second modeling data to a counterpart extended reality device via a communication interface.

[0398] An extended reality device according to one embodiment of the present disclosure can recognize mutual agreement on transmission and reception of at least one of gaze information of the user's two eyes, first modeling data, or second modeling data between the other extended reality device and the user.

[0399] The relative extended reality device may mean an extended reality device worn by an external user who interacts with a user wearing the extended reality device according to one embodiment of the present disclosure.

[0400] External users may refer to users wearing augmented reality devices.

[0401] An extended reality device according to one embodiment of the present disclosure can recognize an opposing extended reality device based on at least one forward-facing camera included in the extended reality device according to one embodiment of the present disclosure.

[0402] An augmented reality device according to one embodiment of the present disclosure can recognize an identifier included in a counterpart augmented reality device. Here, the identifier may refer to an image in the form of a code including a pattern located on the front of the counterpart augmented reality device. The identifier may include, for example, at least one of a Quick Response (QR) code or an Arco Marker, but is not limited thereto.

[0403] An extended reality device according to one embodiment of the present disclosure can assign an identifier to a counterpart extended reality device.

[0404] An extended reality device according to one embodiment of the present disclosure can obtain distance information between at least one camera of the extended reality device of the present disclosure and a relative extended reality device through at least one camera.

[0405] An extended reality device according to one embodiment of the present disclosure can assign an identifier to a relative extended reality device based on acquired distance information.

[0406] For example, when assuming that the front of the camera of the extended reality device according to one embodiment of the present disclosure is 0°, if the relative extended reality device is located 30° to the left and at a distance of about 1m, the identifier of the “first relative extended reality device” can be assigned to the relative extended reality device located 30° to the left and at a distance of about 1m.

[0407] An extended reality device according to one embodiment of the present disclosure can recognize a plurality of relative extended reality devices, respectively.

[0408] For example, an extended reality device according to one embodiment of the present disclosure can recognize a plurality of relative extended reality devices based on at least one of a QR code or an arc marker.

[0409] An extended reality device according to one embodiment of the present disclosure can assign identifiers to each of a plurality of relative extended reality devices.

[0410] An extended reality device according to one embodiment of the present disclosure can obtain distance information between at least one camera of the extended reality device of the present disclosure and a plurality of relative extended reality devices through at least one camera.

[0411] An extended reality device according to one embodiment of the present disclosure can assign an identifier to each of the corresponding extended reality devices based on acquired distance information.

[0412] For example, when there are multiple relative extended reality devices, and assuming that the front of the camera of the extended reality device according to one embodiment of the present disclosure is 0°, when a first relative extended reality device is present at a distance of about 30° to the left and about 1m, and a second relative extended reality device is present at a distance of about 30° to the right and about 2m, an identifier of “relative extended reality device 1” can be assigned to the first relative extended reality device located at a distance of about 30° to the left and about 1m, and an identifier of “relative extended reality device 2” can be assigned to the second relative extended reality device located at a distance of about 30° to the right and about 2m.

[0413] An extended reality device according to one embodiment of the present disclosure can perform mutual agreement on transmission and reception of at least one of the gaze information of the user's two eyes, first modeling data, or second modeling data with at least one recognized relative extended reality device.

[0414] Here, mutual agreement may mean that there is consent from a user for the transmission of at least one of the user's binocular gaze information, first modeling data, or second modeling data by an extended reality device according to an embodiment of the present disclosure to a counterpart extended reality device, and that there is consent from an external user for the reception of at least one of the user's binocular gaze information, first dimensional modeling data, or second modeling data.

[0415] Mutual agreement between an extended reality device and a counterpart extended reality device according to one embodiment of the present disclosure can be performed by a communication interface included in the extended reality device and the counterpart extended reality device according to one embodiment of the present disclosure.

[0416] The communication interface can communicate with an external device or server via at least one wired or wireless communication network. According to one embodiment, the communication interface can include at least one short-range communication module that performs communication according to a communication standard such as Bluetooth, Wi-Fi, Bluetooth Low Energy (BLE), Near Field Communication (NFC) / Radio-Frequency Identification (RFID), Wi-Fi Direct, Ultra-Wide Band (UWB), or ZIGBEE, and a long-range communication module that performs communication with a server for supporting long-range communication according to a long-range communication standard. The long-range communication module can communicate via a communication network according to a 3G, 4G, and / or 5G communication standard, or a network for Internet communication.

[0417] An extended reality device according to one embodiment of the present disclosure may output a notification message to obtain a user's input on whether to transmit at least one of the user's binocular gaze information, first modeling data, or second modeling data when a counterpart extended reality device is recognized.

[0418] An extended reality device according to one embodiment of the present disclosure may output a notification message through a display unit to obtain a user's input regarding whether to transmit at least one of the user's binocular gaze information, first modeling data, or second modeling data. However, the present invention is not limited thereto.

[0419] An extended reality device according to one embodiment of the present disclosure may output a notification message as a voice message through a speaker to obtain a user's input regarding whether to transmit at least one of the user's binocular gaze information, first modeling data, or second modeling data. However, this is not limited thereto.

[0420] An extended reality device according to one embodiment of the present disclosure can obtain a user's input for an outputted notification message.

[0421] User input may include, but is not limited to, at least one of the user's gaze direction, the user's voice, or the user's hand movements.

[0422] An extended reality device according to one embodiment of the present disclosure can obtain information related to whether an external user wearing a relative extended reality device consents to receiving at least one of the user's binocular gaze information, first modeling data, or second modeling data through a communication interface.

[0423] Accordingly, the extended reality device according to one embodiment of the present disclosure can recognize mutual agreement on transmission and reception of at least one of the user's binocular gaze information, first modeling data, or second modeling data with the other extended reality device through a communication interface.

[0424] An extended reality device according to one embodiment of the present disclosure may transmit at least one of the user's binocular gaze information, the first modeling data, or the second modeling data to the counterpart extended reality device through a communication interface when a mutual agreement is recognized regarding transmission and reception of at least one of the user's binocular gaze information, the first modeling data, or the second modeling data between the counterpart extended reality device and the user.

[0425] When an extended reality device according to one embodiment of the present disclosure transmits first modeling data to a counterpart extended reality device, the extended reality device according to one embodiment of the present disclosure can transmit user gaze information acquired through at least one gaze tracking sensor to the counterpart extended reality device.

[0426] Accordingly, limited resources can be efficiently used in transmitting and receiving data between an extended reality device and a counterpart extended reality device according to one embodiment of the present disclosure.

[0427] FIG. 12 is a flowchart illustrating an operation method of an extended reality device according to one embodiment of the present disclosure.

[0428] At step 1210, the augmented reality device can acquire an image of an external user wearing the augmented reality device through at least one camera.

[0429] An extended reality device according to one embodiment of the present disclosure can acquire an image of an external user by photographing an external user wearing an extended reality device through at least one camera.

[0430] A relative augmented reality device may refer to an augmented reality device worn by an external user who interacts with a user wearing an augmented reality device according to one embodiment of the present disclosure. The external user may refer to a user wearing the relative augmented reality device.

[0431] At step 1220, the augmented reality device can identify an area within the acquired image that represents the relative augmented reality device.

[0432] An extended reality device according to one embodiment of the present disclosure can obtain device information corresponding to the corresponding extended reality device from device information previously stored in a memory.

[0433] An extended reality device according to one embodiment of the present disclosure can obtain device information corresponding to a relative extended reality device included in an image of an external user obtained by controlling at least one camera from device information stored in a memory.

[0434] Device information stored in memory can be changed by the user.

[0435] Device information may mean at least one of model information of the extended reality device, size information of the extended reality device, appearance information of the extended reality device, or color information of the extended reality device, but is not limited thereto.

[0436] An extended reality device according to one embodiment of the present disclosure may acquire at least one of model information, size information, appearance information, or color information of a relative extended reality device based on device information corresponding to the acquired relative extended reality device. However, the present invention is not limited thereto.

[0437] An extended reality device according to one embodiment of the present disclosure can identify an area indicated by a relative extended reality device in an image of an external user based on device information corresponding to the acquired relative extended reality device.

[0438] The extended reality device can obtain location information about an area where an image representing a relative extended reality device is located within an image of an external user.

[0439] The location information may include location information of the relative extended reality device relative to the face of an external user wearing the relative extended reality device.

[0440] An extended reality device according to one embodiment of the present disclosure can detect the appearance of a counterpart extended reality device through at least one camera. The appearance of the counterpart extended reality device may refer to an edge of the counterpart extended reality device.

[0441] For example, an extended reality device according to one embodiment of the present disclosure can detect the appearance of a counterpart extended reality device based on a difference in color between a face color of an external user and a color of the counterpart extended reality device through at least one camera.

[0442] For example, an AR device according to one embodiment of the present disclosure may detect the appearance of a corresponding AR device based on segmentation. Segmentation may refer to clustering pixels of an image into a predetermined number of groups. However, the above-described example is merely an example and is not limited thereto.

[0443] An extended reality device according to one embodiment of the present disclosure can obtain location information about an area where an image representing a relative extended reality device is located within an image of an external user based on an appearance of a detected relative extended reality device.

[0444] At step 1230, the augmented reality device may remove an area representing the identified relative augmented reality device from the acquired image of the external user.

[0445] In one embodiment of the present disclosure, another extended reality device may be capable of removing an image region representing a relative extended reality device from an image of an external user based on location information.

[0446] An extended reality device according to one embodiment of the present disclosure can detect the appearance of a counterpart extended reality device through at least one camera.

[0447] Here, the appearance of the relative extended reality device may mean the edge of the relative extended reality device.

[0448] An extended reality device according to one embodiment of the present disclosure can remove an image region representing a relative extended reality device from an image of an external user based on an appearance of a detected relative extended reality device.

[0449] At step 1240, the extended reality device can receive gaze information of the external user's two eyes from the counterpart extended reality device.

[0450] An extended reality device according to one embodiment of the present disclosure can receive gaze information of both eyes of an external user from a counterpart extended reality device through a communication interface.

[0451] In step 1250, the extended reality device can generate first modeling data by synthesizing an eye image corresponding to the gaze information of both eyes of the external user into an area from which an area representing the relative extended reality device has been removed from the entire area of ​​the acquired image.

[0452] The gaze information may include, but is not limited to, at least one of the position of the user's eyes or the direction of the user's gaze.

[0453] The synthesis may be performed based on at least one of a method of forming a layer by superimposing eye images corresponding to the gaze information of both eyes of an external user on an area from which an area representing a relative extended reality device has been removed from the entire area of ​​the acquired image, or a method of synthesis based on an artificial intelligence model, but is not limited thereto.

[0454] At step 1260, the augmented reality device can output the first modeling data to at least one display.

[0455] An extended reality device according to one embodiment of the present disclosure can render binocular images of eyes in a direction corresponding to a gaze direction of an external user obtained from gaze information of both eyes of the external user.

[0456] Rendering can mean converting a two-dimensional image into a three-dimensional image.

[0457] An extended reality device according to one embodiment of the present disclosure can receive second modeling data from a counterpart extended reality device.

[0458] An extended reality device according to one embodiment of the present disclosure can receive first modeling data from a counterpart extended reality device via a communication interface.

[0459] Here, the second modeling data may mean modeling data including the facial expression of an external user.

[0460] An extended reality device according to one embodiment of the present disclosure can generate third modeling data by synchronizing second modeling data with an area representing a relative extended reality device in an entire area of ​​an acquired image of an external user from which an area representing a relative extended reality device has been removed.

[0461] Synthesis may mean combining second modeling data with image areas from which the relative extended reality device has been removed.

[0462] The synthesis may be performed by at least one of a method of forming layers by overlapping a plurality of images, for example, a plurality of second modeling data and an area from which the relative extended reality device has been removed, and adjusting the transparency of each layer for synthesis, a method of segmenting a portion of a user's face image included in the second modeling data, for example, an eye, a nose, or a mouth, and synthesizing each of them at a location from which the relative extended reality device has been removed, or a method of synthesis based on an artificial intelligence model, but is not limited thereto.

[0463] An extended reality device according to one embodiment of the present disclosure can generate third modeling data in which a relative extended reality device is replaced with the first modeling information within an image of an external user based on the acquired location information and the second modeling data.

[0464] An extended reality device according to one embodiment of the present disclosure can output the generated third modeling data to at least one display.

[0465] FIG. 13 is a flowchart illustrating an operation method of an extended reality device according to one embodiment of the present disclosure.

[0466] Steps 1250 and 1260 of FIG. 13 correspond to steps 1250 and 1260 of FIG. 12, respectively.

[0467] In step 1262, the extended reality device can obtain gaze information of the user's two eyes through at least one gaze tracking sensor.

[0468] The gaze information may include, but is not limited to, at least one of the position of the user's eyes or the direction of the user's gaze.

[0469] A specific method for obtaining gaze information of a user's two eyes through at least one gaze tracking sensor by an extended reality device according to an embodiment of the present disclosure has been described above with reference to FIGS. 5 to 7, and therefore, a redundant description thereof will be omitted.

[0470] In step 1264, the extended reality device can determine the output range of the first modeling data or the third modeling data based on the acquired gaze information.

[0471] The output range may mean the extent to which the first modeling data or the third modeling data including the external user's facial expression and gaze information are output.

[0472] For example, an extended reality device according to one embodiment of the present disclosure may determine an output range to output only the outline of a face included in the first modeling data or the third modeling data based on gaze information.

[0473] For example, an extended reality device according to one embodiment of the present disclosure can determine an output range to output the first modeling data or the third modeling data with a lower resolution based on gaze information.

[0474] For example, an extended reality device according to one embodiment of the present disclosure may determine an output range to output the facial outline, eyes, and nose among data included in the first modeling data or the third modeling data based on gaze information. However, the present invention is not limited thereto.

[0475] An extended reality device according to one embodiment of the present disclosure may set a first virtual line in which a user's gaze direction is set to 0°, a second virtual line located to the left of the first virtual line that touches the end point of the first virtual line, and a third virtual line located to the right of the first virtual line that touches the end point of the first virtual line.

[0476] At least one of the angles formed by the first virtual line and the second virtual line or the angles formed by the first virtual line and the third virtual line can be changed based on user input.

[0477] If no user input exists, the angle between the first virtual line and the second virtual line and the angle between the first virtual line and the third virtual line can be set to approximately 15˚ by default.

[0478] An extended reality device according to one embodiment of the present disclosure can determine an output range to output all of the first modeling data or the third modeling data when a relative extended reality device exists between the second virtual line and the third virtual line based on line-of-sight information.

[0479] An extended reality device according to one embodiment of the present disclosure can determine an output range to output part or all of the first modeling data or the third modeling data when a relative extended reality device exists outside the second virtual line and the third virtual line based on line-of-sight information.

[0480] For example, when the relative extended reality device is located to the right of the third virtual line, the extended reality device according to one embodiment of the present disclosure can determine the output range to output the facial outline among the information included in the first modeling data or the third modeling data at a low resolution.

[0481] For example, when the relative augmented reality device is located to the left of the second virtual line, the augmented reality device according to one embodiment of the present disclosure may determine the output range to output only the facial outline among the information included in the first modeling data or the third modeling data. However, the present invention is not limited thereto.

[0482] In step 1266, the extended reality device can output the first modeling data or the third modeling data to at least one display according to the determined output range.

[0483] FIG. 14 is a flowchart illustrating an operation method of an extended reality device according to one embodiment of the present disclosure.

[0484] Steps 1220 and 1240 of FIG. 14 correspond to steps 1220 and 124 of FIG. 12, respectively.

[0485] In step 1235, the extended reality device can identify whether there is a mutual agreement on transmitting and receiving at least one of the gaze information or second modeling data of the counterpart extended reality device and the external user.

[0486] The second modeling data may include modeling data including an external user's facial expression and an external user's gaze information, or modeling data including an external user's facial expression.

[0487] An extended reality device according to one embodiment of the present disclosure can recognize an opposing extended reality device based on at least one forward-facing camera included in the extended reality device according to one embodiment of the present disclosure.

[0488] An augmented reality device according to one embodiment of the present disclosure can recognize an identifier included in a counterpart augmented reality device. Here, the identifier may refer to an image in the form of a code including a pattern located on the front of the counterpart augmented reality device. The identifier may include, for example, at least one of a Quick Response (QR) code or an Arco Marker, but is not limited thereto.

[0489] An extended reality device according to one embodiment of the present disclosure can assign an identifier to a counterpart extended reality device.

[0490] An extended reality device according to one embodiment of the present disclosure can obtain distance information between at least one camera of the extended reality device of the present disclosure and a relative extended reality device through at least one camera.

[0491] An extended reality device according to one embodiment of the present disclosure can assign an identifier to a relative extended reality device based on acquired distance information.

[0492] For example, when the front of the camera of the extended reality device according to one embodiment of the present disclosure is assumed to be 0°, if a relative extended reality device exists at a distance of about 1 m and 30° to the left, the extended reality device according to one embodiment of the present disclosure can assign an identifier of a “first relative extended reality device” to the relative extended reality device located at a distance of about 1 m and 30° to the left.

[0493] An extended reality device according to one embodiment of the present disclosure can recognize a plurality of relative extended reality devices.

[0494] For example, an extended reality device according to one embodiment of the present disclosure can recognize a plurality of relative extended reality devices based on at least one of a QR code or an arcomarker.

[0495] An extended reality device according to one embodiment of the present disclosure can assign identifiers to each of a plurality of relative extended reality devices.

[0496] An extended reality device according to one embodiment of the present disclosure can obtain distance information between at least one camera of the extended reality device of the present disclosure and a plurality of relative extended reality devices through at least one camera.

[0497] An extended reality device according to one embodiment of the present disclosure can assign an identifier to each of the corresponding extended reality devices based on acquired distance information.

[0498] For example, when there are multiple relative extended reality devices, and assuming that the front of the camera of the extended reality device according to one embodiment of the present disclosure is 0°, when a first relative extended reality device is present at a distance of about 30° to the left and about 1 m, and a second extended reality device is present at a distance of about 30° to the right and about 2 m, the extended reality device according to one embodiment of the present disclosure can assign an identifier of “relative extended reality device 1” to the first relative extended reality device located at a distance of about 30° to the left and about 1 m, and an identifier of “relative extended reality device 2” to the second relative extended reality device located at a distance of about 30° to the right and about 2 m, respectively.

[0499] Mutual consent means that there is consent from the user to receive at least one of the external user's binocular gaze information or second modeling data from the other extended reality device, and there is consent from the external user to transmit at least one of the external user's binocular gaze information or second modeling data.

[0500] Mutual agreement between an extended reality device and a counterpart extended reality device according to one embodiment of the present disclosure can be performed by a communication interface included in the extended reality device and the counterpart extended reality device according to one embodiment of the present disclosure.

[0501] The communication interface can communicate with an external device or server via at least one wired or wireless communication network. According to one embodiment, the communication interface can include at least one short-range communication module that performs communication according to a communication standard such as Bluetooth, Wi-Fi, Bluetooth Low Energy (BLE), Near Field Communication (NFC) / Radio-Frequency Identification (RFID), Wi-Fi Direct, Ultra-Wide Band (UWB), or ZIGBEE, and a long-range communication module that performs communication with a server for supporting long-range communication according to a long-range communication standard. The long-range communication module can communicate via a communication network according to a 3G, 4G, and / or 5G communication standard, or a network for Internet communication.

[0502] An extended reality device according to one embodiment of the present disclosure may output a notification message to obtain a user's input regarding whether to receive at least one of the external user's binocular gaze information or second modeling data when a counterpart extended reality device is recognized.

[0503] An extended reality device according to one embodiment of the present disclosure may output a notification message through a display unit to obtain a user's input regarding whether at least one of the external user's binocular gaze information or second modeling data has been received. However, the present invention is not limited thereto.

[0504] An extended reality device according to one embodiment of the present disclosure may output a notification message as a voice message through a speaker to obtain a user's input regarding whether to receive at least one of the external user's binocular gaze information or second modeling data. However, this is not limited thereto.

[0505] An extended reality device according to one embodiment of the present disclosure can obtain a user's input for an outputted notification message.

[0506] User input may include, but is not limited to, at least one of the user's gaze direction, the user's voice, or the user's hand movements.

[0507] An extended reality device according to one embodiment of the present disclosure can obtain, through a communication interface, information on whether an external user wearing a relative extended reality device consents to transmitting at least one of the external user's binocular gaze information or second modeling data.

[0508] Accordingly, the extended reality device according to one embodiment of the present disclosure can identify mutual agreement on transmission and reception of at least one of the gaze information of the two eyes of the external user or the second modeling data between the counterpart extended reality device and the external user through the communication interface.

[0509] In step 1410, if a mutual agreement on the transmission and reception of gaze information between the external user and the counterpart extended reality device and a mutual agreement on the transmission and reception of second modeling data is not identified, the extended reality device may output an image of an external user wearing the counterpart extended reality device or a virtual image set by the user to the display.

[0510] If no mutual agreement is identified, this may mean that there is non-consent from at least one of the user or the external user regarding the transmission and reception of gaze information of the external user's eyes and the second modeling data.

[0511] A user may store a virtual image in a memory included in an extended reality device according to one embodiment of the present disclosure.

[0512] A virtual image set by a user may mean an image stored in the memory of an extended reality device according to one embodiment of the present disclosure.

[0513] A virtual image may refer to an image that includes a human face. The virtual image may include, but is not limited to, at least one of a computer graphic image that includes a human face or an avatar that includes a human face. A human face may include all of the eyes, nose, mouth, ears, and other elements of a human face.

[0514] A user may store a customized virtual image based on the user's settings in a memory included in an extended reality device according to one embodiment of the present disclosure.

[0515] FIG. 15 is a flowchart illustrating an operation method of a first extended reality device and a second extended reality device according to one embodiment of the present disclosure.

[0516] In step 1510, the first extended reality device (200) can obtain gaze information of both eyes of the first user through at least one gaze tracking sensor.

[0517] The first user may refer to a user controlling the first extended reality device (200).

[0518] In step 1520, the first extended reality device (200) can transmit the acquired gaze information of the first user's two eyes to the second extended reality device (800) according to one embodiment of the present disclosure.

[0519] In step 1530, the second extended reality device (800) can acquire an image of the first user wearing the first extended reality device (200) through at least one camera.

[0520] At step 1540, the second augmented reality device (800) can identify an area representing the counterpart augmented reality device within the acquired image of the first user.

[0521] At step 1550, the second augmented reality device (800) may remove an area representing the identified relative augmented reality device from the acquired image of the first user.

[0522] At step 1560, the second extended reality device (800) can receive gaze information of the first user's two eyes from the first extended reality device (200).

[0523] A second extended reality device (800) according to one embodiment of the present disclosure can receive gaze information of both eyes of a first user from a first extended reality device (200) according to one embodiment of the present disclosure through a communication interface.

[0524] In step 1570, the second extended reality device (800) can generate modeling data by synthesizing an eye image corresponding to the gaze information of both eyes of the external user into an area from which an area representing the relative extended reality device has been removed among the entire area of ​​the acquired first user's image.

[0525] At step 1580, the second extended reality device (800) can output modeling data to at least one display.

[0526] Steps 1510, 1520, 1530, 1540, 1550, 1560, 1570 and 1580 may be performed in an extended reality device according to one embodiment of the present disclosure.

[0527] Hereinafter, an example will be described in which a user wearing an extended reality device according to one embodiment of the present disclosure looks into a mirror.

[0528] An extended reality device according to one embodiment of the present disclosure can recognize an identifier included in the extended reality device according to one embodiment of the present disclosure reflected on a mirror surface.

[0529] An extended reality device according to one embodiment of the present disclosure can obtain an image of a user reflected on a mirror surface.

[0530] An extended reality device according to one embodiment of the present disclosure can identify an area representing an extended reality device according to one embodiment of the present disclosure in an acquired image.

[0531] An extended reality device according to one embodiment of the present disclosure can remove an area identified as representing an extended reality device according to one embodiment of the present disclosure from an acquired image.

[0532] An extended reality device according to one embodiment of the present disclosure can obtain gaze information of a user's two eyes through at least one gaze tracking sensor.

[0533] An extended reality device according to one embodiment of the present disclosure can generate modeling data by synthesizing an eye image corresponding to the gaze information of both eyes of a user into an area from which an area representing the extended reality device according to one embodiment of the present disclosure has been removed from the entire area of ​​an acquired image.

[0534] An extended reality device according to one embodiment of the present disclosure can output generated modeling data to at least one display.

[0535] Accordingly, the user can check his / her own appearance before interacting with external users from the output third-party modeling data.

[0536] An extended reality device according to one embodiment of the present disclosure may include at least one display, at least one camera, a communication interface for performing data communication with a counterpart extended reality device, a memory for storing at least one command, and at least one processor for executing at least one command stored in the memory.

[0537] The at least one processor can acquire an image of the external user by controlling the at least one camera to photograph the external user wearing the relative extended reality device by executing the at least one command.

[0538] The at least one processor can identify an area representing the relative extended reality device within the acquired image by executing the at least one command.

[0539] The at least one processor can remove an area representing the identified relative extended reality device from the image by executing the at least one command.

[0540] The at least one processor can control the communication interface by executing the at least one command to receive gaze information of both eyes of the external user from the relative extended reality device.

[0541] The at least one processor can generate first modeling data by synthesizing an eye image corresponding to the gaze information of both eyes of the external user into an area from which an area representing the relative extended reality device has been removed among the entire area of ​​the image by executing the at least one command.

[0542] The at least one processor can output the first modeling data to the at least one display by executing the at least one command.

[0543] The at least one processor can render both eyes of the eye image in a direction corresponding to a gaze direction of the external user obtained from gaze information of both eyes of the external user by executing the at least one command.

[0544] The at least one processor can obtain device information corresponding to the relative extended reality device from device information previously stored in the memory by executing the at least one command.

[0545] The at least one processor can control the communication interface to receive second modeling data from the relative extended reality device by executing the at least one command.

[0546] The at least one processor can generate third modeling data by synthesizing second modeling data into an image area from which an area representing the relative extended reality device has been removed from the entire area of ​​the image by executing the at least one command.

[0547] The at least one processor can output the third modeling data to the at least one display by executing the at least one command.

[0548] The second modeling data may include an expression of the external user.

[0549] An extended reality device according to one embodiment of the present disclosure may further include at least one gaze tracking sensor.

[0550] The at least one processor can control the at least one gaze tracking sensor by executing the at least one command to obtain gaze information of the user's two eyes.

[0551] The at least one processor can determine an output range of the first modeling data or the third modeling data based on the line of sight information by executing the at least one command.

[0552] The at least one processor may output the first modeling data or the third modeling data to the at least one display (810) according to the determined output range by executing the at least one command.

[0553] The at least one processor can identify a mutual agreement on transmission and reception of at least one of the gaze information of the two eyes of the external user or the second modeling data by executing the at least one command.

[0554] The at least one processor may, by executing the at least one command, output an image of an external user wearing the relative extended reality device or a virtual image set by the user to the at least one display when a mutual agreement on transmission and reception of gaze information of both eyes of the relative extended reality device and the external user and a mutual agreement on transmission and reception of the second modeling data are not identified.

[0555] An extended reality device according to one embodiment of the present disclosure may include at least one gaze tracking sensor, a communication interface for performing data communication with a counterpart extended reality device, a memory for storing at least one command, and at least one processor for executing at least one command stored in the memory.

[0556] The at least one processor can control the at least one gaze tracking sensor by executing the at least one command to obtain gaze information of the user's two eyes.

[0557] The at least one processor can control the communication interface by executing the at least one command to transmit gaze information of the user's two eyes to the relative extended reality device.

[0558] An extended reality device according to one embodiment of the present disclosure may further include at least one camera.

[0559] The at least one processor can obtain a facial image of a user not wearing the extended reality device by executing the at least one command.

[0560] The at least one processor can control the at least one camera to obtain facial expression data of the user by executing the at least one command.

[0561] The at least one processor can generate first modeling data including the user's expression based on the facial image and the expression data by executing the at least one command.

[0562] The at least one processor can control the communication interface to transmit the first modeling data to the relative extended reality device by executing the at least one command.

[0563] The above first modeling data may include modeling data obtained based on a virtual face image set by the user and facial expression data of the user.

[0564] The at least one processor can generate second modeling data including gaze information of both eyes of the user based on the first modeling data and the gaze information by executing the at least one command.

[0565] The at least one processor can control the communication interface (250) by executing the at least one command to transmit the second modeling data to the relative extended reality device.

[0566] The at least one processor can recognize a mutual agreement on transmission and reception of at least one of the gaze information of the user's two eyes, the first modeling data, or the second modeling data by executing the at least one command.

[0567] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of acquiring an image of an external user by photographing an external user wearing an extended reality device through at least one camera.

[0568] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of identifying an area representing the relative extended reality device within the acquired image.

[0569] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of removing an area representing the identified relative extended reality device from the image.

[0570] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of receiving gaze information of both eyes of the external user from the counterpart extended reality device through a communication interface.

[0571] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of generating first modeling data by synthesizing an eye image corresponding to the gaze information of both eyes of the external user into an area from which an area representing the relative extended reality device has been removed among the entire area of ​​the image.

[0572] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of outputting the first modeling data to at least one display.

[0573] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of rendering both eyes of the eye image in a direction corresponding to a gaze direction of the external user obtained from gaze information of both eyes of the external user.

[0574] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of obtaining device information corresponding to the corresponding extended reality device from device information previously stored in a memory.

[0575] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of receiving second modeling data from the counterpart extended reality device through the communication interface.

[0576] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of generating third modeling data by synthesizing second modeling data to an image area from which an area representing the relative extended reality device has been removed from the entire area of ​​the image.

[0577] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of outputting the third modeling data to at least one display.

[0578] The second modeling data may include the facial expression of the external user.

[0579] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of acquiring gaze information of both eyes of a user through at least one gaze tracking sensor.

[0580] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of transmitting gaze information of both eyes of the user to a counterpart extended reality device through a communication interface.

[0581] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of acquiring a facial image of a user not wearing the extended reality device.

[0582] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of acquiring facial expression data of the user through at least one camera.

[0583] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of generating first modeling data including an expression of the user based on the facial image and the expression data.

[0584] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of transmitting the first modeling data to the counterpart extended reality device via the communication interface.

[0585] The above first modeling data may include modeling data obtained based on a virtual face image set by the user and facial expression data of the user.

[0586] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of generating second modeling data including gaze information of both eyes of the user based on the first modeling data and the gaze information.

[0587] A method of operating an extended reality device according to one embodiment of the present disclosure may include a step of transmitting the second modeling data to the counterpart extended reality device via a communication interface.

[0588] A method of operating a first extended reality device according to one embodiment of the present disclosure may include a step of acquiring gaze information of both eyes of a first user through at least one gaze tracking sensor.

[0589] A method of operating a first extended reality device according to one embodiment of the present disclosure may include a step of transmitting gaze information of both eyes of the first user to the second extended reality device.

[0590] A method of operating a second extended reality device according to one embodiment of the present disclosure may include a step of acquiring an image of the first user wearing the first extended reality device through at least one camera.

[0591] A method of operating a second extended reality device according to one embodiment of the present disclosure may include a step of identifying an area representing the first extended reality device in the acquired image.

[0592] A method of operating a second extended reality device according to one embodiment of the present disclosure may include a step of removing an area representing the identified first extended reality device from the image.

[0593] A method of operating a second extended reality device according to one embodiment of the present disclosure may include a step of receiving gaze information of both eyes of the first user from the first extended reality device.

[0594] A method of operating a second extended reality device according to one embodiment of the present disclosure may include a step of generating modeling data by synthesizing an eye image corresponding to gaze information of both eyes of the first user into an area from which an area representing the first extended reality device has been removed among the entire area of ​​the acquired image.

[0595] A method of operating a second extended reality device according to one embodiment of the present disclosure may include a step of outputting the modeling data to at least one display.

[0596] The method of operating an extended reality device according to one embodiment of the present disclosure may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0597] Additionally, the operating method of the extended reality device according to the disclosed embodiments may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer.

[0598] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) distributed electronically by an electronic device manufacturer or through an electronic marketplace (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic marketplace server, or a relay server that temporarily stores the software program.

[0599] In a system comprising a server and a client device, the computer program product may include a storage medium of the server or a storage medium of the client device. Alternatively, if a third device (e.g., a smartphone) exists that is communicatively connected to the server or the client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the server to the client device or the third device, or from the third device to the client device.

[0600] In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0601] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on the server, thereby controlling a client device in communication with the server to perform a method according to the disclosed embodiments.

[0602] Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims also fall within the scope of the present invention.

Claims

1. In an extended reality device (800), At least one display (810); At least one camera (830); A communication interface (840) that performs data communication with an extended reality device; A memory (850) storing at least one command; and At least one processor (820) that executes at least one instruction stored in the above memory (850), At least one processor (820) above, By controlling at least one camera (830) to capture an image of an external user wearing the relative extended reality device, Identifying an area representing the relative extended reality device within the acquired image, Remove the area representing the identified relative extended reality device from the above image, Controlling the above communication interface (840) to receive the gaze information of both eyes of the external user from the relative extended reality device, Generate first modeling data by synthesizing an eye image corresponding to the gaze information of both eyes of the external user in an area from which an area representing the relative extended reality device has been removed from the entire area of ​​the image, An extended reality device that outputs the first modeling data to at least one display (810).

2. In the first paragraph, at least one processor (820) An extended reality device that renders both eyes of the eye image in a direction corresponding to the gaze direction of the external user obtained from the gaze information of both eyes of the external user.

3. In the first or second paragraph, at least one processor (820) An extended reality device that obtains device information corresponding to the corresponding extended reality device from device information previously stored in the above memory (850).

4. In any one of the clauses 1 to 3, the at least one processor (820) Controlling the above communication interface (840) to receive second modeling data from the relative extended reality device, Generating third modeling data by synthesizing second modeling data to an image area from which an area representing the relative extended reality device has been removed in the entire area of ​​the image, Outputting the above third modeling data to at least one display (810), An extended reality device, wherein the second modeling data includes the facial expression of the external user.

5. In the extended reality device (200), At least one gaze tracking sensor (210); A communication interface (250) for performing data communication with an extended reality device; A memory (240) storing at least one command; and At least one processor (230) that executes at least one instruction stored in the above memory (240), At least one processor (230) above, Controlling at least one of the above gaze tracking sensors (210) to obtain gaze information of the user's two eyes, An extended reality device that controls the communication interface (250) to transmit gaze information of the user's two eyes to the counterpart extended reality device.

6. In paragraph 5, Including at least one additional camera, At least one processor (230) above, Obtaining a facial image of a user not wearing the above extended reality device, Controlling at least one camera to obtain facial expression data of the user, Generating first modeling data including the user's facial expression based on the facial image and the facial expression data, Controlling the above communication interface (250) to transmit the first modeling data to the relative extended reality device, An extended reality device, wherein the first modeling data includes modeling data obtained based on a virtual face image set by the user and facial expression data of the user.

7. In claim 5 or 6, at least one processor, Generate second modeling data including gaze information of both eyes of the user based on the first modeling data and the gaze information, An extended reality device that controls the communication interface (250) to transmit the second modeling data to the corresponding extended reality device.

8. In the operating method of an extended reality device, A step (S1210) of acquiring an image of an external user by photographing the external user wearing the extended reality device through at least one camera; A step (S1220) of identifying an area representing the relative extended reality device within the acquired image; A step of removing an area representing the identified relative extended reality device from the above image (S1230); A step (S1240) of receiving gaze information of both eyes of the external user from the relative extended reality device through a communication interface; A step (S1250) of generating first modeling data by synthesizing an eye image corresponding to the gaze information of both eyes of the external user into an area from which an area representing the relative extended reality device has been removed from the entire area of ​​the image; A step (S1260) of outputting the first modeling data to at least one display; An operating method, wherein the extended reality device performs data communication with the extended reality device through the communication interface.

9. In the 8th paragraph, the step of generating the first modeling data (S1250) is An operating method, comprising: a step of rendering both eyes of the eye image in a direction corresponding to a gaze direction of the external user obtained from gaze information of both eyes of the external user; 10. In the 8th or 9th paragraph, the step (S1220) of identifying the area representing the relative extended reality device is, An operating method, comprising: a step of obtaining device information corresponding to the relative extended reality device from device information previously stored in a memory.

11. In any one of clauses 8 to 10, the step (S1260) of outputting the first modeling data, A step of receiving second modeling data from the relative extended reality device through the communication interface; A step of generating third modeling data by synthesizing second modeling data into an image area from which an area representing the relative extended reality device has been removed in the entire area of ​​the image; A step of outputting the third modeling data to at least one display; A method of operation, wherein the second modeling data includes an expression of the external user.

12. In the operating method of an extended reality device, Step (S1010) of acquiring gaze information of both eyes of the user through at least one gaze tracking sensor; and A step (S1020) of transmitting the gaze information of both eyes of the user to the relative extended reality device through a communication interface; An operating method, wherein the extended reality device performs data communication with the extended reality device through the communication interface.

13. In the 12th paragraph, the operating method is, Step (S1110) of acquiring a facial image of a user not wearing the extended reality device; A step of acquiring facial expression data of the user through at least one camera (S1120); A step (S1130) of generating first modeling data including the user's facial expression based on the facial image and the facial expression data; and Further comprising a step (S1140) of transmitting the first modeling data to the relative extended reality device through the communication interface; An operating method, wherein the first modeling data includes modeling data obtained based on a virtual face image set by the user and facial expression data of the user.

14. In the 12th or 13th paragraph, the operating method is: A step of generating second modeling data including gaze information of both eyes of the user based on the first modeling data and the gaze information; and An operating method further comprising: a step of transmitting the second modeling data to the relative extended reality device through a communication interface.

15. A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 8 to 14 on a computer.

Images