User terminal and control method thereof

Abstract

The present invention relates to a user terminal which can provide a simple and secure communication connection method between user terminals, and a control method thereof. The user terminal comprises: a camera; a wireless communication unit for communicating with an external user terminal; and a control unit which controls to capture a screen, displayed by a first external user terminal among a plurality of external user terminals, through the camera, transmit information about the captured image to a management server or the first external user terminal, and automatically access the first external user terminal by using access information received from the management server or the first external user terminal.

Description

User terminal and its control method

[0001] The present disclosure relates to a user terminal capable of communicating with another user terminal and a method for controlling the same.

[0002] User terminals can be divided into mobile / portable terminals and stationary terminals based on their mobility. Furthermore, user terminals can be divided into handheld terminals and vehicle-mounted terminals based on whether the user can carry them.

[0003] The functions of user terminals are becoming increasingly diverse. Examples include data and voice communications, camera photography and video recording, audio recording, music playback through speaker systems, and display of images and videos. Some terminals have added electronic game play capabilities or multimedia player functionality. Recent user terminals, in particular, can receive multicast signals that provide visual content such as broadcasts, videos, and television programs.

[0004] As the functions of such user terminals become more diverse, they are being implemented in the form of multimedia players with complex functions such as taking pictures or videos, playing music or video files, playing games, and receiving broadcasts.

[0005] Meanwhile, VR (Virtual Reality) technology provides only CG (Computer Graphic) images of real-world objects or backgrounds, AR (Augmented Reality) technology provides virtual CG images on top of real-world object images, and MR (Mixed) technology is a computer graphics technology that provides virtual objects by mixing and combining them in the real world. The aforementioned VR, AR, and MR are all sometimes simply referred to as extended reality or XR (extended reality) technologies.

[0006] XR technology can be applied to HMD (Head-Mount Display), HUD (Head-Up Display), glasses-type glasses, mobile phones, tablet PCs, laptops, desktops, TVs, digital signage, etc., and devices to which XR technology is applied can be called XR devices.

[0007] The XR device described above may be equipped with a transparent display as a display for displaying information. Users can view real-world objects located across the transparent display through the transparent display, and can also view information provided by the extended reality device through the transparent display.

[0008] XR devices can also be understood as a type of user terminal.

[0009] Recently, XR devices are increasingly being connected and operated by other user devices, such as laptops and mobile devices. For example, XR devices can transmit the multimedia content they play to other user devices or receive and display multimedia content from other user devices.

[0010] From a user perspective, the need to connect XR devices to other user devices is becoming more frequent. Therefore, research is needed to develop a simple and secure communication method between XR devices and other user devices.

[0011] The present disclosure is proposed to solve such problems, and aims to provide a user terminal and a control method thereof that can provide a simple and secure communication connection method between user terminals.

[0012] In order to achieve the above object, the present disclosure can provide a user terminal including a camera, a wireless communication unit for communicating with an external user terminal, and a control unit for capturing a screen displayed by a first external user terminal among a plurality of external user terminals through the camera, transmitting captured image information to a management server or the first external user terminal, and automatically connecting to the first external user terminal using connection information received from the management server or the first external user terminal.

[0013] The control unit may control to receive the connection information from the management server or the first external user terminal when image identity is recognized between the captured image information and the first recorded image information of the first external user terminal.

[0014] The above control unit can control the first external user terminal to recognize the object through the camera and transmit the captured image information to the management server or the first external user terminal when the first external user terminal corresponds to a type of device capable of wireless communication connection.

[0015] The above-mentioned captured video information may include at least one of a captured video during a reference time, a reference number of captured video frames, and an image hash value of the captured video frames.

[0016] The above control unit can control the shooting image information to be transmitted together with surrounding environment information.

[0017] The above surrounding environment information may include a list of searchable wireless Wi-Fi APs (Access Points) in the vicinity.

[0018] The above-mentioned captured image information may include a first characteristic pattern of the first external user terminal.

[0019] The above control unit can control the registration of the user terminal as a user account in the management server.

[0020] The first external user terminal may be registered with the management server with the same user account.

[0021] The above connection information may include an IP (Internet Protocol) address and port number for connection to the first external user terminal.

[0022] The control unit can control to establish a first BLE connection with a first external user terminal based on a first BLE advertising signal received from a first external user terminal, and to establish a second BLE connection with a second external user terminal based on a second BLE advertising signal received from a second external user terminal among the plurality of external user terminals.

[0023] The control unit can control to transmit the captured image information to a first external user terminal through a first BLE connection with a first external user terminal, and to transmit the captured image information to a second external user terminal through a second BLE connection with a second external user terminal.

[0024] The control unit may, when image identity is recognized between the photographed image information and the first recorded image information of the first external user terminal, receive soft AP (Access Point) information for connecting to the first external user terminal as the connection information from the first external user terminal through the first BLE connection, and control to terminate the first BLE connection.

[0025] The above control unit can control to terminate the second BLE connection when the image identity is not recognized between the above-described captured image information and the second recorded image information of the second external user terminal.

[0026] The above user terminal may be an XR (Extended Reality) device.

[0027] In addition, to achieve the above purpose, the present disclosure may provide a method for controlling a user terminal, including a step of capturing a screen displayed by a first external user terminal among a plurality of external user terminals through a camera, a step of transmitting captured image information to a management server or the first external user terminal, and a step of automatically connecting to the first external user terminal using connection information received from the management server or the first external user terminal.

[0028] The effects of the user terminal and its control method according to the present disclosure are described as follows.

[0029] According to at least one aspect of the present disclosure, there is an advantage in that it can provide a simple and secure communication connection method between user terminals.

[0030] FIG. 1 is a block diagram illustrating a user terminal related to the present disclosure.

[0031] FIG. 2 is a block diagram illustrating an XR device related to the present disclosure.

[0032] FIG. 3 illustrates an example of a communication connection between an XR device and another user terminal according to one aspect of the present disclosure.

[0033] FIGS. 4 to 8 are flowcharts showing a communication connection between a user terminal and another user terminal according to one aspect of the present disclosure.

[0034] FIG. 9 is an example of a user terminal that outputs a feature pattern on a screen according to one aspect of the present disclosure.

[0035] FIG. 10 and FIG. 11 are flowcharts showing a communication connection between a user terminal and another user terminal according to one aspect of the present disclosure.

[0036] Hereinafter, embodiments disclosed in the present disclosure will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are given or used interchangeably only for the convenience of writing the disclosure, and do not in themselves have distinct meanings or roles. In addition, when describing embodiments disclosed in the present disclosure, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in the present disclosure, a detailed description thereof will be omitted. In addition, the attached drawings are only for easily understanding the embodiments disclosed in the present disclosure, and the technical idea disclosed in the present disclosure is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0037] These components may each be implemented as separate individual hardware modules, or may be implemented as two or more hardware modules, or two or more components may be implemented as one hardware module, and in some cases, they may also be implemented as software.

[0038] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0039] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0040] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, terms such as "comprises" or "have" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the disclosure, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0041] In this disclosure, the expression “at least one of A and B” may mean “A,” may mean “B,” or may mean both “A” and “B.”

[0042] The user terminals described in the present disclosure may include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), and the like. The user terminals may also be referred to as electronic devices.

[0043] However, it will be readily apparent to those skilled in the art that the configuration according to the embodiments described in the present disclosure may also be applied to fixed terminals such as digital TVs, desktop computers, digital signage, etc., except in cases where it is applicable only to user terminals.

[0044] Referring to FIG. 1, FIG. 1 is a block diagram illustrating a user terminal related to the present disclosure.

[0045] The user terminal (100) may include a wireless communication unit (110), an input unit (120), a sensing unit (140), an output unit (150), an interface unit (160), a memory (170), a control unit (180), and a power supply unit (190). The components illustrated in FIG. 1 are not essential for implementing the user terminal, and thus the user terminal described in this specification may have more or fewer components than the components listed above.

[0046] More specifically, among the above components, the wireless communication unit (110) may include one or more modules that enable wireless communication between a user terminal (100) and a wireless communication system, between a user terminal (100) and another user terminal (100), or between a user terminal (100) and an external server. In addition, the wireless communication unit (110) may include one or more modules that connect the user terminal (100) to one or more networks.

[0047] This wireless communication unit (110) may include at least one of a broadcast reception module (111), a mobile communication module (112), a wireless Internet module (113), a short-range communication module (114), and a location information module (115).

[0048] The input unit (120) may include a camera (121) or a video input unit for inputting a video signal, a microphone (122) or an audio input unit for inputting an audio signal, and a user input unit (123, for example, a touch key, a mechanical key, etc.) for receiving information from a user. Voice data or image data collected by the input unit (120) may be analyzed and processed into a user's control command.

[0049] The sensing unit (140) may include one or more sensors for sensing at least one of information within the user terminal, information about the surrounding environment surrounding the user terminal, and user information. For example, the sensing unit (140) may include at least one of a proximity sensor (141), an illumination sensor (142), a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor), a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor (e.g., a camera (see 121)), a microphone (see 122), a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric recognition sensor, etc.). Meanwhile, the user terminal disclosed in this specification can utilize information sensed by at least two of these sensors in combination.

[0050] The output unit (150) is for generating output related to visual, auditory, or tactile sensations, and may include at least one of a display unit (151), an audio output unit (152), a haptic module (153), and an optical output unit (154). The display unit (151) may be formed as a layer structure with a touch sensor or formed as an integral part, thereby implementing a touch screen. This touch screen may function as a user input unit (123) that provides an input interface between the user terminal (100) and the user, and at the same time, may provide an output interface between the user terminal (100) and the user.

[0051] The interface unit (160) serves as a passageway for various types of external devices connected to the user terminal (100). This interface unit (160) may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, and an earphone port. In response to the external device being connected to the interface unit (160), the user terminal (100) may perform appropriate control related to the connected external device.

[0052] In addition, the memory (170) stores data that supports various functions of the user terminal (100). The memory (170) can store a plurality of application programs (or applications) running on the user terminal (100), data for the operation of the user terminal (100), and commands. At least some of these application programs can be downloaded from an external server via wireless communication. In addition, at least some of these application programs can exist on the user terminal (100) from the time of shipment for the basic functions of the user terminal (100) (e.g., call receiving, calling, message receiving, calling). Meanwhile, the application programs can be stored in the memory (170), installed on the user terminal (100), and driven by the control unit (180) to perform operations (or functions) of the user terminal.

[0053] In addition to the operations related to the above application program, the control unit (180) typically controls the overall operation of the user terminal (100). The control unit (180) can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above or by operating an application program stored in the memory (170).

[0054] In addition, the control unit (180) can control at least some of the components discussed with reference to FIG. 1 to drive an application program stored in the memory (170). Furthermore, the control unit (180) can operate at least two or more of the components included in the user terminal (100) in combination to drive the application program.

[0055] The power supply unit (190) receives external power and internal power under the control of the control unit (180) and supplies power to each component included in the user terminal (100). The power supply unit (190) includes a battery, and the battery may be a built-in battery or a replaceable battery.

[0056] At least some of the above components may cooperate with each other to implement the operation, control, or control method of the user terminal according to the various embodiments described below. In addition, the operation, control, or control method of the user terminal may be implemented on the user terminal by running at least one application program stored in the memory (170).

[0057] Below, before examining various embodiments implemented through the user terminal (100) discussed above, the components listed above will be examined in more detail with reference to FIG. 1.

[0058] First, regarding the wireless communication unit (110), the broadcast reception module (111) of the wireless communication unit (110) receives broadcast signals and / or broadcast-related information from an external broadcast management server via a broadcast channel. The broadcast channel may include a satellite channel or a terrestrial channel. Two or more of the broadcast reception modules may be provided in the mobile terminal (100) for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

[0059] The above broadcast management server may refer to a server that generates and transmits broadcast signals and / or broadcast-related information, or a server that receives previously generated broadcast signals and / or broadcast-related information and transmits them to a terminal. The above broadcast signals may include not only TV broadcast signals, radio broadcast signals, and data broadcast signals, but may also include broadcast signals in the form of TV broadcast signals or radio broadcast signals combined with data broadcast signals.

[0060] The above broadcast signal may be encoded according to at least one of the technical standards (or broadcasting methods, for example, ISO, IEC, DVB, ATSC, etc.) for transmitting and receiving digital broadcast signals, and the broadcast receiving module (111) may receive the digital broadcast signal using a method suitable for the technical specifications set forth in the above technical standards.

[0061] The above broadcast-related information may refer to information related to a broadcast channel, broadcast program, or broadcast service provider. The above broadcast-related information may also be provided via a mobile communications network. In such cases, it may be received by the mobile communications module (112).

[0062] The above broadcast-related information may exist in various forms, such as, for example, the Electronic Program Guide (EPG) of DMB (Digital Multimedia Broadcasting) or the Electronic Service Guide (ESG) of DVB-H (Digital Video Broadcast-Handheld). The broadcast signal and / or broadcast-related information received through the broadcast reception module (111) may be stored in the memory (170).

[0063] The mobile communication module (112) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network constructed according to technical standards or communication methods for mobile communication (e.g., GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G, etc.).

[0064] The above wireless signal may include various forms of data such as voice call signals, video call signals, or text / multimedia message transmission and reception.

[0065] The wireless Internet module (113) refers to a module for wireless Internet access, and can be built into or externally installed in the user terminal (100). The wireless Internet module (113) is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

[0066] Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G, etc., and the wireless Internet module (113) transmits and receives data according to at least one wireless Internet technology, including Internet technologies not listed above.

[0067] From the perspective that wireless Internet access through WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A, 5G, etc. is achieved through a mobile communication network, the wireless Internet module (113) that performs wireless Internet access through the mobile communication network may be understood as a type of the mobile communication module (112).

[0068] The short-range communication module (114) is for short-range communication, and can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies. The short-range communication module (114) can support wireless communication between a user terminal (100) and a wireless communication system, between a user terminal (100) and another user terminal (100), or between a user terminal (100) and a network where another user terminal (100, or an external server) is located through a short-range wireless communication network (Wireless Area Networks). The short-range wireless communication network may be a short-range wireless personal area network (Wireless Personal Area Networks).

[0069] Here, the other user terminal (100) may be a wearable device (e.g., a smartwatch, smart glasses, or a head mounted display (HMD)) capable of exchanging data with (or linking with) the user terminal (100) according to the present disclosure. The short-range communication module (114) may detect (or recognize) a wearable device capable of communicating with the user terminal (100) around the user terminal (100). Furthermore, if the detected wearable device is a device authenticated to communicate with the user terminal (100) according to the present disclosure, the control unit (180) may transmit at least a portion of the data processed in the user terminal (100) to the wearable device via the short-range communication module (114). Accordingly, the user of the wearable device may utilize the data processed in the user terminal (100) via the wearable device. For example, according to this, when a call is received on the user terminal (100), the user can make a phone call through the wearable device, or when a message is received on the user terminal (100), the user can check the received message through the wearable device.

[0070] The location information module (115) is a module for obtaining the location (or current location) of the user terminal, and representative examples thereof include a GPS (Global Positioning System) module or a WiFi (Wireless Fidelity) module. For example, if the user terminal utilizes a GPS module, the user terminal can obtain the location of the user terminal using signals transmitted from GPS satellites. As another example, if the user terminal utilizes a Wi-Fi module, the user terminal can obtain the location of the user terminal based on information from a wireless access point (AP) that transmits or receives wireless signals with the Wi-Fi module. If necessary, the location information module (115) may perform the function of any of the other modules of the wireless communication unit (110) to obtain data regarding the location of the user terminal as a substitute or in addition. The location information module (115) is a module used to obtain the location (or current location) of the user terminal, and is not limited to a module that directly calculates or obtains the location of the user terminal.

[0071] Next, the input unit (120) is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For inputting image information, the user terminal (100) may be equipped with one or more cameras (121). The camera (121) processes image frames such as still images or moving images obtained by an image sensor in a video call mode or a shooting mode. The processed image frames may be displayed on the display unit (151) or stored in the memory (170). Meanwhile, the plurality of cameras (121) equipped in the user terminal (100) may be arranged to form a matrix structure, and through the cameras (121) forming the matrix structure in this way, a plurality of image information having various angles or focal points may be input to the user terminal (100). In addition, the plurality of cameras (121) may be arranged in a stereo structure to obtain left and right images for implementing a three-dimensional image. Additionally, the plurality of cameras (121) may include a depth camera and / or a TOF (Time of Flight) camera for sensing a subject in three dimensions.

[0072] The microphone (122) processes external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function being performed (or the application being executed) on the user terminal (100). Meanwhile, the microphone (122) can implement various noise removal algorithms to remove noise generated during the process of receiving external acoustic signals.

[0073] The user input unit (123) is for receiving information from the user. When information is input through the user input unit (123), the control unit (180) can control the operation of the user terminal (100) to correspond to the input information. The user input unit (123) may include a mechanical input means (or a mechanical key, for example, a button located on the front, back, or side of the user terminal (100), a dome switch, a jog wheel, a jog switch, etc.) and a touch input means. As an example, the touch input means may be composed of a virtual key, a soft key, or a visual key displayed on a touch screen through software processing, or a touch key placed on a part other than the touch screen. Meanwhile, the virtual key or visual key may be displayed on the touch screen in various forms, and may be formed of, for example, graphics, text, icons, videos, or a combination thereof.

[0074] Meanwhile, the sensing unit (140) senses at least one of information within the user terminal, information regarding the surrounding environment surrounding the user terminal, and user information, and generates a sensing signal corresponding thereto. Based on this sensing signal, the control unit (180) can control the operation or behavior of the user terminal (100), or perform data processing, functions, or operations related to an application program installed in the user terminal (100). Representative sensors among the various sensors that can be included in the sensing unit (140) will be examined in more detail.

[0075] First, a proximity sensor (141) refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing nearby without mechanical contact by using the force of an electromagnetic field or infrared rays. This proximity sensor (141) may be placed in an internal area of ​​a user terminal covered by the touch screen described above or near the touch screen.

[0076] Examples of proximity sensors (141) include a transmissive photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high-frequency oscillation proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. In the case where the touch screen is capacitive, the proximity sensor (141) may be configured to detect the proximity of a conductive object by a change in an electric field caused by the object's proximity. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

[0077] Meanwhile, for the convenience of explanation, an act of bringing an object close to the touch screen without touching it so that it is recognized that the object is located on the touch screen is called a "proximity touch," and an act of actually making contact with the touch screen is called a "contact touch." The position at which an object is touched close to the touch screen means the position at which the object corresponds vertically to the touch screen when the object is touched close to the touch screen. The proximity sensor (141) can detect a proximity touch and a proximity touch pattern (e.g., proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement status, etc.). Meanwhile, the control unit (180) processes data (or information) corresponding to the proximity touch action and proximity touch pattern detected through the proximity sensor (141) as described above, and further outputs visual information corresponding to the processed data on the touch screen. Furthermore, the control unit (180) can control the user terminal (100) so that different actions or data (or information) are processed depending on whether a touch on the same point on the touch screen is a proximity touch or a contact touch.

[0078] The touch sensor detects a touch (or touch input) applied to a touch screen (or display unit (151)) using at least one of various touch methods, such as a resistive method, a capacitive method, an infrared method, an ultrasonic method, and a magnetic field method.

[0079] As an example, a touch sensor may be configured to convert changes in pressure applied to a specific portion of a touch screen or electrostatic capacity generated at a specific portion, etc., into an electrical input signal. The touch sensor may be configured to detect a location, area, pressure at the time of touch, electrostatic capacity at the time of touch, etc., of a touch target that applies a touch to the touch screen on the touch sensor. Here, the touch target is an object that applies a touch to the touch sensor, and may be, for example, a finger, a touch pen or a stylus pen, a pointer, etc.

[0080] In this way, when there is a touch input to the touch sensor, the corresponding signal(s) is sent to the touch controller. The touch controller processes the signal(s) and then transmits the corresponding data to the control unit (180). As a result, the control unit (180) can determine which area of ​​the display unit (151) has been touched, etc. Here, the touch controller may be a separate component from the control unit (180) or may be the control unit (180) itself.

[0081] Meanwhile, the control unit (180) may perform different or identical controls depending on the type of touch target that touches the touch screen (or a touch key provided in addition to the touch screen). Whether to perform different or identical controls depending on the type of touch target may be determined based on the current operating status of the user terminal (100) or the running application program.

[0082] Meanwhile, the touch sensor and proximity sensor discussed above can independently or in combination sense various types of touches, such as short (or tap) touch, long touch, multi-touch, drag touch, flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, etc. on the touch screen.

[0083] An ultrasonic sensor can recognize the location information of a detection target using ultrasonic waves. Meanwhile, the control unit (180) can calculate the location of a wave generation source using information detected by a light sensor and multiple ultrasonic sensors. The location of the wave generation source can be calculated by taking advantage of the fact that light is much faster than ultrasonic waves, that is, the time it takes for light to reach the light sensor is much faster than the time it takes for ultrasonic waves to reach the ultrasonic sensor. More specifically, the location of the wave generation source can be calculated by using the time difference between the time at which ultrasonic waves arrive and the time at which light is a reference signal.

[0084] Meanwhile, the camera (121) as seen in the configuration of the input unit (120) includes at least one of a camera sensor (e.g., CCD, CMOS, etc.), a photo sensor (or image sensor), and a laser sensor.

[0085] A camera (121) and a laser sensor can be combined with each other to detect the touch of a detection target on a three-dimensional stereoscopic image. The photo sensor can be laminated on a display element, and the photo sensor is configured to scan the movement of a detection target close to the touch screen. More specifically, the photo sensor scans the contents placed on the photo sensor by using an electrical signal that changes according to the amount of light applied to the photo diode by mounting a photo diode and a TR (transistor) in rows / columns. That is, the photo sensor performs coordinate calculation of the detection target according to the amount of change in light, and through this, the location information of the detection target can be acquired.

[0086] The display unit (151) displays (outputs) information processed in the user terminal (100). For example, the display unit (151) can display execution screen information of an application program running in the user terminal (100), or UI (User Interface) or GUI (Graphical User Interface) information according to such execution screen information.

[0087] Additionally, the display unit (151) may be configured as a stereoscopic display unit that displays a stereoscopic image.

[0088] The above stereoscopic display unit can be applied with a three-dimensional display method such as a stereoscopic method (glasses method), an autostereoscopic method (glasses-free method), or a projection method (holographic method).

[0089] The audio output unit (152) can output audio data received from the wireless communication unit (110) or stored in the memory (170) in a call signal reception mode, call mode or recording mode, voice recognition mode, broadcast reception mode, etc. The audio output unit (152) also outputs audio signals related to functions performed in the user terminal (100) (e.g., call signal reception sound, message reception sound, etc.). The audio output unit (152) may include a receiver, a speaker, a buzzer, etc.

[0090] The haptic module (153) generates various tactile effects that can be felt by the user. A representative example of the tactile effect generated by the haptic module (153) may be vibration. The intensity and pattern of the vibration generated by the haptic module (153) may be controlled by the user's selection or the settings of the control unit. For example, the haptic module (153) may synthesize and output different vibrations or output them sequentially.

[0091] In addition to vibration, the haptic module (153) can generate various tactile effects, such as effects caused by stimulation such as pin arrangements moving vertically with respect to the contact skin surface, air injection or suction force through nozzles or suction ports, brushing against the skin surface, contact with electrodes, electrostatic force, and effects caused by reproduction of hot and cold sensations using elements capable of absorbing or exotherming heat.

[0092] The haptic module (153) can not only transmit tactile effects through direct contact, but can also be implemented so that the user can feel the tactile effects through the muscle senses of the fingers or arms, etc. Two or more haptic modules (153) may be provided depending on the configuration of the user terminal (100).

[0093] The light output unit (154) outputs a signal to notify the occurrence of an event using light from a light source of the user terminal (100). Examples of events occurring in the user terminal (100) may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, and information reception through an application.

[0094] The signal output from the light output unit (154) is implemented by the user terminal emitting light of a single color or multiple colors to the front or rear. The signal output may be terminated when the user terminal detects the user's event confirmation.

[0095] The interface unit (160) acts as a passageway for all external devices connected to the user terminal (100). The interface unit (160) receives data from an external device, supplies power, and transmits it to each component inside the user terminal (100), or allows data inside the user terminal (100) to be transmitted to an external device. For example, a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, an HDMI (High Definition Multimedia Interface) port, a USB (Universal Serial Bus) port, a Thunderbolt port, a DisplayPort, etc. may be included in the interface unit (160). When the user terminal (100) is connected to the external device through the wireless communication unit (110), the wireless communication unit (110) can be understood as a type of interface unit (160).

[0096] Meanwhile, the identification module is a chip that stores various information for authenticating the usage rights of the user terminal (100), and may include a user identity module (UIM), a subscriber identity module (SIM), a universal subscriber identity module (USIM), etc. A device equipped with an identification module (hereinafter referred to as an "identification device") may be manufactured in the form of a smart card. Accordingly, the identification device may be connected to the terminal (100) through the interface unit (160).

[0097] In addition, the interface unit (160) may serve as a passage through which power from the cradle is supplied to the user terminal (100) when the user terminal (100) is connected to an external cradle, or may serve as a passage through which various command signals input by the user from the cradle are transmitted to the user terminal (100). Various command signals or the power input from the cradle may function as signals for recognizing that the user terminal (100) is correctly mounted on the cradle.

[0098] The memory (170) can store a program for the operation of the control unit (180), and can also temporarily store input / output data (e.g., phonebook, messages, still images, videos, etc.). The memory (170) can store data regarding various patterns of vibration and sound output when a touch input is made on the touch screen.

[0099] The memory (170) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The user terminal (100) may also be operated in relation to web storage that performs the storage function of the memory (170) on the Internet.

[0100] Meanwhile, as previously discussed, the control unit (180) controls operations related to the application program and, typically, the overall operation of the user terminal (100). For example, if the state of the user terminal satisfies a set condition, the control unit (180) may execute or release a lock state that restricts the user's input of control commands for applications.

[0101] In addition, the control unit (180) may perform control and processing related to voice calls, data communications, video calls, etc., or may perform pattern recognition processing that can recognize handwriting input or drawing input performed on a touch screen as characters and images, respectively. Furthermore, the control unit (180) may control any one or a combination of the components described above in order to implement various embodiments described below on a user terminal (100) according to the present disclosure.

[0102] The power supply unit (190) receives external power and internal power under the control of the control unit (180) and supplies the power required for the operation of each component. The power supply unit (190) includes a battery, and the battery may be a built-in battery that is rechargeable and may be detachably connected to the terminal body for charging, etc.

[0103] Additionally, the power supply unit (190) may be equipped with a connection port, and the connection port may be configured as an example of an interface (160) to which an external charger that supplies power for charging a battery is electrically connected.

[0104] As another example, the power supply unit (190) may be configured to wirelessly charge the battery without using the above-described connection port. In this case, the power supply unit (190) may receive power from an external wireless power transmission device using at least one of an inductive coupling method based on a magnetic induction phenomenon and a magnetic resonance coupling method based on an electromagnetic resonance phenomenon.

[0105] Meanwhile, the various embodiments described below may be implemented in a recording medium readable by a computer or similar device, for example, using software, hardware, or a combination thereof.

[0106] Referring to FIG. 2, an XR device related to the present disclosure will be described. FIG. 2 is a block diagram for explaining an XR device related to the present disclosure. The XR device can be understood as a type of user terminal of FIG. 1. The drawing reference numeral of the XR device of FIG. 2 will be denoted as 100A, and the drawing reference numeral of the user terminal of FIG. 1 will be denoted as 100B. In order to distinguish the components of the XR device of FIG. 2 from the components of other user terminals, the drawing reference numeral of each component of the XR device will be denoted with “A.”

[0107] The XR device (100A) may include more components than those described in FIG. 2. For example, the XR device (100A) may further include some of the components of the user terminal (100b) of FIG. 1.

[0108] The XR device (100A) of FIG. 2 is exemplified as being implemented as an HMD (Head-Mount Display) type, but it can of course also be configured as an XR glass or goggle type.

[0109] The XR device (100A) may include a wireless communication unit (110A), a control unit (180A), a memory (130A), an input unit (120A), an output unit (150A), and a power supply unit (190A).

[0110] Each component of the XR device (100A) may correspond to the corresponding component described in the user terminal (100b) of Fig. 1. A detailed description of each component of the XR device (100A) will be omitted.

[0111] Hereinafter, with reference to FIG. 3, a communication connection between an XR device (100A) and another user terminal (100B) will be described. FIG. 3 illustrates an example of a communication connection between an XR device and another user terminal according to one aspect of the present disclosure.

[0112] As illustrated in FIG. 3, the wireless communication unit (110A) of the XR device (100A) and the wireless communication unit (110) of the user terminal (100B) can be wirelessly connected to each other. The XR device (100A) and the user terminal (100B) can be wirelessly connected to each other, for example, according to an Internet communication method or a Wi-Fi communication method, but are not limited thereto.

[0113] The XR device (100A) and the user terminal (100B) can share multimedia content with each other through a wireless communication connection between them.

[0114] For example, an XR device (100A) can share multimedia content it plays with a user terminal (100B). Alternatively, the user terminal (100B) can share multimedia content it plays with the XR device (100A).

[0115] Accordingly, the content screen (300A) displayed by the XR device (100A) and the content screen (300B) displayed by the user terminal (100B) may be mirrored screens of each other.

[0116] Hereinafter, with reference to FIGS. 4 and 5, a process for an XR device (100A) to establish communication connection with a user terminal desired by a user among multiple other user terminals will be described. FIGS. 4 and 5 are flowcharts illustrating a process for a user terminal to establish communication connection with another user terminal according to one aspect of the present disclosure.

[0117] Hereinafter, the description will be given assuming that the first user terminal (100A) is an XR device (100A). However, other types of user terminals (e.g., smartphones, tablet PCs, etc.) other than the XR device (100A) may also be used as the first user terminal (100A). The names of each component of the first user terminal (100A) will be indicated by "first", and the drawing symbols of each component of the first user terminal (100A) will be indicated by "A."

[0118] Hereinafter, it will be assumed that there are a second user terminal (100B) and a third user terminal (100C) as multiple other user terminals. However, it will be understood that there may be at least one other user terminal as multiple other user terminals in addition to the second user terminal (100B) and the third user terminal (100C). The second user terminal (100B) and the third user terminal (100C) may each be, for example, any one of a smartphone, a tablet PC, and a laptop. The name of each component of the second user terminal (100B) will be indicated with "second", and the name of each component of the third user terminal (100C) will be indicated with "third". In addition, the drawing reference numeral of each component of the second user terminal (100B) will be indicated with "B", and the drawing reference numeral of each component of the third user terminal (100C) will be indicated with "C".

[0119] In FIGS. 4 and 5, it is assumed that the first user terminal (100A), the second user terminal (100B), and the third user terminal (100C) are each capable of Internet access.

[0120] First, the first user terminal (100A), the second user terminal (100B), and the third user terminal (100C) can each log in to the management server (200) using the same user account [S401, S402, S403]. The management server (200) may be, for example, a cloud server. That is, the first user terminal (100A), the second user terminal (100B), and the third user terminal (100C) can be managed as a single group using user accounts. If multiple user terminals can be managed as a single group, multiple user accounts may be used.

[0121] There is no requirement for a chronological order between steps S401, S402, and S403. Any step may be executed before the others. These steps may also be executed concurrently.

[0122] The first user terminal (100A), the second user terminal (100B), and the third user terminal (100C) can register their devices with the management server (200) by transmitting their respective device information (e.g., device identifiers such as MAC (Media Access Control) addresses) to the management server (200) [S404, S405, S406]. There is no requirement for a temporal order between steps S404, S405, and S406. Any step may be executed before the other steps. These steps may also be executed simultaneously.

[0123] The management server (200) can store registered device information for each user account [S407]. Accordingly, the management server (200) can know which devices (i.e., the first user terminal (100A), the second user terminal (100B), and the third user terminal (100C)) are registered for the same user account.

[0124] The first user terminal (100A) (or the first control unit (180A) of the first user terminal (100A)) can execute a connection application for connection with another user terminal [S411]. The execution of the connection application of the first user terminal (100A) may be performed in response to a user command input through the first input unit (120A), or may be performed automatically in response to the occurrence of a predetermined event (e.g., Internet connection). If the connection application is already running in the foreground or background on the first user terminal (100A), step S411 may be omitted.

[0125] A user of a first user terminal (100A) may wish to wirelessly connect one of a second user terminal (100B) and a third user terminal (100C) (hereinafter, a target device) to the first user terminal (100A). Hereinafter, it will be assumed that the target device is the second user terminal (100B).

[0126] In this case, the user can trigger a wireless communication connection between the first user terminal (100A) and the second user terminal (100B) by wearing the first user terminal (100A), i.e., the XR device, and looking at the second user terminal (100B), which is the target device [S412]. When the wireless communication connection between the first user terminal (100A) and the second user terminal (100B) is triggered, the first user terminal (100A) can display an alarm graphic (and / or text) notifying that the wireless communication connection has been triggered. The alarm graphic can be displayed at least until step S501, which will be described later, is completed.

[0127] Here, the user wearing the XR device (100A) and gazing at the second user terminal (100B) may mean activating the camera of the XR device (100A) and allowing the second user terminal (100B) to be within the field of view of the camera. Accordingly, if the first user terminal (100A) is a device of another type (e.g., a smartphone or a tablet PC) other than the XR device (100A), the step S412 may be performed by activating the camera of the other type of device and directing the camera toward the target device so that the target device is within the field of view of the camera. A wireless communication connection between the first user terminal (100A) and the second user terminal (100B) may be triggered when the target device remains within the field of view of the activated camera for a predetermined period of time (e.g., 1 second or 2 seconds).

[0128] When a wireless communication connection is triggered, the first user terminal (100A) recognizes the appearance of the second user terminal (100B), which is a target device, as an object, and the second user terminal (100B) can recognize that the first user terminal (100A) is a type of device capable of wireless communication connection [S413]. Information regarding the type of device capable of wireless communication connection of the first user terminal (100A) may be pre-stored in the first memory (170A) of the first user terminal (100A).

[0129] If the second user terminal (100B) is recognized as a type of device capable of wireless communication connection, the first user terminal (100A) can request a wireless communication connection with the target device to the management server (200) [S414].

[0130] Then, the management server (200) can transmit a shooting request signal (or message) to the first user terminal (100A) requesting that the first user terminal (100A) shoot (or capture) the second user terminal (100B) (i.e., the display screen displayed on the second user terminal (100B)) which is the target device for a reference time (or the reference frame number) [S415].

[0131] In addition, the management server (200) can transmit a recording request signal (or message) to each of the other user terminals (i.e., the second user terminal (100B) and the third user terminal (100C)) managed as a group such as the first user terminal (100A) to request recording (or capturing) the screen currently being displayed by the other user terminal for a reference time (or for the reference frame number) [S416, S417].

[0132] The time at which the management server (200) transmits a shooting request signal to the first user terminal (100A) and the time at which the management server (200) transmits a recording request signal to other user terminals (100B, 100C) may be substantially the same or within a predetermined margin time.

[0133] Referring to FIG. 5, while the first user terminal (100A) is looking at the second user terminal (100B), in response to a shooting request signal received from the management server (200), the first user terminal (100A) can shoot the display screen currently displayed on the second user terminal (100B) for a reference time (or as many as the reference frame number) [S501].

[0134] The second user terminal (100B) (or the second control unit (180B) of the second user terminal (100B)) can record or capture the screen currently being displayed by the second user terminal (100B) for a reference time (or for the reference frame number) in response to a recording request signal received from the management server (200) [S502].

[0135] The third user terminal (100C) (or the third control unit (180C) of the third user terminal (100C)) can record or capture the screen currently being displayed by the third user terminal (100C) for a reference time (or for the reference frame number) in response to a recording request signal received from the management server (200) [S503].

[0136] As described above, since the transmission time of the shooting request signal and the transmission time of the recording request signal are substantially the same or within a predetermined margin time, steps S501, S502, and S503 can be performed substantially simultaneously (or simultaneously within a predetermined margin time).

[0137] The first user terminal (100A) can transmit the captured video (or the number of captured video frames corresponding to the number of reference frames) to the management server (200) [S504]. To reduce transmission weight, the first user terminal (100A) can also transmit the captured image hash value of the captured video frame to the management server (200) instead of the captured video frame.

[0138] The second user terminal (100B) can transmit the first recorded video (or the number of first captured video frames corresponding to the number of reference frames) recorded by the second user terminal (100B) to the management server (200) [S505]. In order to reduce transmission weight, the second user terminal (100B) can also transmit the first image hash value of the first captured video frame to the management server (200) instead of the first captured video.

[0139] The third user terminal (100C) can transmit the second recorded video (or the number of second captured video frames equal to the number of reference frames) recorded by the third user terminal (100C) to the management server (200) [S506]. In order to reduce the transmission weight, the third user terminal (100C) can also transmit the second image hash value of the second captured video frame to the management server (200) instead of the second captured video frame.

[0140] There is no requirement for a chronological order between steps S504, S505, and S506. Any step may be executed before the others. These steps may also be executed concurrently.

[0141] The management server (200) can compare the captured video received from the first user terminal (100A) with the first recorded video received from the second user terminal (100B) to determine whether there is image identity between the captured video and the second recorded video [S511]. Alternatively, the management server (200) can compare the captured video frame received from the first user terminal (100A) with the first captured video frame received from the second user terminal (100B) to determine whether there is image identity between the captured video frame and the first captured video frame. Alternatively, the management server (200) can compare the captured image hash value received from the first user terminal (100A) with the first image hash value received from the second user terminal (100B) to determine whether there is image identity between the captured image hash value and the first image hash value.

[0142] Since the first user terminal (100A) took a picture or captured the display screen currently being displayed on the second user terminal (100B), there is a high probability that there is image identity between the image information received from the first user terminal (100A) and the image information received from the second user terminal (100B). The explanation will continue assuming that there is image identity between them.

[0143] The existence of image identity between these may mean that the second user terminal (100B) is identified as the target device. If the management server (200) determines that there is image identity, it may transmit an image identity recognition signal to the second user terminal (100B) [S512].

[0144] Meanwhile, the management server (200) can compare the captured video received from the first user terminal (100A) with the second recorded video received from the third user terminal (100C) to determine whether there is image identity between the captured video and the second recorded video [S513]. Alternatively, the management server (200) can compare the captured video frame received from the first user terminal (100A) with the second captured video frame received from the third user terminal (100C) to determine whether there is image identity between the captured video frame and the second captured video frame. Alternatively, the management server (200) can compare the captured image hash value received from the first user terminal (100A) with the second image hash value received from the third user terminal (100C) to determine whether there is image identity between the captured image hash value and the second image hash value.

[0145] Since the first user terminal (100A) photographed or captured the display screen currently being displayed on the second user terminal (100B), the likelihood of image identity between the image information received from the first user terminal (100A) and the image information received from the third user terminal (100C) is quite low. The explanation will continue assuming there is no image identity between them.

[0146] If it is not recognized that there is image identity between them, the management server (200) can transmit an image identity non-recognition signal to the third user terminal (100C) [S514].

[0147] There is no requirement that there be a temporal order between steps S511 and S512 and steps S513 and S514. Steps S511 and S512 may be executed after steps S513 and S514. Alternatively, steps S511 and S512 may be executed substantially simultaneously with steps S513 and S514.

[0148] The second user terminal (100B) can transmit its connection information (e.g., IP (Internet Protocol) address and port number) to the management server (200) in response to receiving the video identity recognition signal [S521].

[0149] The management server (200) can again transmit the connection information of the second user terminal (100B) to the first user terminal (100A) [S522].

[0150] The first user terminal (100A) can automatically connect to the second user terminal (100B), which is the target device, using the connection information of the second user terminal (100B) [S523].

[0151] That is, a wireless communication connection can be established between the first user terminal (100A) and the second user terminal (100B) simply by the user wearing the XR device (100A), which is the first user terminal (100A), and looking at the second user terminal (100B), which is the target device.

[0152] Figures 4 and 5 describe target device identification based on image information received from each user terminal. However, to more accurately identify the target device, environmental information surrounding each user terminal may be further considered in addition to the image information received from each user terminal. This will be further described with reference to Figure 6. Figure 6 is a flowchart illustrating a communication connection between a user terminal and another user terminal according to one aspect of the present disclosure.

[0153] In FIG. 6, steps S401 to S417 of FIG. 4 and steps S501 to S503 of FIG. 5 are executed identically, so description thereof will be omitted for the sake of brevity of the present disclosure.

[0154] The first user terminal (100A) can transmit the captured video (or the number of captured video frames corresponding to the number of reference frames) and its surrounding environment information to the management server (200) [S604]. In order to reduce the transmission weight, the first user terminal (100A) can also transmit the captured image hash value of the captured video frame to the management server (200) instead of the captured video frame. The surrounding environment information can include location information such as GPS of the first user terminal (100A) and / or a list of surrounding wireless Wi-Fi Access Points (APs) that the first user terminal (100A) can search for (for example, a list of SSIDs (Service Set Identifiers) of surrounding Wi-Fi APs that can be searched for).

[0155] The second user terminal (100B) can transmit the first recorded video (or the number of first captured video frames corresponding to the number of reference frames) recorded by the second user terminal (100B) and surrounding environment information to the management server (200) [S605]. In order to reduce transmission weight, the second user terminal (100B) can also transmit the first image hash value of the first captured video frame to the management server (200) instead of the first captured video. The surrounding environment information can include location information such as GPS of the second user terminal (100B) and / or a list of surrounding wireless Wi-Fi Access Points (APs) that the second user terminal (100B) can search for (for example, a list of SSIDs (Service Set Identifiers) of surrounding Wi-Fi APs that can be searched for).

[0156] The third user terminal (100C) can transmit the second recorded video (or the number of second captured video frames corresponding to the number of reference frames) recorded by the third user terminal and surrounding environment information to the management server (200) [S606]. In order to reduce the transmission weight, the third user terminal (100C) can also transmit the second image hash value of the second captured video frame to the management server (200) instead of the second captured video frame. The surrounding environment information can include location information such as GPS of the third user terminal (100C) and / or a list of surrounding wireless Wi-Fi Access Points (APs) that the third user terminal (100C) can search for (for example, a list of SSIDs (Service Set Identifiers) of surrounding Wi-Fi APs that can be searched for).

[0157] There is no requirement for a chronological order between steps S604, S605, and S606. Any step may be executed before the others. These steps may also be executed concurrently.

[0158] The management server (200) can compare the captured video received from the first user terminal (100A) with the first recorded video received from the second user terminal (100B) to determine whether there is image identity between the captured video and the second recorded video [S611]. Alternatively, the management server (200) can compare the captured video frame received from the first user terminal (100A) with the first captured video frame received from the second user terminal (100B) to determine whether there is image identity between the captured video frame and the first captured video frame. Alternatively, the management server (200) can compare the captured image hash value received from the first user terminal (100A) with the first image hash value received from the second user terminal (100B) to determine whether there is image identity between the captured image hash value and the first image hash value.

[0159] The management server (200) may further consider the surrounding environment information received from each of the first user terminal (100A) and the second user terminal (100B) when determining the image identity between the captured video and the first recorded video [S611]. For example, the management server (200) may further consider the surrounding environment information received from each of the first user terminal (100A) and the second user terminal (100B), and may determine that there is image identity between the captured video and the first recorded video when there is a surrounding environment identity between them above a predetermined level.

[0160] The fact that the surrounding environment identity is above a predetermined level may mean that the location information of the first user terminal (100A) and the location information of the second user terminal (100B) are close within a predetermined distance. Alternatively, the fact that the surrounding environment identity is above a predetermined level may mean that the number of identical items between the list of wireless Wi-Fi APs searchable by the first user terminal (100A) and the list of wireless Wi-Fi APs searchable by the second user terminal (100B) is above a predetermined number or above a predetermined ratio.

[0161] Since the first user terminal (100A) has photographed or captured the display screen currently being displayed on the second user terminal (100B), there is a high possibility that there is environmental identity between the environmental information received from each of the first user terminal (100A) and the second user terminal (100B), as well as image identity between the image information received from the first user terminal (100A) and the image information received from the second user terminal (100B). The explanation will continue assuming that there is environmental identity and image identity between them.

[0162] The existence of environmental identity and image identity between these devices may mean that the second user terminal (100B) is identified as the target device. If the management server (200) determines that environmental identity and image identity exist, it may transmit an image identity recognition signal to the second user terminal (100B) [S612].

[0163] Meanwhile, the management server (200) can compare the captured video received from the first user terminal (100A) with the second recorded video received from the third user terminal (100C) to determine whether there is image identity between the captured video and the second recorded video [S613]. Alternatively, the management server (200) can compare the captured video frame received from the first user terminal (100A) with the second captured video frame received from the third user terminal (100C) to determine whether there is image identity between the captured video frame and the second captured video frame. Alternatively, the management server (200) can compare the captured image hash value received from the first user terminal (100A) with the second image hash value received from the third user terminal (100C) to determine whether there is image identity between the captured image hash value and the second image hash value.

[0164] The management server (200) may further consider the surrounding environment information received from each of the first user terminal (100A) and the third user terminal (100C) when determining the image identity between the captured video and the second recorded video [S613]. For example, the management server (200) may further consider the surrounding environment information received from each of the first user terminal (100A) and the third user terminal (100C), and may determine that there is image identity between the captured video and the second recorded video when there is a surrounding environment identity between them above a predetermined level.

[0165] The fact that the surrounding environment identity is above a predetermined level may mean that the location information of the first user terminal (100A) and the location information of the third user terminal (100C) are close within a predetermined distance. Alternatively, the fact that the surrounding environment identity is above a predetermined level may mean that the number of identical items between the list of wireless Wi-Fi APs searchable by the first user terminal (100A) and the list of wireless Wi-Fi APs searchable by the third user terminal (100C) is above a predetermined number or above a predetermined ratio.

[0166] Since the first user terminal (100A) took a picture or captured the display screen currently being displayed on the second user terminal (100B), there is a very low possibility that there is image identity between the image information received from the first user terminal (100A) and the image information received from the third user terminal (100C).

[0167] Even if there is coincidence between the image information received from the first user terminal (100A) and the image information received from the third user terminal (100C), there is a low possibility that there is coincidence between the surrounding environment information received from each of the first user terminal (100A) and the third user terminal (100C). Therefore, if the management server (200) determines that there is no image identity, or that there is coincidence but there is no surrounding environment identity, it can transmit an image identity rejection signal to the third user terminal (100C) [S614].

[0168] There is no requirement that there be a temporal order between steps S611 and S612 and steps S613 and S614. Steps S611 and S612 may be executed after steps S613 and S614. Alternatively, steps S611 and S612 may be executed substantially simultaneously with steps S613 and S614.

[0169] The second user terminal (100B) can transmit its connection information (e.g., IP (Internet Protocol) address and port number) to the management server (200) in response to receiving the video identity recognition signal [S521].

[0170] The management server (200) can again transmit the connection information of the second user terminal (100B) to the first user terminal (100A) [S522].

[0171] The first user terminal (100A) can automatically connect to the second user terminal (100B), which is the target device, using the connection information of the second user terminal (100B) [S523].

[0172] That is, a wireless communication connection can be established between the first user terminal (100A) and the second user terminal (100B) simply by the user wearing the XR device (100A), which is the first user terminal (100A), and looking at the second user terminal (100B), which is the target device.

[0173] Figure 6 describes identifying a target device based on image information and surrounding environment information received from each user terminal. However, by chance, the second user terminal (100B) and the third user terminal (100C) may be displaying the same or similar image or screen, or the second user terminal (100B) and the third user terminal (100C) may be located close to each other. In this case, identifying the target device may be difficult.

[0174] Accordingly, in order to more accurately specify the target device, the second user terminal (100B) and the third user terminal (100C) may each output a characteristic pattern on the screen, and image information including the characteristic pattern may be received from each user terminal to specify the target device. This will be further described with reference to FIGS. 7 to 9. FIGS. 7 and 8 are flowcharts showing a communication connection between a user terminal and another user terminal according to one aspect of the present disclosure. FIG. 9 is an example of a user terminal that outputs a characteristic pattern on the screen according to one aspect of the present disclosure.

[0175] First, referring to FIG. 7, steps S401 to S414 of FIG. 7 are the same as those described in FIG. 4, and therefore, a detailed description thereof will be omitted for the sake of brevity of the present disclosure.

[0176] In response to a wireless communication connection request from a first user terminal (100A), the management server (200) can transmit a shooting request signal (or message) to the first user terminal (100A) requesting that the first user terminal (100A) shoot (or capture) a second user terminal (100B) that is a target device (i.e., a display screen displayed on the second user terminal (100B)) for a reference time (or for a reference frame number) [S701].

[0177] In addition, the management server (200) can transmit a feature pattern output and recording request signal (or message) to each of the other user terminals (i.e., the second user terminal (100B) and the third user terminal (100C)) managed as a group such as the first user terminal (100A) to output a feature pattern on the screen currently being displayed and to request that the other user terminal record (or capture) the output screen of the feature pattern currently being displayed for a reference time (or for the reference frame number) [S702, S703].

[0178] The time at which the management server (200) transmits a characteristic pattern output and recording request signal to the first user terminal (100A) and the time at which the management server (200) transmits a characteristic pattern output and recording request signal to other user terminals (100B, 100C) may be substantially the same or within a predetermined margin time.

[0179] Referring to FIG. 8, the second user terminal (100B) can output the first characteristic pattern on the screen it is currently displaying for a reference time period [S704]. The first characteristic pattern may also be output instead of the screen the second user terminal (100B) is currently displaying.

[0180] Additionally, the third user terminal (100C) can output the second characteristic pattern on the screen it is currently displaying for a reference time period [S705]. The second characteristic pattern may be output instead of the screen that the third user terminal (100C) is currently displaying.

[0181] Refer further to Figure 9 for the first feature pattern and the second feature pattern.

[0182] As illustrated in (9-1) of FIG. 9, the second user terminal (100B) can, in response to the feature pattern output and recording request signal, output a first feature pattern (410B) on the first screen (400B) that it is currently displaying for a reference time. The first feature pattern (410B) may also be output for the reference time instead of the first screen (400B). The first feature pattern is unique to the second user terminal (100B) and is different from the second feature pattern.

[0183] As illustrated in (9-2) of FIG. 9, the third user terminal (100C) can, in response to the feature pattern output and recording request signal, output a second feature pattern (410C) on the second screen (400C) that it is currently displaying for a reference time. The second feature pattern (410C) may also be output for the reference time instead of the second screen (400C). The second feature pattern is unique to the third user terminal (100C) and is different from the first feature pattern.

[0184] Referring again to FIG. 8, while the first user terminal (100A) is looking at the second user terminal (100B), in response to a shooting request signal received from the management server (200), the first user terminal (100A) can capture the output display screen of the first characteristic pattern currently displayed on the second user terminal (100B) for a reference time (or as many reference frames) [S801].

[0185] The second user terminal (100B) can record or capture the output screen of the first characteristic pattern that the second user terminal (100B) is currently displaying for a reference time (or for the number of reference frames) in response to the characteristic pattern output and recording request signal received from the management server (200) [S802].

[0186] The third user terminal (100C) can record or capture the screen on which the second characteristic pattern is currently displayed for a reference time (or for the reference frame number) in response to the characteristic pattern output and recording request signal received from the management server (200) [S803].

[0187] Since the transmission time of the shooting request signal and the transmission time of the characteristic pattern output and recording request signal are substantially the same or within a predetermined margin time, steps S801, S802, and S803 can be performed substantially simultaneously (or simultaneously within a predetermined margin time).

[0188] The first user terminal (100A) can transmit the captured video (or the number of captured video frames corresponding to the number of reference frames) to the management server (200) [S804]. To reduce transmission weight, the first user terminal (100A) can also transmit the captured image hash value of the captured video frame to the management server (200) instead of the captured video frame.

[0189] The second user terminal (100B) can transmit the first recorded video (or the number of first captured video frames equal to the number of reference frames) recorded by the second user terminal (100B) to the management server (200) [S805]. In order to reduce transmission weight, the second user terminal (100B) can also transmit the first image hash value of the first captured video frame to the management server (200) instead of the first captured video.

[0190] The third user terminal (100C) can transmit the second recorded video (or the number of second captured video frames equal to the number of reference frames) recorded by the third user terminal (100C) to the management server (200) [S606]. In order to reduce transmission weight, the third user terminal (100C) can also transmit the second image hash value of the second captured video frame to the management server (200) instead of the second captured video frame.

[0191] There is no requirement for a chronological order between steps S804, S805, and S806. Any step may be executed before the others. These steps may also be executed concurrently.

[0192] The management server (200) can compare the captured video received from the first user terminal (100A) with the first recorded video received from the second user terminal (100B) to determine whether there is image identity between the captured video and the second recorded video [S811]. Alternatively, the management server (200) can compare the captured video frame received from the first user terminal (100A) with the first captured video frame received from the second user terminal (100B) to determine whether there is image identity between the captured video frame and the first captured video frame. Alternatively, the management server (200) can compare the captured image hash value received from the first user terminal (100A) with the first image hash value received from the second user terminal (100B) to determine whether there is image identity between the captured image hash value and the first image hash value.

[0193] Since the first user terminal (100A) has photographed or captured the display screen currently being displayed on the second user terminal (100B), there is a high probability that there is image identity between the image information received from the first user terminal (100A) and the image information received from the second user terminal (100B). This is especially true due to the first and second feature patterns. The explanation will continue assuming that there is image identity between them.

[0194] The existence of image identity between these may mean that the second user terminal (100B) is identified as the target device. If the management server (200) determines that there is image identity, it may transmit an image identity recognition signal to the second user terminal (100B) [S812].

[0195] Meanwhile, the management server (200) can compare the captured video received from the first user terminal (100A) with the second recorded video received from the third user terminal (100C) to determine whether there is image identity between the captured video and the second recorded video [S813]. Alternatively, the management server (200) can compare the captured video frame received from the first user terminal (100A) with the second captured video frame received from the third user terminal (100C) to determine whether there is image identity between the captured video frame and the second captured video frame. Alternatively, the management server (200) can compare the captured image hash value received from the first user terminal (100A) with the second image hash value received from the third user terminal (100C) to determine whether there is image identity between the captured image hash value and the second image hash value.

[0196] Since the first user terminal (100A) captured or photographed the display screen currently being displayed on the second user terminal (100B), the likelihood of image identity between the image information received from the first user terminal (100A) and the image information received from the third user terminal (100C) is significantly low. This is especially true due to the first and second feature patterns. The explanation will continue assuming no image identity between them.

[0197] If it is not recognized that there is image identity between them, the management server (200) can transmit an image identity non-recognition signal to the third user terminal (100C) [S814].

[0198] There is no requirement that there be a temporal order between steps S811 and S812 and steps S813 and S814. Steps S811 and S812 may be executed after steps S813 and S814. Alternatively, steps S811 and S812 may be executed substantially simultaneously with steps S813 and S814.

[0199] The second user terminal (100B) can transmit its connection information (e.g., IP (Internet Protocol) address and port number) to the management server (200) in response to receiving the video identity recognition signal [S821].

[0200] The management server (200) can again transmit the connection information of the second user terminal (100B) to the first user terminal (100A) [S822].

[0201] The first user terminal (100A) can automatically connect to the second user terminal (100B), which is the target device, using the connection information of the second user terminal (100B) [S823].

[0202] That is, a wireless communication connection can be established between the first user terminal (100A) and the second user terminal (100B) simply by the user wearing the XR device (100A), which is the first user terminal (100A), and looking at the second user terminal (100B), which is the target device.

[0203] In Figures 7 and 8, it is described that a feature pattern is used to specify a target device. It goes without saying that not only a feature pattern but also surrounding environmental information may be used to specify a target device.

[0204] In the above, the case where the first user terminal (100A), the second user terminal (100B), and the third user terminal (100C) operate in an environment where Internet access is possible has been described. Hereinafter, with reference to FIGS. 10 and 11, the case where the first user terminal (100A), the second user terminal (100B), and the third user terminal (100C) operate in an environment where Internet access is not possible will be described. FIGS. 10 and 11 are flowcharts showing a user terminal communicating with another user terminal according to one aspect of the present disclosure.

[0205] First, referring to FIG. 10, the second user terminal (100B) and the third user terminal (100C) can each activate a BLE (Bluetooth Low Energy) server function [S1001, S1002]. Here, the BLE server function may include a function capable of transmitting images between user terminals via Bluetooth low energy communication.

[0206] The second user terminal (100B) can periodically transmit a first BLE advertising signal to the surroundings to notify that the BLE (Bluetooth Low Energy) server function has been activated [S1003].

[0207] In addition, the third user terminal (100C) can periodically transmit a second BLE advertising signal to the surroundings to notify that the BLE (Bluetooth Low Energy) server function has been activated [S1004].

[0208] The first user terminal (100A) can execute a connection application for connection with another user terminal [S1011]. The execution of the connection application of the first user terminal (100A) may be performed in response to a user command input through the first input unit (120A), or may be performed automatically in response to the occurrence of a predetermined event (e.g., Internet connection). If the connection application is already running in the foreground or background on the first user terminal (100A), step S1011 may be omitted.

[0209] A user of a first user terminal (100A) may wish to wirelessly connect one of a second user terminal (100B) and a third user terminal (100C) (hereinafter, a target device) to the first user terminal (100A). Hereinafter, it will be assumed that the target device is the second user terminal (100B).

[0210] In this case, the user can trigger a wireless communication connection between the first user terminal (100A) and the second user terminal (100B) by wearing the first user terminal (100A), i.e., the XR device, and looking at the second user terminal (100B), which is the target device [S1012]. When the wireless communication connection between the first user terminal (100A) and the second user terminal (100B) is triggered, the first user terminal (100A) can display an alarm graphic (and / or text) notifying that the wireless communication connection has been triggered. The alarm graphic can be displayed at least until step S1027, which will be described later, is completed.

[0211] When a wireless communication connection is triggered, the first user terminal (100A) recognizes the appearance of the second user terminal (100B), which is a target device, as an object, and the second user terminal (100B) can recognize that the first user terminal (100A) is a type of device capable of wireless communication connection [S1013]. Information regarding the type of device capable of wireless communication connection of the first user terminal (100A) may be pre-stored in the first memory (170A) of the first user terminal (100A).

[0212] When the second user terminal (100B) is recognized as a type of device capable of wireless communication connection, the first user terminal (100A) can search for BLE server devices, i.e., the second user terminal (100B) and the third user terminal (100C), based on the first BLE advertising signal and the second BLE advertising signal [S1014].

[0213] The first user terminal (100A) can transmit a first BLE connection request signal to the second user terminal (100B) based on the first BLE advertising signal [S1021].

[0214] The second user terminal (100B) can transmit a first BLE connection completion signal to the first user terminal (100A) in response to the first BLE connection request signal [S1022].

[0215] When the BLE connection between the first user terminal (100A) and the second user terminal (100B) is completed, the second user terminal (100B) can record or capture the screen that the second user terminal (100B) is currently displaying for a reference time (or for the reference frame number) [S1023].

[0216] The first user terminal (100A) can transmit a second BLE connection request signal to the third user terminal (100C) based on the second BLE advertising signal [S1024].

[0217] The third user terminal (100C) can transmit a second BLE connection completion signal to the first user terminal (100A) in response to the second BLE connection request signal [S1025].

[0218] When the BLE connection between the first user terminal (100A) and the third user terminal (100C) is completed, the third user terminal (100C) can record or capture the screen that the third user terminal (100C) is currently displaying for a reference time (or for the reference frame number) [S1026].

[0219] While the first user terminal (100A) is looking at the second user terminal (100B), when the BLE connection between the first user terminal (100A) and the second user terminal (100B) and the BLE connection between the first user terminal (100A) and the third user terminal (100C) are completed, the display screen currently displayed on the second user terminal (100B) can be captured for a reference time (or for the number of reference frames) [S1027].

[0220] In FIG. 10, the first user terminal (100A) is illustrated as sending a BLE connection signal to the second user terminal (100B) before the third user terminal (100C), but the first user terminal (100A) may send a BLE connection signal to the third user terminal (100C) before the second user terminal (100B). Alternatively, the first user terminal (100A) may send BLE connection signals to the second user terminal (100B) and the second user terminal (100C) simultaneously (or simultaneously within a predetermined margin time).

[0221] Therefore, steps S1023, S1026 and S1027 can be performed virtually simultaneously (or simultaneously within a predetermined margin time).

[0222] Referring to FIG. 11, a first user terminal (100A) can transmit a captured video (or a number of captured video frames equal to the number of reference frames) to a second user terminal (100B) and / or a third user terminal (100C) via a BLE connection [S1101, S1111]. The description will assume that the captured video is first transmitted to the third user terminal (100C).

[0223] The first user terminal (100A) can transmit the captured video (or the number of captured video frames equal to the number of reference frames) to the third user terminal (100C) [S1101].

[0224] The third user terminal (100A) can compare the captured video received from the first user terminal (100A) with the second recorded video recorded by the third user terminal (100A) to determine whether there is video identity between the captured video and the second recorded video [S1102]. As described above, video frames or image hash values ​​may be used to determine video identity.

[0225] Since the first user terminal (100A) photographed or captured the display screen currently displayed on the second user terminal (100B), the possibility of image identity between the image information received from the first user terminal (100A) and the image information recorded or captured by the third user terminal (100C) is quite low. The explanation will continue assuming that there is no image identity between them.

[0226] If it is not recognized that there is image identity between them, the third user terminal (100C) can transmit an image identity non-recognition signal to the first user terminal (100A) via a BLE connection [S1103].

[0227] The first user terminal (100A) can terminate the BLE connection with the third user terminal (100C) in response to the image identity non-recognition signal [S1104].

[0228] The first user terminal (100A) can transmit the captured video (or the number of captured video frames equal to the number of reference frames) to the second user terminal (100B) [S1111].

[0229] The second user terminal (100B) can compare the captured video received from the first user terminal (100A) with the first recorded video recorded by the second user terminal (100B) to determine whether there is video identity between the captured video and the first recorded video [S1112]. As described above, video frames or image hash values ​​may be used to determine video identity.

[0230] Since the first user terminal (100A) photographed or captured the display screen currently displayed on the second user terminal (100B), there is a high probability that there is image identity between the image information received from the first user terminal (100A) and the image information recorded or captured by the second user terminal (100B). The explanation will continue assuming that there is image identity between them.

[0231] The existence of image identity between them may mean that the second user terminal (100B) is identified as the target device. If the existence of image identity is recognized, the second user terminal (100B) may transmit an image identity recognition signal to the second user terminal (100B) via a BLE connection [S1113].

[0232] The first user terminal (100A) can transmit a soft AP connection information request signal to the second user terminal (100B) via a BLE connection in response to the image identity recognition signal [S1114].

[0233] The second user terminal (100B) can activate its own soft AP function in response to the soft AP connection information request signal and transmit information for connecting to the activated soft AP to the first user terminal (100A) via a BLE connection [S1115, S1116]. If the soft AP function is already activated in the second user terminal (100B), step S1115 can be omitted. The soft AP connection information can include the SSID and password of the soft AP. The soft AP connection information can further include an IP (Internet Protocol) address and a port number.

[0234] The first user terminal (100A) can terminate the BLE connection with the second user terminal (100B) [S1117].

[0235] The first user terminal (100A) can automatically connect to the second user terminal (100B), which is a target device, using soft AP connection information [S1118].

[0236] In determining the identity of an image, at least one of the aforementioned surrounding environment information and characteristic patterns may be further utilized.

[0237] Although not shown, if the first user terminal (100A) first transmits its own captured video to the second user terminal (100B) and the first user terminal (100A) connects to the second user terminal (100B), the first user terminal (100A) may terminate the BLE connection with the third user terminal (100C) without transmitting its own captured video to the third user terminal (100C).

[0238] The above-described present disclosure can be implemented as computer-readable code on a program-recorded medium. A computer-readable medium includes any type of recording device that stores data readable by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.

Claims

1. Camera; A wireless communication unit for communicating with an external user terminal; and Taking a picture of the screen displayed by a first external user terminal among multiple external user terminals through the above camera, Transmit the captured video information to the management server or the first external user terminal, A user terminal including a control unit that automatically controls connection to a first external user terminal using connection information received from the management server or the first external user terminal.

2. In the first paragraph, the control unit, A user terminal characterized in that, when image identity is recognized between the above-mentioned shooting image information and the first recorded image information of the first external user terminal, the connection information is controlled to be received from the management server or the first external user terminal.

3. In the first paragraph, the control unit, A user terminal characterized in that the first external user terminal is recognized as an object through the camera and, if the first external user terminal corresponds to a type of device capable of wireless communication connection, the captured image information is controlled to be transmitted to the management server or the first external user terminal.

4. In paragraph 1, the above-mentioned shooting image information is: A user terminal characterized by including at least one of a video shot during a reference time, a reference number of video frames shot, and an image hash value of the video frames shot.

5. In one of paragraphs 1 to 4, the control unit, A user terminal characterized by controlling the above-mentioned shooting image information to be transmitted together with surrounding environment information.

6. In paragraph 5, the surrounding environment information is: A user terminal characterized by including a list of searchable wireless Wi-Fi APs (Access Points) in the vicinity.

7. In paragraph 1, the above-mentioned shooting image information is: A user terminal characterized by including a first characteristic pattern of a first external user terminal.

8. In the first paragraph, the control unit, A user terminal characterized in that it controls the registration of the user terminal with a user account on the management server.

9. In paragraph 8, A user terminal characterized in that the first external user terminal is registered with the management server with the same user account.

10. In paragraph 1, the connection information is: A user terminal characterized by including an IP (Internet Protocol) address and port number for connecting to a first external user terminal.

11. In the first paragraph, the control unit, Connecting a first external user terminal to a first BLE (Bluetooth Low Energy) based on a first BLE advertising signal received from the first external user terminal, A user terminal characterized in that it controls to connect a second BLE with a second external user terminal based on a second BLE advertising signal received from a second external user terminal among the plurality of external user terminals.

12. In the 11th paragraph, the control unit, Transmit the above-described captured image information to the first external user terminal through a first BLE connection with the first external user terminal, A user terminal characterized in that it controls to transmit the captured image information to a second external user terminal through a second BLE connection with the second external user terminal.

13. In the 11th paragraph, the control unit, If the image identity is recognized between the above-mentioned shooting image information and the first recorded image information of the first external user terminal, soft AP (Access Point) information for connecting to the first external user terminal is received as the connection information from the first external user terminal through the first BLE connection, A user terminal characterized by controlling termination of a first BLE connection.

14. In the 11th paragraph, the control unit, A user terminal characterized in that, when the image identity between the above-mentioned captured image information and the second recorded image information of the second external user terminal is not recognized, the second BLE connection is controlled to be terminated.

15. In paragraph 1, A user terminal characterized in that the user terminal is an XR (Extended Reality) device.

16. A step of capturing a screen displayed by a first external user terminal among a plurality of external user terminals through a camera; A step of transmitting the captured video information to the management server or the first external user terminal; and A method for controlling a user terminal, comprising: a step of automatically connecting to a first external user terminal using connection information received from the management server or the first external user terminal.

17. In paragraph 16, A control method for a user terminal, characterized in that it comprises a step of receiving the connection information from the management server or the first external user terminal when image identity is recognized between the above-mentioned shooting image information and the first recorded image information of the first external user terminal.

18. In paragraph 16, A method for controlling a user terminal, comprising: a step of recognizing a first external user terminal as an object through the camera and transmitting the captured image information to the management server or the first external user terminal when the first external user terminal corresponds to a type of device capable of wireless communication connection; 19. In paragraph 16, A step of connecting a first external user terminal and a first BLE (Bluetooth Low Energy) based on a first BLE advertising signal received from the first external user terminal; and A method for controlling a user terminal, characterized in that it comprises a step of connecting a second external user terminal to a second BLE based on a second BLE advertising signal received from a second external user terminal among the plurality of external user terminals.

20. In paragraph 19, The above-described captured image information is transmitted to the first external user terminal through the first BLE connection with the first external user terminal, A control method for a user terminal, characterized in that the captured image information is transmitted to a second external user terminal through a second BLE connection with the second external user terminal.

Images