Visual Content Verification in Extended Reality and Augmented Reality

Abstract

A system and method for detecting unauthorized virtual objects in a mobile device are disclosed. In one aspect, the mobile device may receive an image from a camera. The mobile device may detect virtual objects displayed on a display of the mobile device included in the image. The mobile device may receive data from an inertial measurement unit (IMU) after movement of the mobile device. The mobile device may determine an estimated new position of the virtual object based on the data received from the IMU. The mobile device may determine an actual position of the virtual object after receiving data from the IMU. The mobile device may determine a difference between the estimated new position and the actual position of the virtual object. The mobile device may determine whether the virtual object is an unauthorized virtual object based on the difference.

Description

Technical Field

[0001] 1. Field of Disclosure Aspects of the present disclosure generally relate to extended reality and augmented reality. In some aspects, the present disclosure relates to the verification of objects within extended reality and / or augmented reality.

[0002] 2. Description of Related Art Devices and systems for augmented reality (AR) and extended reality (XR) have become very common. AR / XR devices can be used in various areas such as gaming, web surfing, business software, map navigation, etc. Many devices such as smartphones, wearables, vehicles, and smartwatches may adopt AR / XR technology, and AR / XR virtual objects and AR / XR renderings can be displayed on the displays of these devices. However, the display of virtual objects and virtual renderings may invite unauthorized entities (such as hackers, viruses, malware, etc.) to display false or unauthorized virtual objects and virtual renderings. For example, an unauthorized entity may display a fake vehicle on the display of a vehicle equipped with an AR / XR system to cause an accident, or display a fake bank account number on a smartphone equipped with an AR / XR system to steal money while the user is using the smartphone for online banking.

Summary of the Invention

[0003] The following presents a simplified summary regarding one or more aspects disclosed in this specification. Accordingly, the following summary should not be regarded as an extensive overview of all contemplated aspects, nor should the following summary be regarded as identifying key or critical elements of all contemplated aspects or as delimiting the scope of any particular aspect. Thus, the sole purpose of the following summary is to present, in a simplified form, certain concepts regarding one or more aspects of the mechanisms disclosed in this specification prior to the detailed description presented below.

[0004] In one aspect, a method for detecting an unauthorized virtual object includes receiving an image from a camera, detecting a virtual object displayed on a display of a mobile device included in the image, receiving data from an inertial measurement unit (IMU) after movement of the mobile device, determining an estimated new position of the virtual object based on the data received from the IMU, determining an actual position of the virtual object after receiving data from the IMU, determining a difference between the estimated new position and the actual position of the virtual object, and determining whether the virtual object is an unauthorized virtual object based on the difference.

[0005] In one aspect, a mobile device configured to detect an unauthorized virtual object includes a memory and at least one processor communicatively coupled to the memory. The at least one processor receives an image from a camera, detects a virtual object displayed on a display of the mobile device included in the image, receives data from an inertial measurement unit (IMU) after movement of the mobile device, determines an estimated new position of the virtual object based on the data received from the IMU, determines an actual position of the virtual object after receiving data from the IMU, determines a difference between the estimated new position and the actual position of the virtual object, and determines whether the virtual object is an unauthorized virtual object based on the difference.

[0006] In one aspect, a mobile device configured to detect an unauthorized virtual object includes means for receiving an image from a camera, means for detecting a virtual object displayed on a display of the mobile device included in the image, means for receiving data from an inertial measurement unit (IMU) after movement of the mobile device, means for determining an estimated new position of the virtual object based on the data received from the IMU, means for determining an actual position of the virtual object after receiving data from the IMU, means for determining a difference between the estimated new position and the actual position of the virtual object, and means for determining whether the virtual object is an unauthorized virtual object based on the difference.

[0007] In one aspect, a non-transitory computer-readable medium storing computer-executable instructions that, when executed by a mobile device configured to detect unauthorized virtual objects, cause the mobile device to receive an image from a camera, detect a virtual object displayed on a display of the mobile device included in the image, receive data from an inertial measurement unit (IMU) after movement of the mobile device, determine an estimated new position of the virtual object based on the data received from the IMU, determine an actual position of the virtual object after receiving data from the IMU, determine a difference between the estimated new position and the actual position of the virtual object, and determine whether the virtual object is an unauthorized virtual object based on the difference.

[0008] Other objects and advantages related to the aspects disclosed herein will be apparent to those skilled in the art based on the accompanying drawings and detailed description.

[0009] The accompanying drawings are presented to assist in the description of various aspects of the present invention.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5A

Figure 5B

Figure 5C

[0011] Aspects of the present disclosure are provided in the following description and related drawings directed to various examples provided for illustrative purposes. Alternative aspects may be devised without departing from the scope of the present disclosure. Additionally, well-known elements of the present disclosure are not described in detail or are omitted so as not to obscure the relevant details of the present disclosure.

[0012] As used herein, the terms "exemplary" and / or "example" are used to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" and / or "example" should not necessarily be construed as preferred or advantageous over other aspects. Similarly, the term "aspect of the present disclosure" does not necessarily require that all aspects of the present disclosure include the recited features, advantages, or modes of operation.

[0013] Those skilled in the art will appreciate that the information and signals described below may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the following description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof, depending in part on the particular application, desired design, corresponding technology, and the like.

[0014] Furthermore, many aspects will be described from the perspective of a sequence of actions that, for example, would be performed by elements of a computing device. It will be recognized that the various actions described herein can be implemented by a particular circuit (e.g., application specific integrated circuits (ASICs)), by program instructions executed by one or more processors, or by a combination of both. Additionally, the sequence of actions described herein, when executed, can be considered to be fully embodied within any form of non-transitory computer-readable storage medium that stores a corresponding set of computer instructions that, when executed, cause the relevant processor of the device to perform or cause to be performed the functionality described herein. Accordingly, the various aspects of the present disclosure can be embodied in several different forms, all of which are intended to fall within the scope of the claimed subject matter. Additionally, for each of the aspects described herein, a corresponding form of any such aspect can be described herein, for example, as "logic configured to" perform the recited action.

[0015] As used herein, the terms "user equipment" (UE) and "base station" are not intended to be specific to, or limited to, any particular radio access technology (RAT) unless otherwise specified. Generally, a UE may be any wireless communication device (e.g., a mobile phone, router, tablet computer, laptop computer, consumer asset tracking device, wearable (e.g., smartwatch, glasses, augmented reality (AR) / virtual reality (VR) headset, etc.), vehicle (e.g., automobile, motorcycle, bicycle, etc.), Internet of Things (IoT) device, etc.) used by a user to communicate via a wireless communication network. The UE may be mobile or (e.g., for a particular time) stationary and may communicate with a radio access network (RAN). The term "UE" as used herein may be interchangeably referred to as "access terminal" or "AT", "client device", "wireless device", "subscriber device", "subscriber terminal", "subscriber station", "user terminal" or "UT", "mobile device", "mobile terminal", "mobile station", or variations thereof. Generally, a UE can communicate with a core network via a RAN, and through the core network, the UE can be connected to an external network such as the Internet and to other UEs. Of course, other mechanisms for connecting to the core network and / or the Internet are also possible for the UE, such as via a wired access network, a wireless local area network (WLAN) network (e.g., based on Institute of Electrical and Electronics Engineers (IEEE) 802.11 specifications, etc.).

[0016] The base station may operate according to one of several RATs that the base station is communicating with the UE according to the network in which the base station is deployed. Alternatively, it may be called an access point (AP), network node, Node B, evolved Node B (eNB), next generation eNB (ng-eNB), new radio (NR) Node B (also called gNB or g-node B), etc. The base station may be mainly used to support wireless access by the UE, including supporting data, voice, and / or signaling connections for the supported UEs. In some systems, the base station may provide only the edge node signaling function, while in other systems, the base station may provide additional control and / or network management functions. The communication link through which the UE can send a signal to the base station is called an uplink (UL) channel (e.g., reverse traffic channel, reverse control channel, access channel, etc.). The communication link through which the base station can transmit a signal to the UE is called a downlink (DL) channel or forward link channel (e.g., paging channel, control channel, broadcast channel, forward traffic channel, etc.). The term traffic channel (TCH) used in this specification may refer to either an uplink / reverse traffic channel or a downlink / forward traffic channel.

[0017] The term "base station" may refer to a single physical transmit-receive point (TRP) or multiple physical TRPs that may or may not be collocated. For example, when the term "base station" refers to a single physical TRP, that physical TRP may be the base station's antenna corresponding to the base station's cell (or some cell sectors). When the term "base station" refers to multiple collocated physical TRPs, the physical TRPs may be an array of antennas of the base station (such as in a multiple-input multiple-output (MIMO) system or when the base station employs beamforming). When the term "base station" refers to multiple non-collocated physical TRPs, the physical TRPs may be a distributed antenna system (DAS) (a network of spatially separated antennas connected to a common source via a transport medium), or a remote radio head (RRH) (a remote base station connected to a serving base station). Alternatively, the non-collocated physical TRPs may be the serving base station that receives measurement reports from the UE and neighboring base stations whose reference RF signals the UE is measuring. Since the TRP is the point from which the base station transmits and receives wireless signals, references to transmissions from the base station or receptions at the base station, as used herein, should be understood to refer to a particular TRP of the base station.

[0018] In some implementations that support UE positioning, the base station may not support wireless access by the UE (for example, may not support a data connection, a voice connection, and / or a signaling connection for the UE), but instead may transmit to the UE a reference signal that is to be measured by the UE and / or receive and measure signals transmitted by the UE. Such a base station may be referred to as a positioning beacon (for example, when transmitting a signal to the UE) and / or a location measurement unit (for example, when receiving and measuring signals from the UE).

[0019] An "RF signal" includes electromagnetic waves of a given frequency that propagate information through the space between a transmitter and a receiver. As used herein, a transmitter can transmit a single "RF signal" or multiple "RF signals" to a receiver. However, due to the propagation characteristics of RF signals through a multipath channel, a receiver may receive multiple "RF signals" corresponding to each transmitted RF signal. The same RF signal transmitted along different paths between a transmitter and a receiver may be referred to as a "multipath" RF signal.

[0020] FIG. 1 shows an exemplary wireless communication system 100. The wireless communication system 100 (which may also be referred to as a wireless wide area network (WWAN)) may include various base stations 102 and various UEs 104. The base stations 102 may include macrocell base stations (high-power cellular base stations) and / or small cell base stations (low-power cellular base stations). In one aspect, the macrocell base stations may include an eNB and / or an ng-eNB if the wireless communication system 100 corresponds to an LTE network, or a gNB if the wireless communication system 100 corresponds to an NR network, or a combination of both, and the small cell base stations may include femtocells, picocells, microcells, etc.

[0021] The base stations 102 can collectively form a RAN and interface with a core network 170 (e.g., an evolved packet core (EPC) or a 5G core (5GC)) through a backhaul link 122, and through the core network 170, interface with one or more location servers 172 (which may be part of the core network 170 or may exist outside the core network 170). The base stations 102 can communicate with each other directly or indirectly (e.g., through the EPC / 5GC) via a backhaul link 134 that may be wired or wireless.

[0022] The base station 102 can wirelessly communicate with the UE 104. Each of the base stations 102 can provide communication coverage regarding its respective geographical coverage area 110. In one aspect, one or more cells can be supported by the base stations 102 within each geographical coverage area 110. A "cell" is a logical communication entity used for communication with a base station (e.g., via some frequency resource such as a carrier frequency, a component carrier, a carrier, a band, etc.), and can be associated with an identifier (e.g., a physical cell identifier (PCI), a virtual cell identifier (VCI), a cell global identifier (CGI)) for distinguishing cells operating via the same carrier frequency or different carrier frequencies. In some cases, different cells can be configured according to different protocol types (e.g., machine-type communication (MTC), narrowband IoT (NB-IoT), enhanced mobile broadband (eMBB), or others) that can provide access to different types of UEs. Since a cell is supported by a specific base station, the term "cell" may, depending on the context, refer to one or both of the logical communication entity and the base station that supports it. In some cases, the term "cell" may also refer to the geographical coverage area (e.g., a sector) of a base station as long as a carrier frequency can be detected and used for communication within a certain part of the geographical coverage area 110.

[0023] The geographical coverage area 110 of the neighboring macro cell base station 102 may partially overlap (e.g., in the handover area), and some of the geographical coverage areas 110 may be significantly overlapped by a larger geographical coverage area 110. For example, a small cell (SC) base station 102' may have a geographical coverage area 110' that significantly overlaps with the geographical coverage area 110 of one or more macro cell base stations 102. A network including both small cell base stations and macro cell base stations may be known as a heterogeneous network. The heterogeneous network may also include home eNBs (HeNBs) that may provide services to a limited group known as a closed subscriber group (CSG).

[0024] The communication link 120 between the base station 102 and the UE 104 may include uplink (also referred to as reverse link) transmission from the UE 104 to the base station 102 and / or downlink (also referred to as forward link) transmission from the base station 102 to the UE 104. The communication link 120 may use MIMO antenna technology including spatial multiplexing, beamforming, and / or transmit diversity. The communication link 120 may be through one or more carrier frequencies. The carrier assignment may be asymmetric with respect to the downlink and uplink (e.g., more or fewer carriers may be assigned for the downlink than for the uplink).

[0025] The wireless communication system 100 may further include a WLAN access point (AP) 150 that communicates with WLAN stations (STAs) 152 via a communication link 154 in an unlicensed frequency spectrum (e.g., 5 GHz). When communicating in the unlicensed frequency spectrum, the WLAN STA 152 and / or the WLAN AP 150 may perform a clear channel assessment (CCA) or a listen before talk (LBT) procedure before communicating to determine whether the channel is available.

[0026] The small cell base station 102' may operate in a licensed frequency spectrum and / or an unlicensed frequency spectrum. When operating in the unlicensed frequency spectrum, the small cell base station 102' may utilize LTE technology or NR technology and use the same 5 GHz unlicensed frequency spectrum as used by the WLAN AP 150. A small cell base station 102' that employs LTE / 5G in the unlicensed frequency spectrum may expand the coverage to the access network and / or increase the capacity of the access network. NR in the unlicensed spectrum may be referred to as NR-U. LTE in the unlicensed spectrum may sometimes be referred to as LTE-U, licensed assisted access (LAA), or MulteFire.

[0027] The wireless communication system 100 may further include a mmW base station 180 that communicates with the UE 182 and may operate at millimeter wave (mmW) frequencies and / or near mmW. Extremely high frequency (EHF) is a part of RF in the electromagnetic spectrum. EHF has a range of 30 GHz to 300 GHz and a wavelength of 1 millimeter to 10 millimeters. Radio waves within this band may be referred to as millimeter waves. Near mmW may decline until it reaches a frequency of 3 GHz with a wavelength of 100 millimeters.

[0028] The wireless communication system 100 may further include a UE 164 that may communicate with the macrocell base station 102 via a communication link 120 and / or with the mmW base station 180 via an mmW communication link 184. For example, the macrocell base station 102 may support a PCell and one or more SCell for the UE 164, and the mmW base station 180 may support one or more SCell for the UE 164.

[0029] In the example of FIG. 1, one or more satellite positioning system (SPS) space vehicles (SVs) 112 (e.g., satellites) may be used as an independent source of location information for any of the illustrated UEs (shown in FIG. 1 as a single UE 104 for simplicity). The UE 104 may include one or more dedicated SPS receivers specifically designed to receive an SPS signal 124 for deriving geolocation information from the SV 112. SPS is generally arranged such that a receiver (e.g., UE 104) can determine its location on or above the Earth, at least in part based on signals received from a transmitter (e.g., SPS signal 124), and includes a system of transmitters (e.g., SV 112).

[0030] The use of the SPS signal 124 may be associated with, or otherwise enabled for, use with one or more global and / or regional navigation satellite systems and may be augmented by various satellite-based augmentation systems (SBAS). For example, SBAS may include augmentation systems that provide integrity information, error correction, etc., such as the Wide Area Augmentation System (WAAS), the European Geostationary Navigation Overlay Service (EGNOS), the Multi-functional Satellite Augmentation System (MSAS), the Global Positioning System (GPS)-aided Geo Augmented Navigation, or the GPS and Geo Augmented Navigation system (GAGAN). Thus, as used herein, SPS may include any combination of one or more global and / or regional navigation satellite systems and / or augmentation systems, and the SPS signal 124 may include SPS, signals such as SPS, and / or other signals associated with such one or more SPS.

[0031] The wireless communication system 100 may further include one or more UEs, such as UE190, that are indirectly connected to one or more communication networks via one or more device-to-device (D2D) peer-to-peer (P2P) links (referred to as "side links"). In the example of FIG. 1, UE190 has a D2D P2P link 192 with one of UE104s connected to one of base stations 102 (e.g., through which UE190 may indirectly obtain a cellular connection), and a D2D P2P link 194 with WLAN STA152 connected to WLAN AP150 (through which UE190 may indirectly obtain a WLAN-based Internet connection). In one example, D2D P2P links 192 and 194 may be supported using any well-known D2D RAT, such as LTE Direct (LTE-D), WiFi Direct (WiFi-D), Bluetooth®.

[0032] In some aspects, the UE / mobile device described herein that incorporates AR / XR technology to display AR / XR virtual objects and AR / XR rendering may include one or more aspects of the various UEs described above, such as the ability to directly communicate with various base stations via a cellular network using various technologies (e.g., LTE, 5G, etc.), the ability to communicate via a WLAN network, the ability to directly communicate with another UE, etc. Accordingly, according to the various aspects disclosed herein, the aforementioned UE may be implemented in various forms including, but not limited to, smart watches, smart phones, tablets, and other devices capable of implementing the functions described herein. It will be understood, accordingly, that the above examples are merely illustrative and should not be construed as limiting the various aspects disclosed herein.

[0033] Referring to FIG. 2, a simplified diagram of an exemplary mobile device 200 having a processor 220, an antenna array 210, a transceiver 222, a camera sensor system 230, and a display 234 is shown. In one aspect, the mobile device 200 may employ AR / XR technology to display AR / XR virtual objects and AR / XR rendering on the display 234. The mobile device 200 further includes a memory 240. The camera sensor system 230 includes a camera 232 and an inertial measurement unit (IMU) 236. In one aspect, the IMU 236 may include a gyroscope, an accelerometer, a magnetometer, and / or other sensors necessary to measure and detect the movement of the mobile device 200. The camera sensor system 230 may further include other sensors (not shown), such as a lidar sensor, a radar sensor, a speed sensor, and / or one or more of any other sensors that may assist in the operation of the mobile device 200. Note that the mobile device 200 may be similar to the UE 104, 190, or any other UE shown in FIG. 1 and may further include one or more components known to those skilled in the art but not shown in FIG. 2B.

[0034] The mobile device 200 can be any suitable electronic device that is portable. For example, the mobile device 200 can be a smartphone, a tablet, a laptop, a smartwatch, a shipment tracking device, a wearable, smart glasses, an on-board computer installed in a vehicle, and the like. For example, in one aspect, the mobile device 200 can be in the form of smart glasses such as the smart glasses 400 shown in FIG. 4. As shown in FIG. 4, the smart glasses 400 include frame portions 414, 416, and 412, a camera 418, a display 424, and a nose piece 430. In one aspect, the camera 418 may be similar to the camera 232, and the display 424 may be similar to the display 234. Further, in one aspect, other components similar to the processor 220, the antenna array 210, and the transceiver 222 may be hidden inside the frame portion 412 or any other frame portion.

[0035] As described above, the antenna array 210 includes a plurality of antennas for transmission beamforming and reception beamforming. The antenna array 210 is coupled to the transceiver 222. The processor 220 may control the antenna array 210 and the transceiver 222. The transceiver 222 provides means (e.g., means for transmitting, means for receiving, means for measuring, means for synchronizing, means for refraining from transmitting, etc.) for communicating via one or more wireless communication networks (not shown) such as an NR network, an LTE network, a GSM network, etc. and may include a wireless wide area network (WWAN) transceiver. The WWAN transceiver may be connected to one or more antennas in the antenna array 210 for communicating with other network nodes such as other UEs, access points, base stations (e.g., eNB, gNB) via at least one designated RAT (e.g., NR, LTE, GSM, etc.) via a target wireless communication medium (e.g., some set of time / frequency resources within a particular frequency spectrum). The transceiver 222 may further include a wireless local area network (WLAN) transceiver. The WLAN transceiver may be connected to one or more antennas in the antenna array 210 and may provide means (e.g., means for transmitting, means for receiving, means for measuring, means for synchronizing, means for refraining from transmitting, etc.) for communicating with other network nodes such as other UEs, access points, base stations via at least one designated RAT (e.g., WiFi, LTE-D, Bluetooth (registered trademark), etc.) via a target wireless communication medium.

[0036] In addition, transceiver 222 may include a satellite positioning system (SPS) receiver. The SPS receiver may be connected to one or more of the antennas in antenna array 210 respectively, and may provide means for receiving and / or measuring SPS signals such as Global Positioning System (GPS) signals, Global Navigation Satellite System (GLONASS) signals, Galileo signals, Beidou signals, Indian Regional Navigation Satellite System (NAVIC), Quasi-Zenith Satellite System (QZSS), etc. The SPS receiver may comprise any suitable hardware and / or software for receiving and processing SPS signals.

[0037] The transceiver circuitry in transceiver 222, which includes at least one transmitter and at least one receiver, may in some implementations comprise an integrated device (e.g., implemented as the transmitter circuitry and receiver circuitry of a single communication device), in some implementations may comprise separate transmitter and receiver devices, or in other implementations may be embodied in other ways. In one aspect, the transmitter may include, or be coupled to, a plurality of antennas such as antenna array 210 that enable an individual device to perform transmission “beamforming” as described herein. Similarly, the receiver may include, or be coupled to, a plurality of antennas such as antenna array 210 that enable an individual device to perform receive beamforming as described herein. In one aspect, the transmitter and the receiver may share the same plurality of antennas (e.g., antenna array 210) such that an individual device can perform only reception or transmission at a given time, rather than both reception and transmission simultaneously.

[0038] As shown in FIG. 2, the processor 220 is coupled to the transceiver 222. The processor 220 is also coupled to the memory 240 and the camera sensor system 230, as shown in FIG. 2. The camera sensor system 230 includes a camera 232 and an IMU 236. The camera sensor system 230 detects and measures an individual environment and transmits the measurement values as data to the processor 220. The IMU 236 can measure the movement of the mobile device 200, including movement and rotational movement in a three-dimensional (3D) coordinate system. In one aspect, the processor 220 can control the operation of the camera sensor system 230. Additionally, the processor 220 is coupled to the display 234 and can display objects and renderings, including virtual objects and virtual renderings, on the display 234. As used herein, the term real object is understood to refer to the visual measurement of the actual scene and objects in front of the AR / XR device observed through the camera lens. Real objects are invariant compared to direct observations perceived by the human eye. Virtual objects and virtual renderings refer to the displayed content that can only be viewed through the AR / XR device. Virtual objects / renderings are the result of rendering algorithms, rather than direct optical measurements, and may include multiple aspects of real objects or may be completely computer-generated by the rendering algorithm.

[0039] In one aspect, the processor 220 may control the camera 232 and the IMU 236 within the camera sensor system 230 and receive data from the camera 232 and the IMU 236. The camera 232 and the IMU 236 may provide individual data to the processor 220 as directed by the processor 220. The camera 232 may provide the processor 220 with an image of the environment surrounding the mobile device 200, and the processor 220 may analyze the image from the camera 232 to determine different objects in the image. The IMU 236 may measure the movement and orientation of the mobile device 200, including rotational movement and movement in a three-dimensional (e.g., x, y, z) coordinate system. The IMU 236 may also measure the speed of the movement of the mobile device 200. For example, the IMU may measure angular velocity and acceleration and use these to calculate changes in the device's orientation. The IMU 236 may transmit the measured data to the processor 220.

[0040] The processor 220 may use the data received from the IMU 236 to determine position information of the mobile device 200, including at least one of the orientation, translation, or movement of the mobile device 200. The position information may include the angles and movement of the mobile device 200 in multiple axes as processed by the processor 220 based on the data detected by the IMU 236.

[0041] The processor 220 may use such an algorithm as a rotation vector algorithm to determine the position information of the mobile device 200. Such an algorithm may be stored in the memory 240 and processed by the processor 220. For example, the processor 220 may use a combination of data from the IMU 236 to calculate the position of the mobile device 200 in a 2D coordinate system and / or a 3D coordinate system.

[0042] The processor 220 may combine the image data from the camera 232 and the determined location information of the mobile device 200 to display virtual objects and virtual renderings on the display 234. For example, assume that the mobile device 200 takes the form of the smart glasses 400 shown in FIG. 4. The processor 220 may receive an image captured by the camera 418 on the smart glasses 400. In this example, the camera 418 may capture an image 500 of a bedroom shown in FIG. 5A and transmit the image 500 to the processor 220. Based on the received image 500, the processor 220 may analyze the image 500 and determine real objects in the image 500, such as the clock 520, the nightstand 510, and the bed 540. Based on the determined real objects in the image 500, the processor 220 may display virtual objects, such as the bottle 530 shown in FIG. 5A, on the display 424. To accurately display the bottle 530 on the nightstand 510, the processor 220 may need to calculate the appropriate x coordinate 532, y coordinate 531, and z coordinate 533 of the bottle 530 based on the data received from the image 500 and the IMU 236. The correct positioning and / or orientation of the bottle 530 on the display 424 depends on the movement of the smart glasses 400 (i.e., depends on the movement of the user of the smart glasses 400), and the processor 220 may analyze the data from the IMU 236, including the orientation, movement, and speed of the smart glasses 400, to determine the appropriate coordinates for positioning and displaying the bottle 530.

[0043] However, when the smart glasses 400 move (i.e., the user of the smart glasses 400 moves while using the smart glasses 400), the camera 418 may send a new image, such as the image 550 shown in FIG. 5B, to the processor 220. Since the smart glasses 400 are moving, the IMU 236 may send new data reflecting the movement to the processor 220. Based on the image 550 and the new data from the IMU 236, the processor 220 may determine new appropriate x-coordinate 532, y-coordinate 531, and z-coordinate 533 of the bottle 530 as shown in FIG. 5B. After determining the new coordinates, the processor 220 may display the bottle 530 based on the new coordinates as shown in FIG. 5B.

[0044] One problem related to AR / XR technology is that unauthorized entities may gain unauthorized access to AR / XR devices and insert unauthorized content into the device's display, causing problems. In one aspect, the mobile device 200 detects and prevents such hacking by monitoring virtual objects and virtual renderings shown on the display 234. Based on the images received from the camera 232, the processor 220 can analyze the images to determine which of the objects and renderings shown on the display 234 are real objects or virtual objects and virtual renderings. By comparing the images received from the camera 232 with the images being displayed on the display 234, the processor 220 can determine which objects are real objects or virtual objects and discover virtual objects. The processor 220 can monitor the displayed virtual objects and virtual renderings by comparing the positioning and / or orientation of the virtual objects and virtual renderings with the data from the IMU 236. In other words, the processor 220 can determine whether the positioning and / or orientation of the virtual objects and virtual renderings match or correlate with the data received from the IMU 236. If an unauthorized entity that has hacked the mobile device 200 cannot access the data from the IMU 236, the virtual objects inserted by the unauthorized entity may not accurately correlate or align with the data from the IMU 236.

[0045] In one aspect, when the smart glasses 400 move, the processor 220 can determine a change in the orientation of the smart glasses 400 based on data from the IMU 236, which includes orientation and translational data. Using the data from the IMU 236, new positions or coordinates of the virtual object displayed on the display 234 can be estimated. The estimated new position / coordinates indicate the position where the virtual object should be placed on the display 234 based on the data from the IMU 236. In one aspect, the processor 220 can estimate the new position / coordinates by projecting the previous coordinates of the virtual object onto the current visual frame visible by the smart glasses 400 by using the data from the IMU 236. Additionally, the processor 220 can determine the actual position / coordinates of the virtual object within the current visual frame of the display 234. The processor 220 can compare the estimated new position / coordinates with the actual determined position / coordinates of the virtual object to determine the difference between the estimated new position / coordinates and the actual determined position / coordinates. If the difference between the estimated new position / coordinates and the actual determined position / coordinates exceeds a certain threshold (e.g., exceeds a distance difference threshold), the processor 220 can determine that the virtual object is an unauthorized virtual object. For example, in some aspects, the distance difference is calculated between the location of the predicted object (or key point on the object) and the measured location of the camera. In some aspects, the distance difference threshold can be on the order of a few millimeters (mm). However, it will be understood that larger distance difference thresholds can be used to reduce false alarms. Generally, a smaller distance difference threshold improves security, and a larger distance difference threshold reduces false alarms / mis-detections of unauthorized virtual objects. Additionally, the threshold can be related to the actual detection algorithm, the size of the virtual object, the size of the image (e.g., the actual dimensions of the captured image), the movement of the AR / XR device, etc. Thus, it will be understood that the various aspects are not limited to the examples described above. Also, those skilled in the art will understand the design implications and trade-offs in establishing various thresholds for detecting unauthorized virtual objects.

[0046] Referring back to the examples of FIGS. 5A and 5B, if an unauthorized entity inserts the bottle 530 into the image 500 and cannot access the data from the IMU 236, the positioning and / or orientation of the bottle 530 will not be properly adjusted when the user of the smart glass 400 moves the smart glass 400. For example, the images 500 and 550 indicate that the smart glass 400 has moved towards the bed 540 from the time the image 500 was taken to the time the image 550 was taken. The bottle 530 shown in FIG. 5B shows the appropriate positioning and orientation based on the data from the IMU 236. However, if an unauthorized entity cannot access the data from the IMU 236, the positioning and orientation of the bottle 530 in the image 550 will appear the same as the positioning and orientation shown in the image 500.

[0047] In FIG. 5C, since the unauthorized entity cannot access the data from the IMU 236, unlike the image 550, the image 570 shows the bottle 530 that is not properly adjusted based on the movement of the smart glasses 400. Instead of showing the proper position and orientation as in the image 550, the bottle 530 in the image 570 is offset from the estimated new position 535 of the virtual object and does not show the proper adjusted position and orientation of the bottle 530 when the smart glasses 400 move closer to the bed 540. Thus, when an unauthorized entity cannot access the data from the IMU 236, the unauthorized entity cannot properly adjust the positioning and orientation of the virtual object inserted by the unauthorized entity into the display 234 when the mobile device 200 moves. It will be appreciated that there is a difference between the estimated new position 535 of the virtual object (bottle 530) and the actual position. This difference may be determined as a distance, orientation (e.g., tilt, direction, etc.), time difference (e.g., delay in response to movement), and as shown, may be determined from one or more points of the virtual object (bottle 530) with respect to the estimated new position 535 (e.g., appropriate x coordinate 532, y coordinate 531, and z coordinate 533) of the virtual object (bottle 530) in 3D space. Thus, the processor 220 may monitor all of the virtual objects and virtual renderings shown on the display 234 to determine whether the positioning and orientation of the virtual objects and virtual renderings correlate or are consistent with the data from the IMU 236. In one aspect, if the position and / or orientation of any of the displayed virtual objects do not correlate or are not consistent with the data received from the IMU 236, the processor 220 may determine that a virtual object having an inappropriate uncorrelated positioning and / or orientation may have been inserted into the display 234 by an unauthorized entity without proper authorization. Through this process, the processor 220 may discover any improperly inserted virtual objects inserted without authorization.

[0048] In one aspect, if an unauthorized entity with an unauthorized virtual object can access the data from the IMU 236, the processor 220 can still detect the unauthorized object based on the latency and delay that occur when repositioning the unauthorized object based on the data received from the IMU 236. For example, an unauthorized entity may attempt to reposition an unauthorized virtual object based on the data from the IMU 236, but the unauthorized entity is far from the mobile device 200 and it is likely that the data needs to move through many different gateways to reach the computer of the unauthorized entity, so the unauthorized entity is likely to face delays when acquiring data from the IMU 236, while an authorized app for positioning the virtual object is likely to receive the data from the IMU 236 relatively quickly. Since the unauthorized entity faces delays when acquiring data from the IMU 236, the repositioning of the unauthorized virtual object is delayed even if the repositioning is based on the data from the IMU 236. In this case, if the delay or latency is greater than a certain threshold, the processor 220 may determine that the virtual object in question is an unauthorized virtual object that may have been inappropriately inserted by an unauthorized entity. The processor 220 can determine the delay threshold based on various factors such as the environment and apps operating on the smart glasses 400 or the mobile device 200. Thus, in one aspect, the processor 220 monitors all of the virtual objects and virtual renderings on the display 234 to detect virtual objects and virtual renderings that exhibit delays and latencies while being repositioned based on the data from the IMU 236. In one aspect, if either the virtual object or the rendering exhibits a delay longer than the threshold while being repositioned based on the data from the IMU 236, the processor 220 may determine that the virtual object or virtual rendering showing the delay may be a virtual object that has been inappropriately inserted. Through this process, the processor 220 can discover any inappropriately inserted virtual objects and virtual renderings.In some embodiments, the delay threshold can be in the range of 10 ms to 100 ms. Similar to the distance delay described above, generally, there is a trade-off between a lower delay threshold and a higher probability of detecting virtual objects with higher security against false alarms / unauthorized access.

[0049] In one embodiment, the mobile device 200 may have an application that renders virtual objects on the display 234 and operates on the mobile device 200. The virtual objects rendered by the application operating on the mobile device 200 can be monitored and checked by the processor 220 as described above. If the application has permission from the user of the application to bypass the above-described monitoring of any rendered virtual objects, the application can bypass the above-described monitoring features and security features. In an application with multiple users, the application may request permission from one or more users before bypassing the above monitoring of virtual objects.

[0050] In one embodiment, when the processor 220 detects an unauthorized object and rendering, the processor 220 can take a plurality of different types of actions to strengthen the security of the mobile device 200 and prevent further hacking. Some of the possible actions that the processor 220 can take to strengthen the security of the mobile device 200 and prevent further hacking are listed below. - Generate a warning icon / message to the user of the mobile device 200. The user can determine what action to take. - In certain critical applications, the processor 220 can send an emergency notification to important people such as the police, personal contacts, security guards, etc. - The connection of the mobile device 200 can be terminated if the mobile device 200 is using a public network (e.g., hotspot, public WiFi, etc.). - The processor 220 may set a flag on specific content or applications that may have led to hacking activities. - A request for re - authentication from the user may be initiated by the processor 220 (in case of false alarms).

[0051] After detecting an unauthorized virtual object and / or warning the user about possible malware or security violations, the processor 220 may identify and categorize the risk level of any unauthorized virtual object, such as a low-risk unauthorized virtual object or a high-risk unauthorized virtual object. For example, a low-risk unauthorized virtual object could be an unauthorized advertisement, while a high-risk unauthorized virtual object could be an unauthorized traffic sign. In some aspects, the differentiating factor for low / high-risk content can be based on whether the content changes how the user would act when returning to the real world. For example, the user is sitting and viewing some content, and one or more unauthorized advertisements appear on the display without interacting with the content or the user's surroundings. If the user is likely to notice the advertisement and it is highly likely that nothing will happen or no danger will occur afterwards, the risk level should be categorized as low risk. However, in a similar scenario, if instead of showing an advertisement, fire or smoke is displayed, this is likely a real emergency and may increase the user's stress due to concerns that something is burning. Additionally, it is highly likely to cause the user to take actions such as triggering a fire alarm, which is inappropriate and may increase the risk to others. This type of unauthorized virtual object is classified as a high risk level. Thus, some aspects include identifying an unauthorized virtual object and determining that the risk level is low risk when the unauthorized virtual object is unlikely to change user action, or determining that the risk level is high risk when the unauthorized virtual object is likely to change user action.

[0052] It will be appreciated that various systems / algorithms / techniques may be used to detect / identify / recognize / classify virtual objects within a captured image. Deformable Parts Models (DPM) systems / algorithms use a sliding window approach where classifiers are run at locations evenly spaced across the entire image. Region-Based Convolutional Neural Networks (R-CNNs) systems / algorithms generate potential bounding boxes within an image and then run classifiers against these proposed boxes for object localization and recognition, and YOLO (You Only Look Once) systems / algorithms perform object detection as a regression from image pixels to bounding box coordinates and class probabilities. The various aspects are not limited to the examples described above, and it will be appreciated that other known systems / algorithms / techniques for object detection / recognition / identification / classification may be used.

[0053] In some aspects, the risk level can be determined by at least one of the user's characteristics or the user's context. For example, certain user characteristics can include aspects such as age, mental ability, other risky characteristics, etc. In some aspects, a certain user characteristic can, for example, for a user under a certain age having a restricted mental ability, low visual perception, or any other characteristic that can cause an adverse user reaction, or for a user over a certain age, cause the risk level to be determined as high risk for any unauthorized virtual object. Further, the user's context can also cause the risk level to be determined as high risk for any unauthorized virtual object. For example, if the user is participating in a training session, an important meeting, etc., and the supplied inappropriate training / information disruption and / or potential is considered unacceptable, any unauthorized virtual object can be flagged as high risk. It will be understood that the characteristic / context information can be collected using any suitable technique. For example, in some aspects, the information is manually input by the user such as age, health risk, etc., and can be approved for us in the detection unit through the user's consent. In some aspects, the information can be calculated by another context awareness algorithm (e.g., calculating that the user is driving based on a motion classifier, understanding that the user is in a meeting based on an audio speech profile, etc.). Further, although the above has discussed the risk level as low risk or high risk, it will be understood that additional risk levels can be defined based on various aspects and / or combinations of the foregoing examples. Thus, from the above, it will be understood that these various aspects disclosed are not limited to the specific examples provided herein.

[0054] In some embodiments, the display function may incorporate multiple security levels. The mobile device 200 may operate under normal and / or moderate security under normal circumstances, but if any unauthorized virtual object / hacking is detected by the processor 220, the processor 220 may increase the security level. It will be understood that the actions that the mobile device 200 may take to prevent further hacking are not limited to the various examples provided herein. The mobile device 200 may take any other action to stop current hacking and prevent future hacking, as will be understood by those skilled in the art.

[0055] The components of FIG. 2 can be implemented in various ways. In some implementations, the components of FIG. 2 may be implemented within one or more circuits, such as, for example, one or more processors and / or one or more ASICs (which may include one or more processors). Here, each circuit may use and / or incorporate at least one memory component for storing information or executable code used by the circuit to provide this functionality. For example, some or all of the functions represented by blocks 210-240 may be implemented by the processor and memory component(s) of mobile device 200 (e.g., by execution of appropriate code and / or by appropriate configuration of the processor component). However, as will be understood, such operations, actions, and / or functions may in fact be performed by specific components or combinations of components of mobile device 200, such as antenna array 210, transceiver 222, processor 220, camera sensor system 230, display 234, and memory 240. Further, it will be understood that some of the functions disclosed herein may be distributed across one or more additional devices. For example, mobile device 200 (e.g., smart glasses, UE, etc.) may provide data to one or more other devices that may execute at least a portion of the processing to reduce power consumption and / or improve performance. For example, mobile device 200 (e.g., smart glasses, UE, etc.) may communicate data with another mobile device, a desktop device, one or more servers within the core network and / or on the Internet (using communication systems disclosed herein and known in the art), or combinations thereof.

[0056] Aspects are to be understood to include various ways of performing the processes, functions, and / or algorithms disclosed herein. For example, FIGS. 3A and 3B respectively show methods 300 and 301 for detecting unauthorized virtual objects within a mobile device. These methods can be implemented by a device such as mobile device 200, processor 220, UE 104, 190, or other UEs shown in FIG. 1. In some aspects, methods 300 and 301 described in FIGS. 3A and 3B can be implemented by smart glasses 400 shown in FIG. 4.

[0057] Referring to FIG. 3A, at block 305, method 300 receives an image from a camera. Processor 220 may receive an image captured by camera 232.

[0058] At block 315, method 300 detects virtual objects displayed on the display of the mobile device included in the image. Processor 220 analyzes the image to determine which of the objects displayed on display 234 are real objects or virtual objects, and may discover virtual objects by comparing the image from the camera with the image displayed on the display using the systems / algorithms / techniques described herein.

[0059] At block 325, method 300 receives data from an inertial measurement unit (IMU) after movement of the mobile device. Processor 220 may detect movement of mobile device 200 by using IMU 236.

[0060] At block 335, method 300 determines an estimated new position of the virtual object based on the data received from the IMU. Processor 200 may estimate an estimated new position of the virtual object based on the data received from IMU 236.

[0061] In block 345, the method determines the actual position of the virtual object after receiving data from the IMU. Processor 220 may determine the actual position of the virtual object displayed on display 234 after receiving data from IMU 236.

[0062] In block 355, method 300 determines the difference between the estimated new position and the actual position of the virtual object. Processor 220 may determine the difference between the estimated new position and the actual position of the virtual object.

[0063] In block 365, the method determines whether the virtual object is an unauthorized virtual object based on the difference. Processor 220 may be determined as distance, posture (e.g., inclination, orientation, etc.), time difference (e.g., delay in response to movement), and may be determined based on the difference determined from one or more points of the virtual object with respect to the estimated new position of the virtual object in 3D space, and may determine that the virtual object is an unauthorized virtual object.

[0064] Referring to FIG. 3B, various alternative methods 301 are shown. In some aspects, one or more may be combined with method 300 described above. For example, in block 310, the method performs an action to enhance the security of the mobile device when it is determined that the virtual object is an unauthorized virtual object. In some aspects, the action may include at least one of generating a warning icon, notifying an important person, terminating the connection to the public network, flagging an incident regarding the discovery of an unauthorized virtual object, reducing one or more thresholds for detecting an unauthorized virtual object, or requesting authentication from the user.

[0065] In block 320, method 301 may optionally categorize the risk level of unauthorized virtual objects. In block 330, when an unauthorized virtual object is unlikely to change user actions, the risk level may be determined as a low risk. In block 340, when an unauthorized virtual object is likely to change user actions, the risk level may be determined as a high risk.

[0066] In block 350, method 301 may optionally determine a delay between the time of receiving data from the IMU and the time of positioning a virtual object based on the data from the IMU. In block 370, method 301 may determine that the difference is a delay greater than a delay threshold when determining whether a virtual object is an unauthorized virtual object based on the difference (from block 365).

[0067] In the embodiments for carrying out the above invention, it can be seen in the examples that different features are grouped together. This way of disclosure should not be understood as meaning that the exemplary clauses have more features than are explicitly stated in each clause. Rather, various aspects of the present disclosure may include fewer features than all the features of the individual exemplary clauses disclosed. Therefore, the following clauses should be considered as incorporated into the description, and each clause can be valid separately as a distinct example. Each dependent clause may refer in that clause to a specific combination with one of the other clauses, but the aspect of that dependent clause is not limited to that specific combination. It will be understood that other exemplary clauses may also include combinations of aspects of dependent clauses with the subject matter of any other dependent or independent clause, or any combination of features with other dependent and independent clauses. Although various aspects disclosed herein clearly include these combinations, it is explicitly stated that no specific combination (such as aspects that are contradictory, such as defining an element as both an insulator and a conductor) is intended, or this is not the case if it can be easily inferred. Further, even if a clause is not directly dependent on an independent clause, it is also intended that the aspect of the clause can be included in any other independent clause.

[0068] In the following numbered clauses, implementation examples are described.

[0069] Clause 1. A method for detecting an unauthorized virtual object, comprising receiving an image from a camera, detecting a virtual object displayed on a display of a mobile device included in the image, receiving data from an inertial measurement unit (IMU) after movement of the mobile device, determining an estimated new position of the virtual object based on the data received from the IMU, determining an actual position of the virtual object after receiving data from the IMU, determining a difference between the estimated new position and the actual position of the virtual object, and determining whether the virtual object is an unauthorized virtual object based on the difference.

[0070] The method according to clause 1, wherein the difference is a distance exceeding a distance difference threshold value.

[0071] The method according to clause 1 or 2, further comprising determining a delay between a time of receiving data from the IMU and a time of positioning the virtual object based on the data from the IMU.

[0072] The method according to clause 3, wherein the virtual object is determined as an unauthorized virtual object when the difference is a delay greater than a delay threshold value.

[0073] The method according to any one of clauses 1 to 4, further comprising performing an action to enhance the security of the mobile device when the virtual object is determined to be an unauthorized virtual object.

[0074] The method according to clause 5, wherein the action to enhance security includes at least one of generating a warning icon, notifying an important person, terminating the connection to the public network, setting a flag for an incident related to the discovery of an unauthorized virtual object, reducing one or more thresholds for detecting an unauthorized virtual object, or requesting authentication from the user.

[0075] The method according to any one of clauses 1 to 6, further comprising categorizing the risk level of the unauthorized virtual object.

[0076] The method according to clause 7, further comprising identifying the unauthorized virtual object.

[0077] The method according to clause 8, further comprising determining the risk level as a low risk when the unauthorized virtual object is unlikely to change the user action, and determining the risk level as a high risk when the unauthorized virtual object is likely to change the user action.

[0078] Article 10. The method according to any one of Articles 7 to 9, wherein the risk level of an unauthorized virtual object is based on at least one of user characteristics or user context.

[0079] Article 11. The method according to any one of Articles 1 to 10, wherein detecting a virtual object further includes comparing an image from a camera with an image displayed on a display.

[0080] Article 12. The method according to any one of Articles 1 to 11, wherein an unauthorized virtual object is a virtual object inserted by an unauthorized entity.

[0081] Article 13. A mobile device configured to detect an unauthorized virtual object, comprising a memory and at least one processor communicatively coupled to the memory, wherein the at least one processor receives an image from a camera, detects a virtual object displayed on a display of the mobile device included in the image, receives data from an inertial measurement unit (IMU) after movement of the mobile device, determines an estimated new position of the virtual object based on the data received from the IMU, determines an actual position of the virtual object after receiving data from the IMU, determines a difference between the estimated new position and the actual position of the virtual object, and determines whether the virtual object is an unauthorized virtual object based on the difference.

[0082] Article 14. The mobile device according to Article 13, wherein the difference is a distance exceeding a distance difference threshold.

[0083] Article 15. The mobile device according to Article 13 or 14, wherein the at least one processor is further configured to determine a delay between a time of receiving data from the IMU and a time of positioning the virtual object based on the data from the IMU.

[0084] Clause 16. The mobile device according to Clause 15, wherein the virtual object is determined to be an unauthorized virtual object when the difference is a delay greater than the delay threshold.

[0085] Clause 17. The mobile device according to any one of Clauses 13 to 16, wherein at least one processor is further configured to perform an action to enhance the security of the mobile device when the virtual object is determined to be an unauthorized virtual object.

[0086] Clause 18. The action to enhance security includes at least one of generating a warning icon, notifying important persons, terminating the connection to the public network, setting a flag on an incident related to the discovery of an unauthorized virtual object, reducing one or more thresholds for detecting an unauthorized virtual object, or requesting authentication from the user, for the mobile device according to Clause 17.

[0087] Clause 19. The mobile device according to any one of Clauses 13 to 18, wherein at least one processor is further configured to categorize the risk level of an unauthorized virtual object.

[0088] Clause 20. The mobile device according to Clause 19, wherein at least one processor is further configured to identify an unauthorized virtual object.

[0089] Clause 21. The mobile device according to Clause 20, wherein at least one processor is further configured to determine the risk level as low risk when the unauthorized virtual object is unlikely to change user actions, and determine the risk level as high risk when the unauthorized virtual object is likely to change user actions.

[0090] Clause 22. The mobile device according to any one of Clauses 19 to 21, wherein the risk level of an unauthorized virtual object is based on at least one of the characteristics or context of the user.

[0091] Clause 23. The mobile device according to any one of Clauses 13 to 22, comprising at least one processor configured to detect a virtual object and at least one processor configured to compare an image from a camera with an image displayed on a display.

[0092] Clause 24. The mobile device according to any one of Claims 13 to 23, wherein the unauthorized virtual object is a virtual object inserted by an unauthorized entity.

[0093] Clause 25. A mobile device configured to detect an unauthorized virtual object, comprising means for receiving an image from a camera, means for detecting a virtual object displayed on a display of the mobile device included in the image, means for receiving data from an inertial measurement unit (IMU) after movement of the mobile device, means for determining an estimated new position of the virtual object based on the data received from the IMU, means for determining an actual position of the virtual object after receiving data from the IMU, means for determining a difference between the estimated new position and the actual position of the virtual object, and means for determining whether the virtual object is an unauthorized virtual object based on the difference.

[0094] Clause 26. The mobile device according to Clause 25, wherein the difference is a distance exceeding a distance difference threshold.

[0095] Clause 27. The mobile device according to Clause 25 or 26, further comprising means for determining a delay between the time of receiving data from the IMU and the time of positioning the virtual object based on the data from the IMU.

[0096] Article 28. The mobile device according to Article 27, wherein the virtual object is determined to be an unauthorized virtual object when the difference is a delay greater than the delay threshold value.

[0097] Article 29. The mobile device according to any one of Articles 25 to 28, further comprising means for performing an action for enhancing the security of the mobile device when the virtual object is determined to be an unauthorized virtual object.

[0098] Article 30. The action for enhancing security includes at least one of generating a warning icon, notifying an important person, terminating the connection to the public network, flagging an incident related to the discovery of an unauthorized virtual object, reducing one or more thresholds for detecting an unauthorized virtual object, or requesting authentication from the user, for the mobile device according to Article 29.

[0099] Article 31. The mobile device according to any one of Articles 25 to 30, further comprising means for categorizing the risk level of an unauthorized virtual object.

[0100] Article 32. The mobile device according to Article 31, further comprising means for identifying an unauthorized virtual object.

[0101] Article 33. The mobile device according to Article 32, further comprising means for determining the risk level as a low risk when the unauthorized virtual object is unlikely to change user actions, and means for determining the risk level as a high risk when the unauthorized virtual object is likely to change user actions.

[0102] Article 34. The risk level of the unauthorized virtual object is based on at least one of the characteristics of the user or the context of the user, for the mobile device according to any one of Articles 31 to 33.

[0103] Item 35. The mobile device according to any one of Items 25 to 34, wherein the means for detecting a virtual object further includes means for comparing an image from a camera with an image displayed on a display.

[0104] Item 36. The mobile device according to any one of Claims 25 to 35, wherein the unauthorized virtual object is a virtual object inserted by an unauthorized entity.

[0105] Item 37. A non-transitory computer-readable medium storing computer-executable instructions, which, when executed by a mobile device configured to detect an unauthorized virtual object, cause the mobile device to receive an image from a camera, detect a virtual object displayed on a display of the mobile device included in the image, receive data from an inertial measurement unit (IMU) after the mobile device has moved, determine an estimated new position of the virtual object based on the data received from the IMU, determine an actual position of the virtual object after receiving data from the IMU, determine a difference between the estimated new position and the actual position of the virtual object, and determine whether the virtual object is an unauthorized virtual object based on the difference.

[0106] Item 38. The non-transitory computer-readable medium according to Item 37, wherein the difference is a distance exceeding a distance difference threshold.

[0107] Item 39. The non-transitory computer-readable medium according to Item 37 or 38, further comprising computer-executable instructions which, when executed by the mobile device, cause the mobile device to determine a delay between a time of receiving data from the IMU and a time of positioning the virtual object based on the data from the IMU.

[0108] Clause 40. The non-transitory computer-readable medium according to Clause 39, wherein the virtual object is determined as an unauthorized virtual object when the difference is a delay greater than the delay threshold.

[0109] Clause 41. The non-transitory computer-readable medium according to Clauses 37 to 40, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to perform an action to enhance the security of the mobile device when it is determined that the virtual object is an unauthorized virtual object.

[0110] Clause 42. The action to enhance security includes at least one of generating a warning icon, notifying an important person, terminating the connection to a public network, setting a flag on an incident related to the discovery of an unauthorized virtual object, reducing one or more thresholds for detecting an unauthorized virtual object, or requesting authentication from the user, according to Clause 41.

[0111] Clause 43. The non-transitory computer-readable medium according to any one of Clauses 37 to 42, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to categorize the risk level of an unauthorized virtual object.

[0112] Clause 44. The non-transitory computer-readable medium according to Clause 43, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to identify an unauthorized virtual object.

[0113] Clause 45. The non-transitory computer-readable medium according to Clause 44, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to determine a low risk level when an unauthorized virtual object is unlikely to change user actions, and to determine a high risk level when an unauthorized virtual object is likely to change user actions.

[0114] Clause 46. The non-transitory computer-readable medium according to any one of Clauses 43 to 45, wherein the risk level of an unauthorized virtual object is based on at least one of user characteristics or user context.

[0115] Clause 47. The non-transitory computer-readable medium according to any one of Clauses 37 to 46, which, when executed by a mobile device, includes computer-executable instructions that cause the mobile device to detect a virtual object and, when executed by the mobile device, cause the mobile device to compare an image from a camera with an image displayed on a display.

[0116] Clause 48. The non-transitory computer-readable medium according to any one of Clauses 37 to 47, wherein the unauthorized virtual object is a virtual object inserted by an unauthorized entity.

[0117] One of ordinary skill in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0118] Furthermore, those skilled in the art will understand that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0119] The various exemplary logical blocks, modules, and circuits described in connection with the aspects disclosed herein can be implemented or executed using a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gates or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0120] The methods, sequences, and / or algorithms described in connection with the aspects disclosed in this specification can be embodied directly in hardware, in software modules executed by a processor, or in a combination of the two. The software modules can reside in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can be present in a user terminal (e.g., a UE). Alternatively, the processor and the storage medium can be present in the user terminal as discrete components.

[0121] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. The computer-readable medium includes both a computer storage medium and a communication medium including any medium that facilitates transfer of a computer program from one place to another. The storage medium may be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable record medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray (registered trademark) disc, where disk typically magnetically reproduces data and disc optically reproduces data using a laser. Combinations of the above should also be included within the scope of computer-readable record medium.

[0122] Note that the above disclosure shows exemplary aspects of the present disclosure, but various changes and modifications can be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps, and / or acts of the method claims according to the aspects of the present disclosure described herein need not be performed in any particular order. Further, elements of the present disclosure may be described or claimed in the singular, but the plural is contemplated unless expressly stated to the contrary.

Claims

1. A method for detecting an unauthorized virtual object, comprising: Receiving an image from a camera; Detecting a virtual object displayed on a display of a mobile device included in the image; Receiving data from an inertial measurement unit (IMU) after movement of the mobile device; Determining a newly estimated position of the virtual object based on the data received from the IMU; Determining an actual position of the virtual object after receiving the data from the IMU; Determining a difference between the newly estimated position and the actual position of the virtual object; Determining whether the virtual object is the unauthorized virtual object based on the difference; A method comprising the above.

2. The method according to claim 1, wherein the difference is a distance exceeding a distance difference threshold.

3. Further comprising determining a delay between a time of receiving the data from the IMU and a time of positioning the virtual object based on the data from the IMU The method according to claim 1.

4. The method according to claim 3, wherein the virtual object is determined as the unauthorized virtual object when the difference is a delay greater than a delay threshold.

5. Further comprising performing an action to enhance the security of the mobile device when it is determined that the virtual object is the unauthorized virtual object The method according to claim 1.

6. The action for enhancing the security includes at least one of generating a warning icon, notifying an important person, terminating a connection to a public network, flagging an incident related to the discovery of the unauthorized virtual object, reducing one or more thresholds for detecting the unauthorized virtual object, or requesting authentication from a user. The method according to claim 5.

7. Further comprising categorizing a risk level of the unauthorized virtual object The method according to claim 1.

8. Further comprising identifying the unauthorized virtual object The method according to claim 7.

9. Determining that the risk level is a low risk when the unauthorized virtual object is unlikely to change a user action; Determining the risk level as high risk when the unauthorized virtual object is likely to change user actions, The method according to claim 8, further comprising.

10. The method according to claim 7, wherein the risk level of the unauthorized virtual object is based on at least one of user characteristics or user context.

11. Detecting the virtual object, Further includes comparing the image from the camera with the image displayed on the display. The method according to claim 1, further comprising.

12. The method according to claim 1, wherein the unauthorized virtual object is a virtual object inserted by an unauthorized entity.

13. A mobile device configured to detect an unauthorized virtual object, A memory, At least one processor communicatively coupled to the memory, Comprising, the at least one processor, Receiving an image from a camera, Detecting a virtual object displayed on a display of the mobile device included in the image, Receiving data from an inertial measurement unit (IMU) after movement of the mobile device, Determining an estimated new position of the virtual object based on the data received from the IMU, Determining an actual position of the virtual object after receiving the data from the IMU, Determining a difference between the estimated new position and the actual position of the virtual object, Based on the difference, determining whether the virtual object is the unauthorized virtual object. A mobile device configured as such.

14. The mobile device according to claim 13, wherein the difference is a distance exceeding a distance difference threshold.

15. The at least one processor, The mobile device according to claim 13, further configured to determine a delay between the time of receiving the data from the IMU and the time of positioning the virtual object based on the data from the IMU.

16. The mobile device according to claim 15, wherein the virtual object is determined as the unauthorized virtual object when the difference is a delay greater than a delay threshold.

17. The at least one processor, When it is determined that the virtual object is the unauthorized virtual object, it is further configured to perform an action to enhance the security of the mobile device. The mobile device according to claim 13.

18. The action for enhancing the security includes at least one of generating a warning icon, notifying an important person, terminating the connection to the public network, setting a flag for an incident related to the discovery of the unauthorized virtual object, reducing one or more thresholds for detecting the unauthorized virtual object, or requesting authentication from the user. The mobile device according to claim 17.

19. The at least one processor is further configured to categorize the risk level of the unauthorized virtual object The mobile device according to claim 13.

20. The at least one processor is further configured to identify the unauthorized virtual object The mobile device according to claim 19.

21. The at least one processor is configured to determine the risk level as low risk when the unauthorized virtual object is less likely to change user actions, configured to determine the risk level as high risk when the unauthorized virtual object is more likely to change user actions. The mobile device according to claim 20.

22. The risk level of the unauthorized virtual object is based on at least one of user characteristics or user context. The mobile device according to claim 19.

23. The at least one processor configured to detect the virtual object is configured to compare the image from the camera with the image displayed on the display The mobile device according to claim 13, including the at least one processor configured as such.

24. The unauthorized virtual object is a virtual object inserted by an unauthorized entity. The mobile device according to claim 13.

25. A mobile device configured to detect an unauthorized virtual object, means for receiving an image from a camera, Means for detecting a virtual object displayed on a display of a mobile device included in the image; Means for receiving data from an inertial measurement unit (IMU) after movement of the mobile device; Means for determining an estimated new position of the virtual object based on the data received from the IMU; Means for determining an actual position of the virtual object after receiving the data from the IMU; Means for determining a difference between the estimated new position and the actual position of the virtual object; Means for determining whether the virtual object is an unauthorized virtual object based on the difference; A mobile device comprising the above.

26. The mobile device according to claim 25, wherein the difference is a distance exceeding a distance difference threshold.

27. Means for determining a delay between a time of receiving the data from the IMU and a time of positioning the virtual object based on the data from the IMU The mobile device according to claim 25, further comprising the above.

28. The mobile device according to claim 27, wherein the virtual object is determined as the unauthorized virtual object when the difference is a delay greater than a delay threshold.

29. Means for performing an action to enhance security of the mobile device when the virtual object is determined to be the unauthorized virtual object The mobile device according to claim 25, further comprising the above.

30. The action for enhancing security includes at least one of generating a warning icon, notifying an important person, terminating a connection to a public network, setting a flag on an incident related to discovery of the unauthorized virtual object, reducing one or more thresholds for detecting the unauthorized virtual object, or requesting authentication from a user, according to claim 29. The mobile device described.

31. Means for categorizing a risk level of the unauthorized virtual object The mobile device according to claim 25, further comprising the above.

32. Means for identifying the unauthorized virtual object The mobile device according to claim 31, further comprising the above.

33. Means for determining the risk level as low risk when the unauthorized virtual object is unlikely to change user actions; Means for determining the risk level as high risk when the unauthorized virtual object is likely to change user actions; The mobile device according to claim 32, further comprising: **Claim 34** The mobile device according to claim 31, wherein the risk level of the unauthorized virtual object is based on at least one of user characteristics or user context. **Claim 35** The means for detecting the virtual object Further includes means for comparing the image from the camera with the image displayed on the display The mobile device according to claim 25. **Claim 36** The mobile device according to claim 25, wherein the unauthorized virtual object is a virtual object inserted by an unauthorized entity. **Claim 37** A non-transitory computer-readable medium storing computer-executable instructions, which, when executed by a mobile device configured to detect an unauthorized virtual object, cause the mobile device to Receive an image from a camera, Detect a virtual object displayed on a display of the mobile device included in the image, Receive data from an inertial measurement unit (IMU) after movement of the mobile device, Determine an estimated new position of the virtual object based on the data received from the IMU, Determine an actual position of the virtual object after receiving the data from the IMU, Determine a difference between the estimated new position and the actual position of the virtual object, Determine whether the virtual object is the unauthorized virtual object based on the difference. Non-transitory computer-readable medium. **Claim 38** The non-transitory computer-readable medium according to claim 37, wherein the difference is a distance exceeding a distance difference threshold. **Claim 39** When executed by the mobile device, cause the mobile device to Determine a delay between a time of receiving the data from the IMU and a time of positioning the virtual object based on the data from the IMU The non-transitory computer-readable medium according to claim 37, further comprising computer-executable instructions.

40. The non-transitory computer-readable medium according to claim 39, wherein the virtual object is determined as the unauthorized virtual object when the difference is a delay greater than a delay threshold.

41. When executed by the mobile device, cause the mobile device to perform an action to enhance the security of the mobile device when the virtual object is determined to be the unauthorized virtual object The non-transitory computer-readable medium according to claim 37, further comprising computer-executable instructions.

42. The non-transitory computer-readable medium according to claim 41, wherein the action for enhancing the security includes at least one of generating a warning icon, notifying an important person, ending a connection to a public network, setting a flag on an incident related to the discovery of the unauthorized virtual object, reducing one or more thresholds for detecting the unauthorized virtual object, or requesting authentication from a user.

43. When executed by the mobile device, cause the mobile device to categorize the risk level of the unauthorized virtual object The non-transitory computer-readable medium according to claim 37, further comprising computer-executable instructions.

44. When executed by the mobile device, cause the mobile device to identify the unauthorized virtual object The non-transitory computer-readable medium according to claim 43, further comprising computer-executable instructions.

45. When executed by the mobile device, cause the mobile device to determine that the risk level is a low risk when the unauthorized virtual object is unlikely to change user actions, and determine that the risk level is a high risk when the unauthorized virtual object is likely to change user actions The non-transitory computer-readable medium according to claim 44, further comprising computer-executable instructions.

46. The non - transitory computer - readable medium according to claim 43, wherein the risk level of the unauthorized virtual object is based on at least one of user characteristics or user context. **Claim 47** When executed by the mobile device, the computer - executable instructions that cause the mobile device to detect the virtual object, when executed by the mobile device, cause the mobile device to compare the image from the camera with the image displayed on the display The non - transitory computer - readable medium according to claim 37, comprising computer - executable instructions. **Claim 48** The non - transitory computer - readable medium according to claim 37, wherein the unauthorized virtual object is a virtual object inserted by an unauthorized entity.

Images