Visual content verification in extended and augmented reality

Abstract

Systems and methods for detecting an unauthorized virtual object in a mobile device are disclosed. In one aspect, a mobile device may receive an image from a camera. The mobile device may detect a virtual object displayed on a display of the mobile device included in the image. After movement of the mobile device, the mobile device may receive data from an inertial measurement unit (IMU). The mobile device may determine an estimated new location of the virtual object based on data received from the IMU. The mobile device may determine an actual location of the virtual object after receiving data from the IMU. The mobile device may determine a difference between the estimated new location and the actual location of the virtual object. The mobile device may determine whether the virtual object is an unauthorized virtual object based on the difference.

Description

Background Art

[0001] 1. Field

[0002] Aspects of the present disclosure generally relate to extended and augmented reality. In some aspects, the present disclosure relates to verification of objects within extended and / or augmented reality.

[0003] 2. Description of related technologies

[0004] Augmented reality (AR) and extended reality (XR) devices and systems have become very popular. AR / XR devices can be used in various fields, such as games, Internet surfing, business software, map navigation, etc. Many devices, such as smartphones, wearable devices, vehicles, and smart watches, can adopt AR / XR technology and display AR / XR virtual objects and renderings on the displays of these devices. However, the display of virtual objects and renderings may cause unauthorized entities (e.g., hackers, viruses, malware, etc.) to display false or unauthorized virtual objects and renderings. For example, an unauthorized entity may display a fake vehicle on the display of a vehicle equipped with an AR / XR system, causing an accident, or display a fake bank account number on a smartphone equipped with an AR / XR system when a user uses the smartphone for online banking, thereby stealing money. Summary of the invention

[0005] A brief summary of one or more aspects disclosed herein is given below. Therefore, the following summary should not be regarded as an extensive review of all expected aspects, nor should it be regarded as identifying key or important elements related to all expected aspects or delineating the scope related to any particular aspect. Therefore, the sole purpose of the following summary is to present certain concepts related to one or more aspects related to the mechanisms disclosed herein in a simplified form before the detailed description presented below.

[0006] In one aspect, a method for detecting an unauthorized virtual object includes receiving an image from a camera; detecting a virtual object displayed on a display of a mobile device that is included in the image; receiving data from an inertial measurement unit (IMU) after the mobile device moves; determining an estimated new position of the virtual object based on the data received from the IMU; determining an actual position of the virtual object after receiving the data from the IMU; determining a difference between the estimated new position and the actual position of the virtual object; and determining whether the virtual object is an unauthorized virtual object based on the difference.

[0007] In one aspect, a mobile device configured to detect an unauthorized virtual object includes: a memory; and at least one processor communicatively coupled to the memory, the at least one processor configured to: receive an image from a camera; detect a virtual object displayed on a display of the mobile device included in the image; receive data from an inertial measurement unit (IMU) after the mobile device moves; determine an estimated new position of the virtual object based on the data received from the IMU; determine an actual position of the virtual object after receiving the data from the IMU; determine a difference between the estimated new position and the actual position of the virtual object; and determine whether the virtual object is an unauthorized virtual object based on the difference.

[0008] In one aspect, a mobile device configured to detect unauthorized virtual objects includes: a device for receiving an image from a camera; a device for detecting a virtual object displayed on a display of the mobile device that is included in the image; a device for receiving data from an inertial measurement unit (IMU) after the mobile device moves; a device for determining an estimated new position of the virtual object based on the data received from the IMU; a device for determining the actual position of the virtual object after receiving the data from the IMU; a device for determining a difference between the estimated new position and the actual position of the virtual object; and a device for determining whether the virtual object is an unauthorized virtual object based on the difference.

[0009] In one aspect, a non-transitory computer-readable medium storing computer-executable instructions, when the computer-executable instructions are executed by a mobile device configured to detect an unauthorized virtual object, causes the mobile device to: receive an image from a camera; detect a virtual object displayed on a display of the mobile device included in the image; after the mobile device moves, receive data from an inertial measurement unit (IMU); determine an estimated new position of the virtual object based on the data received from the IMU; after receiving the data from the IMU, determine the actual position of the virtual object; determine a difference between the estimated new position and the actual position of the virtual object; and determine whether the virtual object is an unauthorized virtual object based on the difference.

[0010] Other objects and advantages associated with the aspects disclosed herein will be apparent to those skilled in the art based on the accompanying drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The figures are presented to aid in describing various aspects of the invention.

[0012] Figure 1 An example wireless communication system in accordance with at least one aspect of the present disclosure is shown.

[0013] Figure 2 An exemplary mobile device in accordance with at least one aspect of the present disclosure is shown.

[0014] Figure 3A A flow chart corresponding to one or more methods of detecting unauthorized virtual objects according to at least one aspect of the present disclosure is shown.

[0015] Figure 3B A flow chart corresponding to one or more methods of detecting unauthorized virtual objects according to at least one aspect of the present disclosure is shown.

[0016] Figure 4 An exemplary embodiment of a mobile device in the form of smart glasses according to at least one aspect of the present disclosure is shown.

[0017] Figure 5A Positioning of a virtual object on a display according to at least one aspect of the present disclosure is shown.

[0018] Figure 5B Positioning of a virtual object on a display according to at least one aspect of the present disclosure is shown.

[0019] Figure 5C Positioning of a virtual object on a display according to at least one aspect of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Various aspects of the present disclosure are provided in the following description and related drawings, which are directed to various examples provided for illustrative purposes. Alternative aspects may be designed without departing from the scope of the present disclosure. In addition, well-known elements of the present disclosure will not be described in detail or will be omitted to avoid confusing the relevant details of the present disclosure.

[0021] The words "exemplary" and / or "example" as used herein mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term "aspects of the disclosure" does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

[0022] Those skilled in the art will appreciate that the information and signals described below may be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned in the following description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof, depending in part on the specific application, in part on the desired design, in part on the corresponding technology, etc.

[0023] In addition, many aspects are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be appreciated that the various actions described herein may be performed by specific circuits (e.g., application specific integrated circuits (ASICs)), by program instructions executed by one or more processors, or by a combination of the two. In addition, the sequences of actions described herein may be considered to be fully contained in any form of non-transitory computer-readable storage medium, in which a corresponding set of computer instructions are stored, which, when executed, will cause or instruct the associated processor of the device to perform the functions described herein. Therefore, various aspects of the present disclosure may be embodied in a variety of different forms, all of which are considered to be within the scope of the claimed subject matter. In addition, for each aspect described herein, the corresponding form of any such aspect may be described herein as, for example, "logic" that is "configured to" perform the described actions.

[0024] As used herein, the terms "user equipment" (UE) and "base station" are not intended to be specific or limited to any specific radio access technology (RAT), unless otherwise specified. In general, a UE can be any wireless communication device (e.g., a mobile phone, a router, a tablet, a laptop, a consumer asset tracking device, a wearable (e.g., a smart watch, glasses, augmented reality (AR) / virtual reality (VR) headset, etc.), a vehicle (e.g., a car, a motorcycle, a bicycle, etc.), an Internet of Things (IoT) device, etc.) used by a user to communicate over a wireless communication network. A UE can be mobile or can be (e.g., at certain times) fixed and can communicate with a radio access network (RAN). As used herein, the term "UE" is interchangeably referred to as an "access terminal" or "AT", "client device", "wireless device", "subscriber equipment", "subscriber terminal", "subscriber station", "user terminal" or "UT", "mobile device", "mobile terminal", "mobile station" or variants thereof. Typically, a UE can communicate with a core network via a RAN, and through the core network, the UE can connect to external networks such as the Internet and other UEs. Of course, other mechanisms for connecting to the core network and / or the Internet are also possible for the UE, such as through a wired access network, a wireless local area network (WLAN) network (e.g., based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 specification, etc.), etc.

[0025] Depending on the network in which the base station is deployed, the base station may operate according to one of several RATs for communicating with the UE, and may be alternatively referred to as an access point (AP), a network node, a Node B, an evolved Node B (eNB), a next generation eNB (ng-eNB), a new radio (NR) Node B (also referred to as a gNB or gNodeB), etc. The base station may be primarily used to support wireless access for the UE, including supporting data, voice and / or signaling connections for the supported UE. In some systems, the base station may provide pure edge node signaling functions, while in other systems it may provide additional control and / or network management functions. The communication link through which the UE may send a signal to the base station is referred to as an uplink (UL) channel (e.g., a reverse traffic channel, a reverse control channel, an access channel, etc.). The communication link through which the base station may send a signal to the UE is referred to as a downlink (DL) or forward link channel (e.g., a paging channel, a control channel, a broadcast channel, a forward traffic channel, etc.). The term traffic channel (TCH) used herein may refer to an uplink / reverse or downlink / forward traffic channel.

[0026] The term "base station" may refer to a single physical transmit-receive point (TRP) or to multiple physical TRPs that may or may not be co-located. For example, when the term "base station" refers to a single physical TRP, the physical TRP may be an antenna of the base station corresponding to the cell (or several cell sectors) of the base station. In the case where the term "base station" refers to multiple co-located physical TRPs, the physical TRP may be an antenna array of the base station (for example, in a multiple-input multiple-output (MIMO) system or where the base station employs beamforming). In the case where the term "base station" refers to multiple non-co-located physical TRPs, the physical TRP may be a distributed antenna system (DAS) (a network of spatially separated antennas connected to a common source via a transmission medium) or a remote radio head (RRH) (a remote base station connected to a serving base station). Alternatively, a non-co-located physical TRP may be a serving base station that receives measurement reports from a UE and an adjacent base station whose reference RF signal the UE is measuring. Because a TRP is a point at which a base station transmits and receives wireless signals, as used herein, references to transmissions from a base station or receptions at a base station should be understood to refer to a specific TRP of a base station.

[0027] In some embodiments supporting UE positioning, the base station may not support wireless access for the UE (e.g., may not support data, voice, and / or signaling connections for the UE), but may send reference signals to the UE for measurement by the UE, and / or may receive and measure signals sent by the UE. Such a base station may be referred to as a positioning beacon (e.g., when sending a signal to the UE) and / or a position measurement unit (e.g., when receiving and measuring a signal from the UE).

[0028] An "RF signal" comprises an electromagnetic wave of a given frequency that transmits information through the space between a transmitter and a receiver. As used herein, a transmitter may transmit a single "RF signal" or multiple "RF signals" to a receiver. However, due to the propagation characteristics of RF signals through multipath channels, a receiver may receive multiple "RF signals" corresponding to each transmitted RF signal. The same transmitted RF signal on different paths between a transmitter and a receiver may be referred to as a "multipath" RF signal.

[0029] Figure 1 An exemplary wireless communication system 100 is shown. The wireless communication system 100 (which may also be referred to as a wireless wide area network (WWAN)) may include various base stations 102 and various UEs 104. The base stations 102 may include macro cell base stations (high power cellular base stations) and / or small cell base stations (low power cellular base stations). In one aspect, the macro cell base stations may include eNBs and / or ng-eNBs, where the wireless communication system 100 corresponds to an LTE network, or gNBs, where the wireless communication system 100 corresponds to an NR network, or a combination of the two, and the small cell base stations may include femto cells, pico cells, micro cells, etc.

[0030] The base stations 102 may collectively form a RAN and interface with a core network 170 (e.g., an evolved packet core (EPC) or a 5G core (5GC)) via backhaul links 122, and interface with one or more location servers 172 (which may be part of the core network 170 or may be external to the core network 170) via the core network 170. The base stations 102 may communicate with each other directly or indirectly (e.g., via the EPC / 5GC) via backhaul links 134, which may be wired or wireless.

[0031] Base stations 102 can communicate wirelessly with UEs 104. Each base station 102 can provide communication coverage for a respective geographic coverage area 110. In one aspect, the base stations 102 in each geographic coverage area 110 can support one or more cells. A "cell" is a logical communication entity for communicating with a base station (e.g., through some frequency resources, called carrier frequency, component carrier, carrier, frequency band, etc.), and can be associated with an identifier (e.g., physical cell identifier (PCI), virtual cell identifier (VCI), cell global identifier (CGI)) to distinguish cells operating via the same or different carrier frequencies. In some cases, different cells can be configured according to different protocol types (e.g., machine type communication (MTC), narrowband Internet of Things (NB-IoT), enhanced mobile broadband (eMBB), etc.) that can provide access to different types of UEs. Because a cell is supported by a specific base station, the term "cell" can refer to one or both of the logical communication entity and the base station that supports it, depending on the context. In some cases, the term "cell" can also refer to a geographic coverage area (e.g., sector) of a base station, as long as the carrier frequency can be detected and used for communication within some portion of the geographic coverage area 110.

[0032] Although the geographic coverage areas 110 of adjacent macrocell base stations 102 may partially overlap (e.g., in a handover region), some geographic coverage areas 110 may be substantially overlapped by a larger geographic coverage area 110. For example, a small cell (SC) base station 102' may have a geographic coverage area 110' that substantially overlaps with the geographic coverage areas 110 of one or more macrocell base stations 102. A network including small cell and macrocell base stations may be referred to as a heterogeneous network. A heterogeneous network may also include a home eNB (HeNB), which may provide services to a restricted group referred to as a closed subscriber group (CSG).

[0033] The communication link 120 between the base station 102 and the UE 104 may include uplink (also referred to as reverse link) transmissions from the UE 104 to the base station 102 and / or downlink (also referred to as forward link) transmissions from the base station 102 to the UE 104. The communication link 120 may use MIMO antenna technology, including spatial multiplexing, beamforming, and / or transmit diversity. The communication link 120 may be over one or more carrier frequencies. The allocation of carriers may be asymmetric for the downlink and uplink (e.g., more or fewer carriers may be allocated for the downlink than for the uplink).

[0034] The wireless communication system 100 may also include a wireless local area network (WLAN) access point (AP) 150 that communicates with a WLAN station (STA) 152 in an unlicensed spectrum (e.g., 5 GHz) via a communication link 154. When communicating in the unlicensed spectrum, the WLAN STA 152 and / or the WLAN AP 150 may perform a clear channel assessment (CCA) or a listen before talk (LBT) procedure prior to communication to determine whether a channel is available.

[0035] The small cell base station 102' can operate in a licensed and / or unlicensed spectrum. When operating in an unlicensed spectrum, the small cell base station 102' can adopt LTE or NR technology and use the same 5GHz unlicensed spectrum used by the WLAN AP 150. The small cell base station 102' adopts LTE / 5G in the unlicensed spectrum to expand the coverage of the access network and / or increase the capacity of the access network. NR in the unlicensed spectrum may be referred to as NR-U. LTE in the unlicensed spectrum may be referred to as LTE-U, licensed assisted access (LAA) or multiple access.

[0036] The wireless communication system 100 may also include a millimeter wave (mmW) base station 180 that may operate at mmW frequencies and / or near mmW frequencies for communicating with the UE 182. Extremely high frequency (EHF) is a portion of the RF in the electromagnetic spectrum. The EHF range is 30 GHz to 300 GHz with wavelengths between 1 mm and 10 mm. Radio waves in this frequency band may be referred to as millimeter waves. Near millimeter waves may extend down to frequencies of 3 GHz with wavelengths of 100 mm.

[0037] The wireless communication system 100 may also include a UE 164, which may communicate with the macrocell base station 102 via a communication link 120 and / or communicate with the mmW base station 180 via a mmW communication link 184. For example, the macrocell base station 102 may support a PCell and one or more SCells for the UE 164, and the mmW base station 180 may support one or more SCells for the UE 164.

[0038] exist Figure 1 In the example of FIG. 1 , one or more Earth-orbiting satellite positioning system (SPS) spacecraft (SV) 112 (eg, satellites) may be used as any of the illustrated UEs (for simplicity, in FIG. 1 ). Figure 1104). UE 104 may include one or more dedicated SPS receivers that are specifically designed to receive SPS signals 124 to derive geographic location information from SV 112. SPS generally includes a transmitter system (e.g., SV 112) that is positioned to enable receivers (e.g., UE 104) to determine their location on or above the earth based at least in part on signals received from the transmitters (e.g., SPS signals 124).

[0039] The use of the SPS signal 124 may be enhanced by various satellite-based augmentation systems (SBAS) that may be associated with or otherwise capable of use with one or more global and / or regional navigation satellite systems. For example, an SBAS may include augmentation systems that provide integrity information, differential corrections, etc. (such as the Wide Area Augmentation System (WAAS), the European Geostationary Navigation Overlay Service (EGNOS), the Multifunction Satellite Augmentation System (MSAS), the Global Positioning System (GPS) Assisted Geographic Augmentation Navigation or the GPS and Geographic Augmentation Navigation System (GAGAN), etc.). Thus, as used herein, an SPS may include any combination of one or more global and / or regional navigation satellite systems and / or augmentation systems, and the SPS signal 124 may include an SPS, a quasi-SPS, and / or other signals associated with the one or more SPS.

[0040] The wireless communication system 100 may also include one or more UEs, such as UE 190, which are indirectly connected to one or more communication networks via one or more device-to-device (D2D) peer-to-peer (P2P) links (referred to as "bypass"). Figure 1 In the example of FIG. 1 , UE 190 has a D2D P2P link 192 with one of UEs 104 connected to one of base stations 102 (e.g., through which UE 190 can indirectly obtain a cellular connection), and a D2D P2P link 194 with a WLAN STA 152 connected to a WLAN AP 150 (through which UE 190 can indirectly obtain a WLAN-based Internet connection). In one example, the D2D P2P links 192 and 194 can be supported by any well-known D2D RAT, such as LTE Direct (LTE-D), WiFi Direct (WiFi-D), etc.

[0041] In some aspects, the UE / mobile device discussed herein that combines AR / XR technology to display AR / XR virtual objects and renderings may include one or more aspects of the various UEs described above, such as the ability to communicate directly with various base stations through a cellular network using various technologies (e.g., LTE, 5G, etc.), communicate through a WLAN network, communicate directly with another UE, etc. Therefore, according to the various aspects disclosed herein, the UE described above can be implemented in various forms, including but not limited to smart watches, smart phones, tablets, and other devices that can perform the functions disclosed herein. Therefore, it should be understood that the foregoing description is merely an example and should not be interpreted as limiting the various aspects disclosed herein.

[0042] refer to Figure 2 , shows a simplified schematic diagram of an exemplary mobile device 200 having a processor 220, an antenna array 210, a transceiver 222, a camera sensor system 230, and a display 234. In one aspect, the mobile device 200 can employ AR / XR technology to display AR / XR virtual objects and renderings on the display 234. The mobile device 200 also includes a memory 240. The camera sensor system 230 includes a camera 232 and an inertial measurement unit (IMU) 236. In one aspect, the IMU 236 may include a gyroscope, an accelerometer, a magnetometer, and / or other sensors necessary to measure and detect movement of the mobile device 200. The camera sensor system 230 may also include other sensors (not shown), such as one or more of a lidar sensor, a radar sensor, a velocity sensor, and / or any other sensor that may assist in the operation of the mobile device 200. It should be noted that the mobile device 200 may be similar to Figure 1 The UE 104, 190 shown, or any other UE, may also include one or more components known to those skilled in the art, but these components are not shown in the figure. Figure 2 Shown in.

[0043] The mobile device 200 may be any suitable mobile electronic device. For example, the mobile device 200 may be a smartphone, a tablet computer, a laptop computer, a smart watch, a shipment tracking device, wearable smart glasses, a vehicle-mounted computer, etc. For example, in one aspect, the mobile device 200 may take the form of smart glasses, such as Figure 4 The smart glasses 400 shown. Figure 4As shown, smart glasses 400 include frame portions 414, 416, and 412, camera 418, display 424, and nose bridge 430. In one aspect, camera 418 can be similar to camera 232, and display 424 can be similar to display 234. Furthermore, in one aspect, other components similar to processor 220, antenna array 210, and transceiver 222 can be hidden within frame portion 412 or any other frame portion.

[0044] As described above, the antenna array 210 includes multiple antennas for transmitting and receiving beamforming. The antenna array 210 is coupled to the transceiver 222. The processor 220 can control the antenna array 210 and the transceiver 222. The transceiver 222 may include a wireless wide area network (WWAN) transceiver, which provides a device (e.g., a device for transmitting, a device for receiving, a device for measuring, a device for tuning, a device for suppressing transmission, etc.) for communicating via one or more wireless communication networks (not shown) (such as NR networks, LTE networks, GSM networks, etc.). The WWAN transceiver can be connected to one or more antennas in the antenna array 210 via at least one designated RAT (e.g., NR, LTE, GSM, etc.) via a wireless communication medium of interest (e.g., a certain set of time / frequency resources in a specific spectrum) for communicating with other network nodes, such as other UEs, access points, base stations (e.g., eNBs, GNBs), etc. The transceiver 222 may also include a wireless local area network (WLAN) transceiver. The WLAN transceiver communicates via the wireless communication medium of interest via at least one designated RAT (e.g., WiFi, LTE-D, etc.) can be connected to one or more antennas in the antenna array 210 and provide means for communicating with other network nodes (such as other UEs, access points, base stations, etc.) (e.g., means for sending, means for receiving, means for measuring, means for tuning, means for suppressing sending, etc.).

[0045] In addition, the transceiver 222 may include a satellite positioning system (SPS) receiver. The SPS receiver may be connected to one or more antennas in the antenna array 210, respectively, and may provide a device for receiving and / or measuring SPS signals, such as a global positioning system (GPS) signal, a global navigation satellite system (GLONASS) signal, a Galileo signal, a Beidou signal, an Indian regional navigation satellite system (NAVIC), a quasi-zenith satellite system (QZSS), etc. The SPS receiver may include any suitable hardware and / or software for receiving and processing SPS signals.

[0046] The transceiver circuitry in the transceiver 222, which includes at least one transmitter and at least one receiver, may include an integrated device (e.g., a transmitter circuit and a receiver circuit implemented as a single communication device) in some embodiments, may include separate transmitter devices and separate receiver devices in some embodiments, or may be implemented in other ways in other embodiments. In one aspect, the transmitter may include or be coupled to multiple antennas, such as antenna array 210, which allows the corresponding equipment to perform transmit "beamforming" as described herein. Similarly, the receiver may include or be coupled to multiple antennas, such as antenna array 210, which allows the corresponding equipment to perform receive beamforming as described herein. In one aspect, the transmitter and receiver may share the same multiple antennas (e.g., antenna array 210) so that the corresponding equipment can only receive or transmit at a given time, but not simultaneously.

[0047] like Figure 2 As shown, processor 220 is coupled to transceiver 222. Figure 2 As shown, the processor 220 is also coupled to the memory 240 and the camera sensor system 230. The camera sensor system 230 includes a camera 232 and an IMU 236. The camera sensor system 230 detects and measures its respective environment and sends the measurement results as data to the processor 220. The IMU 236 can measure the movement of the mobile device 200, including movement and rotational movement in a three-dimensional (3D) coordinate system. In one aspect, the processor 220 can control the operation of the camera sensor system 230. In addition, the processor 220 is coupled to the display 234, and objects and renderings, including virtual objects and virtual renderings, can be displayed on the display 234. As used herein, it will be understood that the term real object refers to the visual measurement of the actual scene and objects in front of the AR / XR device observed through the camera lens. Compared to direct observation by the human eye, the real object is unchanged. Virtual objects and virtual renderings refer to display content that is visible only through the AR / XR device. Virtual objects / renderings are the result of rendering algorithms, not direct optical measurements, and can contain various aspects of real objects, or be completely generated by a computer through a rendering algorithm.

[0048] In one aspect, the processor 220 can control the camera 232 and the IMU 236 in the camera sensor system 230, and receive data from the camera 232 and the IMU 236. The camera 232 and the IMU 236 can provide their corresponding data to the processor 220 according to the instructions of the processor 220. The camera 232 can provide the processor 220 with an image of the environment surrounding the mobile device 200, and the processor 220 can analyze the image from the camera 232 to determine different objects in the image. The IMU 236 can measure the movement and orientation of the mobile device 200, including rotational movement and movement in a three-dimensional (e.g., x, y, z) coordinate system. The IMU 236 can also measure the movement speed of the mobile device 200. For example, the IMU measures angular velocity and acceleration, which can be used to calculate changes in the device posture. The IMU 236 can send the measured data to the processor 220.

[0049] Processor 220 may use data received from IMU 236 to determine location information of mobile device 200, including at least one of orientation, translation, or movement of mobile device 200. Location information may include angles and movements of mobile device 200 on multiple axes processed by processor 220 based on data detected by IMU 236.

[0050] The processor 220 may use an algorithm, such as a rotation vector algorithm, to determine position information of the mobile device 200. Such an algorithm may be stored in the memory 240 and processed by the processor 220. For example, the processor 220 may use a combination of data from the IMU 236 to calculate the position of the mobile device 200 in a 2D and / or 3D coordinate system.

[0051] Processor 220 can combine image data from camera 232 and determined location information of mobile device 200 to display virtual objects and virtual renderings on display 234. For example, assuming that mobile device 200 takes Figure 4 In the form of smart glasses 400 shown, processor 220 can receive images captured by camera 418 on smart glasses 400. In this example, camera 418 can capture Figure 5A The image 500 of the bedroom shown in the image 500 is received and sent to the processor 220. Based on the received image 500, the processor 220 can analyze the image 500 and determine the real objects in the image 500, such as the clock 520, the bedside table 510 and the bed 540. Based on the real objects determined in the image 500, the processor 220 can display virtual objects on the display 424, such as Figure 5A530 shown in FIG. 54 . In order to correctly display bottle 530 on nightstand 510, processor 220 may need to calculate the correct x-coordinate 532, y-coordinate 531, and z-coordinate 533 of bottle 530 based on image 500 and data received from IMU 236. The correct positioning and / or pose of bottle 530 on display 424 depends on the movement of smart glasses 400 (i.e., depends on the movement of the user of smart glasses 400), and processor 220 may analyze data from IMU 236, including the orientation, movement, and speed of smart glasses 400, to determine the correct coordinates for positioning and displaying bottle 530.

[0052] However, if the smart glasses 400 move (i.e., the user of the smart glasses 400 moves while using the smart glasses 400), the camera 418 may send a new image to the processor 220, e.g. Figure 5B 550 is shown. Since smart glasses 400 are moving, IMU 236 can send new data to processor 220 that reflects the movement. Based on image 550 and the new data from IMU 236, processor 220 can determine the new correct x-coordinate 532, y-coordinate 531, and z-coordinate 533 of bottle 530, as shown in FIG. Figure 5B After determining the new coordinates, processor 220 may display bottle 530 based on the new coordinates, such as Figure 5B shown.

[0053] One problem with AR / XR technology is that unauthorized entities may gain unauthorized access to AR / XR devices and insert unauthorized content into the device's display, causing problems. In one aspect, the mobile device 200 detects and prevents such intrusions by monitoring virtual objects and renderings displayed on the display 234. Based on the image received from the camera 232, the processor 220 can analyze the image to determine which of the objects and renderings displayed on the display 234 is a real object or a virtual object and rendering. By comparing the image received from the camera 232 with the image displayed on the display 234, the processor 220 can determine which objects are real or virtual and find virtual objects. The processor 220 can monitor the displayed virtual objects and renderings by comparing the positioning and / or posture of the virtual objects and renderings with the data from the IMU 236. In other words, the processor 220 can determine whether the positioning and / or posture of the virtual objects and renderings are aligned or correlated with the data received from the IMU 236. If an unauthorized entity that hacks into the mobile device 200 does not have access to data from the IMU 236 , virtual objects inserted by the unauthorized entity may not be properly associated or aligned with the data from the IMU 236 .

[0054] In one aspect, when the smart glasses 400 move, the processor 220 can determine the change in the posture of the smart glasses 400 based on the data from the IMU 236, wherein the data includes orientation and translation data. The data from the IMU 236 can be used to estimate the new position or coordinates of the virtual object displayed on the display 234. Based on the data from the IMU 236, the estimated new position / coordinates indicate the position where the virtual object should be placed on the display 234. In one aspect, the processor 220 can estimate the new position / coordinates by projecting the previous coordinates of the virtual object to the current visual frame seen by the smart glasses 400 using the data from the IMU 236. In addition, the processor 220 can determine the actual position / coordinates of the virtual object in the current visual frame of the display 234. The processor 220 can compare the estimated new position / coordinates with the actual determined position / coordinates of the virtual object to determine the difference between the estimated new position / coordinates and the actual determined position / coordinates. If the difference between the estimated new position / coordinates and the actual determined position / coordinates is higher than a certain threshold (e.g., exceeds a distance difference threshold), the processor 220 can determine that the virtual object is an unauthorized virtual object. For example, in some aspects, the distance difference between the predicted object (or key point on the object) position and the camera measured position is calculated. In some aspects, the distance difference threshold can be on the order of a few millimeters (mm). However, it should be understood that a larger distance difference threshold can be used to reduce false alarms. Generally, a smaller distance difference threshold increases safety, while a larger distance difference threshold reduces false alarms / false detections of unauthorized virtual objects. In addition, the threshold can be related to the actual detection algorithm, the size of the virtual object, the size of the image (e.g., the real size of the captured image), the movement of the AR / XR device, etc. Therefore, it will be appreciated that the various aspects are not limited to the foregoing examples, and those skilled in the art will understand the design implications and trade-offs in establishing various thresholds for detecting unauthorized virtual objects.

[0055] Return to reference Figure 5A and Figure 5B , if an unauthorized entity inserts bottle 530 into image 500 and does not have access to data from IMU 236, the positioning and / or pose of bottle 530 will not adjust correctly when the user of smart glasses 400 moves smart glasses 400. For example, images 500 and 550 show that smart glasses 400 moved toward bed 540 between the times when images 500 and 550 were taken. Figure 5B The bottle 530 shown in the image 550 shows the correct position and pose based on the data from the IMU 236. However, if the unauthorized entity does not have access to the data from the IMU 236, the position and pose of the bottle 530 in the image 550 will be the same as the position and pose shown in the image 500.

[0056] exist Figure 5C 5, unlike image 550, image 570 shows bottle 530 that has not been properly adjusted based on the movement of smart glasses 400 because the unauthorized entity cannot access the data from IMU 236. Bottle 530 in image 570 deviates from the estimated new position 535 of the virtual object, and when smart glasses 400 move closer to bed 540, the properly adjusted position and pose of bottle 530 are not displayed, instead of being displayed as in image 550. Therefore, if the unauthorized entity cannot access the data from IMU 236, then when mobile device 200 moves, the unauthorized entity cannot properly adjust the positioning and pose of the virtual object that the unauthorized entity has inserted into display 234. It should be understood that there will be a difference between the estimated new position 535 and the actual position of the virtual object (bottle 530). The difference may be determined as a distance, a pose (e.g., tilt, orientation, etc.), a time difference (e.g., a delay in response to movement), and may be determined from one or more points of the virtual object (bottle 530) relative to an estimated new position 535 (e.g., correct x-coordinate 532, y-coordinate 531, and z-coordinate 533) of the virtual object (bottle 530) in 3D space, as shown. Thus, the processor 220 may monitor all virtual objects and renderings displayed on the display 234 to determine whether the positioning and pose of the virtual objects and renderings are correlated or aligned with the data from the IMU 236. In one aspect, if the position and / or pose of any displayed virtual object is not correlated or aligned with the data received from the IMU 236, the processor 220 may determine that the virtual object with an inappropriate, uncorrelated position and / or pose may have been inserted into the display 234 by an unauthorized entity without proper authorization. Through this process, the processor 220 may find any improperly inserted virtual objects that were not authorized to be inserted.

[0057] In one aspect, if the unauthorized entity that inserted the unauthorized virtual object has access to data from the IMU 236, the processor 220 may still detect the unauthorized virtual object based on the latency and delay that occurs when the unauthorized virtual object is repositioned based on the data received from the IMU 236. For example, the unauthorized entity may attempt to reposition the unauthorized virtual object based on the data from the IMU 236, but the unauthorized entity will likely face delays in obtaining the data from the IMU 236 because the unauthorized entity is likely far away from the mobile device 200 and the data may need to go through many different gateways to reach the unauthorized entity's computer, while the authorized application that positioned the virtual object will likely receive the data from the IMU 236 relatively quickly. Because the unauthorized entity faces delays in obtaining data from the IMU 236, the repositioning of the unauthorized virtual object may be delayed even if the repositioning is based on the data from the IMU 236. In this case, if the delay or latency is greater than a certain threshold, the processor 220 can determine that the virtual object in question is an unauthorized virtual object that may have been improperly inserted by an unauthorized entity. The processor 220 can determine the delay threshold based on various factors, such as the environment and the application running on the smart glasses 400 or the mobile device 200. Therefore, in one aspect, the processor 220 monitors all virtual objects and renderings on the display 234, and detects virtual objects and renderings that display delays and latency when they are repositioned based on data from the IMU 236. In one aspect, if any of the virtual objects and renderings exhibits a longer delay than a threshold when being repositioned based on data from the IMU 236, the processor 220 can determine that the virtual object or rendering exhibiting the delay may be an incorrectly inserted virtual object. Through this process, the processor 220 can find any incorrectly inserted virtual objects and renderings. In some aspects, the delay threshold can be in the range of 10ms to 100ms. Similar to the distance delay discussed above, it should be understood that, in general, there is a tradeoff between a lower delay threshold and higher security and a higher probability of false alarms / unauthorized virtual object detection.

[0058] In one aspect, the mobile device 200 may have applications running on the mobile device 200 that render virtual objects on the display 234. As described above, the processor 220 may monitor and inspect virtual objects rendered by applications running on the mobile device 200. If the application has permission from the user of the application to bypass the above-described monitoring of any rendered virtual objects, the application may bypass the above-described monitoring and security features. In applications with multiple users, the application may request permission from one or more users before bypassing the above-described virtual object monitoring.

[0059] In one aspect, when the processor 220 detects unauthorized objects and renderings, the processor 220 can take different types of actions to strengthen the security of the mobile device 200 and prevent further intrusion. Listed below are some possible actions that the processor 220 can take to strengthen the security of the mobile device 200 and prevent further intrusion:

[0060] • Generate a warning icon / message to the user of the mobile device 200. The user can decide what action to take.

[0061] In certain critical applications, the processor 220 may send emergency notifications to key personnel such as the police, personal contacts, security, etc.

[0062] If the mobile device 200 is using a public network (eg, hotspot, public WiFi, etc.), the connection of the mobile device 200 may be terminated.

[0063] • Processor 220 may flag specific content or applications that may have led to intrusive activity.

[0064] • A re-authentication request from the user may be initiated by the processor 220 (in case of a false alarm).

[0065] After detecting an unauthorized virtual object and / or alerting the user to possible malware or security vulnerabilities, the processor 220 can identify the risk level of any unauthorized virtual object and classify it, such as low-risk or high-risk unauthorized virtual objects. For example, a low-risk unauthorized virtual object can be an unauthorized advertisement, but a high-risk unauthorized virtual object can be an unauthorized traffic sign. In some aspects, the low / high-risk content distinguisher can be based on whether the content will change the way the user takes action in the real world. For example, if a user is sitting and watching some content, but does not interact with the content or the user's surroundings, and one or more unauthorized advertisements appear in the display. The user is likely to realize that this is an advertisement and it is likely that no danger will occur, so the risk level should be classified as low risk. However, in a similar scenario, if instead of displaying an advertisement, a fire or smoke is displayed, this is likely to increase the user's stress because of the fear that there is a real emergency and something is burning. In addition, this will likely cause the user to take action, such as triggering a fire alarm, etc. This is inappropriate and may increase the risk of others. This type of unauthorized virtual object is classified as a high risk level. Thus, some aspects include identifying unauthorized virtual objects and determining a risk level as low risk when the unauthorized virtual object is unlikely to alter a user action, or determining a risk level as high risk when the unauthorized virtual object is likely to alter a user action.

[0066] It should be understood that various systems / algorithms / techniques may be used to detect / identify / recognize / classify virtual objects in captured images. The Deformable Part Model (DPM) system / algorithm uses a sliding window approach where the classifier is run at evenly spaced locations across the image. The Region-Based Convolutional Neural Network (R-CNN) system / algorithm generates potential bounding boxes in the image and then runs a classifier on these proposed boxes for object localization and identification, and the You Only Look Once (YOLO) system / algorithm performs object detection as a regression from image pixels to bounding box coordinates and class probabilities. It will be appreciated that the various aspects are not limited to the foregoing examples and that other known systems / algorithms / techniques for object detection / identification / recognition / classification may be used.

[0067] In some aspects, the risk level may be determined by at least one of the user's characteristics or the user's context. For example, the user's characteristics may include aspects such as age, mental ability, other risk features, etc. In some aspects, certain user characteristics may cause the risk level to be determined as high risk for any unauthorized virtual object, for example, for users below a certain age, above a certain age, with limited mental ability, low visual perception, and any other characteristics that may cause adverse user reactions. In addition, the user's context may also cause the risk level to be determined as high risk for any unauthorized virtual object. For example, if the user is attending a training meeting, an important meeting, etc. Any unauthorized virtual object may be marked as high risk because the interruption and / or possibility of incorrect training / information provided is considered unacceptable. It should be understood that any suitable technology can be used to collect characteristic / background information. For example, in some aspects, information can be manually entered by the user, such as age, health risks, etc., and approved for us in the detector through a user agreement. In some aspects, this information can be calculated by another context-aware algorithm (for example, calculating that you are driving based on a mobile classifier, understanding that you are in a meeting based on an audio voice profile, etc.). In addition, although the risk level discussed above is low risk or high risk, it should be understood that additional risk levels can be defined based on various aspects and / or combinations of the foregoing examples. Therefore, from the foregoing, it will be appreciated that the various aspects disclosed herein are not limited to the specific examples provided herein.

[0068] In some aspects, the display function can incorporate multiple security levels. Under normal circumstances, the mobile device 200 can operate under normal and / or medium security, but if the processor 220 detects any unauthorized virtual objects / intrusions, the processor 220 can increase the security level. It should be understood that the actions that the mobile device 200 can take to prevent further intrusions are not limited to the various examples provided herein. As will be appreciated by those skilled in the art, the mobile device 200 can take any other actions to stop the current intrusion and prevent any future intrusions.

[0069] Figure 2 The components can be implemented in various ways. In some embodiments, Figure 2 The components can be implemented in one or more circuits, such as one or more processors and / or one or more ASICs (which may include one or more processors). Here, each circuit may use and / or combine at least one memory component to store information or executable code used by the circuit to provide the function. For example, some or all of the functions represented by boxes 210 to 240 may be implemented by the processor and memory components of the mobile device 200 (e.g., by executing appropriate code and / or by appropriate configuration of the processor components). However, it should be understood that these operations, actions and / or functions may actually be performed by a specific component or combination of components of the mobile device 200, such as the antenna array 210, the transceiver 222, the processor 220, the camera sensor system 230, the display 234 and the memory 240. In addition, it should be understood that some of the functions disclosed herein may be distributed to one or more additional devices. For example, the mobile device 200 (e.g., smart glasses, UE, etc.) may provide data to one or more other devices, which may perform at least a portion of the processing to reduce power consumption and / or improve performance. For example, the mobile device 200 (e.g., smart glasses, UE, etc.) can communicate data with another mobile device, a desktop device, one or more servers in a core network and / or on the Internet (using the communication system disclosed herein and known in the art), or a combination thereof.

[0070] It should be understood that the various aspects include various methods for performing the processes, functions and / or algorithms disclosed herein. For example, Figure 3A and Figure 3B Methods 300 and 301 are shown for detecting unauthorized virtual objects in a mobile device. These methods may be implemented by, for example, a mobile device 200, a processor 220, a UE 104, 190, or Figure 1 In some aspects, Figure 3A and Figure 3B The methods 300 and 301 described in the above can be implemented by Figure 4 The smart glasses 400 shown are used to perform.

[0071] refer to Figure 3A At block 305 , the method 300 receives an image from a camera. The processor 220 may receive an image captured by the camera 232 .

[0072] At block 315, the method 300 detects a virtual object displayed on the display of the mobile device included in the image. The processor 220 can analyze the image to determine which object displayed on the display 234 is a real object or a virtual object, and find the virtual object by comparing the image from the camera with the image displayed on the display using systems / algorithms / techniques as discussed herein.

[0073] At block 325 , the method 300 receives data from an inertial measurement unit (IMU) after the mobile device moves. The processor 220 can detect the movement of the mobile device 200 by using the IMU 236 .

[0074] At block 335 , the method 300 determines an estimated new position of the virtual object based on the data received from the IMU. The processor 200 may estimate the new position of the virtual object based on the data received from the IMU 236 .

[0075] At block 345 , the method determines the actual position of the virtual object after receiving data from the IMU. After receiving data from the IMU 236 , the processor 220 may determine the actual position of the virtual object displayed on the display 234 .

[0076] At block 355, the method 300 determines the difference between the estimated new position and the actual position of the virtual object.The processor 220 may determine the difference between the estimated new position and the actual position of the virtual object.

[0077] At block 365, the method determines that the virtual object is an unauthorized virtual object based on the difference. The processor 220 can determine that the virtual object is an unauthorized virtual object based on a difference that can be determined as a distance, a posture (e.g., tilt, orientation, etc.), a time difference (e.g., a delay in response to a movement), and can be determined from one or more points of the virtual object relative to the estimated new position of the virtual object in 3D space, as discussed herein.

[0078] refer to Figure 3B, various optional methods 301 are shown. In some aspects, one or more can be combined with the method 300 discussed above. For example, at block 310, when the virtual object is determined to be an unauthorized virtual object, the method performs an action to increase the security of the mobile device. In some aspects, the action can include generating a warning icon, notifying key personnel, terminating a connection to a public network, marking an event of finding an unauthorized virtual object, lowering one or more thresholds for detecting unauthorized virtual objects, or requesting at least one of user authentication.

[0079] At block 320, method 301 may optionally classify the risk level of the unauthorized virtual object. At block 330, when the unauthorized virtual object is unlikely to change the user action, the risk level may be determined to be low risk. At block 340, when the unauthorized virtual object is likely to change the user action, the risk level may be determined to be high risk.

[0080] At block 350, the method 301 may optionally determine a delay between the time the data is received from the IMU and the time the virtual object is positioned based on the data from the IMU. At block 370, the method 301 may determine whether the virtual object is an unauthorized virtual object based on the difference (from block 365) to be a delay greater than a delay threshold.

[0081] In the above detailed description, it can be seen that different features are combined together in the examples. This disclosure should not be interpreted as an exemplary clause having more features than those explicitly mentioned in each clause. In contrast, various aspects of the present disclosure may include less than all the features of the disclosed single example clause. Therefore, the following clauses should be considered to be included in the specification, where each clause itself can be used as a separate example. Although each dependent clause can refer to a specific combination with one of the other clauses in a clause, the aspects of the dependent clause are not limited to the specific combination. It should be understood that other exemplary clauses may also include a combination of dependent clause aspects with the subject of any other dependent clause or independent clause, or any combination of features with other dependent and independent clauses. Various aspects disclosed herein explicitly include these combinations, unless it is explicitly expressed or it can be easily inferred that a specific combination is not intended (for example, contradictory aspects, such as defining an element as an insulator and a conductor). In addition, it is also intended that various aspects of the clause may be included in any other independent clause, even if the clause is not directly dependent on the independent clause.

[0082] Example implementations are described in the following numbered clauses:

[0083] Clause 1. A method for detecting an unauthorized virtual object, the method comprising: receiving an image from a camera; detecting a virtual object displayed on a display of a mobile device included in the image; receiving data from an inertial measurement unit (IMU) after the mobile device moves; determining an estimated new position of the virtual object based on the data received from the IMU; determining an actual position of the virtual object after receiving the data from the IMU; determining a difference between the estimated new position and the actual position of the virtual object; and determining whether the virtual object is an unauthorized virtual object based on the difference.

[0084] Clause 2. The method of clause 1, wherein the difference is a distance exceeding a distance difference threshold.

[0085] Clause 3. The method of any one of clauses 1 to 2, further comprising: determining a delay between a time when data is received from the IMU and a time when a virtual object is positioned based on the data from the IMU.

[0086] Clause 4. The method of clause 3, wherein when the difference is a delay greater than a delay threshold, the virtual object is determined to be an unauthorized virtual object.

[0087] Clause 5. The method of any one of clauses 1 to 4, further comprising: when the virtual object is determined to be an unauthorized virtual object, performing an action to increase the security of the mobile device.

[0088] Clause 6. A method according to clause 5, wherein the action to increase security includes generating a warning icon, notifying key personnel, terminating a connection to a public network, marking an event in which an unauthorized virtual object is discovered, lowering one or more thresholds for detecting unauthorized virtual objects, or requesting at least one of user authentication.

[0089] Clause 7. The method according to any one of clauses 1 to 6 further includes: classifying the risk level of unauthorized virtual objects.

[0090] Clause 8. The method according to Clause 7 further includes: identifying unauthorized virtual objects.

[0091] Clause 9. The method according to Clause 8 also includes: determining the risk level as low risk when the unauthorized virtual object is unlikely to change the user's action; and determining the risk level as high risk when the unauthorized virtual object is likely to change the user's action.

[0092] Clause 10. The method of any one of clauses 7 to 9, wherein the risk level of the unauthorized virtual object is based on at least one of a characteristic of the user or a context of the user.

[0093] Clause 11. The method of any one of clauses 1 to 10, wherein detecting the virtual object further comprises comparing an image from a camera with an image displayed on a display.

[0094] Clause 12. The method of any one of clauses 1 to 11, wherein the unauthorized virtual object is a virtual object inserted by an unauthorized entity.

[0095] Clause 13. A mobile device configured to detect an unauthorized virtual object, comprising: a memory; and at least one processor communicatively coupled to the memory, the at least one processor configured to: receive an image from a camera; detect a virtual object displayed on a display of the mobile device included in the image; receive data from an inertial measurement unit (IMU) after the mobile device moves; determine an estimated new position of the virtual object based on the data received from the IMU; determine an actual position of the virtual object after receiving the data from the IMU; determine a difference between the estimated new position and the actual position of the virtual object; and determine whether the virtual object is an unauthorized virtual object based on the difference.

[0096] Clause 14. The mobile device of clause 13, wherein the difference is a distance exceeding a distance difference threshold.

[0097] Clause 15. The mobile device of any one of clauses 13 to 14, wherein the at least one processor is further configured to: determine a delay between a time when data is received from the IMU and a time when a virtual object is positioned based on the data from the IMU.

[0098] Clause 16. The mobile device of clause 15, wherein when the difference is a delay greater than a delay threshold, the virtual object is determined to be an unauthorized virtual object.

[0099] Clause 17. The mobile device of any one of clauses 13 to 16, wherein the at least one processor is further configured to: when the virtual object is determined to be an unauthorized virtual object, perform an action to increase the security of the mobile device.

[0100] Clause 18. A mobile device according to clause 17, wherein the action to increase security includes generating a warning icon, notifying key personnel, terminating a connection to a public network, marking an event in which an unauthorized virtual object is discovered, lowering one or more thresholds for detecting unauthorized virtual objects, or requesting at least one of user authentication.

[0101] Clause 19. The mobile device of any one of clauses 13 to 18, wherein the at least one processor is further configured to: classify a risk level of an unauthorized virtual object.

[0102] Clause 20. The mobile device of clause 19, wherein the at least one processor is further configured to: identify unauthorized virtual objects.

[0103] Clause 21. A mobile device according to Clause 20, wherein the at least one processor is further configured to: determine the risk level as low risk when the unauthorized virtual object is unlikely to change the user action; and determine the risk level as high risk when the unauthorized virtual object is likely to change the user action.

[0104] Clause 22. The mobile device of any of clauses 19 to 21, wherein the risk level of the unauthorized virtual object is based on at least one of a characteristic of the user or a context of the user.

[0105] Clause 23. A mobile device as described in any of clauses 13 to 22, wherein the at least one processor is configured to detect the virtual object includes the at least one processor being configured to: compare an image from a camera with an image displayed on a display.

[0106] Clause 24. The mobile device of any of clauses 13 to 23, wherein the unauthorized virtual object is a virtual object inserted by an unauthorized entity.

[0107] Clause 25. A mobile device configured to detect an unauthorized virtual object, comprising: a device for receiving an image from a camera; a device for detecting a virtual object displayed on a display of the mobile device that is included in the image; a device for receiving data from an inertial measurement unit (IMU) after the mobile device moves; a device for determining an estimated new position of the virtual object based on the data received from the IMU; a device for determining the actual position of the virtual object after receiving the data from the IMU; a device for determining a difference between the estimated new position and the actual position of the virtual object; and a device for determining whether the virtual object is an unauthorized virtual object based on the difference.

[0108] Clause 26. The mobile device of clause 25, wherein the difference is a distance that exceeds a distance difference threshold.

[0109] Clause 27. The mobile device of any of Clauses 25 to 26, further comprising: means for determining a delay between a time when data is received from the IMU and a time when a virtual object is positioned based on the data from the IMU.

[0110] Clause 28. The mobile device of clause 27, wherein when the difference is a delay greater than a delay threshold, the virtual object is determined to be an unauthorized virtual object.

[0111] Clause 29. The mobile device of any one of clauses 25 to 28, further comprising: means for performing an action to increase the security of the mobile device when the virtual object is determined to be an unauthorized virtual object.

[0112] Clause 30. A mobile device according to clause 29, wherein the action to increase security includes generating a warning icon, notifying key personnel, terminating a connection to a public network, marking an event in which an unauthorized virtual object is discovered, lowering one or more thresholds for detecting unauthorized virtual objects, or requesting at least one of user authentication.

[0113] Clause 31. The mobile device of any one of clauses 25 to 30, further comprising: means for classifying a risk level of an unauthorized virtual object.

[0114] Clause 32. The mobile device of clause 31 further comprising: means for identifying unauthorized virtual objects.

[0115] Clause 33. The mobile device according to Clause 32 further includes: a device for determining the risk level as low risk when an unauthorized virtual object is unlikely to change the user's action; and a device for determining the risk level as high risk when an unauthorized virtual object is likely to change the user's action.

[0116] Clause 34. A mobile device as described in any of clauses 31 to 33, wherein the risk level of the unauthorized virtual object is based on at least one of a characteristic of the user or a context of the user.

[0117] Clause 35. A mobile device as described in any of clauses 25 to 34, wherein the means for detecting the virtual object further comprises: means for comparing the image from the camera with the image displayed on the display.

[0118] Clause 36. The mobile device of any of Clauses 25 to 35, wherein the unauthorized virtual object is a virtual object inserted by an unauthorized entity.

[0119] Clause 37. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by a mobile device configured to detect an unauthorized virtual object, cause the mobile device to: receive an image from a camera; detect a virtual object displayed on a display of the mobile device that is included in the image; receive data from an inertial measurement unit (IMU) after the mobile device moves; determine an estimated new position of the virtual object based on the data received from the IMU; determine an actual position of the virtual object after receiving the data from the IMU; determine a difference between the estimated new position and the actual position of the virtual object; and determine whether the virtual object is an unauthorized virtual object based on the difference.

[0120] Clause 38. The non-transitory computer-readable medium of Clause 37, wherein the difference is a distance exceeding a distance difference threshold.

[0121] Clause 39. A non-transitory computer-readable medium according to any one of clauses 37 to 38, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to: determine a delay between a time when data is received from the IMU and a time when a virtual object is positioned based on the data from the IMU.

[0122] Clause 40. The non-transitory computer-readable medium of Clause 39, wherein the virtual object is determined to be an unauthorized virtual object when the difference is a delay greater than a delay threshold.

[0123] Clause 41. The non-transitory computer-readable medium of any one of clauses 37 to 40 further comprises computer-executable instructions which, when executed by a mobile device, cause the mobile device to: perform an action to increase the security of the mobile device when a virtual object is determined to be an unauthorized virtual object.

[0124] Clause 42. A non-transitory computer-readable medium as described in clause 41, wherein the action to increase security includes generating a warning icon, notifying key personnel, terminating a connection to a public network, marking an event in which an unauthorized virtual object is discovered, lowering one or more thresholds for detecting unauthorized virtual objects, or requesting at least one of user authentication.

[0125] Clause 43. The non-transitory computer-readable medium of any one of clauses 37 to 42, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to: classify a risk level of an unauthorized virtual object.

[0126] Clause 44. The non-transitory computer-readable medium of clause 43, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to: identify an unauthorized virtual object.

[0127] Clause 45. The non-transitory computer-readable medium according to Clause 44 further includes computer-executable instructions, which, when executed by a mobile device, cause the mobile device to: determine the risk level as low risk when an unauthorized virtual object is unlikely to change a user's action; and determine the risk level as high risk when an unauthorized virtual object is likely to change a user's action.

[0128] Clause 46. The non-transitory computer-readable medium of any of Clauses 43 to 45, wherein the risk level of the unauthorized virtual object is based on at least one of a characteristic of the user or a context of the user.

[0129] Clause 47. A non-transitory computer-readable medium as described in any of clauses 37 to 46, wherein the computer executable instructions that, when executed by the mobile device, cause the mobile device to detect a virtual object include computer executable instructions that, when executed by the mobile device, cause the mobile device to: compare an image from a camera with an image displayed on a display.

[0130] Clause 48. The non-transitory computer-readable medium of any of Clauses 37 to 47, wherein the unauthorized virtual object is a virtual object inserted by an unauthorized entity.

[0131] Those skilled in the art will appreciate that any of a variety of different techniques and methods may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0132] In addition, it will be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above according to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints on the entire system. A skilled person may implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present disclosure.

[0133] The various illustrative logical blocks, modules, and circuits described in conjunction with the aspects disclosed herein may be implemented or executed with a general purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration.

[0134] The methods, sequences and / or algorithms described in conjunction with the aspects disclosed herein may be directly embodied in hardware, software modules executed by a processor, or a combination of the two. The software module may reside in a random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. An example storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal (e.g., UE). Alternatively, the processor and the storage medium may reside in a user terminal as discrete components.

[0135] In one or more example aspects, the functions described can be implemented with hardware, software, firmware, or any combination thereof. If implemented with software, these functions can be stored or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, including any media that facilitates the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a computer. As an example and not a limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer. Similarly, any connection is properly referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0136] Although the foregoing disclosure shows the illustrative aspects of the present disclosure, it should be noted that various changes and modifications may be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and / or actions of the method claims according to the aspects of the disclosure described herein do not need to be performed in any particular order. In addition, although the elements of the present disclosure may be described or claimed in singular form, plural forms are also contemplated unless explicitly stated to be limited to the singular form.

Claims

1. A method for detecting an unauthorized virtual object, the method comprising: Receive an image from a camera; detecting a virtual object displayed on a display of a mobile device included in the image; receiving data from an inertial measurement unit (IMU) after the mobile device moves; determining an estimated new position of the virtual object based on data received from the IMU; After receiving the data from the IMU, determining the actual position of the virtual object; determining a difference between the estimated new position and the actual position of the virtual object; and A determination is made based on the difference whether the virtual object is the unauthorized virtual object.

2. The method according to claim 1, wherein: The difference is a distance that exceeds a distance difference threshold.

3. The method according to claim 1, further comprising: A delay between a time when the data is received from the IMU and a time of positioning of the virtual object based on the data from the IMU is determined.

4. The method according to claim 3, wherein: When the difference is a delay greater than a delay threshold, the virtual object is determined to be the unauthorized virtual object.

5. The method according to claim 1, further comprising: When the virtual object is determined to be the unauthorized virtual object, an action to increase the security of the mobile device is performed.

6. The method according to claim 5, wherein: The actions to increase the security include generating a warning icon, notifying key personnel, terminating a connection to a public network, marking an event in which the unauthorized virtual object is discovered, lowering one or more thresholds for detecting the unauthorized virtual object, or requesting at least one of user authentication.

7. The method according to claim 1, further comprising: The risk level of the unauthorized virtual object is classified.

8. The method according to claim 7, further comprising: The unauthorized virtual object is identified.

9. The method according to claim 8, further comprising: When the unauthorized virtual object is unlikely to change the user action, determining the risk level as low risk; and When the unauthorized virtual object may change the user action, the risk level is determined to be high risk.

10. The method according to claim 7, wherein: The risk level of the unauthorized virtual object is based on at least one of a characteristic of a user or a context of a user.

11. The method according to claim 1, wherein: Detecting the virtual object further includes: The image from the camera is compared to the image displayed on the display.

12. The method according to claim 1, wherein: The unauthorized virtual object is a virtual object inserted by an unauthorized entity.

13. A mobile device configured to detect an unauthorized virtual object, comprising: Memory; and at least one processor communicatively coupled to the memory, the at least one processor configured to: Receive an image from a camera; detecting a virtual object displayed on a display of a mobile device included in the image; receiving data from an inertial measurement unit (IMU) after the mobile device moves; determining an estimated new position of the virtual object based on data received from the IMU; After receiving data from the IMU, determining the actual position of the virtual object; determining a difference between the estimated new position and the actual position of the virtual object; and A determination is made based on the difference whether the virtual object is the unauthorized virtual object.

14. The mobile device according to claim 13, wherein: The difference is a distance that exceeds a distance difference threshold.

15. The mobile device according to claim 13, wherein: The at least one processor is further configured to: A delay between a time when the data is received from the IMU and a time of positioning of the virtual object based on the data from the IMU is determined.

16. The mobile device according to claim 15, wherein: When the difference is a delay greater than a delay threshold, the virtual object is determined to be the unauthorized virtual object.

17. The mobile device according to claim 13, wherein: The at least one processor is further configured to: When the virtual object is determined to be the unauthorized virtual object, an action to increase the security of the mobile device is performed.

18. The mobile device according to claim 17, wherein: The actions to increase the security include generating a warning icon, notifying key personnel, terminating a connection to a public network, marking an event in which the unauthorized virtual object is discovered, lowering one or more thresholds for detecting the unauthorized virtual object, or requesting at least one of user authentication.

19. The mobile device according to claim 13, wherein: The at least one processor is further configured to: The risk level of the unauthorized virtual object is classified.

20. The mobile device according to claim 19, wherein: The at least one processor is further configured to: The unauthorized virtual object is identified.

21. The mobile device according to claim 20, wherein: The at least one processor is further configured to: When the unauthorized virtual object is unlikely to change the user action, determining the risk level as low risk; and When the unauthorized virtual object may change the user action, the risk level is determined to be high risk.

22. The mobile device of claim 19, wherein: The risk level of the unauthorized virtual object is based on at least one of a characteristic of a user or a context of a user.

23. The mobile device of claim 13, wherein: The at least one processor being configured to detect the virtual object comprises the at least one processor being configured to: The image from the camera is compared to the image displayed on the display.

24. The mobile device of claim 13, wherein: The unauthorized virtual object is a virtual object inserted by an unauthorized entity.

25. A mobile device configured to detect an unauthorized virtual object, comprising: means for receiving an image from a camera; means for detecting a virtual object displayed on a display of a mobile device included in the image; means for receiving data from an inertial measurement unit (IMU) after the mobile device has moved; means for determining an estimated new position of the virtual object based on the data received from the IMU; means for determining an actual position of the virtual object after receiving the data from the IMU; means for determining a difference between said estimated new position and said actual position of said virtual object; and Means for determining whether the virtual object is the unauthorized virtual object based on the difference.

26. The mobile device of claim 25, wherein: The difference is a distance that exceeds a distance difference threshold.

27. The mobile device of claim 25, further comprising: Means for determining a delay between a time when the data is received from the IMU and a time of positioning of the virtual object based on the data from the IMU.

28. The mobile device of claim 27, wherein: When the difference is a delay greater than a delay threshold, the virtual object is determined to be the unauthorized virtual object.

29. The mobile device of claim 25, further comprising: Means for performing an action to increase the security of the mobile device when the virtual object is determined to be the unauthorized virtual object.

30. The mobile device of claim 29, wherein: The actions to increase the security include generating a warning icon, notifying key personnel, terminating a connection to a public network, marking an event in which the unauthorized virtual object is discovered, lowering one or more thresholds for detecting the unauthorized virtual object, or requesting at least one of user authentication.

31. The mobile device of claim 25, further comprising: Means for classifying the risk level of the unauthorized virtual object.

32. The mobile device of claim 31 , further comprising: Means for identifying the unauthorized virtual object.

33. The mobile device of claim 32, further comprising: means for determining the risk level as low risk when the unauthorized virtual object is unlikely to alter a user action; and Means for determining the risk level as high risk when the unauthorized virtual object may alter a user action.

34. The mobile device of claim 31, wherein: The risk level of the unauthorized virtual object is based on at least one of a characteristic of a user or a context of a user.

35. The mobile device of claim 25, wherein: The device for detecting the virtual object also includes: Means for comparing the image from the camera with the image displayed on the display.

36. The mobile device of claim 25, wherein: The unauthorized virtual object is a virtual object inserted by an unauthorized entity.

37. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by a mobile device configured to detect an unauthorized virtual object, cause the mobile device to: Receive an image from a camera; detecting a virtual object displayed on a display of a mobile device included in the image; receiving data from an inertial measurement unit (IMU) after the mobile device moves; determining an estimated new position of the virtual object based on data received from the IMU; After receiving data from the IMU, determining the actual position of the virtual object; determining a difference between the estimated new position and the actual position of the virtual object; and A determination is made based on the difference whether the virtual object is the unauthorized virtual object.

38. The non-transitory computer readable medium of claim 37, wherein: The difference is a distance that exceeds a distance difference threshold.

39. The non-transitory computer-readable medium of claim 37, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to: A delay between a time when the data is received from the IMU and a time of positioning of the virtual object based on the data from the IMU is determined.

40. The non-transitory computer readable medium of claim 39, wherein: When the difference is a delay greater than a delay threshold, the virtual object is determined to be the unauthorized virtual object.

41. The non-transitory computer-readable medium of claim 37, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to: When the virtual object is determined to be the unauthorized virtual object, an action to increase the security of the mobile device is performed.

42. The non-transitory computer readable medium of claim 41, wherein: The actions to increase the security include generating a warning icon, notifying key personnel, terminating a connection to a public network, marking an event in which the unauthorized virtual object is discovered, lowering one or more thresholds for detecting the unauthorized virtual object, or requesting at least one of user authentication.

43. The non-transitory computer-readable medium of claim 37, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to: The risk level of the unauthorized virtual object is classified.

44. The non-transitory computer-readable medium of claim 43, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to: The unauthorized virtual object is identified.

45. The non-transitory computer-readable medium of claim 44, further comprising computer-executable instructions that, when executed by a mobile device, cause the mobile device to: When the unauthorized virtual object is unlikely to change the user action, determining the risk level as low risk; and When the unauthorized virtual object may change the user action, the risk level is determined to be high risk.

46. ​​The non-transitory computer readable medium of claim 43, wherein: The risk level of the unauthorized virtual object is based on at least one of a characteristic of a user or a context of a user.

47. The non-transitory computer readable medium of claim 37, wherein: The computer executable instructions that, when executed by a mobile device, cause the mobile device to detect a virtual object include computer executable instructions that, when executed by the mobile device, cause the mobile device to: The image from the camera is compared to an image displayed on the display.

48. The non-transitory computer readable medium of claim 37, wherein: The unauthorized virtual object is a virtual object inserted by an unauthorized entity.

Images