Calibration of magnetic sensors using the VIO system

Abstract

This disclosure describes a method for calibrating a magnetic sensor of a client device. The method may include receiving a set of magnetic field measurements, each magnetic field measurement including the device position, the orientation of the client device, and the observed magnetic field vector measured by the magnetic sensor. The method may include calculating a device correction vector for the client device based on the set of magnetic field measurements. For each magnetic field measurement, the method includes determining the world magnetic field vector at the device position of the magnetic field measurement, calculating the expected measured magnetic field vector at the device position, accessing an estimated device correction vector for the client device, calculating the expected adjusted vector for the client device, comparing the observed magnetic field vector associated with the magnetic field measurement with the expected adjusted vector, and calculating a device correction vector based on the comparison.

Description

Technical Field

[0001] The subject matter described generally relates to magnetic sensor calibration, and more particularly to calibration of magnetic sensors for client devices.

Background Art

[0002] This application claims the benefit of U.S. Application No. 18 / 297,164, filed Apr. 7, 2023, and U.S. Application No. 18 / 297,179, filed Apr. 7, 2023, which are incorporated herein by reference.

[0003] An online system can track the position and orientation (collectively "pose") of a user device in the physical world to provide services to a user. For example, in an Augmented Reality (AR) application, the online system can overlay virtual elements on a depiction of the real-world environment. Specifically, the online system can display virtual elements on a video feed captured by the user device and presented to the user such that the virtual elements appear to be located within the physical world. The online system uses the pose of the device to display the virtual elements correctly. For example, the online system can use the orientation of the device to determine whether to display a virtual element and, if so, where to display the virtual element on the video feed.

[0004] To determine the orientation of a user device, an online system may use magnetic field measurements captured by the user device to estimate the orientation of the user device relative to the Earth's magnetic field. For example, the online system may use a compass on the user's phone to determine the device's orientation relative to the Earth's magnetic field. However, electronic components or any metallic structures within the user device itself may generate a magnetic field that can be captured by the device's magnetic sensors. Therefore, the magnetic field measured by the magnetic sensors may be the sum of two vectors: the magnetic field from the Earth and the magnetic field from the device itself. Thus, to provide a magnetic field measurement that accurately measures the magnetic field from the Earth, the user device attempts to correct the magnetic field it generates.

[0005] Existing approaches to calibrating magnetic sensors are primarily based on optimizing the measurement of many instantaneous rotations and the readings of the magnetic sensor. For example, one method of calibrating a compass in a telephone requires the user to rotate the telephone to create a pattern of the number 8. However, this technique requires the user to perform the movement several times before the compass is calibrated. Thus, user devices cannot periodically calibrate their magnetic sensors without causing significant inconvenience to the user. [Overview of the Initiative]

[0006] The client device uses visual inertial odometry (VIO) data to calibrate the magnetic sensor on the client device. The client device uses VIO to determine its orientation when it captures magnetic field measurements. The magnetic field measurements may include the observed magnetic field vector representing the magnetic field measured by the client device's magnetic sensor, the position of the client device when it captured the magnetic field vector, and the orientation of the client device determined using VIO. In some embodiments, the client device calibrates the magnetic sensor using a batch of magnetic field measurements collected within a specified geographical area.

[0007] The client device uses this set of magnetic field measurements to calculate a device correction vector. The device correction vector (labeled herein as VCorrection) is a vector representing the error in the magnetic field vector measured by the client device's magnetic sensor. These errors originate from the electronic components within the client device that generate the magnetic field and represent the difference between the actual magnetic field measured by the magnetic sensor and the magnetic field generated by the Earth. Therefore, once the client device calculates the device correction vector, it can subtract the device correction vector from the magnetic field vector measured by the magnetic sensor to determine the magnetic field vector of the Earth.

[0008] To calculate the device correction vector, the client device determines the world magnetic field vector at the device location corresponding to each magnetic field measurement. The world magnetic field vector (labeled herein as VWorld) is a vector representing the Earth's local magnetic field at the corresponding device location and is located in the world reference coordinate system. Based on the world magnetic field vector and the orientation of the client device determined via VIO data, the client device calculates the expected measured magnetic field vector at the device location. The expected magnetic field vector (labeled herein as VExpect) is a vector in the client device's reference coordinate system representing the expected magnetic field measurement at the device location where there is no magnetic field generated by the client device. Thus, the expected magnetic field vector may represent the world magnetic field vector transformed into the client device's reference coordinate system based on the client device's orientation, and if the client device itself did not generate a magnetic field, the expected magnetic field vector would be equal to the observed magnetic field vector measured by the magnetic sensor.

[0009] The client device uses an existing estimated device correction vector to calculate the expected adjusted vector. The expected adjusted vector (labeled herein as VAdjust) is the magnetic field vector in the device's reference coordinate system, representing the expected measured magnetic field vector after correction using the device correction vector. In other words, if the current estimated device correction vector is perfectly accurate, the expected adjusted vector should be equal to the observed magnetic field vector measured by the magnetic sensor. However, if the estimated device correction vector is inaccurate, the client device updates the estimated device correction vector using the difference between the estimated adjusted vector and the observed magnetic field vector. By performing this process for each set of magnetic field measurements, the client device can iteratively update the device correction vector to calculate the final device correction vector to be used for future magnetic field measurements.

[0010] By using VIO to calculate the orientation of the client device, the client device can calibrate its magnetic sensor without requiring the user to perform specific movements on the client device (e.g., Figure 8). Thus, the client device can calibrate its magnetic sensor more regularly than conventional methods. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 shows a networked computing environment according to one embodiment. [Figure 2] Figure 2 shows a series of exemplary magnetic field measurements received from a client device to calibrate a magnetic sensor according to one embodiment. [Figure 3] Figure 3 is a flowchart illustrating a method for calibrating the magnetic sensor of a client device according to one embodiment. [Figure 4] Figure 4 shows an exemplary architecture of a computing device according to one embodiment.

[0012] The drawings and the following description illustrate specific embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structure and method can be adopted without departing from the described principles. Some embodiments are referenced here, examples of which are shown in the accompanying drawings. [Modes for carrying out the invention]

[0013] Various embodiments are described in the context of parallel reality games that include augmented reality content in a virtual world geography parallel to at least a portion of real-world geography, such that the movements and actions of real-world players affect actions in the virtual world, and vice versa. Using the disclosures provided herein, those skilled in the art will understand that the subject matter described is also applicable to other situations where magnetic field measurements are desirable. For example, the methods described herein can be implemented in a location-based application that displays virtual navigation instructions or text labels related to real-world information. In addition, the inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions between the components of the system. For example, systems and methods according to embodiments of this disclosure can be implemented using a single computing device or across multiple computing devices (e.g., connected by a computer network).

[0014] Figure 1 shows a networked computing environment 100 in one or more embodiments. While Figure 1 shows a parallel reality game environment as an example, the described methods / systems may be used in other contexts. In practice, as those skilled in the art will recognize, a networked computing environment may have additional components, fewer components, or variations thereof, in addition to the components shown in Figure 1. Specifically, the described magnetic sensor calibration is applicable to any environment where magnetic sensor calibration is desired. Furthermore, the functionality of each component may be divided in ways different from those described below. For example, a client device may calculate a device correction vector locally or by sending measurements to a remote server (e.g., game server 120).

[0015] The networked computing environment 100 provides interaction with a player in a virtual world having geography parallel to the real world. In particular, geographical regions of the real world may be directly linked to or mapped to corresponding regions of the virtual world. The player may move through the virtual world by moving to various geographical locations in the real world. For example, the player's location in the real world can be tracked and updated to update the player's location in the virtual world. Typically, the player's location in the real world is determined by finding the location of the client device 110 with which the player is interacting with the virtual world and assuming the player is in the same (or nearly the same) location. For example, in various embodiments, the player may interact with a virtual element if the player's location in the real world is within a threshold distance (e.g., 10 meters, 20 meters, etc.) of the real-world location corresponding to the virtual location of the virtual element in the virtual world. For convenience, various embodiments will be described with reference to “player location,” but those skilled in the art will understand that such reference may refer to the location of the player’s client device 110.

[0016] Returning to Figure 1, the networked computing environment 100 uses a client-server architecture, and the game server 120 communicates with client devices 110 via the network 105 to provide players with parallel reality games on the client devices 110. The networked computing environment 100 may also include other external systems, such as sponsor / advertiser systems or business systems. Although only one client device 110 is shown in Figure 1, any number of client devices 110 or other external systems may be connected to the game server 120 via the network 105. Furthermore, the networked computing environment 100 may include different or additional elements, and functionality may be distributed between the client devices 110 and the server 120 in ways different from those described below.

[0017] The client device 110 may be any portable computing device that can be used by a player to interface with the game server 120. For example, the client device 110 may be a wireless device, a personal digital assistant (PDA), a portable gaming device, a mobile phone, a smartphone, a tablet, a navigation system, a handheld GPS system, a wearable computing device, a display having one or more processors, or other such device. In another example, the client device 110 includes a conventional computer system such as a desktop or laptop computer. Furthermore, the client device 110 may be a vehicle having a computing device. In short, the client device 110 may be any computer device or system that can enable a player to interact with the game server 120. As a computing device, the client device 110 may include one or more processors and one or more computer-readable storage media. The computer-readable storage media can store instructions that cause the processors to perform actions. The client device 110 is preferably a portable computing device that can be easily carried with the player, such as a smartphone or tablet, or otherwise can be carried.

[0018] The client device 110 communicates with the game server 120 and provides the game server 120 with sensory data of the physical environment. The client device 110 includes a camera assembly 125 that captures two-dimensional image data of the scene in the physical environment in which the client device 110 resides. In the embodiment shown in Figure 1, each client device 110 includes a magnetic sensor 130 and software components such as a gaming module 135 and a positioning module 140. The client device 110 may also include various other input / output devices for receiving information from and / or providing information to the player. Exemplary input / output devices include a display screen, touchscreen, touchpad, data entry keys, speaker, and microphone suitable for speech recognition. The client device 110 may also include various other sensors for recording data from the client device 110, including, but not limited to, motion sensors, accelerometers, gyroscopes, other inertial measurement units (IMUs), barometers, positioning systems, thermometers, light sensors, depth sensors, etc. The captured images and various sensory data can be used as visual inertial odometry (VIO) data. The client device 110 can transmit the VIO data to the attitude determination module 170 to determine the attitude of the client device 110, for example, the orientation of the client device 110. The client device 110 may further include a network interface for providing communication over the network 105. The network interface may include any suitable components for interfacing with one or more networks, for example, a transmitter, receiver, port, controller, antenna, or other suitable components. The camera assembly 125 captures image data of the scene in the environment where the client device 110 is located. The camera assembly 125 may use various photosensors having various color capture ranges at various capture rates. The camera assembly 125 may include a wide-angle lens or a telephoto lens. The camera assembly 125 may be configured to capture a single image or a video as image data. Furthermore, the orientation of the camera assembly 125 can be parallel to the ground with the camera assembly 125 facing the horizon. The camera assembly 125 captures image data and shares the image data with the computing device on the client device 110. The image data may be accompanied by metadata describing other details of the image data, including sensory data (e.g., temperature, ambient brightness) or capture data (e.g., exposure, warmth, shutter speed, focal length, capture time, etc.). The camera assembly 125 may include one or more cameras capable of capturing image data. In one example, the camera assembly 125 includes one camera and is configured to capture monocular image data. In another example, the camera assembly 125 includes two cameras configured to capture stereoscopic image data. In various other embodiments, the camera assembly 125 comprises multiple cameras, each configured to capture image data.

[0019] The magnetic sensor 130 is a sensor that captures magnetic field measurements. For example, the magnetic sensor 130 may include a compass, a magnetometer, or a magnetic field detector. The magnetic sensor 130 captures magnetic field measurements of the magnetic field generated by the Earth and the magnetic field generated by the electronics within the client device 110 itself. In other words, the measurements captured by the magnetic sensor may reflect the sum of the Earth's magnetic field and the client device's magnetic field. The magnetic sensor may capture magnetic field measurements periodically, or it may capture magnetic field measurements when instructed by the processor of the client device 110. The client device 110 may use the positioning module 140 to determine its position when the magnetic sensor 130 captures magnetic field measurements.

[0020] In one embodiment, the magnetic sensor 130 includes three separate sensors internally aligned separately on the x, y, and z axes of the client device 110. The x, y, and z axes define the body frame of the client device 110, i.e., the device's reference coordinate system. Each of these three sensors measures the strength of the magnetic field at the location of the client device 110. The three measured magnetic field strengths can be combined (e.g., using vector addition) to generate a magnetic field vector (i.e., a local magnetic field vector) at the location of the client device 110. A smartphone is given as an example of the client device 110. The z axis may be perpendicular to the plane of the smartphone, the x axis is to the right along the shorter length, and the y axis is forward along the longer length. When the smartphone is facing upwards and its plane is parallel to the Earth's surface, the z axis of the smartphone points upwards. Assuming that the magnetic sensor 130 measures a large strength along the y axis and a small (or zero) strength along the x axis, the vector addition of the strengths calculates a local magnetic field vector aligned with the y axis of the smartphone. If the local magnetic field vector is the same as the Earth's magnetic field and points to geographical north, the smartphone's y-axis may point to geographical north (i.e., magnetic south), and the smartphone's x-axis may point to geographical east. Therefore, based on the smartphone's device reference coordinate system, the front of the smartphone points to geographical north. When the smartphone is rotated in the horizontal plane, the measured local magnetic field vector rotates according to the device reference coordinate system. As the device reference coordinate system is known, the magnetic sensor 130 can convert the measured local magnetic vector from the device reference coordinate system to the Earth coordinate system (i.e., the world reference coordinate system) to determine the orientation of the client device 110 in physical space.

[0021] The gaming module 135 provides the player with an interface for participating in a parallel reality game. The game server 120 transmits game data to the client device 110 via the network 105 for use by the gaming module 135 of the client device 110, providing a local version of the game to players located away from the game server 120. The game server 120 may include a network interface for providing communication via the network 105. The network interface may include any suitable components for interfacing with one or more networks, such as transmitters, receivers, ports, controllers, antennas, or other suitable components.

[0022] The gaming module 135, run by the client device 110, provides an interface between the player and the parallel reality game. The gaming module 135 can display a virtual world associated with the game (e.g., rendering images of the virtual world) and present a user interface on a display device associated with the client device 110, allowing the user to interact with the virtual world and perform various game objectives. In some embodiments, the gaming module 135 presents real-world image data (e.g., captured by the camera assembly 125) augmented with virtual elements from the parallel reality game. In these embodiments, the gaming module 135 may generate and / or adjust virtual content according to other information received from other components of the client device 110. For example, the gaming module 135 may adjust virtual objects displayed on the user interface according to a depth map of the scene captured in the image data. In other embodiments, the gaming module 135 generates virtual objects for display on a translucent display (e.g., an AR headset, AR glasses, etc.) that allows the user to view the real world. Thus, the virtual objects can be overlaid on the user's view of the real world.

[0023] The gaming module 135 can also control various other outputs to enable a player to interact with the game without the player needing to view a display screen. For example, the gaming module 135 can control various audio, vibration, or other notifications that enable a player to play the game without viewing a display screen. The gaming module 135 can access the game data received from the game server 120 to provide an accurate representation of the game to the player. The gaming module 135 can receive and process player input and provide updates to the game server 120 via the network 105. The gaming module 135 can also generate and / or adjust game content displayed by the client device 110. For example, the gaming module 135 can generate virtual elements based on depth information. In another example, the gaming module 135 can update virtual elements based on the orientation of the camera assembly.

[0024] In one embodiment, the rough positioning of the client device 110 may be performed on the client device 110. The client device 110 includes a positioning module 140 which includes any device or circuitry for monitoring the position of the client device 110. For example, the positioning module 140 can determine the actual or relative position by using a satellite navigation positioning system (e.g., GPS system, Galileo positioning system, Global Navigation Satellite System (GLONASS), Beidou satellite navigation and positioning system), an inertial navigation system, a dead reckoning system, triangulation and / or proximity to a cellular tower or Wi-Fi hotspot based on an IP address, and / or by using other suitable techniques. The positioning module 140 may also use measured magnetic field data from a magnetic sensor 130 when determining the position of the client device 110 in physical space. The positioning module 140 may further include various other sensors which may help to precisely position the client device 110. The positioning module 140 may be used to determine the course position of the client device 110, but the relocalization of the client device 110 (e.g., to determine the fine-grained position and orientation of the device) is performed by the attitude determination module 170 on the game server 120, as described below. For example, the coarse position (e.g., GPS coordinates) identified by the positioning module 140 may be used to identify a three-dimensional (3D) model of the environment in which the client device 110 is located, and the attitude determination module 170 localizes it to the retrieved model using images captured by the camera assembly 125 and the magnetic field measured by the magnetic sensor 130 on the client device 110.

[0025] In an embodiment where the approximate location of the client device is determined on the client side, the positioning module 140 tracks the player's location as the player moves in the real world with the client device 110 and provides player location information to the gaming module 135. The gaming module 135 updates the player's position in the virtual world associated with the game based on the player's actual position in the real world. Thus, the player can interact with the virtual world simply by carrying or transporting the client device 110 in the real world. In particular, the player's position in the virtual world can correspond to the player's position in the real world. The gaming module 135 may provide player location information to the game server 120 via the network 105. In response, the game server 120 may implement various techniques to verify the location of the client device 110 to prevent fraudsters from disguising the location of the client device 110. It should be understood that the location information associated with the player is only used when permission is granted after the player is notified of accessing the player's location information and how the location information is used in the context of the game (e.g., to update the player's position in the virtual world). Further, any location information associated with the player is stored and maintained in a manner that protects the player's privacy.

[0026] The game server 120 can be any computing device and may include one or more processors and one or more computer-readable storage media. The computer-readable storage media may store instructions that cause the processor to execute operations. The game server 120 may include or communicate with a game database 115. The game database 115 stores game data that is provided to the client 110 via the network 105 or used in a parallel reality game that is provided.

[0027] The game data stored in the game database 115 includes: (1) data associated with the virtual world in the parallel reality game (e.g., image data used to render the virtual world on a display device, geographical coordinates of locations within the virtual world, etc.); (2) data associated with the player of the parallel reality game (e.g., player profile including, but not limited to, player information, player experience level, player currency, current player location in the virtual / real world, player energy level, player preferences, team information, faction information, etc.); (3) data associated with game objectives (e.g., data associated with current game objectives, game objective status, past game objectives, future game objectives, desired game objectives, etc.); (4) data associated with virtual elements in the virtual world (e.g., location of virtual elements, location of virtual elements, etc.). (1) Game data may include (2) game objectives associated with virtual elements, corresponding real-world location information of virtual elements, behavior of virtual elements, relationships of virtual elements, etc., (3) data associated with real-world objects, landmarks, and locations linked to virtual world elements (e.g., location of real-world objects / landmarks, description of real-world objects / landmarks, relationships of virtual elements linked to real-world objects, etc.), (4) game status (e.g., current number of players, current status of game objectives, player leaderboard, etc.), (5) data associated with player actions / inputs (e.g., current player location, past player locations, player movements, player inputs, player queries, player communications, etc.), or (6) other data used, related to, or acquired during the implementation of a parallel reality game. Game data stored in the game database 115 may be entered by the system administrator either offline or in real time, or may be entered by data received from users / players of environment 100, such as from client devices 110 via the network 105.

[0028] The game server 120 may be configured to receive requests for game data from client devices 110 (for example, via remote procedure calls (RPCs)) and to respond to those requests via the network 105. For example, the game server 120 may encode game data in one or more data files and provide the data files to the client devices 110. Furthermore, the game server 120 may be configured to receive game data (for example, player position, player actions, player input, etc.) from client devices 110 via the network 105. For example, the client device 110 may be configured to periodically send player input and other updates to the game server 120, which then updates the game data in the game database 115 to reflect any and all changed conditions of the game.

[0029] In the embodiments shown, the game server 120 includes a universal gaming module 145, a commercial gaming module 150, a data acquisition module 155, an event module 160, a posture determination module 170, and a calibration module 180. As described above, the game server 120 may be part of the game server 120 or interact with a game database 115 that can be accessed remotely (for example, the game database 115 may be a distributed database accessed via a network 105). In other embodiments, the game server 120 includes different and / or additional elements. Furthermore, functionality may be distributed among the elements in ways different from those described. For example, the game database 115 may be integrated into the game server 120. Furthermore, although the posture determination module 170 and the calibration module 180 are described as being located on the game server 120, in other embodiments, as described above, posture determination and / or magnetic sensor calibration are performed on the client device 110.

[0030] The Universal Gaming Module 145 hosts a parallel reality game for all players and serves as the authoritative source of the current status of the parallel reality game for all players. As a host, the Universal Gaming Module 145 generates game content to present to players, for example, via each client device 110. When hosting a parallel reality game, the Universal Gaming Module 145 may access the game database 115 to retrieve and / or store game data. The Universal Gaming Module 145 also receives game data from the client devices 110 (e.g., depth information, player input, player position, player actions, landmark information, etc.) and incorporates the received game data into the overall parallel reality game for all players of the parallel reality game. The Universal Gaming Module 145 may also manage the distribution of game data to the client devices 110 via the network 105. The universal gaming module 145 may also manage security aspects of the client devices 110, including, but not limited to, ensuring connections between the client devices 110 and the game server 120, establishing connections between various client devices 110, and verifying the locations of various client devices 110.

[0031] In embodiments where one is included, the commercial gaming module 150 may be separate from the universal gaming module 145 or may be part of the universal gaming module 145. The commercial gaming module 150 may manage the inclusion of various game features within a parallel reality game linked to commercial activities in the real world. For example, the commercial gaming module 150 may receive requests from external systems, such as sponsors / advertisers, corporations, or other entities, via the network 105 (via the network interface), for game features linked to commercial activities in the parallel reality game. The commercial gaming module 150 may then arrange to include these game features in the parallel reality game.

[0032] The game server 120 may further include a data collection module 155. In embodiments where one data collection module 155 is included, it may be separate from or part of the universal gaming module 145. The data collection module 155 can manage to incorporate various game features within a parallel reality game that are linked to data collection activities in the real world. For example, the data collection module 155 may modify game data stored in the game database 115 to include game features linked to data collection activities within the parallel reality game. The data collection module 155 may also analyze data collected by players according to data collection activities and provide data for access by various platforms.

[0033] Event Module 160 manages player access to events within a parallel reality game. While the term “event” is used for convenience, it should be understood that this term does not necessarily refer to a specific event at a specific place or time. Rather, it may refer to any provision of access-controlled game content where one or more access criteria are used to determine whether a player is permitted to access that content. Such content may be part of a larger parallel reality game containing game content with little or no access control, or it may be a standalone access-controlled parallel reality game.

[0034] The attitude determination module 170 may estimate the attitude of the client device 110 using VIO data from the client device 110. The VIO data may include images captured by the camera assembly 125 and other sensor data. In some embodiments, the attitude determination module 170 uses a computer vision algorithm to estimate the attitude of one or more cameras of the camera assembly 125. The attitude determination module 170 is trained to determine the relative attitude of the cameras from one or more images captured by the cameras to one or more existing images of the physical environment around the client device 110. In one embodiment, the attitude determination module 170 uses GPS coordinates (e.g., received from a positioning module 140 on the client device 110) to obtain a 3D model of the environment in which the client device 110 is located. In various implementations, the 3D model may be a point cloud or mesh topology generated from previously captured images of the environment. The attitude determination module 170 then compares one or more images captured by the camera assembly 125 to the obtained 3D model to generate an attitude estimate of the client device 110.

[0035] The calibration module 180 is configured to calibrate the magnetic sensor 130 of the client device 110 based on magnetic field measurements received from the client device 110. To calibrate the magnetic sensor 130, the calibration module 180 estimates a device correction vector for the magnetic sensor 130. The device correction vector is a vector representing the error in the magnetic field measurement caused by the magnetic field introduced by the client device 110. Thus, the magnetic sensor 130 can be calibrated by offsetting its measured magnetic vector by the device correction vector.

[0036] The magnetic field vector at the device location within the world reference coordinate system is referred to as the world magnetic field vector and may be labeled as VWorld. The world magnetic field vector VWorld (sometimes referred to herein as the local magnetic field vector) is a vector representing the local magnetic field at the device location due to the Earth's magnetic field. The expected measured magnetic field vector is the world magnetic field vector transformed into the device's reference coordinate system, labeled as VExpect. Therefore, the expected measured magnetic field vector is a vector in the device's reference coordinate system representing the expected magnetic field measurement at the device location, assuming the magnetic sensor is not affected by the magnetic field introduced by the client device 110. The device correction vector is a vector labeled as VCorrection, also presented in the device's reference coordinate system, representing the error in the magnetic field measurement caused by the magnetic field introduced by the client device 110. The magnetic sensor 130 may observe the local magnetic field vector within the device's reference coordinate system, and the observed magnetic field vector (VObserved) represents the magnetic field actually measured by the magnetic sensor 130 at the device location. The magnetic sensor 130 measures both the local magnetic field generated by the Earth and the magnetic field introduced by the client device 110. Therefore, the expected adjusted vector (VAdjust) is the vector sum of the expected measured magnetic field vector (VExpect) and the device correction vector (VCorrrection), i.e., VAdjust = VExpect + VCorrrection. The expected adjusted vector is a vector in the device's reference coordinate system that represents the expected magnetic field measurement at the device position after adjusting for the magnetic field introduced by the client device 110.

[0037] The calibration module 180 may receive the global magnetic field vector (VWorld) from the positioning module 140 or the data acquisition module 155. Alternatively, the calibration module 180 may calculate the local magnetic field vector based on the device's position and based on a map of the Earth's magnetic field at different locations. The calibration module 180 may receive a map of the Earth's magnetic field from a third-party system.

[0038] The magnetic sensor 130 measures the magnetic field vector in the device's reference coordinate system. To calibrate the magnetic sensor 130, the calibration module 180 can convert the world magnetic field vector (VWorld) to the device's reference coordinate system. The calibration module 180 can perform the conversion from the world reference coordinate system to the device's reference coordinate system by using the attitude estimation of the client device 110. The attitude estimation of the client device 110 includes the orientation of the client device 110 in the world reference coordinate system and therefore provides a rotation matrix (transformation matrix) that converts vectors between the world reference coordinate system and the device's reference coordinate system.

[0039] After the transformation from the world reference coordinate system to the device's reference coordinate system, the local magnetic field vector is the expected measured magnetic field vector, represented as VExpect. The expected measured magnetic field vector is a vector in the device's reference coordinate system that represents the expected magnetic field measurement at the device location where there is no magnetic field introduced by the client device 110.

[0040] As described above, the magnetic field generated by the client device causes a difference between the expected measured magnetic field vector VExpect and the actually observed magnetic field vector VObserved. This difference is corrected by a device correction vector (VCorrection). To find the difference between the expected measured magnetic field vector (VExpect) and the observed magnetic field vector (VObserved), i.e., the device correction vector, the calibration module 180 may receive and analyze a set of magnetic field measurements. Each magnetic field measurement may include the device position of the client device 110 on which the magnetic field measurement is performed, and the observed magnetic field vector (VObserved) measured by the magnetic sensor 130 at the device position. In some embodiments, the calibration module 180 may receive the orientation of the client device 110 when the magnetic field measurements were captured. The calibration module 180 may use bin measurements based on the device position to group the received data, including the magnetic field measurements and VIO data.

[0041] Calibration module 180 can calculate a device correction vector (VCorrection) based on a set of magnetic field measurements. Calibration module 180 can access the estimated device correction vector (VCorrection). The device correction vector (VCorrection) of the magnetic sensor 130, represented in the device's reference coordinate system, is the same for all magnetic field measurements. Calibration module 180 calculates an expected adjusted vector (VAdjust) based on the expected measured magnetic field vector (VExpect) and the estimated device correction vector (VCorrection), where VAdjust = VExpect + VCorrection. The expected adjusted vector (VAdJust) is a vector in the device's reference coordinate system that represents the expected magnetic field measurement at the device position after adjusting the magnetic field introduced by the client device 110. Calibration module 180 can obtain the device correction vector (VCorrection) by comparing the observed magnetic field (VObserved) with the expected adjusted vector (VAdjust). Calibration module 180 optimizes the device correction vector (VCorrection) using an optimization algorithm. The optimization algorithm may use a set of magnetic field measurements to optimize the device correction vector (VCorrection) and obtain a minimum cost value based on a cost function. In some embodiments, the estimated device correction vector associated with the minimum value may be determined as the device correction vector (VCorrection). The calibration module 180 then stores the optimized device correction vector (VCorrection) for calibrating the magnetic sensor 130 of the client device 110.

[0042] Referring back to Figure 1, once the device correction vector is determined, the calibration module 180 can calibrate the magnetic sensor 130 and adjust the measured magnetic field vector using the device correction vector as the magnetic sensor 130 measures the magnetic field vector. In some embodiments, the calibration module 180 can continuously receive additional sets of magnetic field measurements from the client device 110 and use these additional sets of magnetic field measurements to continuously update the estimated device offset magnetic field.

[0043] In some embodiments, the calibration module 180 iterates over a set of magnetic field measurements to calculate a device correction vector. Each magnetic field measurement may be collected within a specific geographical area to ensure that the global magnetic field vector is relatively stable among the magnetic field measurements. The calibration module 180 continuously adjusts the device correction vector estimated based on each magnetic field measurement (e.g., using a process such as gradient descent) until the calibration module 180 has processed all of the magnetic field measurements in the set of magnetic field measurements. The calibration module 180 may use the final estimated device correction vector as the device correction vector for the client device.

[0044] Alternatively, the calibration module 180 may perform a batch process and calculate a device correction vector for a set of magnetic field measurements. As described above, each magnetic field measurement includes the magnetic field vector observed at a given location (VObserved). Furthermore, the calibration module 180 calculates an expected magnetic field vector (VExpect) based on the location where the magnetic field measurement was taken. In some embodiments, each magnetic field measurement includes the location where the measurement was taken, and the calibration module 180 calculates the expected magnetic field vector for each magnetic field measurement. Alternatively, if the set of magnetic field measurements is located within a particular geographical area such that the expected magnetic field vectors for each magnetic field measurement are substantially similar, the calibration module 180 may use the same expected magnetic field vector for each magnetic field measurement.

[0045] As described above, if the device correction vector VCorrection is perfectly calibrated, the observed magnetic field vector should be equal to the sum of the device correction vector and the expected magnetic field vector. In other words, the following relationship should hold:

[0046]

number

[0047] The calibration module 180 can use this expected linear relationship to calculate a device correction vector using a set of magnetic field measurements. For example, the calibration module may calculate the mean of all "VObserved-VExpect" to calculate "VCorrection". Alternatively, since the calibration module 180 has a set of observed magnetic field vectors and has calculated a set of expected magnetic field vectors for each magnetic field measurement, the calibration module 180 can calculate the device correction vector using an equation to calculate the constant term in a linear regression analysis. In other words, the device correction vector may be calculated based on the following equation:

[0048]

number

[0049] In the formula, n represents the number of magnetic field measurements in the set of magnetic field measurements. In some embodiments, the calibration module 180 may update the device correction vector using the iterative process described above.

[0050] Network 105 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. The network may also include a direct connection between the client device 110 and the game server 120. In general, communication between the game server 120 and the client device 110 can be transmitted over a network interface using any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or format (e.g., HTML, XML, JSON), and / or protection scheme (e.g., VPN, Secure HTTP, SSL).

[0051] The technologies discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as actions performed and information transmitted to and from such systems. Those skilled in the art will recognize that the inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions between components. For example, the server processes discussed herein may be implemented using a single server or multiple servers operating in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0052] In addition, in situations where the systems and methods discussed herein access and analyze personal information relating to a user, or use personal information such as location information, the user may be given the opportunity to control whether a program or function collects information, and whether and / or how content is received from the system or other applications. Such information or data will not be collected or used until the user is provided with meaningful notice about what information is collected and how that information is used. No information will be collected or used unless the user consents, and the user can withdraw or change their consent at any time. Thus, the user can control how information about them is collected and used by the application or system. Furthermore, certain information or data may be processed in one or more ways so that personally identifiable information is removed before it is stored or used. For example, a user's identity may be treated in such a way that personally identifiable information cannot be determined for the user.

[0053] According to aspects of this disclosure, a player can interact with a parallel reality game by carrying a client device 110 in the real world. For example, a player may simply access an application associated with a parallel reality game on a smartphone and play the game by moving through the real world with the smartphone. In this respect, the player does not need to continuously view a visual representation of the virtual world on a display screen to play a location-based game. As a result, the user interface may include several non-visual elements that enable the user to interact with the game. For example, the game interface may provide the player with audible notifications when the player is approaching a virtual element or object in the game, or when an important event occurs in the parallel reality game. The player can control these audible notifications using audio controls. Different types of audible notifications may be provided to the user depending on the type of virtual element or event. The frequency or volume of the audible notifications may be increased or decreased depending on the player's proximity to the virtual element or object. Other non-visual notifications and signals, such as vibration notifications or other appropriate notifications or signals, may be provided to the user.

[0054] A person skilled in the art will understand, in light of this disclosure, that numerous game interface configurations and underlying functionalities are apparent. This disclosure is not intended to be limited to any one specific configuration.

[0055] Figure 2 shows a set of exemplary magnetic field measurements received from a client device to calibrate a magnetic sensor, according to several embodiments. For each measurement 200, the corresponding expected magnetic field vector 210, observed magnetic field vector 220, estimated device correction vector 230, and expected adjusted vector 240 are shown. For "Measurement 1" 200a, the estimated device correction vector 230 is at its default or starting value. Therefore, the expected adjusted vector 240 calculated based on that estimated device correction vector 230 may have a relatively large error 250. The estimated device correction vector 230 is adjusted based on the error 250 of "Measurement 2" 200b. Because the estimated device correction vector 230 is close to the true value of the device correction vector, the error 260 between the expected adjusted vector 240 and the observed magnetic field vector 220 is smaller than the first error 250. The estimated device correction vector 230 is adjusted again based on the second error 260 so that for "measurement 3" 200c there is little or no difference between the expected adjusted vector 240 and the observed magnetic field vector 220. In some embodiments, if the difference between the expected adjusted vector 240 and the expected adjusted vector 240 is less than a threshold, the associated estimated device correction vector 230 is stored as the device correction vector for the client device.

[0056] Figure 3 is a flowchart illustrating a method 300 for calibrating a magnetic sensor 130 of a client device 110 according to one embodiment. The steps in Figure 3 are shown from the perspective of a game server 120 performing method 300. However, some or all of the steps may be performed by other entities and / or components. In addition, in some embodiments, the steps may be performed in parallel, in a different order, or different steps may be performed.

[0057] In the embodiments shown, method 300 begins with a game server 120 receiving a set of magnetic field measurements from a client device 110 305. Each magnetic field measurement may include the device location of the client device, the observed magnetic field vector measured at the device location by the client device's magnetic sensor, and the orientation of the client device when the magnetic field measurement was captured. In some implementations, the game server 120 may receive both magnetic field measurements and VIO data from the client device 110 305. A calibration module 180 within the game server 120 may group the received magnetic field measurements and VIO data using bin measurements based on the device location.

[0058] Calibration module 180 calculates a device correction vector for the client device 310 based on a set of magnetic field measurements. The device correction vector is a vector representing the error in the magnetic field measurements caused by the magnetic field introduced by the client device 110.

[0059] For each magnetic field measurement, the calibration module 180 determines the world magnetic field vector at the device location of the magnetic field measurement 315. The world magnetic field vector is a vector representing the local magnetic field at the device location in the world reference coordinate system. The world magnetic field vector may refer to the magnetic field surrounding the client device 110 at the device location. For example, the world magnetic field vector may be the Earth magnetic field vector at the device location. The local magnetic field vector (VWorld) is expressed in the world reference coordinate system.

[0060] The calibration module 180 calculates the expected measured magnetic field vector at the device location based on the orientation of the client device 110 associated with the world magnetic field vector and the magnetic field measurement. The expected measured magnetic field vector is a vector in the device's reference coordinate system that represents the expected magnetic field measurement at the device location where there is no magnetic field introduced by the client device 110.

[0061] In some implementations, the calibration module 180 uses the attitude estimation of the client device 110 to convert the local magnetic field vector (VWorld) from the world reference coordinate system to the device's reference coordinate system. The attitude estimation of the client device 110 includes the orientation of the client device 110 in the world reference coordinate system and therefore provides a rotation matrix that converts vectors between the world reference coordinate system and the device's reference coordinate system.

[0062] The calibration module 180 accesses the estimated device correction vector for the client device 110.

[0063] Calibration module 180 calculates the expected adjusted vector for the client device 330 based on the expected measured magnetic field vector and the estimated device correction vector. The expected adjusted vector is a vector in the device's reference coordinate system that represents the expected magnetic field measurement at the device position after adjusting the magnetic field introduced by the client device 110.

[0064] Next, the calibration module 180 compares the observed magnetic field vector associated with the magnetic field measurement with the expected adjusted vector 335.

[0065] Calibration module 180 calculates a device correction vector 340 based on comparison. The readings from magnetic sensor 130 include errors due to the magnetic field introduced by client device 110, and calibration module 180 may optimize the expected adjusted vector using an optimization algorithm. The optimization algorithm uses a set of magnetic field measurements to optimize the expected adjusted vector and obtain a minimum value. In some embodiments, the minimum value may be determined as the expected adjusted vector.

[0066] The calibration module 180 stores a device correction vector 345 for calibrating the magnetic sensor 130 of the client device 110.

[0067] The calibration module 180 can calibrate the magnetic sensor 130 and offset the measured magnetic field vector using a device correction vector. For example, the calibration module 180 can calculate a corrected magnetic field vector by summing the measured magnetic field vector with the device correction vector. The game server 120 can use the corrected magnetic field vector to determine the orientation of the client device 110. In some embodiments, the calibration module 180 can continuously receive additional sets of magnetic field measurements from the client device 110 and continuously update the estimated device offset magnetic field using these additional sets of measurements. Exemplary alternative embodiments In the above embodiment, the game server 120 uses a world magnetic field vector that is accurate enough to calculate the device correction vector for the client device 110. For example, the game server 120 may use the position of the client device when the magnetic field measurement was captured to determine the world magnetic field vector of the client device at the time the magnetic field measurement was captured. As described above, the game server 120 may use these world magnetic field vectors to calculate a device correction vector for calibrating the magnetometer of the client device.

[0068] However, in some embodiments, the game server 120 may not be able to estimate the world magnetic field vector with sufficient accuracy to effectively calculate the device correction vector. In these embodiments, the game server 120 may rely on the assumption that each set of measurements used to calculate the device correction vector has approximately the same world magnetic field vector. For example, the game server 120 may have access to a set of magnetic field measurements all captured within a specific geographical area (e.g., within 1-2 meters of several geographical locations). Since the magnetic field measurements were captured in relatively close proximity to each other, the world magnetic field vector is likely to be relatively constant among these magnetic field measurements. This assumption can be used to calculate the device correction vector without first receiving or determining the world magnetic field vector.

[0069] The game server 120 may use a device transformation matrix to calculate a device correction vector. The device transformation matrix is ​​a transformation matrix that transforms the world magnetic field vector into an expected magnetic field vector; that is, the device transformation matrix transforms the world magnetic field vector from the world reference coordinate system into a vector in the device's reference coordinate system. The game server 120 generates a device transformation matrix for each measurement in a set of magnetic field measurements, and the device transformation matrix transforms the world magnetic field vector into the device's reference coordinate system based on the orientation of the client device when each magnetic field measurement is taken. The game server 120 generates a device transformation matrix for magnetic field measurements based on VIO data captured by the client device 110 when the magnetic field measurement is captured. For example, the game server 120 may use the VIO data to determine the orientation of the client device 110 when the magnetic field measurement is captured, and generate a device transformation matrix for magnetic field measurements based on the determined orientation of the client device. In some embodiments, the transformation matrix applies a rotation to the world magnetic field vector to transform it into an expected magnetic field vector.

[0070] To calculate the device correction vector, the game server 120 minimizes an objective function using a device transformation matrix from a set of magnetic field measurements and the observed magnetic field vector. The objective function takes the world magnetic field vector and the device correction vector as inputs and calculates a loss score based on the input and the set of magnetic field measurements. Thus, by minimizing its objective function, the game server 120 calculates estimates of both the world magnetic field vector and the device correction vector. In some embodiments, the game server 120 minimizes an objective function f, where f is:

[0071]

number

[0072] It is defined as follows.

[0073] In the equation, Mi is the device transformation matrix for the magnetic field measurement i. The game server 120 can minimize the objective function by applying an optimization algorithm to the objective function.

[0074] The above description primarily describes a game server 120 that performs the step of calculating the device correction vector for the client device 110, although some or all of these steps may be performed by the client device 110 itself. Exemplary computing system Figure 4 is an exemplary architecture of a computing device according to one or more embodiments. Figure 4 shows a high-level block diagram illustrating the physical components of a computer used as some or all of the entities described herein, although according to one embodiment, the computer may have additional, fewer, or variations of the components provided in Figure 4. Figure 4 shows computer 400, but the figure is intended more as a functional description of various features that may be present in a computer system than as a schematic diagram of the structure of the implementation described herein. In practice, as will be recognized by those skilled in the art, items shown individually can be combined or some items can be separated.

[0075] Figure 4 shows at least one processor 402 coupled to a chipset 404. The chipset 404 is also coupled to memory 406, storage device 408, keyboard 410, graphics adapter 412, pointing device 414, and network adapter 416. A display 418 is coupled to the graphics adapter 412. In one embodiment, the functionality of the chipset 404 is provided by a memory controller hub 420 and an I / O hub 422. In another embodiment, the memory 406 is directly coupled to the processor 402 instead of the chipset 404. In some embodiments, the computer 400 includes one or more communication buses for interconnecting these components. The one or more communication buses optionally include circuits (also called chipsets) that interconnect and control communication between system components.

[0076] The storage device 408 is any non-temporary computer-readable storage medium such as a hard drive, compact disc read-only memory (CD-ROM), DVD, or solid-state memory device or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, magnetic disk storage device, optical disk storage device, flash memory device, or other non-volatile solid-state storage device. Such a storage device 408 may also be referred to as persistent memory. The pointing device 414 may be a mouse, trackball, or other type of pointing device, used in conjunction with the keyboard 410 to input data to the computer 400. The graphics adapter 412 displays images and other information on the display device 418. The network adapter 416 connects the computer 400 to a local or wide area network.

[0077] Memory 406 holds instructions and data used by the processor 402. Memory 406 can be non-persistent memory, and examples include high-speed random-access memory such as DRAM, SRAM, DDR RAM, ROM, EEPROM, and flash memory.

[0078] As is known in the art, the computer 400 may have components different from those shown in Figure 4 and / or other components. In addition, the computer 400 may lack certain exemplary components. In one embodiment, the computer 400, which functions as a server, may lack a keyboard 410, a pointing device 414, a graphics adapter 412, and / or a display 418. Furthermore, the storage device 408 may be local and / or remote from the computer 400 (e.g., implemented within a storage network (SAN)).

[0079] As is known in the art, the computer 400 is adapted to run a computer program module for providing the functions described herein. As used herein, the term “module” refers to the computer program logic used to provide the specified functions. Thus, a module can be implemented in hardware, firmware, and / or software. In one embodiment, the program module is stored in storage device 408, loaded into memory 406, and executed by processor 402. Additional considerations Some of the above sections describe embodiments in terms of algorithmic processes or operations. These descriptions and representations of algorithms are commonly used by those skilled in the art of data processing technology to effectively communicate the nature of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, they are understood to be implemented by computer programs, including instructions for execution by a processor or equivalent electrical circuit, microcode, etc. Furthermore, it has proven that it is sometimes convenient to refer to these arrangements of functional operations as modules without losing universality.

[0080] Where used herein, a reference to “one embodiment” means that certain elements, features, structures, or characteristics described in relation to that embodiment are included in at least one embodiment. The phrase “in one embodiment” appearing in various parts of this specification does not necessarily refer to the same embodiment.

[0081] Some embodiments may be described using the expressions “joined” and “connected” along with their derivatives. It should be understood that these terms are not intended to be synonymous with one another. For example, one embodiment may be described using the term “connected,” indicating that two or more elements are directly physically or electrically connected to one another. In another example, one embodiment may be described using the term “joined,” indicating that two or more elements are directly physically or electrically connected. However, the term “joined” can also mean that two or more elements are not in direct contact with one another but still cooperate or interact with one another. Embodiments are not limited to this context.

[0082] Where used herein, “comprises,” “comprising,” “includes,” “including,” “has,” “having,” or any other variation thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to those elements alone, and may include other elements not expressly described in or inherent to such process, method, article, or apparatus. Furthermore, unless expressly stated otherwise, “or” means an inclusive “or” and not an exclusive “or.” For example, condition A or B is satisfied by one of the following: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).

[0083] Furthermore, the use of “a” or “an” is employed to describe elements and components of embodiments herein. This is done solely for convenience and to give a general meaning to the disclosure. This description should be read as including one or at least one, and singular forms include plural forms unless it is clear that this is not the case. When a value is described as “about” or “substantially” (or their derivatives), such a value should be interpreted as exactly + / - 10% unless another meaning is clear from the context. From the example, “about 10” should be understood to mean “within the range of 9 to 11.”

[0084] A person skilled in the art will understand, upon reading this disclosure, further additional alternative structural and functional designs for systems and processes for determining the orientation of a client device. Therefore, although specific embodiments and applications have been illustrated and described, it should be understood that the subject matter described is not limited to the exact structures and components disclosed herein, and various modifications, changes, and variations that will be apparent to a person skilled in the art may be made in the arrangement, operation, and details of the disclosed methods and apparatus. The scope of protection should be limited only by the following claims.

Claims

1. A computer implementation method, The step of accessing a set of magnetic field measurements, wherein each magnetic field measurement includes an observed magnetic field vector measured by a magnetic sensor of a client device, The steps of generating a device transformation matrix for each magnetic field measurement in the set of magnetic field measurements, based on visual inertial odometry (VIO) data captured by the client device, wherein the device transformation matrix is ​​a transformation matrix for transforming the world magnetic field vector from the world reference coordinate system to the client device's reference coordinate system, A step of calculating a device correction vector for the client device, wherein the device correction vector is a vector representing the error of the magnetic field vector measured by the magnetic sensor of the client device, and the step of calculating the device correction vector for the client device includes a step of minimizing an objective function over the set of magnetic field vectors, wherein the objective function calculates a loss score of the magnetic field measurement based on the device transformation matrix corresponding to the magnetic field measurement and the observed magnetic field vector of the magnetic field measurement, The steps include storing the device correction vector for calibrating the magnetic sensor of the client device, Methods that include...

2. The method according to claim 1, wherein the set of magnetic field measurements consists of magnetic field measurements captured within a geographical area.

3. The method according to claim 1, wherein the step of accessing the set of magnetic field measurements includes the step of an online server receiving the set of magnetic field measurements from the client device.

4. The method according to claim 3, wherein the online server is a game server.

5. The step of generating the device transformation matrix is: The steps include determining the orientation of the client device in the world reference coordinate system for the magnetic field measurement based on the VIO data, The method according to claim 1, comprising the step of generating the device transformation matrix based on the determined orientation of the client device.

6. The method according to claim 1, wherein the VIO data includes images and sensor data captured by the client device.

7. The method according to claim 1, wherein the step of minimizing the objective function includes the step of applying an optimization algorithm to the objective function.

8. The step of minimizing the aforementioned objective function is: The observed magnetic field vector, The device correction vector and, The product of a device transformation matrix and a world magnetic field vector, wherein the world magnetic field vector represents the magnetic field from the world within the world reference coordinate system, and the product of the device transformation matrix and the world magnetic field vector, The sum and The method according to claim 1, comprising the step of minimizing the sum of the differences between.

9. The steps include receiving the magnetic field vector observed from the magnetic sensor of the client device, The steps include: calculating the sum of the observed magnetic field vector and the stored device correction vector; The method according to claim 1, further comprising the step of determining the orientation of the client device based on the calculated total.

10. When executed by the processor, the processor will Accessing a set of magnetic field measurements, each of which includes an observed magnetic field vector measured by a magnetic sensor of a client device; The process involves generating a device transformation matrix for each magnetic field measurement in the set of magnetic field measurements based on visual inertial odometry (VIO) data captured by the client device, wherein the device transformation matrix is ​​a transformation matrix for transforming the world magnetic field vector from the world reference coordinate system to the client device's reference coordinate system. The calculation of a device correction vector for the client device, wherein the device correction vector is a vector representing the error of the magnetic field vector measured by the magnetic sensor of the client device, and the calculation of the device correction vector for the client device includes minimizing an objective function over the set of magnetic field vectors, wherein the objective function calculates a loss score of the magnetic field measurement based on the device transformation matrix corresponding to the magnetic field measurement and the observed magnetic field vector of the magnetic field measurement, and the calculation The device correction vector for calibrating the magnetic sensor of the client device is stored, A non-temporary, computer-readable medium that stores instructions for performing actions that include [specific actions].

11. The set of magnetic field measurements comprises magnetic field measurements captured within a geographical area, in the non-temporary computer-readable medium according to claim 10.

12. Accessing the set of magnetic field measurements includes an online server receiving the set of magnetic field measurements from the client device, the non-temporary computer-readable medium according to claim 10.

13. The non-temporary computer-readable medium according to claim 12, wherein the online server is a game server.

14. Generating the aforementioned device transformation matrix means Based on the VIO data, the orientation of the client device in the world reference coordinate system for the magnetic field measurement is determined, A non-temporary computer-readable medium according to claim 10, comprising generating the device transformation matrix based on the determined orientation of the client device.

15. The non-temporary computer-readable medium according to claim 10, wherein the VIO data includes images and sensor data captured by the client device.

16. The non-temporary computer-readable medium according to claim 10, wherein minimizing the objective function includes applying an optimization algorithm to the objective function.

17. Minimizing the aforementioned objective function means The observed magnetic field vector, The device correction vector and, The product of a device transformation matrix and a world magnetic field vector, wherein the world magnetic field vector represents the magnetic field from the world within the world reference coordinate system, and the product of the device transformation matrix and the world magnetic field vector, The sum and A non-temporary computer-readable medium according to claim 10, comprising minimizing the sum of the differences between the two.

18. When executed by the processor, the processor will The client device receives the magnetic field vector observed from the magnetic sensor, The sum of the observed magnetic field vector and the stored device correction vector is calculated, The orientation of the client device is determined based on the calculated total, A non-temporary computer-readable medium according to claim 10, further storing instructions for performing an operation including the above.

19. A non-temporary computer-readable medium for storing device correction vectors, wherein the device correction vectors are: Accessing a set of magnetic field measurements, each of which includes an observed magnetic field vector measured by a magnetic sensor of a client device; The process involves generating a device transformation matrix for each magnetic field measurement in the set of magnetic field measurements based on visual inertial odometry (VIO) data captured by the client device, wherein the device transformation matrix is ​​a transformation matrix for transforming the world magnetic field vector from the world reference coordinate system to the client device's reference coordinate system. The calculation of the device correction vector for the client device, wherein the device correction vector is a vector representing the error of the magnetic field vector measured by the magnetic sensor of the client device, and the calculation of the device correction vector for the client device includes minimizing an objective function over the set of magnetic field vectors, wherein the objective function calculates a loss score of the magnetic field measurement based on the device transformation matrix corresponding to the magnetic field measurement and the observed magnetic field vector of the magnetic field measurement, and the calculation The device correction vector for calibrating the magnetic sensor of the client device is stored, A non-temporary, computer-readable medium generated by a process that includes [a specific type of processing].

20. Minimizing the aforementioned objective function means The observed magnetic field vector, The device correction vector and, The product of a device transformation matrix and a world magnetic field vector, wherein the world magnetic field vector represents the magnetic field from the world within the world reference coordinate system, and the product of the device transformation matrix and the world magnetic field vector, The sum and A non-temporary computer-readable medium according to claim 19, comprising minimizing the sum of the differences between the two.

21. A computer implementation method, The receiving step of receiving a set of magnetic field measurements, each magnetic field measurement including the device position of a client device, an observed magnetic field vector measured at the device position by a magnetic sensor of the client device, and the orientation of the client device when the magnetic field measurement was captured. A step of calculating a device correction vector for the client device based on the set of magnetic field measurements, wherein the device correction vector is a vector representing the magnetic field vector error measured by the magnetic sensor of the client device, and the step of calculating the device correction vector is performed for each magnetic field measurement in the set of magnetic field measurements, A step of determining the world magnetic field vector at the device location of the magnetic field measurement, wherein the world magnetic field vector is a vector representing the local magnetic field at the device location, and the world magnetic field vector is located in the world reference coordinate system, and the step of determining A step of calculating an expected measured magnetic field vector at the device location based on the world magnetic field vector and the orientation of the client device associated with the magnetic field measurement, wherein the expected measured magnetic field vector is a vector in the device's reference coordinate system representing the expected magnetic field measurement at the device location of the client device without magnetic field error, The steps include: accessing the estimated device correction vector of the client device; A step of calculating a predicted adjusted vector for the client device based on the predicted measured magnetic field vector and the estimated device correction vector, wherein the predicted adjusted vector is a vector in the device's reference coordinate system that represents the predicted magnetic field measurement at the device position after adjusting for the magnetic field vector error of the client device, The steps include comparing the observed magnetic field vector associated with the measured magnetic field value with the expected adjusted vector, A step of calculating the device correction vector based on the above comparison, The calculation step includes, The steps include storing the device correction vector for calibrating the magnetic sensor of the client device, Methods that include...

22. The method according to claim 21, wherein the step of calculating the expected measured magnetic field vector includes the step of transforming the global magnetic field vector from the world reference coordinate system to the device's reference coordinate system using an attitude estimate of the client device.

23. The method of claim 22, wherein the attitude estimate of the client device includes the orientation of the client device in the world reference coordinate system.

24. The method according to claim 22, wherein the pose estimate of the client device is determined using visual inertial odometry (VIO) data received from the client device.

25. The method according to claim 24, wherein the VIO data includes captured images and sensor data from the client device associated with a location-based application.

26. The method of claim 24, wherein the step of calculating the expected measured magnetic field vector includes the step of grouping the received VIO data and the magnetic field measurements from the client device using bin measurements based on the device position.

27. The step of calculating the device correction vector is: The steps include optimizing the device correction vector, The method according to claim 21, comprising the step of obtaining a minimum value corresponding to the estimated device correction vector.

28. The step of receiving a second magnetic field measurement from the client device, wherein the second magnetic field measurement includes a second observed magnetic field vector measured by the magnetic sensor of the client device, The method according to claim 21, further comprising the step of estimating a local magnetic field vector by adjusting the second observed magnetic field vector using the device correction vector of the client device.

29. The steps include receiving an additional set of magnetic field measurement values ​​from the client device, The method according to claim 21, further comprising the step of updating the device correction vector of the client device based on an additional set of magnetic field measurements.

30. A non-temporary computer-readable storage medium containing instructions executable by a processor, wherein the instructions are: Receiving a set of magnetic field measurements, each magnetic field measurement including the device position of a client device, the observed magnetic field vector measured at the device position by the magnetic sensor of the client device, and the orientation of the client device when the magnetic field measurement was captured, The calculation of a device correction vector for the client device based on the set of magnetic field measurements, wherein the device correction vector is a vector representing the magnetic field vector error measured by the magnetic sensor caused by the client device, and the calculation of the device correction vector is performed for each magnetic field measurement in the set of magnetic field measurements, The determination of the world magnetic field vector at the device position of the magnetic field measurement, wherein the world magnetic field vector is a vector representing the local magnetic field at the device position, and the world magnetic field vector is located within the world reference coordinate system, and the determination of the world magnetic field vector is... The calculation involves calculating an expected measured magnetic field vector at the device location based on the world magnetic field vector and the orientation of the client device associated with the magnetic field measurement, wherein the expected measured magnetic field vector is a vector in the device's reference coordinate system representing the expected magnetic field measurement at the device location of the client device without magnetic field error. Accessing the estimated device correction vector of the client device, The calculation of a predicted adjusted vector for the client device based on the predicted measured magnetic field vector and the estimated device correction vector, wherein the predicted adjusted vector is a vector in the device's reference coordinate system that represents the predicted magnetic field measurement at the device position after adjusting for the magnetic field error of the client device, Comparing the observed magnetic field vector associated with the measured magnetic field value with the expected adjusted vector, Based on the above comparison, the correction vector of the device is calculated, The calculations mentioned above, The device correction vector for calibrating the magnetic sensor of the client device is stored, A non-temporary, computer-readable storage medium that is executable to perform operations including [specific actions].

31. The non-temporary computer-readable storage medium according to claim 30, wherein calculating the expected measured magnetic field vector includes converting the world magnetic field vector from the world reference coordinate system to the device's reference coordinate system using the attitude estimate of the client device.

32. The non-temporary computer-readable storage medium according to claim 31, wherein the attitude estimate of the client device includes the orientation of the client device in the world reference coordinate system.

33. The non-temporary computer-readable storage medium according to claim 31, wherein the attitude estimate of the client device is determined using visual inertial odometry (VIO) data received from the client device.

34. The non-temporary computer-readable storage medium according to claim 33, wherein the VIO data includes captured images and sensor data from the client device associated with a location-based application.

35. The non-temporary computer-readable storage medium according to claim 33, wherein calculating the expected measured magnetic field vector includes grouping the received VIO data and the magnetic field measurements from the client device using bin measurements based on the device position.

36. Calculating the device correction vector is, Optimizing the correction vector of the aforementioned device, A non-temporary computer-readable storage medium according to claim 30, comprising obtaining a minimum value as the estimated device correction vector.

37. The aforementioned operation is, Receiving a second magnetic field measurement from the client device, wherein the second magnetic field measurement includes a second observed magnetic field vector measured by the magnetic sensor of the client device, The non-temporary computer-readable storage medium according to claim 30, further comprising estimating a local magnetic field vector by adjusting the second observed magnetic field vector using the device correction vector of the client device.

38. The aforementioned operation is, Receiving an additional set of magnetic field measurement values ​​from the aforementioned client device, A non-temporary computer-readable storage medium according to claim 30, further comprising updating the device correction vector of the client device based on an additional set of magnetic field measurements.

39. At least one processor, A computer system comprising a non-temporary computer-readable medium containing stored instructions, wherein when an instruction is executed by the at least one processor, the at least one processor receives Receiving a set of magnetic field measurements, each magnetic field measurement including the device position of a client device, the observed magnetic field vector measured at the device position by the magnetic sensor of the client device, and the orientation of the client device when the magnetic field measurement was captured, The calculation of a device correction vector for the client device based on the set of magnetic field measurements, wherein the device correction vector is a vector representing the magnetic field vector error measured by the magnetic sensor caused by the client device, and the calculation of the device correction vector is performed for each magnetic field measurement in the set of magnetic field measurements, The determination of the world magnetic field vector at the device position of the magnetic field measurement, wherein the world magnetic field vector is a vector representing the local magnetic field at the device position, and the world magnetic field vector is located within the world reference coordinate system, and the determination of the world magnetic field vector is... The calculation involves calculating an expected measured magnetic field vector at the device location based on the world magnetic field vector and the orientation of the client device associated with the magnetic field measurement, wherein the expected measured magnetic field vector is a vector in the device's reference coordinate system representing the expected magnetic field measurement at the device location of the client device without magnetic field error. Accessing the estimated device correction vector of the client device, The calculation of a predicted adjusted vector for the client device based on the predicted measured magnetic field vector and the estimated device correction vector, wherein the predicted adjusted vector is a vector in the device's reference coordinate system that represents the predicted magnetic field measurement at the device position after adjusting for the magnetic field error of the client device, Comparing the observed magnetic field vector associated with the measured magnetic field value with the expected adjusted vector, Based on the above comparison, the correction vector of the device is calculated, The calculations mentioned above, The device correction vector for calibrating the magnetic sensor of the client device is stored, A computer system that executes a command.

40. When the instruction is executed by the at least one processor, the at least one processor further... Receiving a second magnetic field measurement from the client device, wherein the second magnetic field measurement includes a second observed magnetic field vector measured by the magnetic sensor of the client device, The local magnetic field vector is estimated by adjusting the second observed magnetic field vector using the device correction vector of the client device, A computer system according to claim 39, which causes to perform the following.

Images