Smooth Object Correction for Augmented Reality Devices

Abstract

An augmented reality (「AR」) device applies a smoothing correction method to correct the position of a virtual object presented to a user. The AR device may apply an angular threshold to determine whether the virtual object can be moved from its original position to a target position. The angular threshold is the maximum angle by which the line from the AR device to the virtual object can change within a time step. Similarly, the AR device may apply a motion threshold, which may be the maximum value of the distance by which the position of the virtual object can be corrected based on the movement of the virtual object. Further, the AR device may apply a pixel threshold to the correction of the position of the virtual object. The pixel threshold is the maximum value of the distance by which the pixel projection of the virtual object can change based on the change in the position of the virtual object.

Description

Technical Field

[0001] The present disclosure relates to an augmented reality device.

Background Art

[0002] An augmented reality ("AR") device displays virtual objects to a user on a display that superimposes the virtual objects on the real world. For example, an AR device may display image data showing the real-world environment around the user, along with virtual objects placed on a real-world table, to the user. Since the virtual objects need to be integrated into the real world in order for the user to "feel" that the virtual objects are real, augmented reality is typically a more complex technical task than full virtual reality (where the entire image presented to the user is virtual). For example, an AR device may need to acquire sensor data of the environment around the user and a method for displaying the virtual objects so that they appear as if they are actual objects in the real world.

[0003] There are many challenges in displaying virtual objects in AR. For example, the virtual objects may appear to be in a location other than where they are supposed to be displayed. For example, an AR device may have inaccurate or incorrect position data that describes its position relative to an object within the environment of the AR device, or orientation data that describes the orientation of the AR device within that environment. This can cause the AR device to display the virtual objects inappropriately. Similarly, the AR device may receive inaccurate or incorrect sensor data that describes the environment of the AR device, and thus may display the virtual objects based on an inaccurate estimate of where other objects are within the environment.

[0004] The AR device can update how virtual objects are displayed based on the updated data received by the AR device. However, if the position of the virtual object is updated to be at a significantly different position, the user may notice that the position of the virtual object has been updated. For example, conventional methods of updating the position of a virtual object may cause the virtual object to suddenly appear at the correct position or move quickly or unnaturally to their correct positions. These methods often alert the user when the position of the virtual object is updated, and thus, conventional correction methods cause the user to lose immersion within the experience of the AR device.

Summary of the Invention

[0005] The AR device disclosed herein can improve conventional object position correction methods by applying an angular threshold when correcting the target position of a virtual object. The target position of the virtual object is the position within the real world that is targeted for the virtual object to be displayed when the AR device is presented to the user. However, as described above, when the AR device presents the virtual object to the user, the actual appearance position of the virtual object may be different from the target position. The AR device can determine whether the position of the virtual object is different from the target position and can correct the position of the virtual object.

[0006] The AR device can determine whether the target position is within the angular threshold of the original position of the virtual object. The angular threshold is the maximum value of the angle from the line of sight of the AR device to the position of the virtual object and the angle from the line of sight of the AR device to the updated position of the virtual object. If the target position is within the angular threshold, the position of the virtual object is updated to the target position. If the target position is outside the angular threshold, the position of the virtual object is updated to the position closest to the target position while not exceeding the angular threshold.

[0007] Furthermore, the AR device may apply a motion threshold when correcting the position of the virtual object. The virtual object may be displayed to the user such that the virtual object appears to be moving relative to the AR device. The motion threshold is the maximum distance by which the position of the virtual object can be adjusted based on the movement of the virtual object. For example, the motion threshold may enable the position of the virtual object to be adjusted up to a certain percentage of the distance the virtual object moves in a time step. If the target position of the virtual object is within the motion threshold of the original position of the virtual object, the position of the virtual object is updated to the target position. If the target position is outside the motion threshold, the position of the virtual object may be updated to the position closest to the target position while not exceeding the motion threshold. Also, when moving the virtual object, the AR device may move the virtual object to the next step of its movement based on the updated position of the virtual object rather than the original position of the virtual object.

[0008] Furthermore, the AR device may apply a pixel threshold when correcting the position of the virtual object. The pixel threshold limits the distance by which the position of the virtual object can be corrected and sets a maximum value for the distance (in pixel units) by which the pixel projection of the virtual object can be adjusted. For example, the AR device may compare the 2D pixel projection of the virtual object at the position of the virtual object with the 2D pixel projection of the virtual object at the target position. If the distance between the pixel projections does not exceed the pixel threshold, the AR device may update the position of the virtual object to the target position. If the distance between the pixel projections exceeds the pixel threshold, the AR device may update the position of the virtual object to the position closest to the target position that does not exceed the pixel threshold.

[0009] By using an angle threshold, the AR device can enable correction of the virtual object's position in the depth direction rather than the lateral direction. Thus, the AR device can enable corrections that may be less noticeable to the user, which brings the virtual object as close as possible to the correct position while maintaining the desired level of immersion or believability for the user. Similarly, by limiting how much the position of the virtual object can be adjusted based on a motion threshold, the user can ignore any adjustment of the virtual object's position as part of the intended movement of the virtual object rather than as a correction of the virtual object's position. Thus, the AR device can more easily correct the position of the virtual object when the virtual object is moving faster because the adjustment is less noticeable to the user. Further, by limiting the adjustment to the position of the virtual object based on a pixel threshold, the AR device reduces how much the pixels corresponding to the virtual object change, and thus, more directly, makes the correction to the position of the virtual object less noticeable to the user.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0011] FIG. 1 shows a system environment of an augmented reality (AR) device according to some embodiments. The system environment illustrated in FIG. 1 includes an AR device 100, a network 110, and one or more online servers 120. Alternative embodiments may include more, fewer, or different components than those shown in FIG. 1, and the functions of the components may be divided among different components than those described below. For example, some or all of the functions of the AR device 100 may be performed by one or more online servers 120, and vice versa.

[0012] The AR device 100 can be any computing device that can be used by a user to interact with AR content presented by the AR device 100. For example, the AR device 100 can be a wireless device, a personal digital assistant (PDA), a portable gaming device, a mobile phone, a smartphone, a tablet, a navigation system, a handheld GPS system, a wearable computing device, a display having one or more processors, or other such devices. In some embodiments, the AR device 100 includes a conventional computer system such as a desktop computer or a laptop computer (e.g., a desktop computer wirelessly connected to an AR headset). Further, the AR device 100 can be a vehicle having a computing device. The AR device 100 can include one or more processors and one or more computer-readable storage media. The computer-readable storage media can store instructions that cause the processor to execute operations. The AR device 100 may further include various other input / output devices for receiving information from or providing information to a player. Exemplary input / output devices include a display screen, a touch screen, a touch pad, a data entry key, a speaker, and a microphone suitable for voice recognition. The AR device 100 can further include a network interface for providing communication via the network 110. The network interface can include any suitable components for interfacing with one or more networks, including, for example, a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0013] Figure 1 shows an exemplary system architecture of an AR device 100 according to some embodiments. The AR device 100 may include a camera assembly 130, one or more sensors 140, an object position module 150, a content display module 160, a smooth correction module 170, and a data store 180. In alternative embodiments, the AR device 100 may include components different from those shown in Figure 1, and the functions of each component may be divided in a different way from the functions in Figure 1.

[0014] The camera assembly 130 acquires image data of the environment around the AR device 100. The camera assembly 130 may include an optical sensor and may acquire image data of the environment. The camera assembly 130 may include a wide-angle lens or a telephoto lens. The camera assembly 130 may be configured to acquire a single image or video as image data. The camera assembly 130 may include metadata that describes other details of the image data, including sensory data (e.g., temperature, ambient brightness) or acquisition data (e.g., exposure, warmth, shutter speed, focal length, capture time, etc.). The camera assembly 130 may include one or more cameras capable of acquiring image data. In one example, the camera assembly 130 includes one camera and is configured to acquire monocular image data. In another example, the camera assembly 130 includes two cameras and is configured to acquire stereo image data. In various other embodiments, the camera assembly 130 includes a plurality of cameras each configured to acquire image data.

[0015] The AR device 100 includes a sensor 140 that can obtain measurement values of the AR device 100 or the environment around the AR device 100. For example, the AR device 100 may include a motion sensor, an accelerometer, a gyroscope, an inertial measurement unit (IMU), a barometer, a positioning system, a thermometer, a light sensor, a photosensor (e.g., a camera), etc. The sensor 140 can obtain position data describing the position of the AR device 100. The sensor 140 can use a satellite navigation positioning system (e.g., a GPS system, a Galileo positioning system, a Global Navigation Satellite System (GLONASS), a Beidou satellite navigation and positioning system), an inertial navigation system, a dead reckoning system based on an IP address, triangulation or proximity to a cellular tower or a Wi-Fi hotspot, or other appropriate techniques for determining position to determine the actual or relative position of the AR device 100.

[0016] The object position module 150 determines the target position of a virtual object in the real world. The virtual object is a computer-generated object that is displayed to the user. For example, the virtual object can be a computer-generated person, avatar, creature, or item. The target position of the virtual object is the position in the real world where the AR device 100 targets the object so that it can be seen when displayed to the user. The target position can be mainly described herein as the position determined relative to the AR device (i.e., the location where the AR device 100 functions as a starting point), but the target position can also be determined as an absolute position. In some embodiments, the target position includes the target direction of the virtual object. Further, the target position can be a 2D position or a 3D position.

[0017] The object position module 150 can determine the target position of a virtual object based on content instructions. The content instructions are instructions for presenting content to the user by the AR device 100. For example, the content instructions may include a 3D model that the AR device 100 uses to display the virtual object. Further, the content instructions may specify where in the real world the virtual object should be placed. For example, the content instructions for the virtual object may specify the absolute position in the real world where the virtual object should be placed, or the position relative to the user where the virtual object should be placed. The content instructions may further specify the target position of the virtual object based on real-world objects. For example, the content instructions may specify that the virtual object should be placed at the center of a table or within a window frame. In some embodiments, the content instructions include motion instructions that specify how the virtual object should move. For example, the motion instructions may specify the direction of movement of the virtual object and the distance the virtual object should move within a time step. The object position module 150 can receive content instructions from the online system 120 or from the data store 180.

[0018] The object position module 150 can determine the target position of a virtual object based on the sensor data received from the sensor 140. For example, the object position module 150 can determine the position of the AR device 100 based on the position data received from the sensor 140, and can determine the target position of the virtual object based on the position of the AR device 100. In some embodiments, the object position module 150 can use the sensor data from the sensor 140 to identify real-world objects in the environment around the AR device 100, and can determine the target position of the object based on the identities or positions of those real-world objects. For example, the object position module 150 can identify a table and the position of the table, and can determine the target position of a virtual object at the center of the top of the table.

[0019] The content display module 160 displays content to the user. The content display module 160 can include a display 165 that displays content to the user. The display 165 is a device that can project a 2D array of pixels to the user. For example, the display 165 may include a flat display (e.g., an LCD screen, an OLED screen, or a flat touch screen), or display content to the user through one or more lenses of a headset configured to project light to one or both of the user's eyes, and the user views the real-world environment through the lenses, and one or more virtual objects are superimposed and displayed in the user's field of view of the real-world environment. The content display module 160 can display image data to the user via the display 165. For example, the content display module 160 can display the image data acquired by the camera assembly 130 and display the image data to the user via the display 165.

[0020] The content display module 160 displays virtual objects to the user. The content display module 160 may generate a 2D projection of the 3D model of the virtual object and display the virtual object to the user via the display 165. The content display module 160 may generate a 2D projection of the virtual object so that the object appears to the user to be at the target position of the virtual object. For example, the content display module 160 may adjust the size, orientation, or placement of the 2D projection of the virtual object within the display 165 so that the virtual object appears to be located at the target position. The content display module 160 may further adjust the 2D projection of the virtual object using sensor data from the AR device 100. For example, the content display module 160 may receive position data describing the position of the AR device 100 and adjust the size of the 2D projection of the virtual object so that the virtual object appears to be at the correct distance from the AR device 100 relative to the target position of the virtual object. Similarly, the content display module 160 may receive orientation data describing the orientation of the AR device 100 and adjust the position of the 2D projection of the virtual object so that the virtual object appears to be located at the target position of the virtual object.

[0021] The smooth correction module 170 detects whether a virtual object is correctly displayed assuming that the virtual object is at the target position, and corrects the position of the virtual object if the virtual object is not correctly displayed. For example, as the AR device 100 collects more data (e.g., from the sensor 140), the smooth correction module 170 may determine that the position of the virtual object is incorrect and needs to be updated. Similarly, the smooth correction module 170 may determine that the virtual object has been displayed to the user based on an inaccurate estimation of the position of the AR device 100 and that the virtual object needs to be displayed in a different way so that it appears to be located at the target position. The smooth correction module 170 is shown herein as a separate module from the object position module 150, but in an alternative embodiment, the smooth correction module 170 is part of the object position module 150.

[0022] The smooth correction module 170 may smoothly correct the display of a virtual object when the display position of the virtual object does not correspond to the target position of the virtual object. For example, when the AR device 100 receives sensor data from the sensor 140, the smooth correction module 170 may determine the original display position of the virtual object. The original display position of the virtual object is the position in the real world where the virtual object was originally displayed. The smooth correction module 170 may determine that the original position of the virtual object does not correspond to the target position of the virtual object and update the display of the virtual object so that it appears to be located at the updated position in the real world.

[0023] The smooth correction module 170 can continuously update the displayed position of the virtual object. For example, the smooth correction module 170 can update the display of the virtual object based on updated sensor data from the sensor 140 or based on new content commands executed on the virtual object. In some embodiments, the smooth correction module 170 periodically updates the target position of the virtual object based on time steps. A time step is the time interval used by the AR device 100 relative to how frequently the smooth correction module 150 updates the display of the virtual object. For example, the time step may be based on the refresh rate of the display. The AR device 100 can adjust the time step. For example, the AR device 100 may shorten the time step when the virtual object is moving or may extend the time step when the virtual object is relatively stationary.

[0024] Methods for smoothly correcting the display of a virtual object are described in detail below. For brevity, these methods for smoothly correcting the display of a virtual object can be described mainly in the context of updating the position of the virtual object from the reference frame of the non-moving AR device 100. However, the methods described herein can be applied to smoothly update the display of the virtual object based on the updated target position of the virtual object or the updated position of the AR device 100.

[0025] In some embodiments, the smooth correction module 170 updates the position of the virtual object based on an angular threshold. FIG. 2 shows the position of the virtual object updated based on the angular threshold according to some embodiments. The angular threshold 200 is the maximum value of the angle between (1) the line of sight 220 from the AR device 100 to the original position 230 of the virtual object and (2) the line of sight 240 from the AR device 100 to the updated position 250 of the virtual object. For example, if the angle 210 between the line of sight from the AR device 100 to the original position 230 and the target position 260 of the virtual object is 30 degrees and the angular threshold 200 is 15 degrees, the smooth correction module 170 can update the displayed position of the virtual object to the real-world position 250 between the original position 230 and the target position 260 so that the angle between the original position 230 and the updated position 250 does not exceed 15 degrees. However, if the target position 260 is 10 degrees away from the original position 230, since the target position 260 is within the angular threshold of the original position 230, the smooth correction module 170 can display the virtual object at the target position 260.

[0026] The smooth correction module 170 can select, as the updated position 250, the position closest to the target position 260 without exceeding the angular threshold 200. For example, the smooth correction module 170 can use the Euclidean distance to select the updated position 250 closest to the target position 260. In some embodiments, the smooth correction module 170 selects the updated position 250 based on the position along the line of sight 240 corresponding to the angular threshold 200. The smooth correction module 170 can select the position along the line of sight 240 closest to the target position 260 as the updated position 250. The smooth correction module 170 can consider the set of positions along that line of sight 240 and select the position closest to the target position 260.

[0027] The smooth correction module 170 can also identify the position 250 that is closest to the target position 260 using triangulation. Since the updated position 250 is the point closest to the target position 260 along its line of sight, the line 270 from the target position 260 to the updated position 250 intersects the line of sight 240 that corresponds to the angular threshold 200 at a right angle. Thus, the smooth correction module 170 can identify the updated position 250 based on the angular threshold 200, the original position 230, and the target position 260 using triangulation. Similarly, the smooth correction module 170 can calculate the parameters (e.g., slope and intercept) of the line of sight 240 that corresponds to the angular threshold 200 and calculate the parameters of the line 270 between the target position 260 and the updated position 250 based on the slope of the line of sight 240 and the target position to identify the updated position 250. For example, since the line 270 from the target position 260 to the updated position 250 intersects the line of sight 240 that corresponds to the angular threshold 200 at a right angle, the slope of the line 270 between the target position 260 and the updated position 250 is the negative reciprocal of the slope of the line of sight 240. Next, the smooth correction module 170 can determine the updated position 250 based on where the line of sight 240 intersects the line 270 between the target position and the updated position 250.

[0028] The smooth correction module 170 can determine how much the displayed position of the virtual object can be updated within a time step using the angular threshold 200. For example, if the target position 260 is not within the angular threshold of the original position 230, the smooth correction module can continuously update the displayed position of the virtual object through the sequence of updated positions 250 until the virtual object reaches the target position 260 at each time step.

[0029] Furthermore, the smooth correction module 170 may continuously update the target position 260 based on the updated data from the sensor 140 or the camera assembly 130. The smooth correction module 170 may determine the updated position 250 of the virtual object at each time step based on the updated target position 130.

[0030] In some embodiments, the smooth correction module 170 updates the position of the virtual object based on a motion threshold. FIG. 3 shows the position of a virtual object updated based on a motion threshold according to some embodiments. The AR device 100 may determine the original position 300 of the virtual object. The AR device 100 may also determine that the virtual object should be displayed such that the virtual object moves a certain amount 310 within a time step. The AR device 100 may then determine the next original position 320 of the virtual object based on the original position 300 of the virtual object and the movement 310 of the object.

[0031] The smooth correction module 170 may determine that the original position 300 is not the target position 330. For example, the smooth correction module 170 may determine that the AR device 100 is positioned at a location different from that used to determine the original position 300. The smooth correction module 170 may determine an updated position 340 of the virtual object based on a motion threshold 350. The motion threshold 350 is the maximum distance by which the position of the virtual object may be adjusted based on the motion 310 of the virtual object. For example, the motion threshold 350 may be a certain percentage of the distance 310 that the virtual object advances within a time step. If the target position 330 is within the motion threshold 350 of the original position 300, the smooth correction module 170 updates the displayed position of the virtual object from the original position 300 to the target position 330. However, if the target position 330 is outside the motion threshold 350, the smooth correction module 170 may determine an updated position 340 of the virtual object that is in the direction of the target position 330 but within the motion threshold 350 of the original position 300 of the virtual object.

[0032] The smooth correction module 170 determines an updated next position 360 of the virtual object. The updated next position 360 is the position at which the virtual object is to be displayed based on the updated position 340 of the virtual object and the motion 310 of the virtual object. In some embodiments, the smooth correction module 170 updates the position of the virtual object from the original position 300 to the updated next position 360 in one step 370. Alternatively, the smooth correction module 170 may update the position of the virtual object from the original position 300 to the updated next position 360 in two steps 380. In the first time step, the smooth correction module 170 may update the position of the virtual object from its original position 300 to the updated position 340, and in the second time step, the position of the virtual object may be updated from the updated position 340 to the updated next position 360.

[0033] As described above, the above explanation mainly describes correcting the displayed position of the virtual object based on the movement of the object. However, the smooth correction module 170 can update the position of the virtual object based on the movement of the AR device 100 using a movement threshold. For example, the smooth correction module 170 can detect that the AR device 100 is moving or changing direction. The smooth correction module 170 can convert these movements of the AR device 100 into movements of the virtual object relative to the AR device, and as described above, can apply a movement threshold to update the position of the virtual object.

[0034] In some embodiments, the smooth correction module 170 updates the displayed position of the virtual object based on a pixel threshold. FIG. 4 shows the displayed position of a virtual object updated based on a pixel threshold according to some embodiments. The pixel threshold is the maximum number of pixels by which the virtual object moves when the smooth correction module 170 updates the position of the virtual object. In FIG. 4, the virtual object is depicted at an original position 410 within the display 400 of the AR device 100. The smooth correction module 170 may determine that the original position 400 of the virtual object is not the target position 420 of the virtual object. If the target position 420 is within the pixel threshold 430 of the original position 410, the smooth correction module 170 updates the position of the virtual object to the target position 420. If the target position 420 is outside the pixel threshold 430 from the original position, the smooth correction module 170 may determine an updated position 440 of the virtual object that is in the direction of the target position 420 but within the movement threshold 430 of the original position 410 of the virtual object.

[0035] To determine whether the target position 420 is within the pixel threshold 430 of the original position 410, the smooth correction module 170 may compare the pixel projection of the virtual object at the original position 410 with the pixel projection of the virtual object at the correct target position 420. These pixel projections may be pixel projections generated by a pixel projection function on a 2D array of pixels representing the display of the AR device 100. The smooth correction module 170 may then compare the pixels within each pixel projection to determine the distance between the pixel projections. If the distance between the pixel projection of the original position 410 and the pixel projection of the target position 420 is less than the pixel threshold 430, the smooth correction module 170 updates the position of the virtual object to the target position 420. If the distance is not within the pixel threshold 430, the smooth position module 170 may determine an updated position 440 between the target position 420 and the original position 410 that is within the pixel threshold 430 and has the shortest distance to the correct target position 420.

[0036] To determine the distance between the pixel projections, the smooth correction module 170 may identify which pixels within each pixel projection correspond to a part of the virtual object (e.g., which pixels within each pixel projection correspond to the corners of a box) and compare the positions of the pixels within each pixel projection. The smooth correction module 170 may also compare the pixels within each pixel projection that are closest to each other to determine the distance between the pixel projections. Similarly, the smooth correction module 170 may compare the pixels within each pixel projection that are farthest from each other to determine the distance between the pixel projections. In some embodiments, the smooth correction module 170 identifies the center pixels of each pixel projection and compares the distances between those center pixels. For example, the smooth correction module 170 may determine the centroid of each pixel projection and identify the center pixel of the pixel projection.

[0037] In some embodiments, the pixel threshold 430 is based on the movement of the virtual object. If the content command of the virtual object indicates that the virtual object is depicted as moving, the smooth correction module 170 may adjust the pixel threshold 430 based on the movement of the virtual object. For example, the pixel threshold 430 may be adjusted based on a certain percentage of the distance that the virtual object moves within a time step. The pixel threshold 430 may be based only on the movement of the virtual object, or may have a baseline value that increases based on the movement of the virtual object.

[0038] The data store 180 stores data used by the AR device 100 and provides an AR experience to the user. For example, the data store 180 may store content data that describes content that may be presented to the user by the AR device 100. For example, the data store 180 may store 3D models of virtual objects that may be presented to the user. Additionally, the data store 180 may include content commands for the virtual objects. The data store 180 may also store image data acquired by the camera assembly 130 or sensor data acquired by the sensor 140.

[0039] The network 110 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., Internet), or some combination thereof. The network can also include a direct connection between the AR device 100 and the game server 120. Generally, communication between the game server 120 and the AR device 100 can be transmitted via a network interface using any type of wired or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML, JSON), or security schemes (e.g., VPN, secure HTTP, SSL).

[0040] The online server 120 can be any computing device and can include one or more processors and one or more computer-readable storage media. The computer-readable storage media can store instructions that cause the processor to execute operations. The online server 120 can be configured to receive requests for data from the AR device 100 (e.g., via a remote procedure call (RPC)) and respond to those requests via the network 110. For example, the online server 120 can encode content data in one or more data files and provide the data files to the AR device 100. Further, the online server 120 can be configured to receive data (e.g., the location of the AR device 100, user actions, user input, etc.) from the AR device 100 via the network 110. The AR device 100 can periodically transmit data from the AR device 100 to the game server 120.

[0041] In some embodiments, the online server 120 transmits content data to the AR device 100 for presentation to the user. The content data can include 3D models of virtual objects and content instructions for presenting the 3D models to the user. In some embodiments, the online server 120 receives location data, sensor data, image data, or input data from the AR device 100 and transmits updated content instructions to the AR device 100 based on the received data.

[0042] The techniques discussed in this specification refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted between such systems. One of ordinary skill in the art will recognize that the inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions between components. For example, the server processes discussed in this specification may be implemented using a single server or multiple servers operating in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0043] Furthermore, when the systems and methods described herein access and analyze personal information about a user, or utilize personal information such as location information, the user may be provided with an opportunity to control whether a program or function collects information, whether the system or other applications receive content, or how such content is received. Such information or data is not collected or used until the user is provided with a meaningful notice about what information is being collected and how that information will be used. Information is not collected or used without the user's consent, and the user may revoke or change their consent at any time. Thus, the user may control how information about the user is collected and used by an application or system. Additionally, certain information or data may be processed in one or more ways such that personally identifiable information is removed before the information is stored or used. For example, a user identifier may be handled in such a way that it cannot be used to determine personally identifiable information about the user.

[0044] FIG. 5 is a flowchart showing an exemplary method of correcting the position of a virtual object based on an angle threshold according to some embodiments. Alternative embodiments may include more steps, fewer steps, or different steps than those shown in FIG. 5, and the steps may be performed in an order different from that shown in FIG. 5. Further, each of these steps may be automatically performed by an AR device or an online server, with or without human intervention.

[0045] The AR device receives sensor data from one or more sensors of the AR device at 500 and determines a target position of the virtual object at 510. The AR device displays the virtual object at the original position based on the sensor data at 520. If the received sensor data is inaccurate or old, the original position may be different from the target position. The AR device receives updated sensor data at 530 and may determine based on the sensor data that the original position is not the target position.

[0046] The AR device determines at 540 whether the target position is within the angle threshold of the original position. The angle threshold may be the maximum angle by which the line from the AR device to the virtual object can change within a time step. If the target position is within the angle threshold of the original position, the AR device displays the virtual object at the target position at 550. If the target position is not within the angle threshold of the original position, the AR device displays the virtual object at the updated position at 560. The updated position may be the position closest to the target position without being outside the angle threshold. The AR device may update the position of the virtual object to the target position or the updated position within one time step. The AR device may continuously adjust the position of the virtual object in a similar manner in subsequent time steps.

[0047] FIG. 6 is a flowchart showing an exemplary method of correcting the position of a virtual object based on a motion threshold according to some embodiments. Alternative embodiments may include more steps, fewer steps, or different steps than those shown in FIG. 6, and the steps may be performed in an order different from that shown in FIG. 6. Further, each of these steps may be automatically performed by an AR device or an online server, with or without human intervention.

[0048] The AR device receives sensor data from one or more sensors of the AR device 600 and determines a target position of the virtual object 610. The AR device displays the virtual object at the original position based on the sensor data 620. If the received sensor data is inaccurate or old, the original position may be different from the target position. The AR device receives updated sensor data 630 and may determine based on the sensor data that the original position is not the target position.

[0049] The AR device determines whether the target position is within a motion threshold of the original position 640. The motion threshold may be the maximum distance by which the position of the virtual object can be adjusted from the original position to the target position. The motion threshold may be based on the movement of the virtual object. If the target position is within the motion threshold of the original position, the AR device updates the virtual object based on the target position 650. If the target position is not within the angular threshold of the original position, the AR device determines an updated position of the virtual object 660. The updated position may be the position closest to the target position without being outside the motion threshold. The AR device updates the virtual object based on the updated position 670.

[0050] The AR device can update a virtual object by moving the position of the virtual object to a target position or an updated position in a single time step. The AR device can then, in subsequent time steps, move the virtual object from the target or updated position to the next position during the movement of the virtual object. The AR device can also update the virtual object by directly moving the position of the virtual object from its original position to the next position in a single time step.

[0051] FIG. 7 is a flowchart showing an exemplary method of correcting the position of a virtual object based on a pixel threshold according to some embodiments. Alternative embodiments may include more steps, fewer steps, or different steps than those shown in FIG. 7, and the steps may be performed in an order different from that shown in FIG. 7. Further, each of these steps may be automatically performed by the AR device or an online server, with or without human intervention.

[0052] The AR device receives sensor data from one or more sensors of the AR device 700 and determines a target position of the virtual object 710. The AR device displays the virtual object at the original position based on the sensor data 720. If the received sensor data is inaccurate or outdated, the original position may be different from the target position. The AR device receives updated sensor data 730 and may determine based on the sensor data that the original position is not the target position.

[0053] The AR device determines whether the target position is within the pixel threshold of the original position. The pixel threshold limits the distance by which the position of the virtual object can be corrected and places a maximum value on the distance (in pixels) by which the pixel projection of the virtual object can be adjusted. To determine whether the target position is within the pixel threshold of the pixel position, the AR device can compare the pixel projection of the virtual object at the original position with the pixel projection of the virtual object at the target position 740. The AR device determines the distance between the pixel projections 750, compares the distance with the pixel threshold 760, and can determine whether the original position is within the pixel threshold of the target position.

[0054] If the target position is within the pixel threshold of the original position, the AR device displays the virtual object at the target position 770. If the target position is not within the angular threshold of the original position, the AR device displays the virtual object at the updated position 780. The updated position can be the position closest to the target position without being outside the angular threshold. The AR device can update the position of the virtual object to the target position or the updated position within one time step. The AR device can continuously adjust the position of the virtual object in a similar manner in subsequent time steps. Example of a computing system FIG. 8 is an exemplary architecture of a computing device according to one embodiment. FIG. 8 shows a high-level block diagram of the physical components of a computer that may be used as part or all of one or more of the entities described herein. According to one embodiment, however, the computer may have additional, fewer, or variations of the components provided in FIG. 8. FIG. 8 shows a computer 800, but the figure is intended as a functional explanation of the various features that may exist in a computer system rather than as a structural schematic of the implementation described herein. In practice, as will be recognized by those skilled in the art, the individually shown items can be combined or some items separated.

[0055] FIG. 8 shows at least one processor 802 coupled to a chipset 804. Also coupled to the chipset 804 are a memory 806, a storage device 808, a keyboard 810, a graphics adapter 812, a pointing device 814, and a network adapter 816. A display 818 is coupled to the graphics adapter 812. In one embodiment, the functions of the chipset 804 are provided by a memory controller hub 820 and an I / O hub 822. In another embodiment, the memory 806 is directly coupled to the processor 802 instead of the chipset 804. In some embodiments, the computer 800 includes one or more communication buses for interconnecting these components. One or more communication buses optionally include circuitry (also called a chipset) for interconnecting and controlling communications between system components.

[0056] The storage device 808 is any non-transitory computer-readable storage medium such as a hard drive, a compact disc read-only memory (CD-ROM), a DVD, or a solid-state memory device or other optical storage device, a magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, a magnetic disk storage device, an optical disk storage device, a flash memory device, or other non-volatile solid-state storage device. Such a storage device 808 may also be referred to as persistent memory. The pointing device 814 can be a mouse, a trackball, or other type of pointing device and can be used in combination with the keyboard 810 to input data into the computer system 800. The graphics adapter 812 displays images and other information on the display 818. The network adapter 816 couples the computer 800 to a local or wide area network.

[0057] Memory 806 holds the instructions and data used by processor 802. Memory 806 can be non-persistent memory, examples of which include high-speed random access memory such as DRAM, SRAM, DDR RAM, ROM, EEPROM, flash memory, etc.

[0058] As is known in the art, computer 800 can have components different from those shown in FIG. 8 or other components. Further, computer 800 may lack certain specific components shown. In one embodiment, computer 800 functioning as a server may lack keyboard 810, pointing device 814, graphic adapter 812, or display 818. Further, storage device 808 can be located locally or remotely with respect to computer 800 (such as being incorporated within a storage area network (SAN)).

[0059] As is known in the art, computer 800 is adapted to execute computer program modules for providing the functions described herein. As used herein, the term "module" refers to computer program logic utilized to provide a specified function. Thus, a module can be implemented in hardware, firmware, or software. In one embodiment, program modules are stored in storage device 808, loaded into memory 806, and executed by processor 302. Additional Considerations The foregoing description of the embodiments has been presented for purposes of illustration and is not intended to be exhaustive or to limit the patent rights to the exact forms disclosed. Those skilled in the art will appreciate that many modifications and changes are possible in light of the above disclosure.

[0060] Certain portions of this description describe embodiments from the perspective of algorithms and symbolic representations of operations on information. The descriptions and representations of these algorithms are commonly used by those skilled in the data processing arts to effectively convey the spirit of their work to other skilled artisans. While these operations are described functionally, computationally, or logically, it is understood that they may be implemented by a computer program, an equivalent electrical circuit, or microcode, etc. Further, and without loss of generality, it can be seen that it is sometimes convenient to refer to sequences of these operations as modules. The operations described and the modules associated therewith can be embodied in software, firmware, hardware, or any combination thereof.

[0061] Any step, operation, or process described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In some embodiments, a software module may be implemented by a computer program product including one or more computer-readable media that include computer program code or instructions executable by a computer processor to perform any or all of the steps, operations, or processes described. In some embodiments, the computer-readable media includes one or more computer-readable media that, when executed by one or more processors, include instructions to cause the one or more processors to individually or together execute the steps of instructions stored on the one or more computer-readable media. Similarly, a processor includes one or more processors or processing units that individually or together execute the steps of instructions stored on a computer-readable media.

[0062] Embodiments may also relate to products generated by the computing processes described herein. Such products may be composed of information resulting from the computing process, which information may be stored on a non-transitory tangible computer-readable storage medium and may include any embodiment of a combination with the computer program product or other data described herein.

[0063] Finally, the language used in the specification has been selected primarily for readability and for purposes of explanation and may not have been selected to depict or limit the subject matter of the invention. Accordingly, the scope of the patent rights is intended to be limited not by this detailed description of the invention but by any claims that issue from an application based on this specification. Thus, the disclosure of the embodiments is intended to exemplify, and not to limit, the scope of the patent rights described in the following claims.

[0064] As used herein, terms such as "comprise", "comprising", "include", "including", "have", "having", or other variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive "or" and not an exclusive "or". For example, the condition "A or B" is satisfied by any of the following: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present). Similarly, the condition "A, B, or C" is satisfied by any combination of A, B, and C that has at least one element in the combination that is true (or present). By way of non-limiting example, the condition "A, B, or C" is satisfied where A and B are true (or present) and C is false (or absent). Similarly, as another non-limiting example, the condition "A, B, or C" is satisfied where A is true (or present) and B and C are false (or absent).

Claims

1. When executed by a computing system, the computing system Receiving sensor data from one or more sensors of an augmented reality device, Determining the target position of a virtual object, wherein the target position of the virtual object includes a real-world position that targets the virtual object so that the augmented reality device appears to be located when viewed by a user through the display of the augmented reality device, Based on the target position and the received sensor data, the virtual object is displayed to the user of the augmented reality device at its original position. Receiving updated sensor data from one or more sensors of the augmented reality device, Based on the updated sensor data, determine whether the target position is within an angular threshold of the original position, wherein the angular threshold is the maximum angle that the line from the augmented reality device to the virtual object can change within a time step. In response to the target position being outside the angular threshold of the original position, displaying the virtual object to the user at the updated position, wherein the updated position is within the angular threshold of the original position and is the position closest to the target position, A non-temporary, computer-readable storage medium that stores instructions for executing a program.

2. The computer-readable storage medium according to claim 1, further storing instructions causing a processor to determine, based on the updated sensor data, whether the original position of the virtual object is substantially similar to the target position.

3. The computer-readable storage medium according to claim 1, wherein the one or more sensors comprises one or more of the following: a motion sensor, an accelerometer, a gyroscope, an inertial measurement unit, a barometer, a positioning system, a thermometer, or an optical sensor.

4. The computer-readable storage medium according to claim 1, further storing an instruction for the processor to display the virtual object to the user at the target position in response that the target position is within the angular threshold of the original position.

5. The command for determining whether the target position is within the angular threshold of the original position is: Identifying a first line of sight from the position of the augmented reality device to the original position, Identifying a second line of sight from the position of the augmented reality device to the target position, To determine whether the angle between the first line of sight and the second line of sight exceeds the angle threshold, A computer-readable storage medium according to claim 1, including the following:

6. Displaying the virtual object to the user at the updated position includes determining the parameters of a first line that intersects the target position and intersects a second line at a right angle, wherein the second line corresponds to the angle threshold, according to the computer-readable storage medium of claim 1.

7. Displaying the virtual object to the user at the updated position includes determining the updated position based on one or more trigonometric functions and the angle threshold, according to claim 1, for the computer-readable storage medium.

8. The computer-readable storage medium according to claim 1, wherein the updated position is the position closest to the target position within the angular threshold, based on the Euclidean distance between the updated position and the target position.

9. The augmented reality device is a computer-readable storage medium according to claim 1, comprising a smartphone.

10. The augmented reality device comprises a headset, as described in claim 1, for the computer-readable storage medium.

11. When executed by the processor, the processor will Receiving sensor data from one or more sensors of an augmented reality device, Determining the target position of a virtual object, wherein the target position of the virtual object includes a real-world position that targets the virtual object so that the augmented reality device appears to be located when viewed by a user through the display of the augmented reality device, Based on the target position and the received sensor data, the virtual object is displayed to the user of the augmented reality device at its original position. Receiving updated sensor data from one or more sensors of the augmented reality device, Based on the updated sensor data, determine whether the target position is within a motion threshold of the original position, wherein the motion threshold is the maximum distance at which the position of the virtual object can be adjusted from the original position to the target position, and the motion threshold is based on the movement of the virtual object. In response to the target position being outside the motion threshold of the original position, Determining the updated position of the virtual object, wherein the updated position is within the motion threshold of the original position and is the position closest to the target position, Updating the virtual object based on the updated position, A non-temporary, computer-readable storage medium that stores instructions for executing a program.

12. The computer-readable storage medium according to claim 11, further storing instructions causing a processor to determine, based on the updated sensor data, whether the original position of the virtual object is substantially similar to the target position.

13. The computer-readable storage medium according to claim 11, wherein the one or more sensors comprises one or more of the following: a motion sensor, an accelerometer, a gyroscope, an inertial measurement unit, a barometer, a positioning system, a thermometer, or an optical sensor.

14. The computer-readable storage medium according to claim 11, further storing an instruction for the processor to display the virtual object to the user at the target position in response that the target position is within the motion threshold of the original position.

15. Displaying the virtual object in its original position further includes displaying the virtual object based on a movement command for the virtual object, wherein the movement command includes the direction of the movement of the virtual object and the distance the virtual object moves within a time step, according to claim 11.

16. The computer-readable storage medium according to claim 15, wherein the motion threshold is based on the motion distance the virtual object moves within a time step.

17. The computer-readable storage medium according to claim 15, wherein, when executed by a processor, it further stores in the processor an instruction to cause the virtual object to move to a next position based on the updated position and the movement instruction for the virtual object, the next position being the position of the virtual object after it has moved the distance of the movement in the direction of the movement from the updated position.

18. Displaying the virtual object at the next position includes moving the virtual object directly from the original position to the next position within a single time step, according to claim 17, for the computer-readable storage medium.

19. Displaying the virtual object at the following position means that Display the virtual object at the updated position in the first time step, Display the virtual object at the following position in the second time step following the first time step, A computer-readable storage medium according to claim 17, including the following:

20. The augmented reality device is a computer-readable storage medium according to claim 11, comprising a smartphone.

Images