Smooth object correction for augmented reality devices

Abstract

An augmented reality ("AR") device employs a smoothing correction method to correct a location of a virtual object presented to a user. The AR device may apply the angle threshold to determine whether the virtual object may move from the original position to the target position. The angle threshold is a maximum angle at which a line from the AR device to the virtual object may vary within a time step. Similarly, the AR device may apply a motion threshold that is a maximum distance by which the position of the virtual object may be corrected based on the motion of the virtual object. In addition, the AR device may apply a pixel threshold to the position correction of the virtual object. The pixel threshold is based on a change in the position of the virtual object, the maximum distance at which the pixel projection can vary.

Description

Background Art

[0001] Augmented reality ("AR") devices display virtual objects to a user on a user display that overlays the virtual objects on the real world. For example, an AR device may display image data to a user that shows the real-world environment around the user, but with virtual objects positioned on a real-world table. Augmented reality is generally a more complex technical task than full virtual reality (where the entire image displayed to the user is virtual) because it must be used to integrate the virtual objects into the real world in order for the virtual objects to "feel" real to the user. For example, an AR device may have sensor data that captures the user's surroundings and determines how to display a virtual object so that the virtual object appears to be a real object in the real world.

[0002] There may be numerous challenges in displaying virtual objects in AR. For example, a virtual object may be displayed such that it appears to be in an incorrect location, rather than where it is expected to appear. For example, an AR device may have inaccurate or imprecise position data describing its location relative to objects of the AR device environment, or orientation data describing the orientation of the AR device in its environment. This may cause the AR device to fail to display virtual objects correctly. Similarly, the AR device may receive inaccurate or imprecise sensor data describing the AR device environment, and may therefore display virtual objects based on inaccurate estimates of the locations of other objects in the environment.

[0003] The AR device may update the display of the virtual object based on the updated data received by the AR device. However, if the position of the virtual object is updated to a completely different position, the user may notice that the position of the virtual object is being updated. For example, conventional methods of updating the position of a virtual object may cause the virtual object to suddenly appear in the correct position, or to move quickly or unnaturally to its correct position. These methods often cause the user to notice that the position of the virtual object is being updated, and therefore, conventional correction methods cause the user to lose immersion in the AR device experience. Summary of the invention

[0004] The AR device disclosed herein can improve the traditional object position correction method by applying an angle threshold when correcting the target position of a virtual object. The target position of the virtual object is the position that the AR device makes the virtual object appear to be in the real world when displayed to the user. However, as noted above, when the AR device displays the virtual object to the user, the actual presentation position of the virtual object may be different from the target position. The AR device can determine whether the position of the virtual object is different from the target position, and can correct the position of the virtual object.

[0005] The AR device may determine whether the target position is within an angle threshold of the original position of the virtual object. The angle threshold is the maximum angle between the line of sight of the AR device to the position of the virtual object and the line of sight of the AR device to the updated position of the virtual object. If the target position is within the angle threshold, the position of the virtual object is updated to the target position. If the target position is outside the angle threshold, the position of the virtual object is updated to the position closest to the target position and not exceeding the angle threshold.

[0006] Additionally, the AR device may apply a motion threshold when correcting the position of a virtual object. The virtual object displayed to the user may make the virtual object appear to be moving relative to the AR device. The motion threshold is the maximum distance by which the position of the virtual object can be adjusted based on the motion of the virtual object. For example, the motion threshold may allow the position of the virtual object to be adjusted to within a certain proportion of the distance the virtual object moves within a time step. If the target position for the virtual object is within the motion threshold of its original position, the position of the virtual object is updated to the target position. If the target position is outside the motion threshold, the position of the virtual object may be updated to the position closest to the target position and not exceeding the motion threshold. When moving a virtual object, the AR device may also move the virtual object to the next step of its motion based on the updated position of the virtual object rather than its original position.

[0007] In addition, the AR device may apply a pixel threshold when correcting the position of the virtual object. The pixel threshold limits the distance at which the position of the virtual object can be corrected, that is, the maximum pixel distance at which the pixel projection of the virtual object can be adjusted. For example, the AR device may compare the 2D pixel projection of the virtual object at the position of the virtual object with the 2D pixel projection of the virtual object at the target position. If the distance between the pixel projections does not exceed the pixel threshold, the AR device may update the position of the virtual object to the target position. If the distance between the pixel projections exceeds the pixel threshold, the AR device may update the position of the virtual object to the position closest to the target position and not exceeding the pixel threshold.

[0008] By using an angle threshold, the AR device can allow more depth corrections to the position of the virtual object rather than lateral corrections. Therefore, the corrections allowed by the AR device can be less obvious to the user, which enables the virtual object to be as close to the correct position as possible while retaining the desired degree of immersion or credibility for the user. Similarly, by limiting the degree to which the position of the virtual object can be adjusted based on a motion threshold, the user can regard any adjustment of the position of the virtual object as part of the expected movement of the virtual object rather than a correction to the position of the virtual object. Therefore, when the virtual object moves faster, the AR device can more easily correct its position because the adjustment will be less obvious to the user. In addition, by limiting the adjustment of the position of the virtual object based on a pixel threshold, the AR device reduces the amount of pixel changes corresponding to the virtual object, and thus more directly makes the correction of the position of the virtual object less obvious to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A network computing environment is illustrated in accordance with some embodiments.

[0010] Figure 2 Illustrated is updating the position of a virtual object based on an angle threshold in accordance with some embodiments.

[0011] Figure 3 Illustrated is updating the position of a virtual object based on a motion threshold in accordance with some embodiments.

[0012] Figure 4 Illustrated is updating a target position of a virtual object based on a pixel threshold in accordance with some embodiments.

[0013] Figure 5 is a flow chart illustrating an example method of correcting the position of a virtual object based on an angle threshold in accordance with some embodiments.

[0014] Figure 6 is a flow chart illustrating an example method of correcting the position of a virtual object based on a motion threshold in accordance with some embodiments.

[0015] Figure 7 is a flow chart illustrating an example method of correcting the position of a virtual object based on a pixel threshold in accordance with some embodiments.

[0016] Figure 8 is an example architecture for a computing device in accordance with some embodiments. DETAILED DESCRIPTION

[0017] Figure 1 A system environment of an augmented reality (AR) device according to some embodiments is illustrated. Figure 1The illustrated system environment includes an AR device 100, a network 110, and one or more online servers 120. Alternative embodiments may include Figure 1 More, fewer, or different components are illustrated, and the functionality of the components may be divided among components different than described below. For example, some or all of the functionality of the AR device 100 may be performed by one or more online servers 120, and vice versa.

[0018] The AR device 100 can be any computing device that can be used by a user to interact with AR content presented by the AR device 100. For example, the AR device 100 can be a wireless device, a personal digital assistant (PDA), a portable gaming device, a cellular phone, a smart phone, a tablet computer, a navigation system, a handheld GPS system, a wearable computing device, a display with one or more processors, or other such devices. In some embodiments, the AR device 100 includes a traditional computer system, such as a desktop computer or a laptop computer (e.g., a desktop computer wirelessly connected to an AR headset). Additionally, the AR device 100 can be a vehicle with a computing device. The AR device 100 can include one or more processors and one or more computer-readable storage media. The computer-readable storage medium can store instructions that cause the processor to perform operations.

[0019] The AR device 100 may include various other input / output devices for receiving information from a player or providing information to a player. Example input / output devices include a display screen, a touch screen, a touch pad, data entry keys, a speaker, and a microphone suitable for speech recognition. The AR device 100 may also include a network interface for providing communication via the network 110. The network interface may include any suitable components for interfacing with one or more networks, including, for example, a transmitter, a receiver, a port, a controller, an antenna, or other suitable components.

[0020] Figure 1 1 illustrates an example system architecture of an AR device 100 according to some embodiments. The AR device 100 may include a camera assembly 130, one or more sensors 140, an object location module 150, a content display module 160, a smoothing correction module 170, and a data storage 180. In an alternative embodiment, the AR device 100 may include a camera assembly 130, one or more sensors 140, an object location module 150, a content display module 160, a smoothing correction module 170, and a data storage 180. Figure 1 More, fewer or different components may be shown, and the functionality of each component may be different. Figure 1 The functions are divided differently.

[0021] The camera assembly 130 captures image data of the environment around the AR device 100. The camera assembly 130 may include a light sensor for capturing environmental image data. The camera assembly 130 may include a wide-angle lens or a telephoto lens. The camera assembly 130 may be configured to capture a single image or video as image data. The camera assembly 130 may include metadata with image data, which describes other details of the image data, including sensory data (e.g., temperature, ambient brightness) or capture data (e.g., exposure, color temperature, shutter speed, focal length, capture time, etc.). The camera assembly 130 may include one or more cameras capable of capturing image data. In one example, the camera assembly 130 includes one camera and is configured to capture monocular image data. In another example, the camera assembly 130 includes two cameras and is configured to capture stereoscopic image data. In various other implementations, the camera assembly 130 includes multiple cameras, each of which is configured to capture image data.

[0022] The AR device 100 includes a sensor 140 that can capture measurements of the AR device 100 or the environment surrounding the AR device 100. For example, the AR device 100 may include a motion sensor, an accelerometer, a gyroscope, an inertial measurement unit (IMU), a barometer, a positioning system, a thermometer, a light sensor, a light-sensitive sensor (e.g., a camera), etc. The sensor 140 can capture location data describing the location of the AR device 100. The sensor 140 can be used to determine the actual or relative location of the AR device 100 by using a satellite navigation positioning system (e.g., a GPS system, a Galileo positioning system, a global navigation satellite system (GLONASS), a Beidou satellite navigation and positioning system), an inertial navigation system, a dead reckoning system, based on an IP address, by using cellular towers or Wi-Fi hotspots to triangulate or proximity measurements, or other suitable techniques for determining location.

[0023] The object position module 150 determines the target position for the virtual object in the real world. A virtual object is an object generated by a computer and displayed to a user. For example, a virtual object can be a computer-generated character, avatar, creature, or object. The target position of a virtual object is the position where the AR device 100 makes the object appear to be in the real world when displayed to the user. The target position is mainly described herein as a position determined relative to the AR device (i.e., the AR device 100 is used as the origin), but the target position can also be determined as an absolute position. In some embodiments, the target position includes a target orientation of the virtual object. Additionally, the target position can be a 2D position or a 3D position.

[0024] The object position module 150 can determine the target position for the virtual object based on the content instructions. The content instructions are instructions for presenting content to the user through the AR device 100. For example, the content instructions may include a 3D model for the AR device 100 to present the virtual object. Additionally, the content instructions may specify the position of the virtual object in the real world. For example, the content instructions of the virtual object may be used to specify the absolute position of the real world where the virtual object should be located, or may specify the position of the virtual object relative to the user. The content instructions may further specify the target position of the virtual object based on the real-world object. For example, the content instructions may specify that the virtual object should be located in the center of the table or in the window frame. In some embodiments, the content instructions include motion instructions that specify how the virtual object should move. For example, the motion instructions may specify the direction of movement for the virtual object and the distance that the virtual object should move within the time step. The object position module 150 may receive the content instructions from the online system 120 or the data storage 180.

[0025] The object location module 150 may determine a target location for a virtual object based on the sensor data received from the sensor 140. For example, the object location module 150 may determine the location of the AR device 100 based on the location data received from the sensor 140, and determine the target location of the virtual object based on the location of the AR device 100. In some embodiments, the object location module 150 uses the sensor data from the sensor 140 to identify real-world objects of the environment surrounding the AR device 100, and may determine the target location of the object based on the identification or location of these real-world objects. For example, the object location module 150 may identify a table and the location of the table, and may determine the target location so that the virtual object is centered on top of the table.

[0026] The content display module 160 displays content to the user. The content display module 160 may include a display 165 that displays content to the user. The display 165 is a device capable of projecting a 2D pixel array to the user. For example, the display 165 may include a flat display (e.g., an LCD screen, an OLED screen, or a flat touch screen), or may display content to the user through one or more lenses of a headset configured to project light to one or both eyes of the user, so that the user views the real world environment through the lens with one or more virtual objects superimposed on the user's view of the real world environment. The content display module 160 may display image data to the user through the display 165. For example, the content display module 160 may display image data captured by the camera assembly 130, and display the image data to the user through the display 165.

[0027] The content display module 160 displays a virtual object to the user. The content display module 160 may generate a 2D projection of a 3D model of a virtual object to display the virtual object to the user through the display 165. The content display module 160 may generate a 2D projection of the virtual object so that the object seen by the user appears to be at a target position of the virtual object. For example, the content display module 160 may adjust the size, orientation, or position of the 2D projection of the virtual object on the display 165 so that the virtual object appears to be positioned at the target position. The content display module 160 may also additionally use sensor data from the AR device 100 to adjust the 2D projection of the virtual object. For example, the content display module 160 may receive position data describing the position of the AR device 100, and may adjust the size of the 2D projection of the virtual object so that the virtual object is presented at the correct distance from the AR device 100 for the target position of the virtual object. Similarly, the content display module 160 may receive orientation data describing the orientation in the AR device 100, and may adjust the position of the 2D projection of the virtual object so that the virtual object appears to be positioned at the target position for the virtual object.

[0028] The smooth correction module 170 detects whether the virtual object is correctly displayed at the target position, and if the virtual object is not correctly displayed, corrects the position of the virtual object. For example, as the AR device 100 collects more data (e.g., from the sensor 140), the smooth correction module 170 may determine that the position of the virtual object is incorrect and needs to be updated. Similarly, the smooth correction module 170 may determine that the virtual object is displayed to the user based on an inaccurate estimate of the position for the AR device 100, and the virtual object needs to be displayed in a different way so that the virtual object appears to be positioned in the target position. Although the smooth correction module 170 is depicted as a separate module from the object position module 150 in this article, in an alternative embodiment, the smooth correction module 170 is part of the object position module 150.

[0029] When the display position of the virtual object does not correspond to the target position for the virtual object, the smooth correction module 170 can smoothly correct the display of the virtual object. For example, when the AR device 100 receives sensor data from the sensor 140, the smooth correction module 170 can determine the original display position for the virtual object. The original display position of the virtual object is the position where the virtual object was originally displayed in the real world. The smooth correction module 170 can determine that the original position of the virtual object does not correspond to the target position of the virtual object, and update the display of the virtual object so that the virtual object appears to be positioned at the updated position in the real world.

[0030] The smooth correction module 170 can continuously update the display position for the virtual object. For example, the smooth correction module 170 can update the display of the virtual object based on updated sensor data from the sensor 140 or based on new content instructions for the virtual object to execute. In some embodiments, the smooth correction module 170 periodically updates the target position of the virtual object based on a time step. The time step is the time interval of the frequency at which the AR device 100 uses the smooth correction module 150 to update the display of the virtual object. For example, the time step can be based on the refresh rate of the display. The AR device 100 can adjust the time step. For example, when the virtual object is moving, the AR device 100 can shorten the time step, or when the virtual object is relatively stationary, the time step can be extended.

[0031] Methods for smoothly correcting the display of virtual objects are discussed in detail below. For simplicity of explanation, these methods for smoothly correcting the display of virtual objects will be described primarily in the context of updating the position of virtual objects in the reference frame of a non-mobile AR device 100. However, the methods described herein are applicable to smoothly updating the display of virtual objects based on an updated target position for a virtual object or an updated position of the AR device 100.

[0032] In some embodiments, the smoothing correction module 170 updates the position of the virtual object based on an angle threshold. Figure 2 The diagram illustrates that, according to some embodiments, the virtual object position is updated based on an angle threshold. The angle threshold 200 is the maximum of the angles between: (1) the line of sight 220 from the AR device 100 to the original position 230 of the virtual object; and (2) the line of sight 240 from the AR device 100 to the updated position 250 of the virtual object. For example, if the angle 210 between the lines of sight from the AR device 100 to the original position 230 and the target position 260 of the virtual object is 30 degrees, and the angle threshold 200 is 15 degrees, the smooth correction module 170 may update the display position of the virtual object to a position 250 between the original position 230 and the target position 260 in the real world, so that the angle between the original position 230 and the updated position 250 does not exceed 15 degrees. However, if the target position 260 is 10 degrees away from the original position 230, the smooth correction module 170 may display the virtual object at the target position 260 because the target position 260 is within the angle threshold of the original position 230.

[0033] The smooth correction module 170 may select the position that is closest to the target position 260 and does not exceed the angle threshold 200 as the updated position 250. For example, the smooth correction module 170 may use the Euclidean distance to select the updated position 250 that is closest to the target position 260. In some embodiments, the smooth correction module 170 may select the updated position 250 based on the position along the line of sight 240 corresponding to the angle threshold 200. The smooth correction module 170 may select the position that is closest to the target position 260 along the line of sight 240 as the updated position 250. The smooth correction module 170 may consider a set of positions along the line of sight 240 and select the position that is closest to the target position 260.

[0034] The smooth correction module 170 can also use trigonometric functions to identify the position 250 closest to the target position 260. Since the updated position 250 is the point closest to the target position 260 along the line of sight, the line 270 from the target position 260 to the updated position 250 intersects the line of sight 240 corresponding to the angle threshold 200 at a right angle. Therefore, the smooth correction module 170 can use trigonometric functions to identify the updated position 250 based on the angle threshold 200, the original position 230, and the target position 260. Similarly, the smooth correction module 170 can identify the updated position 250 by calculating the parameters (e.g., slope and intercept) of the line of sight 240 corresponding to the angle threshold 200 and calculating the parameters of the line 270 between the target position 260 and the updated position 250 based on the slope of the line of sight 240 and the target position. For example, since the line 270 from the target position 260 to the updated position 250 intersects the sight line 240 corresponding to the angle threshold 200 at a right angle, the slope of the line 270 between the target position 260 and the updated position 250 will be the negative inverse of the slope of the sight line 240. The smoothing correction module 170 can then determine the updated position 250 based on the intersection of the sight line 240 with the line 270 between the target position and the updated position 250.

[0035] The smooth correction module 170 can use the angle threshold 200 to determine how much the display position of the virtual object can be updated within a time step. For example, if the target position 260 is not within the angle threshold of the original position 230, the smooth correction module can continue to update the display position of the virtual object through a series of updated positions 250 in each time step until the virtual object reaches the target position 260.

[0036] Additionally, the smoothing correction module 170 can continuously update the target position 260 based on updated data from the sensor 140 or the camera assembly 130. The smoothing correction module 170 can be used to determine an updated position 250 for the virtual object based on the updated target position 130 in each time step.

[0037] In some embodiments, the smoothing correction module 170 updates the position of the virtual object based on a motion threshold. Figure 3 The diagram illustrates updating the position of a virtual object based on a motion threshold according to some embodiments. The AR device 100 can be used to determine an original position 300 for the virtual object. The AR device 100 can also determine that the virtual object should be displayed so that the virtual object moves 310 a certain distance within a time step. Then, the AR device 100 can determine an original next position 320 for the virtual object based on the original position 300 of the virtual object and the object motion 310.

[0038] The smoothing correction module 170 may determine that the original position 300 is not at the target position 330. For example, the smoothing correction module 170 may determine that the AR device 100 is located at a different position than the position used to determine the original position 300. The smoothing correction module 170 may determine an updated position 340 for the virtual object based on a motion threshold 350. The motion threshold 350 is a maximum distance by which the position of the virtual object can be adjusted based on the motion 310 of the virtual object. For example, the motion threshold 350 may be some proportion of the distance 310 covered by the virtual object within a time step. If the target position 330 is within the motion threshold 350 of the original position 300, the smoothing correction module 170 updates the display position of the virtual object from the original position 300 to the target position 330. However, if the target position 330 is outside the motion threshold 350, the smoothing correction module 170 may be used to determine an updated position 340 for the virtual object that is oriented toward the target position 330 but is within the motion threshold 350 for the virtual object in the original position 300.

[0039] The smooth correction module 170 is used to determine an updated next position 360 for the virtual object. The updated next position 360 displays the position of the virtual object based on the updated position 340 of the virtual object and the movement 310 of the virtual object. In some embodiments, the smooth correction module 170 may update the position of the virtual object from its original position 300 to the updated next position 360 in one step 370. Alternatively, the smooth correction module 170 may update the position of the virtual object from its original position 300 to the updated next position 360 in two steps 380. In a first time step, the smooth correction module 170 may update the position of the virtual object from its original position 300 to the updated position 340, and in a second time step, may update the position of the virtual object from the updated position 340 to the updated next position 360.

[0040] As noted above, although the description above mainly describes correcting the display position of a virtual object based on the motion of the object, the smooth correction module 170 may also use a motion threshold to update the position of the virtual object based on the movement of the AR device 100. For example, the smooth correction module 170 may detect that the AR device 100 is moving or changing direction. The smooth correction module 170 may convert these movements of the AR device 100 into movements of the virtual object relative to the AR device, and may apply the motion threshold to update the position of the virtual object, as described above.

[0041] In some embodiments, the smoothing correction module 170 updates the display position of the virtual object based on a pixel threshold. Figure 4 The diagram illustrates updating the display position of a virtual object based on a pixel threshold according to some embodiments. The pixel threshold is the maximum number of pixels that the virtual object moves when the smoothing correction module 170 updates the position of the virtual object. Figure 4 , the virtual object is depicted in the display 400 of the AR device 100 at the original position 410. The smoothing correction module 170 can determine the original position 400 of the virtual object instead of at the target position 420 for the virtual object. If the target position 420 is within the pixel threshold 430 of the original position 410, the smoothing correction module 170 updates the position of the virtual object to the target position 420. If the target position 420 is outside the pixel threshold 430 of the original position, the smoothing correction module 170 can be used to determine an updated position 440 for the virtual object, which is oriented toward the target position 420, but within the motion threshold 430 for the original position 410 of the virtual object.

[0042] To determine whether the target position 420 is within the pixel threshold 430 of the original position 410, the smooth correction module 170 may compare the pixel projection of the virtual object at the original position 410 with the pixel projection of the virtual object at the correct target position 420. These pixel projections may be pixel projections generated by a pixel projection function on a 2D pixel array representing the display of the AR device 100. The smooth correction module 170 may then compare the pixels within each pixel projection to determine the distance between the pixel projections. If the distance between the pixel projection of the original position 410 and the pixel projection of the target position 420 is less than the pixel threshold 430, the smooth correction module 170 updates the position for the virtual object to the target position 420. If the distance is not within the pixel threshold 430, the smooth position module 170 may determine an updated position 440 between the target position 420 and the original position 410, the updated position being within the pixel threshold 430 and having the closest distance to the correct target position 420.

[0043] To determine the distance between pixel projections, the smooth correction module 170 may identify which pixels of each pixel projection correspond to a portion of a virtual object (e.g., which pixels of each pixel projection correspond to a corner of a box) and compare the positions of the pixels in each pixel projection. The smooth correction module 170 may also compare the pixels closest to each other in each pixel projection to determine the distance between the pixel projections. Similarly, the smooth correction module 170 may compare the pixels farthest from each other in each pixel projection to determine the distance between the pixel projections. In some embodiments, the smooth correction module 170 identifies the center pixel of each pixel projection and compares the distances between these center pixels. For example, the smooth correction module 170 may determine the center of mass of each pixel projection to identify the center pixel of the pixel projection.

[0044] In some embodiments, the pixel threshold 430 is based on the motion of the virtual object. If the content instructions for the virtual object indicate that the virtual object should be depicted as moving, the smoothing correction module 170 can adjust the pixel threshold 430 based on the motion of the virtual object. For example, the pixel threshold 430 can be adjusted based on some proportion of the distance the virtual object moves within the time step. The pixel threshold 430 can be based only on the motion of the virtual object, or can have a baseline value that is increased based on the motion of the virtual object.

[0045] The data storage 180 stores data used by the AR device 100 to provide an AR experience to the user. For example, the data storage 180 may store content data describing the content that the AR device 100 can present to the user. For example, the data storage 180 may store a 3D model of a virtual object that can be presented to the user. Additionally, the data storage 180 may include content instructions for the virtual object. The data storage 180 may also store image data captured by the camera assembly 130 or sensor data captured by the sensor 140.

[0046] The network 110 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof. The network can also include a direct connection between the AR device 100 and the game server 120. In general, the communication between the game server 120 and the AR device 100 can be carried using any type of wired or wireless connection via a network interface, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML, JSON), or protection schemes (e.g., VPN, secure HTTP, SSL).

[0047] The online server 120 can be any computing device and can include one or more processors and one or more computer-readable storage media. The computer-readable storage medium can store instructions that cause the processor to perform operations. The online server 120 can be configured to receive requests for data from the AR device 100 (e.g., via remote procedure calls (RPCs)) and respond to these requests via the network 110. For example, the online server 120 can encode content data in one or more data files and provide these data files to the AR device 100. In addition, the online server 120 can be configured to receive data from the AR device 100 via the network 110 (e.g., the position of the AR device 100, user actions, user input, etc.). The AR device 100 can periodically send data from the AR device 100 to the game server 120.

[0048] In some embodiments, the online server 120 transmits content data to the AR device 100 for presentation to the user. The content data may include a 3D model of a virtual object and content instructions for presenting the 3D model to the user. In some embodiments, the online server 120 receives position data, sensor data, image data, or input data from the AR device 100, and transmits updated content instructions to the AR device 100 based on the received data.

[0049] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent in these systems. Those skilled in the art will recognize that the inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and division of tasks and functions between and among components. For example, the server processes discussed herein can be implemented using a single server or multiple servers working in combination. Databases and applications can be implemented on a single system or distributed on multiple systems. Distributed components can run sequentially or in parallel.

[0050] In addition, in cases where the systems and methods discussed herein access and analyze user personal information or use personal information (such as location information), the user may have the opportunity to control whether the program or feature collects information and whether or how content is received from the system or other applications. No such information or data will be collected or used until the user is provided with clear notice of what information will be collected and how the information will be used. Information will not be collected or used unless the user provides permission, and the user can revoke or modify the permission at any time. Therefore, the user can control how the application or system collects information about the user and how the information is used. In addition, before certain information or data is stored or used, it may be processed in one or more ways to remove personally identifiable information. For example, the user's identity may be processed so that personally identifiable information specific to the user cannot be determined.

[0051] Figure 5 is a flow chart illustrating an example method for correcting the position of a virtual object based on an angle threshold according to some embodiments. Alternative embodiments may include: Figure 5 More, fewer or different steps may be shown, and the steps may be performed differently. Figure 5 Additionally, each of these steps can be automatically performed by the AR device or online server, with or without human intervention.

[0052] The AR device receives 500 sensor data from one or more sensors of the AR device and determines 510 a target position of a virtual object. The AR device displays 520 the virtual object at an original position based on the sensor data. If the received sensor data is inaccurate or outdated, the original position may be different from the target position. The AR device receives 530 updated sensor data and may determine, based on the sensor data, that the original position is not at the target position.

[0053] The AR device determines 540 whether the target position is within an angle threshold of the original position. The angle threshold may be the maximum angle that a line from the AR device to the virtual object may change within a time step. If the target position is within the angle threshold of the original position, the AR device displays 550 the virtual object at the target position. If the target position is not within the angle threshold of the original position, the AR device displays 560 the virtual object at an updated position. The updated position may be a position that is closest to the target position and is not outside the angle threshold. The AR device may update the position of the virtual object to the target position or the updated position within one time step. The AR device may continue to adjust the position of the virtual object in a similar manner in subsequent time steps.

[0054] Figure 6 is a flow chart illustrating an example method for correcting the position of a virtual object based on a motion threshold according to some embodiments. Alternative embodiments may include: Figure 6 More, fewer or different steps may be shown, and the steps may be performed differently. Figure 6 Additionally, each of these steps can be automatically performed by the AR device or online server, with or without human intervention.

[0055] The AR device receives 600 sensor data from one or more sensors of the AR device and determines 610 a target position for a virtual object. The AR device displays 620 the virtual object at an original position based on the sensor data. If the received sensor data is inaccurate or outdated, the original position may be different from the target position. The AR device receives 630 updated sensor data and may determine, based on the sensor data, that the original position is not at the target position.

[0056] The AR device determines 640 whether the target position is within a motion threshold of the original position. The motion threshold may be a maximum distance that the position of the virtual object can be adjusted from the original position to the target position. The motion threshold may be based on the motion of the virtual object. If the target position is within the motion threshold of the original position, the AR device updates 650 the virtual object based on the target position. If the target position is not within the angle threshold of the original position, the AR device determines 660 an updated position for the virtual object. The updated position may be a position that is closest to the target position and is not outside the motion threshold. The AR device updates 670 the virtual object based on the updated position.

[0057] The AR device may update the virtual object by moving the position of the virtual object to a target position or an updated position within a single time step. The AR device may then move the virtual object from the target position or the updated position to a next position during the movement of the virtual object in a subsequent time step. The AR device may also update the virtual object by directly moving the position of the virtual object from its original position to a next position within a single time step.

[0058] Figure 7 is a flow chart illustrating an example method for correcting the position of a virtual object based on a pixel threshold according to some embodiments. Alternative embodiments may include Figure 7 More, fewer or different steps may be shown in the figure, and the order of execution of the steps may be different. Figure 7 Additionally, each of the steps may be automatically performed by the AR device or the online server, regardless of whether there is human intervention.

[0059] The AR device receives 700 sensor data from one or more sensors of the AR device and determines 710 a target position for a virtual object. The AR device displays 720 the virtual object at an original position based on the sensor data. If the received sensor data is inaccurate or outdated, the original position may be different from the target position. The AR device receives 730 updated sensor data and may determine, based on the sensor data, that the original position is not at the target position.

[0060] The AR device determines whether the target position is within a pixel threshold of the original position. The pixel threshold limits the distance that the virtual object position can be corrected (i.e., the maximum pixel distance that the pixel projection of the virtual object can be adjusted). To determine whether the target position is within the pixel threshold of the pixel position, the AR device can compare 740 the pixel projection of the virtual object at the original position with the pixel projection of the virtual object at the target position. The AR device can determine 750 the distance between the pixel projections, and can compare 760 the distance to the pixel threshold to determine whether the original position is within the pixel threshold of the target position.

[0061] If the target position is within the pixel threshold of the original position, the AR device displays 770 the virtual object at the target position. If the target position is not within the pixel threshold of the original position, the AR device displays 780 the virtual object at an updated position. The updated position may be a position that is closest to the target position and is not outside the angle threshold. The AR device may update the position of the virtual object to the target position or the updated position within one time step. The AR device may continue to adjust the position of the virtual object in subsequent time steps in a similar manner.

[0062] Example computing system

[0063] Figure 8 is an example of a computing device architecture according to an embodiment. Figure 8 Depicts a high-level block diagram illustrating the physical components of a computer that may be part or all of one or more entities described herein, according to an embodiment. According to an embodiment, a computer may have Figure 8 additional, fewer, or variant components provided in the Figure 8 A computer 800 is shown, but the figure is intended as a functional description of various features that may be present in a computer system rather than a structural schematic diagram of the implementations described herein. In practice, and as recognized by one of ordinary skill in the art, items shown separately may be combined, and some items may be separated.

[0064] Figure 8 804. Memory 806, storage device 808, keyboard 810, graphics adapter 812, pointing device 814, and network adapter 816 are also coupled to chipset 804. Display 818 is coupled to graphics adapter 812. In one embodiment, the functionality of the chipset 804 is provided by a memory controller hub 820 and an I / O hub 822. In another embodiment, the memory 806 is directly coupled to the processor 802, rather than the chipset 804. In some embodiments, the computer 800 includes one or more communication buses for interconnecting these components. The one or more communication buses optionally include circuits (sometimes referred to as chipsets) that interconnect and control communications between system components.

[0065] The storage device 808 is any non-transitory computer-readable storage medium, such as a hard disk, a compact disk-read only memory (CD-ROM), a DVD, a solid-state storage device or other optical storage, a cassette, a magnetic tape, a magnetic disk storage or other magnetic storage device, a magnetic disk storage device, an optical disk storage device, a flash memory device or other non-volatile solid-state storage device. Such storage devices 808 may also be referred to as persistent memory. The pointing device 814 may be a mouse, a trackball, or other type of pointing device, and is used in conjunction with the keyboard 810 to input data into the computer 800. The graphics adapter 812 displays images and other information on a display 818. The network adapter 816 couples the computer 800 to a local area network or a wide area network.

[0066] The memory 806 stores instructions and data used by the processor 802. The memory 806 may be a non-persistent memory, examples of which include high-speed random access memory such as DRAM, SRAM, DDR RAM, ROM, EEPROM, flash memory.

[0067] As is known in the art, the computer 800 may have Figure 8 800. In addition, the computer 800 may lack certain components shown in the drawings. In one embodiment, the computer 800 as a server may lack a keyboard 810, a pointing device 814, a graphics adapter 812, or a display 818. Moreover, the storage device 808 may be local or remote from the computer 800 (such as, embodied in a storage area network (SAN)).

[0068] As is known in the art, the computer 800 is adapted to execute computer program modules for providing the functions described herein. As used herein, the term "module" refers to computer program logic that is used to provide a specified function. Therefore, a module can be implemented in hardware, firmware, or software. In one embodiment, the program module is stored on a storage device 808, loaded into the memory 806, and executed by the processor 302.

[0069] Additional Notes

[0070] The foregoing description of the embodiments is for illustrative purposes; it is not intended to be exhaustive or to limit the patent rights to the precise pages disclosed. It will be appreciated by those skilled in the relevant art that many modifications and variations are possible in light of the disclosure described above.

[0071] Some parts of the present embodiment describe the embodiment in terms of the algorithm and symbolic representation of the operation of information. The description and representation of these algorithms are usually used by the technical personnel in the field of data processing to effectively convey the essence of their work to other technical personnel in the field. Although these operations are described functionally, computationally or logically, it should be understood that they are implemented by computer programs or equivalent circuits, microcodes or the like. In addition, sometimes for convenience, the arrangement of these operations is referred to as a module without loss of generality. The described operations and their associated modules can be embodied in software, firmware, hardware or any combination thereof.

[0072] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In some embodiments, a software module is implemented by a computer program product comprising one or more computer-readable media containing computer program codes or instructions, which codes or instructions may be executed by a computer processor to perform any or all of the steps, operations or processes described herein. In some embodiments, a computer-readable medium comprises one or more computer-readable media, which individually or collectively comprise instructions, which, when executed by one or more processors, cause one or more processors to individually or collectively perform the steps of instructions stored on one or more computer-readable media. Similarly, a processor comprises one or more processors or processing units, which individually or collectively perform the steps of instructions stored on a computer-readable medium.

[0073] Embodiments may also relate to products produced by the computing process described herein. This product may include information produced by the computing process, wherein the information is stored on a non-transitory, tangible computer-readable storage medium, and may include any embodiment of a computer program product or other data combination described herein.

[0074] In addition, the language used in the specification is primarily selected for readability and didactic purposes, and may not be selected to delimit or limit the subject matter of the invention. Therefore, the scope of the patent right is intended not to be limited by this specific embodiment, but by any claims published based on the application. Accordingly, the disclosure of the embodiment is intended to illustrate rather than limit the scope of the patent right, which is set forth in the claims below.

[0075] As used herein, the terms "consisting of," "consisting of," "including," "comprising," "having," "having," or any other variation thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or device that includes a list of elements is not necessarily limited to only those elements, but may include other elements that are not explicitly listed or inherent to the process, method, article, or device. In addition, unless otherwise expressly provided to the contrary, "or" refers to an inclusive "or" rather than an exclusive "or." For example, the condition "A or B" satisfies any of the following situations: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and A and B are both true (or exist). Similarly, the condition "A, B, or C" satisfies any combination of A, B, and C in which at least one element is true (or exists). As a non-limiting example, the condition "A, B, or C" is satisfied when A and B are true (or exist) and C is false (or does not exist). Similarly, as another non-limiting example, the condition "A, B, or C" is satisfied when A is true (or exists) and B and C are false (or do not exist).

Claims

1. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to: receiving sensor data from one or more sensors of an augmented reality device; determining a target position for a virtual object, wherein the target position for the virtual object comprises a real-world position at which the augmented reality device targets the virtual object to appear to be positioned when viewed by a user through a display of the augmented reality device; displaying the virtual object to a user of the augmented reality device at an original location based on the target location and the received sensor data; receiving updated sensor data from the one or more sensors of the augmented reality device; determining whether the target position is within an angle threshold of the original position based on the updated sensor data, wherein the angle threshold is a maximum angle that a line from the augmented reality device to the virtual object can change within a time step; In response to the target position being outside the angular threshold of the original position, displaying the virtual object to the user at an updated position, wherein the updated position is within the angular threshold of the original position and the updated position is the position closest to the target position.

2. The computer-readable storage medium of claim 1, further storing instructions that cause a processor to determine whether the original position of the virtual object is substantially similar to the target position based on the updated sensor data.

3. The computer-readable storage medium of claim 1, wherein the one or more sensors include one or more of: a motion sensor, an accelerometer, a gyroscope, an inertial measurement unit, a barometer, a positioning system, a thermometer, or a light sensor.

4. The computer-readable storage medium of claim 1 , further storing instructions that cause a processor to: in response to the target position being within the angle threshold of the original position, display the virtual object to the user at the target position.

5. The computer-readable storage medium of claim 1 , wherein the instructions for determining whether the target position is within an angular threshold of the original position comprise: Identifying a first line of sight from the location of the augmented reality device to the original location; identifying a second line of sight from the location of the augmented reality device to the target location; as well as It is determined whether an angle between the first sight line and the second sight line exceeds the angle threshold.

6. The computer-readable storage medium of claim 1 , wherein displaying the virtual object to the user at the updated position comprises: Parameters of a first line are determined, the first line intersecting the target location and intersecting a second line at a right angle, wherein the second line corresponds to the angle threshold.

7. The computer-readable storage medium of claim 1 , wherein displaying the virtual object to the user at the updated position comprises: The updated position is determined based on one or more trigonometric functions and the angle threshold.

8. The computer-readable storage medium of claim 1, wherein the updated position is a position closest to the target position within the angle threshold based on a Euclidean distance between the updated position and the target position.

9. The computer-readable storage medium of claim 1, wherein the augmented reality device comprises a smartphone.

10. The computer-readable storage medium of claim 1, wherein the augmented reality device comprises a headset.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to: receiving sensor data from one or more sensors of an augmented reality device; determining a target position for a virtual object, wherein the target position for the virtual object comprises a real-world position at which the augmented reality device targets the virtual object to appear to be positioned when viewed by a user through a display of the augmented reality device; displaying the virtual object to a user of the augmented reality device at an original location based on the target location and the received sensor data; receiving updated sensor data from the one or more sensors of the augmented reality device; determining, based on the updated sensor data, whether the target position is within a motion threshold of the original position, wherein the motion threshold is a maximum distance by which the position of the virtual object can be adjusted from the original position to the target position, wherein the motion threshold is based on the motion of the virtual object; In response to the target position being outside the motion threshold of the original position: determining an updated position for the virtual object, wherein the updated position is within the motion threshold of the original position and the updated position is a position closest to the target position; The virtual object is updated based on the updated position.

12. The computer-readable storage medium of claim 11, further storing instructions that cause a processor to determine whether the original position of the virtual object is substantially similar to the target position based on the updated sensor data.

13. The computer-readable storage medium of claim 11, wherein the one or more sensors include one or more of: a motion sensor, an accelerometer, a gyroscope, an inertial measurement unit, a barometer, a positioning system, a thermometer, or a light sensor.

14. The computer-readable storage medium of claim 11, further storing instructions that cause a processor to: in response to the target position being within the motion threshold of the original position, display the virtual object to the user at the target position.

15. The computer-readable storage medium of claim 11, wherein displaying the virtual object at the original position further comprises displaying the virtual object based on a motion instruction for the virtual object, wherein the motion instruction comprises a motion direction of the virtual object and a motion distance that the virtual object moves within a time step.

16. The computer-readable storage medium of claim 15, wherein the motion threshold is based on the motion distance that the virtual object moves within a time step.

17. The computer-readable storage medium of claim 15, further storing instructions which, when executed by a processor, cause the processor to display the virtual object at a next position based on the updated position and the motion instruction for the virtual object, wherein the next position is a position of the virtual object after it moves the motion distance from the updated position in the motion direction.

18. The computer-readable storage medium of claim 17, wherein displaying the virtual object at the next location comprises moving the virtual object directly from the original location to the next location within one time step.

19. The computer-readable storage medium of claim 17, wherein displaying the virtual object at the next location comprises: displaying the virtual object at the updated position at a first time step; as well as The virtual object is displayed at the next position at a second time step, the second time step being subsequent to the first time step.

20. The computer-readable storage medium of claim 11, wherein the augmented reality device comprises a smartphone.

Images