Method and system for surface detection and tracking in an augmented reality session

Abstract

Techniques for detecting and tracking surface planes in an AR session are described. In an example, a computer system generates a sparse 3D representation of a real-world environment in an AR session and performs a surface detection process based on the sparse 3D representation. Points in the sparse 3D representation are tracked in a next sparse 3D representation. The number of these points that belong to the detected surface plane is determined. If the number of points exceeds a predefined threshold, the surface plane is tracked based on the points. Otherwise, the surface plane is no longer tracked and the next sparse 3D representation can be input to the surface detection process to perform a new surface detection in the AR session.

Description

Technical Field

[0001] The present invention generally relates to the field of augmented reality (AR), and more particularly to a method and system for detecting and tracking surface planes in an AR session. Background Art

[0002] Augmented reality (AR) overlays virtual content on the user's view of the real world. With the development of AR software development kits (SDKs), the mobile industry has brought smartphone AR into the mainstream. AR SDKs typically provide six degrees-of-freedom (6DoF) tracking capabilities. Users can scan the environment using the smartphone's camera, and the smartphone performs visual inertialodometry (VIO) in real time. After continuously tracking the camera pose, virtual objects can be placed into the AR scene to create an illusion that real and virtual objects are fused together.

[0003] Presenting virtual objects in an AR scene may involve detecting a surface plane on which to place the virtual objects. Such surface planes may be tracked over time to update the relative positions of the virtual objects. Summary of the invention

[0004] In an example, a method is implemented by a computer system in an AR session. The method includes determining a plurality of points belonging to a surface plane, the surface plane being detected based on a first multidimensional representation of a real-world environment, the first multidimensional representation being generated in the AR session based on a first image of the real-world environment. The method also includes determining a second multidimensional representation of the real-world environment, the second multidimensional representation being generated in the AR session based on a second image of the real-world environment, the second image being generated after the first image. The method also includes determining the number of points in the plurality of points that are also included in the second multidimensional representation. The method also includes comparing the number with a threshold, and when the number is greater than the threshold, tracking the surface plane in the AR session based on the second multidimensional representation.

[0005] In an example, the first multidimensional representation includes a first point cloud, and the first point cloud corresponds to a feature detected in the real-world environment. Each point in the first point cloud corresponds to a feature and is associated with a point identifier. In this example, determining the number includes: determining that a first point belonging to the surface plane is associated with a first point identifier; based on the first point identifier, determining that the first point is included in a second point cloud, and the second point cloud corresponds to the second multidimensional representation; when determining that the first point is included in the second point cloud, adding 1 to the number. In this example, the method may also include: in a first feature detection iteration, detecting the feature based on the first image and the first inertial measurement unit (IMU) data; generating the first point cloud based on the detected feature; and assigning a different point identifier to each point in the first point cloud. In addition, the method may also include: in a second feature detection iteration, detecting a first feature among the above features based on the above second image and the second IMU data; in the above second feature detection iteration, detecting a second feature that was not detected in the above first feature detection iteration based on the above second image and the above second IMU data; generating a second point cloud, the above second point cloud including a first point corresponding to the above first feature and a second point corresponding to the above second feature, the above first point also being included in the above first point cloud; maintaining a first point identifier assigned to the above first point; and assigning a second point identifier to the above second point.

[0006] In the example, the method further includes: generating the first multidimensional representation based on the execution of a simultaneous localization and mapping (SLAM) process, wherein the first image is input to the SLAM process; and inputting the first multidimensional representation into a random sampling consensus (RANSAC) process. The surface plane is detected based on the execution of the RANSAC process using the first multidimensional representation. In the example, the method may further include: assigning a different point identifier to each point in the first multidimensional representation; determining that the first point in the first multidimensional representation belongs to the surface plane; and associating the surface plane with the first point identifier of the first point. In addition, determining the quantity includes: determining that the first point is included in the second multidimensional representation based on the first point identifier; and adding 1 to the quantity when determining that the first point is included in the second multidimensional representation.

[0007] In an example, tracking the surface plane includes: determining a first point set from the plurality of points belonging to the surface plane, wherein the first point set is present in the second multidimensional representation; and updating a plane function of the surface plane based on the first point set. In this example, tracking the surface plane also includes: determining a second point set belonging to the surface plane, wherein the second point set is present in the second multidimensional representation but not in the first multidimensional representation; and associating the second point set with the surface plane.

[0008] In an example, a computer system includes one or more processors and one or more memories, storing a program, which, when executed by the one or more processors, configures the computer system to perform an operation. The operation includes determining a plurality of points belonging to a surface plane, the surface plane being detected based on a first multidimensional representation of a real-world environment, the first multidimensional representation being generated in an augmented reality (AR) session based on a first image of the real-world environment. The operation also includes determining a second multidimensional representation of the real-world environment, the second multidimensional representation being generated in the AR session based on a second image of the real-world environment, the second image being generated after the first image. The operation also includes determining the number of points in the plurality of points that are also included in the second multidimensional representation. The operation also includes comparing the number with a threshold, and when the number is greater than the threshold, tracking the surface plane in the AR session based on the second multidimensional representation.

[0009] In the example, the program is also executed to configure the computer system: determine a plurality of second points belonging to the surface plane from the second multidimensional representation; determine a third multidimensional representation of the real-world environment, the third multidimensional representation being generated in the AR session based on a third image of the real-world environment, the third image being generated after the first image; determine that the plurality of second points also include a second number of second points in the first multidimensional representation; compare the second number with the threshold; and determine that the surface plane is no longer tracked in the AR session based on the second number being less than the threshold. In the example, the program is also executed to configure the computer system: determine that the total number of planar surfaces tracked based on the comparison with the threshold is less than a second threshold; and based on the total number being less than the second threshold, detect the second surface plane by inputting at least the third multidimensional representation into a random sampling consensus (RANSAC) process. In addition, the second surface plane is detected based on a set of constraints of the RANSAC process, the set of constraints specifying a minimum number of points in the third multidimensional representation. In addition, the set of constraints further specifies a maximum distance between a point belonging to the second planar surface and a camera of the computer system, wherein the camera is used to generate the third image.

[0010] In an example, one or more non-transitory computer storage media store instructions that, when executed on a computer system, cause the computer system to perform operations. The operations include determining a plurality of points belonging to a surface plane, the surface plane being detected based on a first multidimensional representation of a real-world environment, the first multidimensional representation being generated in an augmented reality (AR) session based on a first image of the real-world environment. The operations also include determining a second multidimensional representation of the real-world environment, the second multidimensional representation being generated in the AR session based on a second image of the real-world environment, the second image being generated after the first image. The operations also include determining the number of points in the plurality of points that are also included in the second multidimensional representation. The operations also include comparing the number with a threshold, and when the number is greater than the threshold, tracking the surface plane in the AR session based on the second multidimensional representation.

[0011] In an example, the operation further includes generating the first multidimensional representation based on execution of a simultaneous localization and mapping (SLAM) process, wherein the first image is input to the SLAM process; and inputting the first multidimensional representation into a random sampling consensus (RANSAC) process. The surface plane is detected based on execution of the RANSAC process using the first multidimensional representation. In the example, the operation further includes: assigning a different point identifier to each point in the first multidimensional representation; determining that a first point in the first multidimensional representation belongs to the surface plane; and associating the surface plane with the first point identifier of the first point. In addition, determining the quantity includes: determining that the first point is included in the second multidimensional representation based on the first point identifier; and adding 1 to the quantity when determining that the first point is included in the second multidimensional representation.

[0012] In the example, the above threshold is set to a value greater than or equal to 3.

[0013] The present invention has many advantages over conventional techniques. For example, embodiments of the present disclosure relate to methods and systems for surface detection and tracking techniques, wherein these techniques provide substantial processing improvements over conventional techniques. Depending on the configuration of the computer system implementing the techniques of the present disclosure, the delay associated with the detection and tracking of surface planes can be reduced by a factor of two or more. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0015] Figure 1 An example of a computer system for AR applications according to at least one embodiment of the present disclosure is shown, the computer system including a camera and an inertial measurement unit (IMU).

[0016] Figure 2 An example of tracking a surface plane in accordance with at least one embodiment of the present disclosure is shown.

[0017] Figure 3 An example of a computing component for detecting and tracking a surface plane in accordance with at least one embodiment of the present disclosure is shown.

[0018] Figure 4 An example of initial surface detection and tracking iterations in an AR session in accordance with at least one embodiment of the present disclosure is shown.

[0019] Figure 5 An example of a current surface detection and tracking iteration in an AR session is shown in accordance with at least one embodiment of the present disclosure.

[0020] Figure 6 An example of surface plane tracking based on point identifiers and associations with surface planes is shown in accordance with at least one embodiment of the present disclosure.

[0021] Figure 7 An example of determining whether to repeat a surface detection process according to at least one embodiment of the present disclosure is shown.

[0022] Figure 8 An example of a process for surface plane detection and tracking according to at least one embodiment of the present disclosure is shown.

[0023] Fig. 9 An example of components of a computer system in accordance with at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] In the following description, various embodiments will be described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may also be implemented without these specific details. In addition, well-known features may be omitted or simplified in order not to confuse the described embodiments.

[0025] Embodiments of the present disclosure are particularly related to detecting and tracking surface planes in augmented reality (AR) sessions. In an example, a computer system includes a camera, an inertial measurement unit (IMU) sensor, and an AR module. The AR module is responsible for feature detection and tracking processes, surface detection processes, and surface tracking processes. During an AR session, the camera generates an image of a real-world environment. Image data of the image and IMU data generated by the IMU sensor are input to a feature detection and tracking process, which in turn outputs a multi-dimensional representation of the real-world environment. The multi-dimensional representation of the real-world environment is input to a surface detection process, which in turn outputs data identifying the detected surface plane. The data includes a plurality of points belonging to the above-mentioned plane in the above-mentioned multi-dimensional representation. The tracking process associates the detected plane with an identifier of the point. Such an identifier is referred to herein as a point identifier. Subsequently, the next image is generated and input to the feature detection and tracking process together with the next IMU data, which in turn outputs the next multi-dimensional representation of the real-world environment. Using the point identifier, the tracking process determines a specific point that was previously determined to belong to the above-mentioned surface plane and exists in the next multi-dimensional representation. If the number of such specific points exceeds a first predefined threshold, the tracking system updates the plane function of the surface plane based on the next multi-dimensional representation. Otherwise, the surface tracking process outputs a decision to no longer track the surface plane. In this case, the total number of tracked surface planes is updated (e.g., reduced by at least one) and compared with a second predefined threshold. Only if it is less than the second predefined threshold, the next multi-dimensional representation is input to the surface detection process to perform a new detection on one or more surface planes.

[0026] Illustratively, consider an example of a real-world environment that includes an open cabinet with four horizontal shelves. In an AR session executed on a smartphone, different virtual objects representing books, vases, and / or other items will be placed on these shelves. The images generated by the camera of the AR session can be input to a simultaneous localization and mapping (SLAM) process at a frame rate of 20-30 frames per second (FPS). The SLAM process is an example of a feature detection and tracking process. The processing of each input image corresponds to a surface detection and tracking iteration (e.g., the surface detection and tracking iterations have the same 20-30 iterations per second rate (iterations persecond, ITS)). In the initial surface detection and tracking iteration, the SLAM process outputs an initial point cloud of the real-world environment. The initial point cloud is an example of a sparse three-dimensional representation of the real-world environment and includes points corresponding to features of the open cabinet. The number of points in the initial point cloud can be between one hundred and one thousand. Each point is assigned a point identifier, which is a string that uniquely identifies the point. The initial point cloud is input to a random sample consensus (RANSAC) process, which is an example of a feature detection and tracking process. The RANSAC process outputs parameters that define four horizontal surface planes, each corresponding to one of the four shelves. For each horizontal surface plane, the points belonging to the horizontal surface plane are identified, and the point identifiers of these points are associated with the horizontal surface plane. In the next surface detection and tracking iteration (e.g., an iteration using the next image input), the SLAM process outputs the next point cloud of the real world environment. The SLAM process tracks the features of the above-mentioned open cabinet. Some of the previously detected features will be detected again. Some of the previously detected features are no longer detected. In addition, some new features that were not previously detected are currently detected. Therefore, some points in the next point cloud correspond to previously detected features and have existing point identifiers. The remaining points in the next point cloud correspond to newly detected features and are assigned new point identifiers. Some points in the initial point cloud no longer exist in the next point cloud. For each horizontal surface plane, the surface tracking process determines point identifiers associated with the horizontal surface plane, compares the point identifiers with point identifiers of points in the next point cloud, and determines a match. If the number of matching point identifiers (e.g., matching points are points that were determined to belong to the horizontal surface plane in the initial surface detection and tracking iteration and are included in the next point cloud) exceeds a first predefined threshold (e.g., five), the surface tracking process updates the plane function of the horizontal surface plane using the matching points.New point identifiers for points belonging to the horizontal surface plane are associated with the horizontal surface plane. However, if the number of matching point identifiers is less than a first predefined threshold, the horizontal surface plane is no longer tracked. In this case, if the total number of horizontal surface planes to be tracked in the next surface detection and tracking iteration drops below a second predefined threshold (e.g., three), the next point cloud is input to the RANSAC process for a new surface plane detection. The surface detection and tracking iterations are repeated, where the input and output of each iteration depends on whether to continue using the point identifiers to track the surface plane or whether to perform the RANSAC process again.

[0027] Embodiments of the present disclosure provide many technical advantages over conventional techniques for detecting and tracking surfaces. By using a multi-dimensional representation of the real-world environment, particularly a sparse representation such as a point cloud as input to a surface detection process (such as a RANSAC process), the processing of detecting surface planes is greatly reduced. In addition, by using point identifiers to intelligently determine when to perform the surface detection process and avoid its execution when it is not necessary, additional processing can be saved. For example, depending on the configuration of a computer system implementing the technology of the present disclosure, the delay associated with the detection and tracking of surface planes can be reduced by a factor of two or more. A reduction of at least five has been observed.

[0028] Figure 1 An example of a computer system 110 according to at least one embodiment of the present disclosure is shown, the computer system 110 including a camera 112 and an inertial measurement unit (IMU) sensor 114 for an AR application. The AR application may be implemented by an AR module 116 of the computer system 110. Typically, the camera 112 generates an image of a real-world environment, which includes, for example, a real-world object 130. The camera 112 may also include a depth sensor that generates depth data about the real-world environment, wherein the data includes, for example, a depth map showing the depth of the real-world object 130 (e.g., the distance between the depth sensor and the real-world object 130). The IMU sensor 114 may include a gyroscope and an accelerometer, among other components, and may output IMU data, which includes, for example, the orientation of the computer system 110. The image data generated by the camera 112 in an AR session and the IMU data generated by the IMU sensor 114 in an AR session may be used to determine the 6DoF pose (e.g., position along the X, Y, and Z axes and rotation around each of the above axes) of the computer system 110 relative to the real-world environment.

[0029] After initialization of the AR session (which may include calibration and tracking), the AR module 116 renders an AR scene 120 of the real-world environment in the AR session, which may be presented as a graphical user interface (GUI) on the display of the computer system 110. The AR scene 120 displays real-world object representations 122 of real-world objects 130. In addition, the AR scene 120 also displays virtual objects 124 that do not exist in the real-world environment. In order to place the virtual objects 124 on the real-world object representations 122 in an appropriate manner, the AR module 116 may detect a surface plane 126 corresponding to the real-world object representations 122 and position the virtual objects relative to the surface plane 126 (e.g., place the virtual objects 124 on the surface plane 126). The surface plane 126 may be a horizontal plane, a vertical plane, or any other plane that may be angled and may correspond to a visible surface of the real-world object 130.

[0030] In the example, the computer system 110 represents a suitable user device, which includes one or more graphics processing units (GPUs), one or more general purpose processors (GPPs), and one or more memories storing computer-readable instructions in addition to the camera 112 and the IMU sensor 114. These instructions can be executed by at least one processor to perform various functions of the embodiments of the present disclosure. For example, the computer system 110 can be any one of a smartphone, a tablet computer, an AR headset, or a wearable AR device.

[0031] The AR module 116 may be implemented as dedicated hardware and / or a combination of hardware and software (e.g., a general purpose processor and computer readable instructions stored in a memory and executable by the general purpose processor). In addition to initiating an AR session and performing VIO, the AR module 116 may detect features of the real world environment, detect surface planes based on the detected features, and track the detected features and detected surface planes to correctly render the AR scene 120. For example, the AR module 116 implements the feature detection and tracking process, the surface detection process, and the surface tracking process as a set of program codes.

[0032] Figure 2An example of tracking a surface plane 202 according to at least one embodiment of the present disclosure is shown. As shown, the surface plane 202 is tracked based on different images generated in an AR session. The first image 210 shows the surface plane 202. The second image 220 also shows the surface plane 202, but the surface plane 202 is partially occluded by another object. In the third image 230, the surface plane 202 is completely or almost completely occluded by another object.

[0033] Different techniques can be used to track the surface plane 202. In one technique, a surface detection process can be performed for each of the images 210-230. In other words, the surface detection process is repeated for each of the images 210-230. However, because the surface detection process is constantly repeated, this technique may require a large amount of processing. In addition, some processing may be wasted because, for example, the surface plane 202 is completely occluded in the third image 230 and does not need to be tracked.

[0034] In another example technique, a surface detection process may be performed on the first image 210 to detect the surface plane 202. From this point on, a high-resolution representation of the real-world environment may be generated from the remaining images 220-230 to track the surface plane 202. Such a high-resolution representation is typically not the output of a feature detection and tracking process (rather, it outputs a low-resolution representation). Therefore, a significant amount of processing may still be required.

[0035] In contrast, embodiments of the present disclosure allow for the reuse of low-resolution representations of feature detection and tracking processes to detect surface plane 202 and associate point identifiers with surface plane 202. Point identifiers are used to track surface planes. Therefore, a significant amount of processing is saved in the detection and tracking of surface plane 202.

[0036] Figure 3 An example of a computing component for detecting and tracking a surface plane according to at least one embodiment of the present disclosure is shown. The computing component can be implemented as an AR module of a computer system (e.g. Figure 1 The program code in the AR module 116).

[0037] As shown, the computing components include a feature detection and tracking process 310, a surface detection process 320, and a surface tracking process 330. The feature detection and tracking process 310 detects and tracks features of the real world environment. The surface detection process 320 detects surface planes. The surface tracking process 330 tracks detected surface planes.

[0038] In an example, the input of the feature detection and tracking process 310 includes image data and IMU data. The output of the feature detection and tracking process 310 includes a sparse multi-dimensional representation (e.g., a three-dimensional representation) of the real world environment, such as a point cloud. For example, the feature detection and tracking process 310 is implemented as a SLAM process involving a particle filter, an extended Kalman filter, a covariance intersection, and / or other SLAM algorithms.

[0039] The input of the surface detection process 320 includes the sparse multidimensional representation output from the feature detection and tracking process 310. The output of the surface detection process 320 includes the parameters of the detected surface plane. For example, the surface plane is defined as a plane function with the following equation: Ax+By+Cz+D=0, and the output includes A, B, C, D parameters. In the example, the surface detection process 320 is implemented as a RANSAC process, which involves iterative estimation of these parameters. Specifically, the RANSAC process selects three points (e.g., x, y, and z) from the sparse multidimensional representation as candidate surface planes, solves the plane function equation to determine the A, B, C, and D parameters, and calculates the distance of each remaining point of the sparse multidimensional representation to the candidate surface plane. For points whose distance is less than a predefined threshold distance, these points are considered to belong to the candidate surface plane. If the number of points belonging to the candidate surface plane exceeds a minimum number (which can be set as a constraint of the RANSAC process), such as 5 or other values, the candidate surface plane is declared as a detected surface plane. Otherwise, the candidate surface plane is removed from the set of candidate surface planes and is not declared as a detected surface plane.

[0040] The input of the surface tracking process 330 includes the output of the feature detection and tracking process 310 (e.g., the sparse multidimensional representation) and the output of the surface detection process 320 (e.g., the parameters of the detected surface planes). The output of the surface tracking process 330 includes point identifiers of the points in the sparse multidimensional representation, associations of some of the point identifiers with the detected surface planes, and a tracking decision indicating whether to track the detected surface planes using the associated point identifiers or by performing the surface detection process again. An example implementation of the surface tracking process 330 is further described in conjunction with the following figure.

[0041] Figure 4 An example of an initial surface detection and tracking iteration 400 in an AR session according to at least one embodiment of the present disclosure is shown. Here, feature detection and tracking process 310, surface detection process 320, and surface tracking process 330 may be used.

[0042] Prior to the initial surface detection and tracking iteration 400, no features of the real-world environment have been detected in the AR session. At the end of the initial surface detection and tracking iteration 400, one or more surface planes are detected based on the detected features.

[0043] In the example, an image of the real world environment is generated and input to a feature detection and tracking process 310 (in Figure 4 402). IMU data 404 is also input to feature detection and tracking process 310. Feature detection and tracking process 310 generates and outputs a sparse three-dimensional (3D) representation 412 of the real-world environment, such as a point cloud comprising one hundred to one thousand points. Each point corresponds to a detected feature. Surface tracking process 330 (or another process of the AR session) assigns a point identifier to each point in the sparse 3D representation 412. For example, the point identifier can be the same as an identifier of the corresponding feature (referred to herein as a feature identifier), or can be derived from the corresponding feature identifier, where the feature identifier is used by the feature detection and tracking process to track the detected features in subsequent iterations.

[0044] The sparse 3D representation 412 is input to the surface detection process 320 and the surface tracking process 330. The surface detection process 320 detects one (or more) surface planes based on the points in the sparse 3D representation 412 and outputs a plane function 422 (e.g., A, B, C, and D parameters) for each detected surface plane. The plane function 422 is input to the surface tracking process 330.

[0045] The surface tracking process 330 (or another process of the AR session) determines points that belong to the surface plane and are included in the sparse 3D representation 412, determines point identifiers for those points, and associates the point identifiers with the surface plane (in Figure 4 4. As shown in FIG. 4A and 4B as surface planes and point identifier associations 432. For example, the point identifiers and A, B, C, and D parameters are stored in a data structure, where entries of the data structure indicate associations 432 (e.g., using a table, where a row of the table indicates association 432 by listing the point identifiers and A, B, C, and D parameters).

[0046] Figure 5 An example of a current surface detection and tracking iteration 500 in an AR session according to at least one embodiment of the present disclosure is shown. The current surface detection and tracking iteration 500 is an iteration after a previous surface detection and tracking iteration (e.g., initial surface detection and tracking iteration 400). Here, feature detection and tracking process 310, surface detection process 320, and surface tracking process 330 may continue to be used.

[0047] Before the current surface detection and tracking iteration 500 starts, one (or similarly multiple) surface planes have been detected in a previous surface detection and tracking iteration. Also before the iteration 500 starts, there already exists an association between the surface plane and the point identifiers of the points belonging to the surface plane in the previous sparse 3D representation (e.g., sparse 3D representation 412).

[0048] The current surface detection and tracking iteration 500 may begin by inputting image data 502 and IMU data 504 into a feature detection and tracking process 310, which in turn outputs a sparse 3D representation 512 of the real-world environment (e.g., a point cloud, where each point corresponds to a detected feature). As described above, the surface detection and tracking iteration may be performed at a specific ITS rate corresponding to the FPS rate. The image data 502 and IMU data are input at the FPS rate.

[0049] The sparse 3D representation 512 is input to the surface tracking process 330. For features that have been previously detected by the feature detection and tracking process 310 and have existing feature identifiers, the surface tracking process 330 determines corresponding points and their point identifiers (e.g., these points existed in the 3D representation 412 and still exist in the 3D representation 512). For features newly detected by the feature detection and tracking process 310 in the current surface detection and tracking iteration 500, the surface tracking process 330 determines corresponding new points in the sparse 3D representation 512 (e.g., these points did not exist in the 3D representation 412 and now appear in the 3D representation 512) and assigns new point identifiers to these new points. These new point identifiers are not yet associated with surface planes. The surface tracking process 330 continues to perform surface plane tracking 532 using the point identifiers (from the previous and current surface detection and tracking iterations) and the existing associations with surface planes (e.g., from the previous surface detection and tracking iteration), which will be further illustrated in the following figure.

[0050] Figure 6An example of surface plane tracking based on point identifiers and associations with surface planes according to at least one embodiment of the present disclosure is shown. Surface plane tracking can be performed by a surface tracking process 330 to track a surface plane (or similarly, multiple surface planes) in a current surface detection and tracking iteration (e.g., current surface detection and tracking iteration 500). In particular, multiple data are input to the surface tracking process 330. The first input includes a current sparse 3D representation of a real-world environment in the current surface detection and tracking iteration (e.g., sparse 3D representation 512 in the current surface detection and tracking iteration 500). For clarity of explanation, a point belonging to the current sparse 3D representation and its associated point identifier are referred to herein as a current point and a current point identifier, respectively. The second input includes a surface plane and point identifier association 620 from a previous surface detection and tracking iteration (e.g., surface plane and point identifier association 432 from initial surface detection and tracking iteration 400), which indicates an existing association between a surface plane and a point identifier corresponding to a point included in a previous sparse 3D representation from a previous surface detection and tracking iteration (e.g., sparse 3D representation 412 from initial surface detection and tracking iteration 400). For clarity of explanation, points belonging to a previous sparse 3D representation and their point identifiers are referred to herein as previous points and previous point identifiers, respectively. A previous point identifier corresponding to a previous point belonging to a surface plane is already associated with the surface plane, and for clarity of explanation, is referred to herein as an existing association.

[0051] In the example, the surface tracking process 330 performs point tracking 630 based on the input. In particular, point tracking 630 involves determining the number of existing points that belong to the surface plane and are included in the current sparse 3D representation (e.g., also the current point). For example, point tracking 630 initializes a counter and uses a point identifier. Point tracking 630 determines whether a previous point identifier matches one of the current point identifiers, wherein the previous point identifier has an existing association with the surface plane (thereby indicating that the corresponding previous point was determined to belong to the surface plane in the previous surface detection and tracking iteration). If there is a match, point tracking 630 increments the counter by one. If there is no match, the counter is not incremented, and point tracking 630 determines that the corresponding previous point no longer exists in the current sparse 3D representation. The matching process is repeated for previous point identifiers that have an existing association with the surface plane. At the end of the matching, the value of the counter indicates the number of previous points that belong to the surface and still exist in the current 3D sparse representation (e.g., also the current point).

[0052] The surface tracking process 330 performs a comparison 640 of the number (e.g., the value of the counter) to a predefined threshold (e.g., set to a value of at least 3; in a specific example, the predefined threshold is set to 5). If the number is greater than (greater means higher than or equal to) the predefined threshold, the surface tracking process 330 determines that the surface plane is to be tracked in the current surface detection and tracking iteration and performs a surface plane update 650 accordingly. Otherwise, the surface tracking process 330 determines that the surface plane no longer needs to be tracked in the current surface detection and tracking iteration and therefore removes tracking of the surface plane. In this case, a determination is made as to whether the surface detection process is to be repeated, which will be performed in Figure 7 Further described in .

[0053] The surface plane update 650 may include updating a plane function of the surface plane. For example, the surface tracking process 330 performs multiple point determinations. The first point determination is related to a previous point. In particular, a previous point (or similarly, multiple previous points) no longer exists in the current sparse 3D representation and therefore no longer belongs to the surface plane. For this previous point (and similarly other previous points, if applicable), the surface plane update 650 involves removing an existing association between the surface plane and the previous point identifier corresponding to the previous point (e.g., see above in conjunction with Figure 4 The data structure described above is used to delete the previous point identifier from the data structure). The second point determination involves a new point. In particular, the new point is a current point (e.g., or similarly, multiple current points) that does not exist in the previous sparse 3D representation but is included in the current sparse 3D representation. The new point has a new point identifier, where the new point identifier has not yet been associated with a surface plane. For this new point (e.g., or similarly, multiple new points), the surface tracking process 330 determines whether the new point belongs to the surface plane (e.g., based on the distance of the new point to the surface plane being less than a predefined threshold). If so, the surface tracking process 330 generates a new association between the new point identifier and the surface plane (e.g., refer to the above in conjunction with Figure 4 The data structure described by , adds the new point identifier to the data structure).

[0054] In an example, after the first point is determined and before the second point is determined, the surface plane update 650 may involve updating the plane function (e.g., A, B, C, and D parameters) of the surface plane based on the remaining previous points determined to still belong to the surface plane. Additionally or alternatively, after the second point is determined, the surface plane update 650 may involve updating the plane function based on the new point determined to belong to the surface plane and / or based on all points determined to belong to the surface plane (e.g., previous points and new points).

[0055] Figure 7 An example of determining whether to repeat the surface detection process according to at least one embodiment of the present disclosure is shown. Figure 6 In the current surface detection and tracking iteration described, when the tracking of a surface plane is removed, the surface tracking process 330 can determine whether to repeat the surface detection process. If no surface plane is removed in the current surface detection and tracking iteration, the surface tracking process 330 does not need to make the above determination. In general, if the total number of surface planes whose tracking is removed in the current surface detection is below a predefined threshold, the surface detection process is repeated. Otherwise, the surface detection process is not repeated.

[0056] As shown, multiple inputs are used. The first input includes the current sparse 3D representation 710 in the current surface detection and tracking iteration. The second input includes the surface planes detected from the previous surface detection and tracking iteration (in Figure 7 720 ), for example, a planar function of the surface plane. Based on the input, the surface tracking process 330 performs surface plane tracking 730 .

[0057] Surface plane tracking 730 involves maintaining a counter indicating the total number of surface planes to be tracked in the current surface detection and tracking iteration. In particular, the value of the counter is equal to the total number of surface planes tracked in the previous surface detection and tracking iteration. Figure 6 The output of the determination performed in indicates whether to remove tracking of individual surface planes in the current surface detection and tracking iteration. For each surface plane to be removed, the surface plane tracking 730 decrements a counter by one. The result value of the counter indicates the total number of surface planes to be tracked in the current detection and tracking iteration. The surface tracking process 330 performs a comparison 740 of the total number (e.g., the result value of the counter) with a predefined threshold. The threshold can be set to a percentage (e.g., eighty percent) of the total number of surface planes detected by the surface detection process 320 the last time the process 320 was performed. If it is greater than the predefined threshold, the surface tracking process 330 determines that the surface detection process 320 does not need to be repeated. In this case, the surface tracking process 330 sets the previous surface plane to be tracked as the current surface plane 750 (and these current surface planes 750 can be as shown in the figure). Figure 6 Otherwise, the surface tracking process 330 may ignore the previous surface plane 720 and continue to trigger 760 new surface detections by repeating the surface detection process 320. Thus, the 3D representation 710 may be input to the surface detection process 320.

[0058] In an example, when the surface detection process 320 is repeated, a set of constraints may be imposed on the process 320. For example, the set of constraints includes a first constraint that specifies a minimum number of points (e.g., five points) that should belong to a candidate surface plane in the sparse 3D representation 710 if the surface detection process 320 is to set the candidate surface plane as the detected surface plane. The set of constraints includes a first constraint that specifies a maximum distance between a point belonging to the candidate surface plane and a camera of a computer system, wherein the camera (e.g., camera 112) generates an image corresponding to the sparse 3D representation 710.

[0059] Figure 8 An example of a process for surface plane detection and tracking according to at least one embodiment of the present disclosure is shown. The process is described in conjunction with a computer system, the computer system being Figure 1 An example of a computer system 110 of the present invention may be provided. Some or all of the operations of the process may be implemented by specific hardware on the computer system, and / or may be implemented as computer-readable instructions stored on a non-transitory computer-readable medium of the computer system. As stored, the computer-readable instructions represent a programmable module that includes code that can be executed by a processor of the computer system. Execution of such instructions configures the computer system to perform the corresponding operation. Each programmable module in conjunction with the processor represents a device for performing the corresponding operation. Although the operations are shown in a particular order, it should be understood that this particular order is not necessary, and one or more operations may be omitted, skipped, performed in parallel, and / or reordered.

[0060] In an example, the process begins at operation 802, where a computer system generates an initial multi-dimensional representation of a real-world environment in an AR session. For example, a camera of the computer system generates an image of the real-world environment. Image data of the image and IMU data of an IMU system of the computer system are input to a feature detection and tracking process (e.g., a SLAM process) of an AR module of the computer system. In an initial surface detection and tracking iteration, the feature detection and tracking process outputs a sparse 3D representation of the real-world environment, such as a point cloud, as an example of an initial multi-dimensional representation.

[0061] In an example, the process includes operation 804, where the computer system assigns a point identifier. For example, each point included in the initial multidimensional representation corresponds to a feature included in the real-world environment and detected by the feature detection and tracking process. Points corresponding to the features are assigned point identifiers, where the point identifiers are derived from feature identifiers of the features.

[0062] In an example, the process includes operation 806, where the computer system detects a surface plane (or similarly, a plurality of surface planes). For example, the initial multi-dimensional representation is input to a surface detection process, such as a RANSAC process. The output of the process indicates a plane function of the surface plane.

[0063] In an example, the process includes operation 808, in which the computer system associates the surface plane (or similarly, multiple surface planes) with the point identifier. For example, for each point belonging to the surface plane and included in the initial multidimensional representation, the computer associates the corresponding point identifier with the surface plane. This association can be stored in a data structure.

[0064] In an example, the process includes operation 810, in which the computer system generates a current multi-dimensional representation. This operation is similar to operation 802, except that new images and new IMU data are input to the feature detection and tracking process and correspond to the current surface detection and tracking iteration.

[0065] In an example, the process includes operation 812, in which the computer system tracks point identifiers in the current multidimensional representation. Tracking involves determining whether a feature in a previous (e.g., initial) multidimensional representation is no longer included in the current multidimensional representation (e.g., a removed feature) or is still included in the current multidimensional representation (e.g., a remaining feature). Tracking also involves determining whether a feature not included in the previous multidimensional representation is included in the current multidimensional representation (e.g., a new feature). The point identifiers of the points corresponding to the removed features (e.g., the removed points) are also removed. The point identifiers of the points corresponding to the remaining features (e.g., the remaining points) are retained. New point identifiers are assigned to points corresponding to the new features (e.g., the new points).

[0066] In an example, the process includes an operation 814, in which the computer system determines the number of points that belong to the surface plane (or similarly, belong to each surface plane detected in a previous surface detection and tracking iteration) and are included in the current multi-dimensional representation. For example, point identifiers associated with the surface plane are matched with point identifiers corresponding to points of the current multi-dimensional representation. The total number of matches represents the number of points.

[0067] In an example, the process includes operation 816, where the computer system compares the number of points to a first predefined threshold. If the number is greater than the first predefined threshold, operation 818 is performed. Otherwise, operation 820 is performed.

[0068] In an example, the process includes an operation 818, in which the computer system updates the surface plane (or similarly, the plurality of surface planes detected in the previous surface detection and tracking iteration, as applicable). For example, the computer system determines the points that belonged to the surface plane and were removed, the points that belonged to the surface plane and were retained, and the new points that now belong to the surface plane. For the point identifiers of the removed points that previously belonged to the surface plane, their association with the surface plane is removed. For the point identifiers of the new points that belonged to the surface plane, their association with the surface plane is added.

[0069] In an example, the process includes operation 820, in which the computer system stops tracking the surface plane in the current surface detection and tracking iteration. Therefore, no surface plane update is performed.

[0070] In an example, the process includes operation 822, where the computer system determines the total number of surface planes tracked in the current surface detection and tracking iteration. For example, each time it is determined that a surface plane is no longer tracked as shown in operation 816, a counter representing the total number is decremented.

[0071] In the example, the process includes operation 824, where the computer system compares the total number to a second predefined threshold. If the number is greater than the second predefined threshold, the computer system continues to track the surface plane based on the point identifiers, as indicated by looping back to operation 810. Otherwise, the computer system repeats the surface detection process, as indicated by looping back to operation 806.

[0072] Fig. 9 An example of components of a computer system 900 according to at least one embodiment of the present disclosure is shown. The computer system 900 is an example of the computer system described above. Although these components are shown as belonging to the same computer system 900, the computer system 900 may also be distributed.

[0073] The computer system 900 includes at least a processor 902, a memory 904, a storage device 906, an input / output (I / O) peripheral device 908, a communication peripheral device 910, and an interface bus 912. The interface bus 912 is used to communicate, send, and transmit data, control, and commands between the various components of the computer system 900. The memory 904 and the storage device 906 include computer-readable storage media, such as RAM, ROM, electrically erasable programmable read-only memory (EEPROM), hard disk drives, CD-ROMs, optical storage devices, magnetic storage devices, electronic non-volatile computer storage devices (such as flash memory), and other tangible storage media. Any such computer-readable storage medium can be used to store instructions or program codes for implementing various aspects of the present disclosure. The memory 904 and the storage device 906 also include computer-readable signal media. Computer-readable signal media include propagated data signals containing computer-readable program codes. Such propagated signals take any of a variety of forms, including but not limited to electromagnetic, optical, or any combination thereof. Computer readable signal media include any computer readable media that is not a computer readable storage media and that can communicate, propagate, or transport a program for use in conjunction with the computer system 900 .

[0074] In addition, the memory 904 includes an operating system, a program, and an application program. The processor 902 is used to execute stored instructions, and includes, for example, a logic processing unit, a microprocessor, a digital signal processor, and other processors. The memory 904 and / or the processor 902 can be virtualized and can be hosted in another computer system such as a cloud network or a data center. The I / O peripherals 908 include user interfaces, such as keyboards, screens (e.g., touch screens), microphones, speakers, other input / output devices, and computing components, such as graphics processing units, serial ports, parallel ports, universal serial buses, and other input / output peripherals. The I / O peripherals 908 are connected to the processor 902 by any port coupled to the interface bus 912. The communication peripherals 910 are used to facilitate communication between the computer system 900 and other computer systems through a communication network, and include, for example, a network interface controller, a modem, wireless and wired interface cards, antennas, and other communication peripherals.

[0075] Although the subject matter has been described in detail for its specific embodiments, it should be understood that those skilled in the art, after obtaining an understanding of the foregoing, can easily produce changes, variations and equivalents to these embodiments. Therefore, it should be understood that the present disclosure is presented for the purpose of illustration rather than limitation, and does not exclude the inclusion of modifications, variations and / or additions to the subject matter that are obvious to ordinary technicians. Skills in this field. In fact, the methods and systems described herein can be embodied in a variety of other forms. In addition, without departing from the spirit of the present disclosure, various omissions, substitutions and changes can be made to the form of the methods and systems described herein. The attached claims and their equivalents are intended to cover such forms or modifications that fall within the scope and spirit of the present disclosure.

[0076] Unless expressly stated otherwise, it should be understood that terms such as "process," "compute," "determine," and "identify" used in the discussion of this specification refer to the actions or processes of a computer system (e.g., one or more computers or similar electronic computer systems or devices) that manipulates or transforms data represented as physical electronic or magnetic quantities in the memory, registers, or other information storage devices, transmission devices, or display devices of a computing platform.

[0077] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computer system may include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computer systems include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computer system from a general-purpose computing device to a dedicated computing device that implements one or more embodiments of the present subject matter. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software used to program or configure a computer system.

[0078] Embodiments of the methods disclosed herein can be performed in the operation of such a computer system. The order of the blocks presented in the above examples can be changed, for example, the blocks can be reordered, combined and / or decomposed into sub-blocks. Certain blocks or processes can be executed in parallel.

[0079] Conditional language used herein, such as "may," "could," "might," "for example," and the like, unless expressly stated otherwise or otherwise understood in the context of use, is generally intended to convey that some examples include and other examples do not include certain features, elements, and / or steps. Thus, such conditional language generally does not mean that one or more examples require features, elements, and / or steps in any way, or that one or more examples must include features, elements, and / or steps for determining whether such features, elements, and / or steps are included or will be performed in any particular example, with or without author input or prompting.

[0080] The terms "including", "having", etc. are synonymous and are used inclusively in an open manner and do not exclude other elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusiveness (not exclusiveness), for example, when used to connect a list of elements, the term "or" means one, some or all of the elements in the list. "For" or "configured to" used herein refers to open and inclusive language, and does not exclude devices used or configured to perform additional tasks or steps. In addition, the use of "based on" means openness and inclusiveness, because the process, step, calculation or other action "based on" one or more listed conditions or values ​​may actually be based on additional conditions or beyond the listed values. Similarly, the use of "based at least in part" means openness and inclusiveness, because the process, step, calculation or other action "based at least in part on" one or more listed conditions or values ​​can be based on additional conditions or values ​​outside the listed in practice. The titles, lists and numbers included here are only for ease of explanation and are not meant to be limiting.

[0081] The various features and processes described above can be used independently of each other, or can be used in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular order, and the blocks or states associated therewith may be performed in other appropriate orders. For example, the blocks or states described may be performed in an order different from that specifically disclosed, or multiple blocks or states may be combined in a single block or state. Example blocks or states may be performed serially, in parallel, or in some other manner. Blocks or states may be added to or deleted from the disclosed examples. Similarly, the example systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed examples.

Claims

1. A method implemented by a computer system in an augmented reality (AR) session, the method comprising: determining a plurality of points belonging to a surface plane, wherein the surface plane is detected based on a first multi-dimensional representation of a real-world environment, the first multi-dimensional representation generated in an AR session based on a first image of the real-world environment; determining a second multi-dimensional representation of the real-world environment, the second multi-dimensional representation generated in the AR session based on a second image of the real-world environment, the second image generated after the first image; determining a number of points in the plurality of points that are also included in the second multidimensional representation; comparing the amount to a threshold value; and When the number is greater than the threshold, tracking the surface plane in the AR session based on the second multi-dimensional representation; Wherein, the method further comprises: determining that the total number of surface planes tracked based on the comparison with the threshold is less than a second threshold; and Based on the total being less than the second threshold, detecting a second surface plane by inputting at least a third multidimensional representation into a random sample consensus (RANSAC) process, wherein the third multidimensional representation is generated in the AR session based on a third image of the real-world environment, the third image being generated after the first image.

2. The method according to claim 1, wherein: The first multi-dimensional representation includes a first point cloud corresponding to features detected in the real-world environment, and each point in the first point cloud corresponds to a feature and is associated with a point identifier.

3. The method according to claim 2, wherein: Determining a number of points in the plurality of points that are also included in the second multidimensional representation comprises: determining that a first point belonging to the surface plane is associated with a first point identifier; determining, based on the first point identifier, that the first point is included in a second point cloud, the second point cloud corresponding to the second multi-dimensional representation; and When it is determined that the first point is included in the second point cloud, the number is increased by 1.

4. The method according to claim 2, further comprising: In a first feature detection iteration, detecting the feature based on the first image and first inertial measurement unit (IMU) data; generating the first point cloud based on the detected features; as well as A different point identifier is assigned to each point in the first point cloud.

5. The method according to claim 4, further comprising: In a second feature detection iteration, detecting a first feature of the features based on the second image and the second IMU data; In the second feature detection iteration, based on the second image and the second IMU data, detecting a second feature that was not detected in the first feature detection iteration; generating a second point cloud, the second point cloud comprising a first point corresponding to the first feature and a second point corresponding to the second feature, the first point also being included in the first point cloud; maintaining a first point identifier assigned to the first point; and A second point identifier is assigned to the second point.

6. The method according to claim 1, further comprising: generating the first multi-dimensional representation by performing a simultaneous localization and mapping (SLAM) process, wherein the first image is input to the SLAM process; and The first multi-dimensional representation is input into a random sample consensus (RANSAC) process, wherein the surface plane is detected by performing the RANSAC process using the first multi-dimensional representation.

7. The method according to claim 6, further comprising: assigning a distinct point identifier to each point in the first multidimensional representation; determining that a first point in the first multidimensional representation belongs to the surface plane; as well as The surface plane is associated with a first point identifier of the first point.

8. The method according to claim 7, wherein: Determining a number of points in the plurality of points that are also included in the second multidimensional representation comprises: Based on the first point identifier, determining that the first point is included in the second multidimensional representation; and Upon determining that the first point is included in the second multidimensional representation, the number is increased by 1.

9. The method according to claim 1, wherein: Tracking the surface plane includes: determining a first set of points from the plurality of points belonging to the surface plane, wherein the first set of points is present in the second multidimensional representation; and A planar function of the surface plane is updated based on the first set of points.

10. The method according to claim 9, wherein: Tracking the surface plane further comprises: determining a second set of points belonging to the surface plane, the second set of points being in the second multidimensional representation but not in the first multidimensional representation; and The second set of points is associated with the surface plane.

11. A computer system comprising: one or more processors; as well as One or more memories storing a program, which, when executed by the one or more processors, configures the computer system to: determining a plurality of points belonging to a surface plane, wherein the surface plane is detected based on a first multi-dimensional representation of a real-world environment, the first multi-dimensional representation generated in an augmented reality (AR) session based on a first image of the real-world environment; determining a second multi-dimensional representation of the real-world environment, the second multi-dimensional representation generated in the AR session based on a second image of the real-world environment, the second image generated after the first image; determining a number of points in the plurality of points that are also included in the second multidimensional representation; comparing the amount to a threshold value; and When the number is greater than the threshold, tracking the surface plane in the AR session based on the second multi-dimensional representation; Wherein, the program is also executed to configure the computer system: determining that the total number of surface planes tracked based on the comparison with the threshold is less than a second threshold; and Based on the total being less than the second threshold, detecting a second surface plane by inputting at least a third multidimensional representation into a random sample consensus (RANSAC) process, wherein the third multidimensional representation is generated in the AR session based on a third image of the real-world environment, the third image being generated after the first image.

12. The computer system according to claim 11, wherein: The program is also executed to configure the computer system to: determining a plurality of second points belonging to the surface plane from the second multidimensional representation; determining a third multi-dimensional representation of the real-world environment; determining a second number of second points in the plurality of second points that also include second points in the first multidimensional representation; comparing the second number to the threshold; Based on the second number being less than the threshold, determining that the surface plane is no longer tracked in the AR session.

13. The computer system according to claim 12, wherein: The second surface plane is detected based on a set of constraints of the RANSAC process, the set of constraints specifying a minimum number of points in the third multidimensional representation.

14. The computer system of claim 13, wherein: The set of constraints also specifies a maximum distance between a point belonging to the second surface plane and a camera of the computer system, wherein the camera is used to generate the third image.

15. One or more non-transitory computer storage media storing instructions that, when executed on a computer system, cause the computer system to perform operations comprising: Determine a number of points that belong to a surface plane, where detecting the surface plane based on a first multi-dimensional representation of a real-world environment, the first multi-dimensional representation generated in an augmented reality (AR) session based on a first image of the real-world environment; determining a second multi-dimensional representation of the real-world environment, the second multi-dimensional representation generated in the AR session based on a second image of the real-world environment, the second image generated after the first image; determining a number of points in the plurality of points that are also included in the second multidimensional representation; comparing the amount to a threshold value; as well as When the number is greater than the threshold, tracking the surface plane in the AR session based on the second multi-dimensional representation; The operation further includes: determining that the total number of surface planes tracked based on the comparison with the threshold is less than a second threshold; and Based on the total being less than the second threshold, detecting a second surface plane by inputting at least a third multidimensional representation into a random sample consensus (RANSAC) process, wherein the third multidimensional representation is generated in the AR session based on a third image of the real-world environment, the third image being generated after the first image.

16. The one or more non-transitory computer storage media of claim 15, wherein: The operations also include: generating the first multi-dimensional representation by performing a simultaneous localization and mapping (SLAM) process, wherein the first image is input to the SLAM process; and The first multi-dimensional representation is input into a random sample consensus (RANSAC) process, wherein the surface plane is detected by performing the RANSAC process using the first multi-dimensional representation.

17. The one or more non-transitory computer storage media of claim 16, wherein: The operations also include: assigning a distinct point identifier to each point in the first multidimensional representation; determining that a first point in the first multidimensional representation belongs to the surface plane; and The surface plane is associated with a first point identifier of the first point.

18. The one or more non-transitory computer storage media of claim 17, wherein: Determining a number of points in the plurality of points that are also included in the second multidimensional representation comprises: Based on the first point identifier, determining that the first point is included in the second multidimensional representation; and Upon determining that the first point is included in the second multidimensional representation, the number is increased by 1.

19. The one or more non-transitory computer storage media of claim 15, wherein: The threshold is set to a value greater than or equal to 3.

Images