Progressive 3D point cloud segmentation of objects and background from tracking sessions

By using a fuzzy segmentation method in augmented reality applications, the object-background segmentation of the tracking session is progressively learned, solving the automation problem of point cloud object-background segmentation and achieving efficient and reliable point cloud object-background segmentation and object tracking without human intervention.

CN113811919BActive Publication Date: 2024-11-15INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080034965.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-05-21
Filing Date
2020-05-06
Publication Date
2024-11-15
Estimated Expiration
2040-05-06

AI Technical Summary

Technical Problem

Existing technologies struggle to automate and reliably perform point cloud object-background segmentation tasks, resulting in solutions that require manual intervention and are not scalable.

Method used

By using a fuzzy segmentation method in augmented reality applications, the object-background segmentation is progressively learned by tracking the session, and the object and background points in the point cloud are automatically segmented using a probabilistic update mechanism for point cloud points.

Benefits of technology

It achieves automatic and reliable point cloud object-background segmentation without human intervention, improving the accuracy of object detection and tracking, and continuously improving over time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113811919B_ABST
    Figure CN113811919B_ABST
Patent Text Reader

Abstract

Embodiments of the present systems and methods may provide techniques that provide automatic, reliable performance of point cloud cleanup tasks. Embodiments may provide the ability to incrementally learn object-background segmentation from tracking sessions in augmented reality (AR) applications. 3D point clouds may be used for tracking in AR applications by matching points from the cloud to regions in live video. 3D points with many matches are more likely to be part of an object, while 3D points with few matches are more likely to be part of the background. Advantages of this approach include that no manual work is required to perform the segmentation, and because objects are tracked in multiple environments, the results may continue to improve over time.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] The present invention relates to techniques for generating augmented reality content by automating point cloud cleanup using fuzzy demarcation of points corresponding to objects or background.

[0002] A 3D point cloud is a standard format for saving 3D models, which can include thousands of points, each with its position and color. Typically, a point cloud is created during 3D reconstruction (a computational pipeline that computes the 3D geometry of a model of an object or scene given a video of the object or scene). Typically, when reconstructing a 3D object from a video, the resulting point cloud contains not only the object, but also some background elements, such as the table on which the object is placed, the wall, the floor, and other objects seen on the video. For many applications of 3D models, it is important to have a clean point cloud that lists only the points that belong to the object. Therefore, the task of clearing the parts of the model that do not belong to the object is very important. This task is sometimes called "segmentation of 'object' and 'background'".

[0003] One method for segmentation is to manually delete background points using an interactive application that can mark background areas and remove them. Tools such as Blender and MeshLab are examples of this approach, which require investment in manual work and time and are therefore not scalable. A second approach is to use some kind of heuristic, for example, thresholding the distance from the camera or calculating compact blobs. These methods will fail in the case where the object matches assumptions about its size and distance from background elements.

[0004] Therefore, there is a need for techniques that provide automatic and reliable performance of the point cloud object-background segmentation task. Summary of the invention

[0005] Embodiments of the present systems and methods may provide techniques for providing automatic, reliable performance of point cloud segmentation tasks. Embodiments may provide the ability to incrementally learn object-background segmentation from tracking sessions in augmented reality (AR) applications. 3D point clouds may be used for tracking in AR applications by matching points from the cloud to regions in live video. 3D points with many matches are more likely to be part of an object, while 3D points with few matches are more likely to be part of the background. Advantages of this approach include that no manual work is required to perform the segmentation, and because objects are tracked in multiple environments, the results can continue to improve over time.

[0006] For example, in one embodiment, a method for generating augmented reality content may be implemented in a computer, the computer comprising a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, and the method may include: tracking a 3D model of an object in a scene in a video stream at a computer system as a camera moves around the scene, wherein the tracking comprises determining, for each point in each frame in the video stream, whether the point corresponds to an object or whether the point corresponds to a background, at the computer system, dividing each of a plurality of points into a point corresponding to an object or a point corresponding to the background based on a probability that the point corresponds to the object and a probability that the point corresponds to the background, wherein for each point, the higher the probability that the point corresponds to the object, the more frequently the tracking identifies the point as an inlier, and for each point, the higher the probability that the point corresponds to the background, the more frequently the tracking identifies the point as an outlier, and generating augmented reality content based on the divided 3D model of the object at the computer system.

[0007] In an embodiment, tracking may include using a partitioned 3D model of the object to estimate the pose of the camera relative to the 3D model of the object. The 3D model of the object may be initially partitioned using a default setting of probabilities for each point, and for each frame, the probability of each point is updated based on whether the tracking determines whether the point corresponds to the object or to the background. The default setting for the probability of each point may be .5, and the probability may be increased when the tracking matches the point to a pixel in the video frame, and the probability may be decreased when the tracking fails to match the point to any of the pixels in the video frame. The 3D model of the object in the scene may include a 3D point cloud including a plurality of points including at least some points corresponding to the object and at least some points corresponding to the background of the scene. The video stream may be obtained from a camera moving around the scene.

[0008] In one embodiment, a system for generating augmented reality content may include a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform tracking of a 3D model of an object in a scene in a video stream as a camera moves around the scene, wherein the tracking includes determining, for each point in each frame, whether the point corresponds to an object or whether the point corresponds to a background, dividing each of a plurality of points into a point corresponding to an object or a point corresponding to the background based on a probability that the point corresponds to the object and a probability that the point corresponds to the background, wherein for each point, the higher the probability that the point corresponds to the object, the higher the frequency with which the tracking determines that the point corresponds to the object, and for each point, the higher the probability that the point corresponds to the background, the higher the frequency with which the tracking determines that the point corresponds to the background, and generating augmented reality content based on the divided 3D model of the object.

[0009] In an embodiment, a computer program product for generating augmented reality content may include non-transitory computer-readable storage having program instructions embodied therein, which are executable by a computer to cause the computer to perform a method, the method comprising: tracking a 3D model of an object in a scene in a video stream as a camera moves around the scene, wherein the tracking comprises determining, for each point in each frame, whether the point corresponds to an object or whether the point corresponds to a background, dividing each of a plurality of points into a point corresponding to an object or a point corresponding to the background based on a probability that the point corresponds to the object and a probability that the point corresponds to the background, wherein for each point, the higher the probability that the point corresponds to the object, the higher the frequency with which the tracking determines that the point corresponds to the object, and for each point, the higher the probability that the point corresponds to the background, the higher the frequency with which the tracking determines that the point corresponds to the background, and generating augmented reality content based on the divided 3D model of the object.

[0010] BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The details of the present invention, both as to its structure and operation, may best be understood by reference to the accompanying drawings in which like reference numerals and designations refer to like elements.

[0012] Figure 1 An exemplary diagram showing a system in which embodiments of the systems and methods of the present invention may be implemented.

[0013] Figure 2 is an exemplary flow chart of an embodiment of a process according to an embodiment of the present system and method.

[0014] Figure 3 An example of a 3D model according to an embodiment of the present systems and methods is shown.

[0015] Figure 4 is an exemplary diagram of using segmentation information to estimate the pose of a camera relative to a model according to an embodiment of the present system and method.

[0016] Figure 5 An exemplary diagram of using segmentation information to estimate a pose of a camera relative to a model according to an embodiment of the present system and method, wherein the segmentation information is updated.

[0017] Figure 6 is an exemplary block diagram of a computer system in which the processes involved in the embodiments described herein may be implemented. DETAILED DESCRIPTION

[0018] Embodiments of the present system and method may provide techniques for providing automatic and reliable execution of point cloud object-background segmentation tasks. Embodiments may provide the ability to perform automatic segmentation of 3D point clouds divided into object and background portions by progressively learning object-background segmentation from tracking sessions in augmented reality (AR) applications. In AR applications, a 3D model of one or more objects to be displayed in a scene in a video stream may be generated and displayed on the scene or on an object in the scene. In order to properly display AR content, the position of the camera relative to the scene and relative to the object in the scene on which the AR content will be displayed may be determined and tracked as the camera moves. A model of an object in a scene may be generated using 3D reconstruction of an image or video stream to generate a 3D point cloud of the object. The 3D point cloud may be used for tracking in AR applications by matching points from the cloud with areas in a live video.

[0019] Typically, a generated 3D model including a 3D point cloud cannot be used as is because the point cloud may include points that are located on the background of the scene rather than on the object to be tracked. A form of "cleaning" of the image or video stream may be applied to remove the background or to divide the points into points that are part of the object to be tracked and points that are not part of the object to be tracked. Embodiments of the systems and methods of the present invention may automatically perform this cleaning by dividing the points into points that are part of the object to be tracked and points that are not part of the object to be tracked. Such automatic division may provide improved performance.

[0020] For example, an embodiment may provide a method for progressive segmentation of a 3D point cloud for objects and background. An embodiment may use a tracking session to progressively associate points to object and background classes. An embodiment may use fuzzy segmentation instead of binary segmentation, and the segmentation may be considered a "side effect" of the tracking process. 3D points that have many matches to pixels in the video are considered to be inliers for tracking an object over time and are more likely to be part of the object, while 3D points that have few matches to pixels in the video are more likely to be part of the background. Advantages of this approach may enable improved object detection and tracking when objects are relocated in different environments, and may further include that no manual work is required to perform the segmentation, and because objects are tracked in multiple environments, the results may continue to improve over time.

[0021] Figure 1 An exemplary system 100 in which embodiments of the present systems and methods may be implemented is shown in FIG. Figure 1As shown in the example of , system 100 may include platform 101, which may be or include a camera that captures video images of three-dimensional (3D) objects 104 as the camera moves through multiple viewpoints 102A-H to form a video stream 103. Platform 101 can move around along a path or trajectory 108. Video stream 103 can be transmitted to image processing system 106, where the processes involved in embodiments of the present systems and methods can be implemented. For example, according to embodiments of the present systems and methods, image processing system 106 can perform 3D reconstruction by generating a 3D point cloud of a scene and / or objects in the scene. In Figure 3 An example of such a model 300 of an object is shown in . After generating a 3D point cloud of a scene including the model 300, the points in the 3D point cloud may be divided into points 302 in the object represented by the model 300 and points 304 belonging to the background.

[0022] Figure 2 An embodiment of a process 200 of operation of an embodiment of the system and method of the present invention is shown. Process 200 begins at 202, where a 3D point cloud may be generated. For example, a video stream 103 may be received, and a 3D point cloud of a model for an object may be reconstructed from the video stream 103, having n points. Each point may be assigned a probability of being part of an object in a scene in the video stream 103. In an embodiment of fuzzy partitioning, each point i in the 3D point cloud may be assigned a probability of belonging to an object, denoted as p i Then, 1-p i is the probability of belonging to the background. The probability pi of point i being part of an object can be initialized to, for example, 50%. The probabilities of all points in the point cloud can be formed as a vector. At 204, given a point cloud and a probability vector P = [p0, p1 ... p n ], can be obtained by replacing i All points >T (threshold) are considered to belong to the object to perform segmentation.

[0023] At 206, in order to use the point cloud for tracking in an augmented reality (AR) session, each point may be assigned some descriptors of its visual appearance from different viewpoints. The descriptors may be computed from the video used to reconstruct the point cloud. At 208, the 3D point cloud may be used to track a model of an object in an AR session. Figure 4As shown, for each frame 402, the tracker 404 can estimate the pose 406 of the camera relative to the model 410 using the segmentation information 408, and can compute matches with points in the AR video stream 103 from the descriptors of the points. For example, the estimation can be based on matching visual descriptors, such as scale-invariant feature transform (SIFT), which can associate 3D points with image points. The segmentation can be used to select which 3D points to consider. For example, only p can be used. i >T. Alternatively, for example, tracking may be performed based on sampled points (based on the probability of the sampled points). Because the tracker may output an estimated camera pose, the result of the tracking may be the position of the object in the video stream 103. Matches that support the estimated pose may be considered as inliers of the pose, while other matches may be considered as outliers.

[0024] At 210, the 3D points belonging to the inner point match can make their p i The value increases, and the points that belong to the outlier matching can make their p i At 212, over time, the process adjusts p i value, so that points belonging to the object can have high p i value, while background points may have low p i At 214, the determined partitioning may be updated to improve the partitioning. In an embodiment, 3D-2D matching statistics, which may be internal data in the tracker, may be used to improve the partitioning. For example, 3D points belonging to object parts may be considered generally "accurately" matched, while points belonging to the background may be considered "poorly" matched. Figure 5 As shown, in an embodiment, for each point, the model tracker 502 may output its matching error 504 and whether it contributes to the pose estimation (inlier point - outlier point). That is, the model tracker 502 may indicate which points are actually used for tracking (inlier points) and which points are not (outliers). The partition update 506 may be performed by increasing the inlier point p i The current partition 508 is updated by adding the outlier pi value and reducing the outlier pi value, and this can be used to update the partition information 510, which can then be used in the model tracker 502. Thus, information about which points are actually used for tracking and which points are not used for tracking can be used to update the partition, which can then be used to improve tracking.

[0025] Figure 6An exemplary block diagram of a computer system 600 is shown in which the processes involved in the embodiments described herein may be implemented. The computer system 600 may be implemented using one or more programmed general-purpose computer systems, such as embedded processors, systems on a chip, personal computers, workstations, server systems, and minicomputers or mainframe computers, or in a distributed networked computing environment. The computer system 600 may include one or more processors (CPUs) 602A-602N, input / output circuits 604, network adapters 606, and memory 608. The CPUs 602A-602N execute program instructions to perform the functions of the present communication systems and methods. Typically, the CPUs 602A-602N are one or more microprocessors, such as INTEL processor. Figure 6 An embodiment in which the computer system 600 is implemented as a single multi-processor computer system is shown, wherein multiple processors 602A-602N share system resources, such as memory 608, input / output circuits 604, and network adapter 606. However, the present communication systems and methods also include embodiments in which the computer system 600 is implemented as multiple networked computer systems, which may be single-processor computer systems, multi-processor computer systems, or a combination thereof.

[0026] Input / output circuitry 604 provides the ability to input data to or output data from computer system 600. For example, input / output circuitry may include input devices such as keyboards, mice, touch pads, trackballs, scanners, analog-to-digital converters, etc., output devices such as video adapters, monitors, printers, etc., and input / output devices such as modems, etc. Network adapter 606 interfaces device 600 with a network 610. Network 610 may be any public or private LAN or WAN, including but not limited to the Internet.

[0027] The memory 608 stores program instructions executed by the CPU 602 and data used and processed by the CPU 602 to perform the functions of the computer system 600. The memory 608 may include, for example, electronic memory devices such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc., and electromechanical storage such as a disk drive, a tape drive, an optical drive, etc., which may use an integrated drive electronics (IDE) interface or a variant or enhancement thereof, such as enhanced IDE (EIDE) or ultra direct memory access (UDMA), or an interface based on a small computer system interface (SCSI) or a variant or enhancement thereof, such as fast SCSI, wide SCSI, fast and wide SCSI, etc., or a serial advanced technology attachment (SATA), or a variant or enhancement thereof, or a fiber channel arbitrated loop (FC-AL) interface.

[0028] The contents of memory 608 may vary depending on the functions that computer system 600 is programmed to perform. Figure 6 In the illustrated example, exemplary memory contents of routines and data representing implementations of the above-described processes are shown. However, those skilled in the art will recognize that these routines, along with memory contents associated with these routines, may not be included in one system or device, but may be distributed among multiple systems or devices based on well-known engineering considerations. The present communication systems and methods may include any and all such arrangements.

[0029] exist Figure 6 In the example shown in , the memory 608 may include a point cloud generation routine 612, a partition determination routine 614, a tracking routine 616, a partition update 618, video stream image data 620, and an operating system 622. The point cloud routine 612 may include a software routine to generate a 3D point cloud for a model of an object, which can be reconstructed from the video stream image data 620, which can assign a probability that each point in the point cloud is part of the object, and which can assign descriptors of the visual appearance from different viewpoints. The partition determination routine 614 may include a software routine to determine and update the probability that each point is part of the object based on information received from the tracking routine 616. The tracking routine 616 may include a software routine to track the object position in the video stream image data 620, output the estimated camera pose, and output information indicating whether each point is used to track the object, which can be used by the partition determination routine 614 to determine and update the probability that each point is part of the object. The operating system 634 may provide overall system functionality.

[0030] like Figure 6As shown, the communication system and method may include implementations on one or more systems that provide multi-processor, multi-tasking, multi-process and / or multi-threaded computing, as well as implementations on systems that only provide single-processor, single-threaded computing. Multi-processor computing involves using more than one processor to perform computing. Multi-tasking computing involves using more than one operating system task to perform computing. A task is an operating system concept that refers to a combination of an executing program and bookkeeping information used by the operating system. Whenever a program is executed, the operating system creates a new task for it. The task is like an envelope for the program because it identifies the program with a task number and attaches other bookkeeping information to it. Including Linux, and Many operating systems are capable of running many tasks simultaneously and are referred to as multitasking operating systems. Multitasking is the ability of an operating system to execute more than one executable file at the same time. Each executable file runs in its own address space, meaning that the executable files cannot share any memory in their memory. This has advantages because it is impossible for any program to disrupt the execution of any other program running on the system. However, programs cannot exchange any information except through the operating system (or by reading files stored on the file system). Multiprocess computing is similar to multitasking computing in that the terms task and process are often used interchangeably, although some operating systems make a distinction between the two.

[0031] The present invention may be a system, method and / or computer program product of any possible degree of integration of technical details. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to perform various aspects of the present invention. The computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device.

[0032] The computer-readable storage medium may be, for example but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a protruding structure in a groove with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal emitted by a wire.

[0033] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0034] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming languages ​​or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can perform computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit so as to perform various aspects of the present invention.

[0035] The present invention will be described below with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer readable program instructions.

[0036] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which causes the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium in which the instructions are stored includes a manufactured product containing instructions for implementing aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0037] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in or in multiple boxes in the flowchart and / or block diagram.

[0038] The flow charts and block diagrams in the accompanying drawings show the architecture, functions and operations of the possible implementations of the system, method and computer program product according to different embodiments of the present invention. To this end, each box in the flow chart or block diagram may represent a module, segment or part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may not occur in the order marked in the figure. For example, depending on the functions involved, the two blocks shown in succession can actually be executed substantially simultaneously, or these blocks can sometimes be executed in reverse order. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0039] Although specific embodiments of the present invention have been described, those skilled in the art will appreciate that there are other embodiments that are equivalent to the described embodiments. Therefore, it should be understood that the present invention is not limited to the specific illustrated embodiments, but only to the scope of the appended claims.

Claims

1. A method for dividing a 3D point cloud into object and background parts implemented in a computer, the computer comprising a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method comprising: tracking, at the computer, a 3D model of an object in the scene in the video stream as the camera moves about the scene, wherein the tracking includes determining, for each point in each frame of the video stream, whether the point corresponds to the object or whether the point corresponds to the background; At the computer, each of the plurality of points is divided into a point corresponding to the object or a point corresponding to the background based on a probability that the point corresponds to the object and a probability that the point corresponds to the background, wherein for each point, the higher the probability that the point corresponds to the object, the more frequently the point is identified as an inlier point by the tracking, and for each point, the higher the probability that the point corresponds to the background, the more frequently the point is identified as an outlier by the tracking, and wherein the 3D model of the object is initially divided using a default setting of the probability for each point, and for each frame, the probability of each point is updated based on whether the tracking determines that the point corresponds to the object or the background; as well as At the computer, augmented reality content is generated based on the segmented 3D model of the object. 2 . The method of claim 1 , wherein the tracking comprises using a segmented 3D model of the object to estimate a pose of the camera relative to the 3D model of the object.

3. The method of claim 1, wherein a default setting of the probability of each point is 0.5, and the probability is increased when the tracking determines that the point corresponds to the object, and the probability is decreased when the tracking determines that the point corresponds to the background.

4. The method according to claim 1, wherein: The 3D model of an object in the scene includes a 3D point cloud including a plurality of points including at least some points corresponding to the object and at least some points corresponding to a background of the scene.

5. The method according to claim 4, wherein: The video stream is obtained from a camera moving around the scene.

6. A system for generating augmented reality content, comprising a processor, a memory accessible by the processor, and computer program instructions stored in the memory and operable by the processor to: tracking a 3D model of an object in a scene in a video stream as the camera moves about the scene, wherein the tracking includes determining, for each point in each frame, whether the point corresponds to the object or whether the point corresponds to background; dividing each of the plurality of points into a point corresponding to the object or a point corresponding to the background based on a probability that the point corresponds to the object and a probability that the point corresponds to the background, wherein for each point, the higher the probability that the point corresponds to the object, the more frequently the tracking determines that the point corresponds to the object, and for each point, the higher the probability that the point corresponds to the background, the more frequently the tracking determines that the point corresponds to the background, and wherein the 3D model of the object is initially divided using a default setting of the probability for each point, and for each frame, the probability for each point is updated based on whether the tracking determines that the point corresponds to the object or to the background; as well as Augmented reality content is generated based on the segmented 3D model of the object. 7 . The system of claim 6 , wherein the tracking comprises using a segmented 3D model of the object to estimate a pose of the camera relative to the 3D model of the object.

8. The system of claim 6, wherein a default setting of the probability of each point is 0.5, and the probability is increased when the tracking determines that the point corresponds to the object, and the probability is decreased when the tracking determines that the point corresponds to the background.

9. The system according to claim 6, wherein: The 3D model of an object in the scene includes a 3D point cloud including a plurality of points including at least some points corresponding to the object and at least some points corresponding to a background of the scene.

10. The system according to claim 9, wherein: The video stream is obtained from a camera moving around the scene.

11. A computer program product for generating augmented reality content, the computer program product comprising a non-transitory computer readable storage having program instructions executable by a computer to cause the computer to perform a method comprising: tracking, at the computer, a 3D model of an object in the scene in the video stream as the camera moves about the scene, wherein the tracking includes determining, for each point in each frame in the video stream, whether the point corresponds to the object or whether the point corresponds to background; at the computer, partitioning each of the plurality of points into a point corresponding to the object or a point corresponding to the background based on a probability that the point corresponds to the object and a probability that the point corresponds to the background, wherein for each point, the higher the probability that the point corresponds to the object, the more frequently the point is identified as an inlier point by the tracking, and for each point, the higher the probability that the point corresponds to the background, the more frequently the point is identified as an outlier by the tracking, and wherein the 3D model of the object is initially partitioned using a default setting of the probability for each point, and for each frame, the probability for each point is updated based on whether the tracking determines whether the point corresponds to the object or the background; as well as At the computer, augmented reality content is generated based on the segmented 3D model of the object.

12. The computer program product of claim 11, wherein the tracking comprises using a segmented 3D model of the object to estimate a pose of the camera relative to the 3D model of the object.

13. The computer program product of claim 11, wherein a default setting for the probability of each point is 0.5, and the probability is increased when the tracking determines that the point corresponds to the object, and the probability is decreased when the tracking determines that the point corresponds to the background.

14. The computer program product of claim 11, wherein the 3D model of the object in the scene comprises a 3D point cloud comprising a plurality of points including at least some points corresponding to the object and at least some points corresponding to a background of the scene.

15. The computer program product of claim 14, wherein the video stream is obtained from a camera moving about the scene.