Methods, storage media, and systems for generating a three-dimensional line segment

Abstract

A method generates a three-dimensional (3D) line segment from a two-dimensional (2D) image. A 3D coordinate space along with a first plane in the coordinate space are received. An image with an associated camera pose in the 3D coordinate space is also received. A 2D line segment is detected in the image and then projected into the 3D coordinate space based on the camera pose. The projected 2D line segment is intersected with the first plane, which generates a 3D line segment defined by the intersection. By leveraging the camera pose and first plane, the 3D line segment can be generated from a single 2D image, eliminating the need to triangulate line segments across stereo image pairs.

Description

cross reference to related applications

[0001] The present application claims priority to provisional patent application 63 / 612,102 entitled “METHODS, STORAGE MEDIA, AND SYSTEMS FOR GENERATING A THREE-DIMENSIONAL LINE SEGMENT,"’ filed December 19, 2023 and provisional patent application 63 / 735,304 entitled “METHODS, STORAGE MEDIA. AND SYSTEMS FOR GENERATING A THREE-DIMENSIONAL LINE SEGMENT,” filed December 17, 2024, and which are hereby incorporated by reference in their entirety and made part of the present application for all purposes. BACKGROUND FIELD OF THE DISCLOSURE

[0002] The present disclosure relates to methods, storage media, and systems for threedimensional (3D) line generation. DESCRIPTION OF THE RELATED ART

[0003] Reconstructing three-dimensional (3D) line segments from two-dimensional (2D) images is an important task in computer vision with applications in mapping, navigation, and augmented reality. A common approach is to use stereo image pairs with a wide baseline to triangulate matching 2D line segments into 3D. However, this method can be sensitive to errors in detecting the 2D line segments, leading to incorrect 3D line estimation. Robust techniques are needed to accurately generate 3D line segments from 2D images. SUMMARY

[0004] Previous methods have relied on triangulating pairs of two-dimensional (2D) line segments detected across 2D image pairs to reconstruct a three-dimensional (3D) line segment. However, when considering 2D image pairs with a small baseline, small errors in the 2D line segment positions in the 2D images can lead to large triangulation errors in the resulting 3D line segment. These issues may stem from geometric uncertainty in triangulation. As the baseline between cameras becomes smaller, the intersection angle between projected rays through the 2D line segments becomes more acute, making 3D position estimation more sensitive to small errors in the 2D line segment detection.

[0005] The problem of triangulating a 3D line segment from a pair of 2D images with a small baseline being sensitive to errors in a 2D line segment in each of the 2D images is solved by generating 3D line segment based on an intersection of a 3D planar surface and a projected 2D line segment projected from a single 2D image.

[0006] The disclosure eliminates sensitivities in 2D line segment triangulation by leveraging a single 2D image and a detected 3D planar surface. Once a 2D line segment in the single 2D image is detected, it is projected into 3D space using the image's camera pose within a defined coordinate system. This projection generates a vertical 3D plane that is intersected with a detected 3D planar surface, such as horizontal 3D plane (e.g., a ground plane). The resulting intersection defines the 3D position of endpoints of a 3D line segment.

[0007] This approach provides a robust 3D line segment estimation that does not require detection of 2D line segments across multiple images. By utilizing a single 2D image, camera pose, and a 3D planar surface, a precise 3D line segment can be reconstructed without triangulation and with a single 2D image.

[0008] A 2D line segment detected in a 2D image and projected into 3D space based on a camera pose associated with the 2D image, and the projected 2D line segment can be intersected with a detected 3D planar surface, such as a horizontal ground plane, to anchor a 3D line segment. The camera pose associated with the 2D image describes the position and orientation of an imaging device that captured the 2D image in 3D coordinate space, for example in terms of its translation (x,y,z location) and rotation (roll, pitch, yaw angles). Using intrinsic calibration parameters, extrinsic calibration parameters, or a combination of both, of the imaging device, pixels in the 2D image can be mapped into 3D rays originating from the imaging device center and projecting out into the 3D coordinate space. By connecting the start and end points of a detected 2D line segment in the image plane, the corresponding 3D rays define a 3D planar surface, such as a vertical plane. This vertical plane projection from the 2D image is oriented relative to the camera pose, and thus aligned to 3D coordinate system. By intersecting the vertical plane containing the projected 2D line segment with a detected 3D planar surface (e.g., horizontal 3D ground plane), the intersection line between the vertical plane and the horizontal plane forms a 3D line segment. The camera pose allows 2D image detections to be projected into 3D space in a geometrically consistent manner, while 3D planar surfaces like the ground provide anchor points to locate 3D lines in the world.

[0009] By utilizing a single 2D image, camera pose, and a 3D planar surface to generate a 3D line segment, the disclosure addresses the sensitivity to errors in 2D line segment positions across images that may lead to triangulation errors. The camera pose may enable mapping from 2D image space to 3D space, for example by using intrinsic and / or extrinsic camera parameters. The camera pose may be used to project 2D lines of the 2D image into 3D space in a geometrically consistent manner. The 3D planar surface may provide anchor points that locate 3D lines accurately in 3D space. Using a single 2D image instead of pairs of images may reduce computational resources and complexity while improving accuracy and robustness and enable various practical applications such as, for example, architectural reconstruction, augmented and / or mixed reality7 reconstruction, and the like. The disclosed techniques address a technology-specific challenge of dimensional metrology by providing single 2D image measurement solutions for objects in the image through projection and intersection with geometric planes. These solutions are rooted in computer technologies, for example computer vision systems, and improve the capabilities of computer vision systems, for example where triangulation across 2D image pairs with a small baseline is inaccurate.

[0010] These and other features, and characteristics of the present technology , as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention. As used in the specification and in the claims, the singular form of 'a', 'an', and 'the' include plural referents unless the context clearly dictates otherwise. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure (FIG.) 1 illustrates an example system 100 for generating three-dimensional (3D) line segments from two-dimensional (2D) images, in accordance with one or more implementations.

[0012] FIG. 2 illustrates an example method 200 for generating a 3D line segment, in accordance with one or more implementations.

[0013] FIG. 3 illustrates an example 3D coordinate system and an example building object, in accordance with one or more implementations.

[0014] FIG. 4 illustrates an example ground plane, in accordance with one or more implementations.

[0015] FIG. 5 illustrates an example image and an example associated camera pose, in accordance with one or more implementations.

[0016] FIG. 6 illustrates an example 2D line segment of an image, in accordance with one or more implementations.

[0017] FIG. 7 illustrates an example projected 2D line segment, in accordance with one or more implementations.

[0018] FIG. 8 illustrates an example intersection of a projected 2D line segment and a ground plane, in accordance with one or more implementations.

[0019] FIG. 9 illustrates an example 3D line segment based on an intersection of a projected 2D line segment and a ground plane, in accordance with one or more implementations. DETAILED DESCRIPTION

[0020] FIG. 1 illustrates an example system 100 for generating three-dimensional (3D) line segments from two-dimensional (2D) images, in accordance with one or more implementations. The system 100 includes an imaging device 110, a pose estimation system 120, a line segment detection system 130, a line projection system 140, a plane detection system 150, an intersection determination system 160, and a 3D line generation system 170 interconnected via a network 180. In some implementations, system 100 may include one or more computing platforms 102. Computing platform(s) 102 may be configured to communicate with one or more remote platforms 104 according to a client / server architecture, a peer-to-peer architecture, and / or other architectures. Remote platform(s) 104 may be configured to communicate with other remote platforms via computing platform(s) 102 and / or according to a client / server architecture, a peer-to-peer architecture, and / or other architectures. Users may access system 100 via remote platform(s) 104.

[0021] Computing platform(s) 102 may be configured by machine-readable instructions 106. Machine-readable instructions 106 may include one or more instruction modules. The instruction modules may include computer program modules or systems. The instruction modules or systems may include one or more of a pose estimation system 120, a line segment detection system 130, a line projection system 140, a plane detection system 150, an intersection determination system 160, and a 3D line generation system 170, and / or other instruction modules or systems.

[0022] The imaging device 110 can include any camera or imaging system capable of capturing 2D images depicting objects or scenes. In some implementations, the imaging device 110 includes, but is not limited to, conventional digital cameras, stereo camera systems, smartphones, tablet computers, augmented reality headsets, virtual reality headsets, drones, aerial platforms, or other imaging systems. The imaging device 110 may have one or more lenses with different focal lengths and fields of view. Typical image resolutions may range from 1 to 40 megapixels, but any resolution capable of discerning lines and edges in a scene may be utilized. The 2D images may be stored in one or more databases, for example electronic storage 126.

[0023] The pose estimation system 120 can be configured to determine a camera pose associated with the 2D images captured by the imaging device 110. The camera pose situates the captured images within a defined 3D coordinate space by providing the 3D position and orientation of the imaging device 110 used to capture the images. In some embodiments, camera intrinsics and / or extrinsics may be known. In these embodiments, the pose estimation system 120 may use the known camera intrinsics and / or extrinsics to determine the camera pose. In some embodiments, camera intrinsics and / or extrinsics may not be known. In these embodiments, the pose estimation system 120 may determine the camera intrinsics and / or extrinsics and may use the determined camera intrinsics and / or extrinsics to determine the camera pose.

[0024] The line segment detection system 130 is configured to detect one or more 2D line segments depicted in 2D images captured by the imaging device 110. Line segment detection can leverage edge detection techniques to identify potential edges. Line segments can then be fit to the edges using probabilistic transforms, incremental algorithms, splitting / merging methods, or other line detection techniques. Noise filtering and line validation based on direction, length, contrast and other factors may also be employed to discard spurious lines.

[0025] The line projection system 140 projects 2D line segments detected by the line segment detection system 130 into 3D space based on the camera pose associated with the 2D image. This projection generates a 3D planar surface, such as a vertical plane, in 3D space corresponding to each 2D line segment. In some embodiments, a 2D line segment may include first and second end points. Projecting the 2D line may include casting 3D rays from a camera center and through points of the 2D line segment, for example the first and second end points. The corresponding 3D rays, for example ones that pass through the first and second end points, may define the vertical plane. In other words, the vertical plane may contain the projected 2D line segment. The vertical plane may be defined by an angle between a first ray connecting the camera center and the first end point and a second ray connecting the camera center and the second end point.

[0026] The plane detection system 150 receives, detects, or otherwise generates representations of 3D planes, such as horizontal ground planes, within the defined 3D coordinate space. In some embodiments, plane detection may leverage simultaneous localization and mapping (SLAM) systems to orient and align ground and floor planes based on registered point clouds. In some embodiments, augmented reality frameworks, such as ARKit and ARCore, may facilitate plane detection using feature point anchors, plane alignment based on IMU sensor data, and real-time plane estimation from point tracks. In some embodiments, plane fitting, refinement, and smoothing techniques such as RANSAC may be utilized. In some embodiments, one or more networks, such as segmentation networks and depth estimation networks, may be used to segment the 3D planes within the image and to estimate depth values (e.g., 3D coordinate space values) of the 3D planes.

[0027] The intersection determination system 160 analyzes projected 2D line segments from the line projection system 140 and the 3D planes from the plane detection system 150 to identify intersection points between the vertical planes of the projected 2D line segments and the horizontal planes. The intersection determination system 160 may apply computational geometry techniques to determine intersection points between projected line segments and detected 3D planes. This can include solving systems of parametric plane and line equations. Alternatively, ray-casting or ray-plane intersection tests can iteratively sample along line segment rays to find plane intersection points. Numerical analysis methods may generate precise coordinate values.

[0028] The 3D line generation system 170 generates 3D line segments defined by the intersection points identified by the intersection determination system 160. By leveraging a single 2D image, camera pose, and 3D planes, the generated 3D line segments avoid triangulation errors associated with typical stereo vision techniques, especially when considering images with a small baseline. The 3D line generation system 170 represents resulting 3D line segments using explicit start and end point coordinates, vector / point parametric forms, or implicit equations. The lines may be further processed by smoothing splines, Kalman filters, piecewise fittings, or other line refinement algorithms. Resulting 3D line segment data can be stored and exported as point clouds, polylines, meshes, or other reusable 3D representations.

[0029] In some implementations, computing platform(s) 102, remote platform(s) 104, and / or external resources 124 may be operatively linked via one or more electronic communication links. For example, such electronic communication links may be established, at least in part, via a netw ork such as the Internet and / or other networks. It will be appreciated that this is not intended to be limiting, and that the scope of this disclosure includes implementations in which computing platform(s) 102. remote platform(s) 104, and / or external resources 124 may be operatively linked via some other communication media.

[0030] A given remote platform 104 may include one or more processors configured to execute computer program modules or systems. The computer program modules or systems may be configured to enable an expert or user associated with the given remote platform 104 to interface with system 100 and / or external resources 124, and / or provide other functionality attributed herein to remote platform(s) 104. By way of non-limiting example, a given remote platform 104 and / or a given computing platform 102 may include one or more of a server, a desktop computer, a laptop computer, a handheld computer, a tablet computing platform, a NetBook, a Smartphone, a gaming console, and / or other computing platforms.

[0031] External resources 124 may include sources of information outside of system 100, external entities participating with system 100, and / or other resources. In some implementations, some or all of the functionality attributed herein to external resources 124 may be provided by resources included in system 100.

[0032] Computing platform(s) 102 may include electronic storage 126, one or more processors 128, and / or other components. Computing platform(s) 102 may include communication lines, or ports to enable the exchange of information with a netw ork and / or other computing platforms. Illustration of computing platform(s) 102 in FIG. 1 is not intended to be limiting. Computing platform(s) 102 may include a plurality of hardware, software, and / or firmw are components operating together to provide the functionality attributed herein to computing platform(s) 102. For example, computing platform(s) 102 may be implemented by a cloud of computing platforms operating together as computing platform(s) 102.

[0033] Electronic storage 126 may comprise non-transitory storage media that electronically stores information. The electronic storage media of electronic storage 126 may include one or both of system storage that is provided integrally (i.e., substantially nonremovable) with computing platform(s) 102 and / or removable storage that is removably connectable to computing platform(s) 102 via. for example, a port (e.g.. a USB port, a firewire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage 126 may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and / or other electronically readable storage media. Electronic storage 126 may include one or more virtual storage resources (e.g., cloud storage, a virtual private network, and / or other virtual storage resources). Electronic storage 126 may store software algorithms, information determined by processor(s) 128, information received from computing platform(s) 102, information received from remote platform(s) 104, and / or other information that enables computing platform(s) 102 to function as described herein.

[0034] Processor(s) 128 may be configured to provide information processing capabilities in computing platform(s) 102. As such, processor(s) 128 may include one or more of a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and / or other mechanisms for electronically processing information. Although processor(s) 128 is shown in FIG. 1 as a single entity, this is for illustrative purposes only. In some implementations, processor(s) 128 may include a plurality of processing units. These processing units may be physically located within the same device, or processor(s) 128 may represent processing functionality of a plurality of devices operating in coordination. Processor(s) 128 may be configured to execute modules or systems 120. 130, 140, 150, 160, and / or, 170 and / or other modules or systems. Processor(s) 128 may be configured to execute modules or systems 120, 130, 140, 150, 160, and / or 170, and / or other modules or systems by software; hardware; firmware; some combination of software, hardware, and / or firmware; and / or other mechanisms for configuring processing capabilities on processor(s) 128. As used herein, the term “module” or “system” may refer to any component or set of components that perform the functionality attributed to the module or the system. This may include one or more physical processors during execution of processor readable instructions, the processor readable instructions, circuitry, hardware, storage media, or any other components.

[0035] It should be appreciated that although modules or systems 120, 130. 140, 150, 160, and / or 170, are illustrated in FIG. 1 as being implemented within a single processing unit, in implementations in which processor(s) 128 includes multiple processing units, one or more of modules or system 120, 130, 140, 150, 160, and / or 170 may be implemented remotely from the other modules or systems. The description of the functionality provided by the different modules or system 120. 130, 140, 150, 160. and / or 170 described below is for illustrative purposes, and is not intended to be limiting, as any of modules or systems 120, 130, 140, 150, 160, and / or 170 may provide more or less functionality than is described. For example, one or more of modules or systems 120, 130, 140, 150, 160, and / or 170 may be eliminated, and some or all of its functionality may be provided by other ones of modules or systems 120, 130, 140, 150, 160. and / or 170. As another example, processor(s) 128 may be configured to execute one or more additional modules that may perform some or all of the functionality attributed below to one of modules or systems 120, 130, 140, 150, 160, and / or 170.

[0036] FIG. 2 illustrates an example method 200 for generating a 3D line segment, in accordance with one or more implementations. The operations of the method 200 are described below as being performed by the system 100 for convenience. However, the method 200 may be applied to any suitable system.

[0037] At operation 210. the system receives or otherwise obtains access to a 3D coordinate space defining a reference frame for situating a captured image and reconstructing 3D structures. In some embodiments, operation 210 may be performed by one or more processors and / or systems. Referring briefly to FIG. 3, it illustrates an example 3D coordinate space 300 and an example building object 302. in accordance with one or more implementations.

[0038] At operation 220. the system receives, detects, or otherwise obtains access to a first 3D planar surface (hereinafter referred to as '’first 3D plane" generally) within the 3D coordinate space. In some embodiments, the first 3D plane is a horizontal plane. In some embodiments, the horizontal plane is a ground plane, such as in an outdoor scene. In some embodiments, the horizontal plane is a floor plane, such as in an indoor scene. In some embodiments, the horizontal plane is a ceiling plane, such as in an indoor scene. In some embodiments, the first 3D plane is detected with augmented reality frameworks such as ARKit or ARCore. In some embodiments, plane fitting, refinement, and smoothing techniques such as RANSAC may be utilized. In some embodiments, one or more networks, such as segmentation networks and depth estimation networks, may be used to segment the first 3D plane within the image and to estimate depth values of the first 3D plane. In some embodiments, operation 220 may be performed by one or more processors and / or one or more systems. For example, the plane detection system 150 may receive, detect, or otherwise obtain access to the first 3D plane within the defined 3D coordinate space. Referring briefly to FIG. 4, it illustrates an example ground plane 402 of the building object 302, in accordance with one or more implementations.

[0039] At operation 230, the system receives or otherwise obtains an image and an associated camera pose, where the camera pose is in the 3D coordinate space. The image may be captured by an imaging device along with camera pose information situating the image within the 3D coordinate space based on the position and orientation of the imaging device used to capture the image. In some embodiments, an image plane of the image and the camera pose are in the 3D coordinate space. In some embodiments, operation 230 may be performed by one or more processors and / or one or more systems. For example, the imaging device 110 may receive or otherwise obtain an image and an associated camera pose. Referring briefly to FIG. 5. it illustrates an example image 502 and an example associated camera pose 504, in accordance with one or more implementations. As illustrated in FIG. 5, the camera pose 504 is in the 3D coordinate space 300.

[0040] In some embodiments, the image is captured by one or more of a smartphone, a tablet computer, an augmented reality headset, a virtual reality headset, a drone, and an aerial platform. In some embodiments, the image depicts a building object. In some embodiments, the image depicts an interior of the building object. In some embodiments, the image depicts an exterior of the building object.

[0041] In some embodiments, at operation 230, the system determines the camera pose associated with the 2D image. The camera pose situates the captured image within a defined 3D coordinate space by providing the 3D position and orientation of the imaging device used to capture the image. In some embodiments, camera intrinsics and / or extrinsics may be known. In these embodiments, the system may use the known camera intrinsics and / or extrinsics to determine the camera pose. In some embodiments, camera intrinsics and / or extrinsics may not be known. In these embodiments, the system may determine the camera intrinsics and / or extrinsics and may use the determined camera intrinsics and / or extrinsics to determine the camera pose. In some embodiments, camera intrinsics may include optical center, focal length, etc. and camera extrinsics may include position and orientation of the camera. In some embodiments, camera intrinsics may be used to transform between pixel coordinates and camera coordinates, and camera extrinsics may be used to transform between camera coordinates and world coordinates. In some embodiments, operation 230 may be performed by one or more processors and / or one or more systems. For example, the pose estimation system 120 may determine a camera pose associated with the 2D images captured by the imaging device.

[0042] At operation 240, the system detects a 2D line segment in the image. In some embodiments, the 2D line segment is a vertical line segment. In some embodiments, the vertical line segment corresponds to edges of buildings, such as walls, edges of features of buildings, such as doors and windows. In some embodiments, operation 230 may be performed by one or more processors and / or one or more systems. For example, the line projection system 140 may detect the 2D line segment in the image. Referring briefly to FIG. 6. it illustrates an example 2D line segment 602 in the image 502, in accordance with one or more implementations.

[0043] At operation 250, the system projects the detected 2D line segment into the 3D coordinate space based on the camera pose. In some embodiments, projecting the 2D line segment into the 3D coordinate space includes generating a second 3D plane defined by an angle between a first ray connecting a camera center of the camera pose and a first end point of the 2D line segment and a second ray connecting the camera center of the camera pose and a second end point of the 2D line segment. In some embodiments, the first 3D plane is a horizontal plane and the second 3D planar surface (hereinafter referred to as “second 3D plane” generally) is a vertical plane. In some embodiments, operation 250 may be performed by one or more processors and / or one or more systems. For example, the line projection system 140 may project the detected 2D line segment into the 3D coordinate space based on the camera pose. In some embodiments, a 2D line segment may include first and second end points. Projecting the 2D line may include casting 3D rays from a camera center and through points of the 2D line segment, for example the first and second end points. The corresponding 3D rays, for example ones that pass through the first and second end points, may define the vertical plane. In other words, the vertical plane may contain the projected 2D line segment. The vertical plane may be defined by an angle between a first ray connecting the camera center and the first end point and a second ray connecting the camera center and the second end point. Referring briefly to FIG. 7, it illustrates an example projected 2D line segment 702, in accordance with one or more implementations. The 2D line segment 602 is projected into the 3D coordinate space 300 based on the camera pose 504. The projected 2D line segment 702 is a vertical plane defined by an angle between a first ray 704 connecting a camera center of the camera pose 504 and a first end point of the 2D line segment 602 (top end point) and a second ray 706 connecting the camera center of the camera pose 504 and a second end point of the 2D line segment 602 (bottom end point).

[0044] At operation 260, the system determines at least one intersection of the projected 2D line segment and the first 3D plane. In some embodiments, operation 250 may be performed by one or more processors and / or one or more systems. For example, the intersection determination system 160 may determine at least one intersection of the projected 2D line segment and the first 3D plane. Referring briefly to FIG. 8, it illustrates an example intersection 802 of the projected 2D line segment 702 and the ground plane 402, in accordance with one or more implementations.

[0045] At operation 270. the system generates a 3D line segment based on the intersection of the projected 2D line segment and the first 3D plane. In some embodiments, a first end point of the 3D line segment corresponds to the intersection of the proj ected 2D line segment and the first 3D plane. In some embodiments, operation 250 may be performed by one or more processors and / or one or more systems. For example, the 3D line generation system 170 may generate a 3D line segment based on the intersection of the projected 2D line segment and the first 3D plane. Referring briefly to FIG. 9, it illustrates an example 3D line segment based on the intersection 802 of the projected 2D line segment 702 and the ground plane 402, in accordance with one or more implementations. A first end point (bottom end point) of the 3D line segment 902 corresponds to the intersection 802 of the projected 2D line segment 702 and the ground plane 402.

[0046] In some embodiments, the method 200 may include generating an AR session using the augmented reality frameworks such as ARKit and ARCore. Operations of the method 200, such as operations 210 through 270, may be performed during the AR session and with respect to the same 3D coordinate space that is received or otherwise obtained in operation 210. Performing operations 210 through 270 during the AR session enables placement of the first 3D plane of operation 220, the image and camera pose of operation 230, the 2D line segment of operation 240, the projected 2D line segment of operation 250, the intersection of the projected line segment and the first 3D plane of operation 260, and the 3D line segment of operation 270 in a common coordinate space and relative to one another.

[0047] The 3D line segment may be used in a variety of settings or applications. With respect to architectural reconstruction, the 3D line segment may be used to generate 3D models of buildings, houses, or infrastructure from images to aid digital reconstruction and modeling. The 3D line segment may help align and scale models. With respect to augmented reality frameworks, the 3D line segment may be used to anchor virtual objects or overlays in augmented reality applications, by snapping and aligning to the 3D line segment tied to real world geometries. With respect to robotics / autonomous vehicles, the 3D line segment may be used to represent architectural features such as walls from on-board cameras to aid in path planning and navigation. With respect to scene understanding, the 3D line segment may be used to provide geometric cues in images to help algorithms understand and make sense of a scene. With respect to dimensional metrology, the 3D line segment may be used to measure dimensions of real-world object by projecting and intersecting imaged lines with known geometric planes or surfaces. In some embodiments, since the 3D line segment in the 3D coordinate space is associated the 2D line segment in the image, a dimensional value (e.g., measurement) of the 3D line segment in the 3D coordinate space may be used to deriving a scaling / calibration factor for the image. With respect to camera calibration, the 3D line segment may provide feedback on estimated camera poses and help refine calibrations.

[0048] In general, the technique is valuable in aligning 2D image understanding with real 3D spaces to improve spatial perception in applications like reconstruction, robotics, augmented reality, and more. The plane anchored lines provide key constraints.

[0049] One or more implementations described herein provide a system for generating 3D line segments from 2D images that eliminates triangulation errors associated with typical stereo vision techniques. By leveraging a single 2D image, camera pose, and known planes, precise 3D line segments can be reconstructed without relying on accurate detection of 2D line positions across multiple images. The described techniques provide robust 3D line estimation useful for mapping, localization, navigation, and augmented reality systems.

[0050] The methods disclosed herein comprise one or more actions for achieving the described method. The method and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions may be modified without departing from the scope of the claims.

[0051] The functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable medium or non-transitory processor-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium may be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, include CD, laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0052] The methods disclosed herein comprise one or more actions for achieving the described method. The method and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions may be modified without departing from the scope of the claims.

[0053] Although the present technology has been described in detail for the purpose of illustration based on what is currently considered to be the most practical and preferred implementations, it is to be understood that such detail is solely for that purpose and that the technology is not limited to the disclosed implementations, but, on the contrary , is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present technology contemplates that, to the extent possible, one or more features of any implementation can be combined with one or more features of any other implementation.

Claims

1. A method of generating a three-dimensional (3D) line segment, the method comprising:receiving a 3D coordinate space;receiving a first plane in the 3D coordinate space;receiving an image and an associated camera pose, wherein the camera pose is in the 3D coordinate space;detecting a two-dimensional (2D) line segment in the image;projecting the 2D line segment into the 3D coordinate space based at least in part on the camera pose;determining an intersection of the projected 2D line segment and the first plane; and generating a 3D line segment based on the intersection of the projected 2D line segment and the first plane.

2. The method of claim 1, wherein the first plane is a horizontal plane.

3. The method of claim 2, wherein the horizontal plane is a ground plane.

4. The method of claim 1, wherein the first plane is detected with augmented realityframeworks.

5. The method of claim 1, wherein the image is captured by one or more of a smartphone, a tablet computer, an augmented reality headset, a virtual reality headset, a drone, and an aerial platform.

6. The method of claim 1, wherein the image depicts a building object.

7. The method of claim 6, wherein the image depicts an interior of the building object.

8. The method of claim 6, wherein the image depicts an exterior of the building object.

9. The method of claim 1, wherein an image plane of the image and the camera pose are in the 3D coordinate space.

10. The method of claim 1, wherein the 2D line segment is a vertical line segment.

11. The method of claim 1, wherein projecting the 2D line segment into the 3Dcoordinate space comprises generating a second plane defined by an angle between a first ray connecting a camera center of the camera pose and a first end point of the 2D line segment and a second ray connecting the camera center of the camera pose and a second end point of the 2D line segment.

12. The method of claim 11, wherein the first plane is a horizontal plane and the second plane is a vertical plane.

13. The method of claim 1, wherein a first end point of the 3D line segment corresponds to the intersection of the projected 2D line segment and the first plane.

14. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:receiving a three-dimensional (3D) coordinate space;receiving a first plane in the 3D coordinate space; receiving an image and an associated camera pose, wherein the camera pose is in the 3D coordinate space;detecting a two-dimensional (2D) line segment in the image;proj ecting the 2D line segment into the 3D coordinate space based at least in part on the camera pose;determining an intersection of the projected 2D line segment and the first plane; and generating a 3D line segment based on the intersection of the projected 2D line segment and the first plane.

15. The non-transitory computer-readable medium of claim 14, wherein the first plane is a horizontal plane.

16. The non-transitory computer-readable medium of claim 15, wherein the horizontal plane is a ground plane.

17. The non-transitory computer-readable medium of claim 14, wherein the first plane is detected with augmented reality frameworks.

18. The non-transitory computer-readable medium of claim 14. wherein the image is captured by one or more of a smartphone, a tablet computer, an augmented reality headset, a virtual reality headset, a drone, and an aerial platform.

19. The non-transitory computer-readable medium of claim 14, wherein the image depicts a building object.

20. The non-transitory computer-readable medium of claim 19, wherein the image depicts an interior of the building object.

21. The non-transitory computer-readable medium of claim 19, wherein the image depicts an exterior of the building object.

22. The non-transitory computer-readable medium of claim 14. wherein an image plane of the image and the camera pose are in the 3D coordinate space.

23. The non-transitory computer-readable medium of claim 14, wherein the 2D line segment is a vertical line segment.

24. The non-transitory computer-readable medium of claim 14, wherein projecting the 2D line segment into the 3D coordinate space comprises generating a second plane defined by an angle between a first ray connecting a camera center of the camera pose and a first end point of the 2D line segment and a second ray connecting the camera center of the camera pose and a second end point of the 2D line segment.

25. The non-transitory computer-readable medium of claim 24, wherein the first plane is a horizontal plane and the second plane is a vertical plane.

26. The non-transitory computer-readable medium of claim 14, wherein a first end point of the 3D line segment corresponds to the intersection of the projected 2D line segment and the first plane.

27. A system comprising:one or more processors;memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:receiving a three-dimensional (3D) coordinate space;receiving a first plane in the 3D coordinate space;receiving an image and an associated camera pose, wherein the camera pose is in the 3D coordinate space;detecting a two-dimensional (2D) line segment in the image;projecting the 2D line segment into the 3D coordinate space based at least in part on the camera pose;determining an intersection of the projected 2D line segment and the first plane; andgenerating a 3D line segment based on the intersection of the projected 2D line segment and the first plane.

28. The system of claim 27, wherein the first plane is a horizontal plane.

29. The system of claim 28, wherein the horizontal plane is a ground plane.

30. The system of claim 27, wherein the first plane is detected with augmented realityframeworks.

31. The system of claim 27, wherein the image is captured by one or more of a smartphone, a tablet computer, an augmented reality headset, a virtual reality headset, a drone, and an aerial platform.

32. The system of claim 27, wherein the image depicts a building object.

33. The system of claim 32, wherein the image depicts an interior of the building object.

34. The system of claim 32, wherein the image depicts an exterior of the building object.

35. The system of claim 27, wherein an image plane of the image and the camera pose arein the 3D coordinate space.

36. The system of claim 27, wherein the 2D line segment is a vertical line segment.

37. The system of claim 27, wherein projecting the 2D line segment into the 3Dcoordinate space comprises generating a second plane defined by an angle between a first ray connecting a camera center of the camera pose and a first end point of the 2D line segment and a second ray connecting the camera center of the camera pose and a second end point of the 2D line segment.

38. The system of claim 37, wherein the first plane is a horizontal plane and the second plane is a vertical plane.

39. The system of claim 27, wherein a first end point of the 3D line segment corresponds to the intersection of the projected 2D line segment and the first plane.

Images