METHODS AND SYSTEMS FOR CARRYING OUT OBJECT DIMENSIONING

Abstract

Aspects of the present disclosure include isolating a target scene object from multiple images of a time-of-flight (TOF) sensor. Multiple images from a TOF sensor can be transformed into multiple three-dimensional (3D) point clouds. In each of the multiple 3D point clouds, one or more large planes can be identified with a minimum threshold size. A common 3D point cloud can be generated by aligning rotated and translated point clouds corresponding to at least a subset of the multiple 3D point clouds. The common 3D point cloud can be rotated and translated to identify one of the large planes as the ground plane in a common coordinate system of the common 3D point cloud, and a target object can be isolated.

Description

Priority claim

[0001] This patent application claims priority to Non-Provisional Patent Application No. 18 / 189,934, entitled "METHODS AND SYSTEMS FOR PERFORMING OBJECT DIMENSIONING," filed March 24, 2023, Provisional Patent Application No. 63 / 324,042, entitled "PALLET DIMENSIONING SYSTEM USING A TIME-OF-FLIGHT CAMERA," filed March 26, 2022, and Provisional Patent Application No. 63 / 324,043, entitled "OBJECT DIMENSIONING SYSTEM AND ALGORITHM," filed March 26, 2022, which are assigned to the assignee of the present specification and are hereby expressly incorporated by reference herein for all purposes. BACKGROUND

[0002] In the shipping, warehousing, or moving industries, companies often quote prices based on the size of one or more items to be shipped, stored, or relocated. Manually measuring the sizes of multiple items, such as with a tape measure or other measuring mechanism, can be time-consuming and / or inaccurate. Additionally, some items may not be box-shaped or otherwise symmetrical, which can complicate measuring such items. SUMMARY

[0003] What follows is a simplified summary of one or more aspects to provide a basic understanding of such aspects. This summary is not a comprehensive overview of all aspects considered and is not intended to identify important or critical elements of all aspects, nor to delimit the scope of protection of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as an introduction to the more detailed description presented later.

[0004] Aspects of the present disclosure include a method for isolating a target scene object from multiple images from a time-of-flight (TOF) sensor. The method includes transforming the multiple images from a TOF sensor into multiple three-dimensional (3D) point clouds, identifying, in each of the multiple 3D point clouds, one or more large planes having a minimum threshold size, generating a common 3D point cloud by aligning rotated and translated point clouds corresponding to at least a portion of the multiple 3D point clouds, rotating and translating the common 3D point cloud to identify one of one or more large planes as a ground plane in a common coordinate system of the common 3D point cloud, and isolating a target object from the rotated and translated common 3D point cloud.

[0005] In another aspect, a system is provided that includes a memory and a processor coupled to the memory. The processor is configured to transform the plurality of images from a TOF sensor into a plurality of 3D point clouds, identify, in each of the plurality of 3D point clouds, one or more large planes having a minimum threshold size, generate a common 3D point cloud by aligning rotated and translated point clouds corresponding to at least a portion of the plurality of 3D point clouds, rotate and translate the common 3D point cloud to identify one of one or more large planes as a ground plane in a common coordinate system of the common 3D point cloud, and isolate a target object from the rotated and translated common 3D point cloud.

[0006] In another aspect, a computer-readable medium is provided including code executable by one or more processors for isolating a target scene object from multiple images of a TOF sensor. The code includes code for transforming the multiple images from a TOF sensor into multiple 3D point clouds, identifying, in each of the multiple 3D point clouds, one or more large planes having a minimum threshold size, generating a common 3D point cloud by aligning rotated and translated point clouds corresponding to at least a portion of the multiple 3D point clouds, rotating and translating the common 3D point cloud to identify one of one or more large planes as a ground plane in a common coordinate system of the common 3D point cloud, and isolating a target object from the rotated and translated common 3D point cloud.

[0007] To achieve the foregoing and related objectives, the one or more aspects comprise the features fully described below and particularly pointed out in the claims. In the following description and the accompanying drawings, certain illustrative features of the one or more aspects are set forth in detail. However, these features illustrate only a few ways of implementing the principles of various aspects, and this description is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The disclosed aspects are described below in conjunction with the accompanying drawings, which are provided to illustrate and not limit the disclosed aspects, in which like designations refer to like elements and in which: Fig. 1 is a block diagram of an example sensor system according to aspects described herein. Fig. 2 illustrates beam directions of pixels of the time-of-flight (TOF) sensor according to aspects described herein. Fig. 3 illustrates an example of a multi-cloud fusion component according to aspects described herein. Fig. 4 illustrates an example of a method for validating TOF images for performing object sizing in accordance with aspects described herein. Fig. 5 illustrates an example of a method for acquiring TOF images to perform object sensing in accordance with aspects described herein. Fig. 6 illustrates an example of a method for associating point clouds from different acquisition positions to perform object dimensioning in accordance with aspects described herein. Fig. Figure 7 illustrates an example of point clouds and corresponding feature spaces according to aspects described here. Fig. Figure 8 illustrates an example of point clouds and corresponding feature spaces as well as point clouds and a diagrammatic representation of tiepoint consistency according to aspects described here. Fig. 9 illustrates an example of a method for isolating a target object in multiple TOF images to perform its sizing in accordance with aspects described herein. Fig. 10 illustrates an example of a method for isolating the target object from the rotated and translated joint three-dimensional (3D) point cloud in accordance with aspects described herein. DETAILED DESCRIPTION

[0009] The following is a description of various aspects with reference to the drawings. In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it may be understood that such aspect(s) may be practiced without these specific details.

[0010] The described features generally relate to an object sizing system, an object sizing method, and / or an object sizing algorithm for measuring the dimensions of an object observed using a time-of-flight (TOF) sensor or other camera, e.g., a depth camera, or other sensor. In one example, the described features may enable the measurement of irregularly shaped objects, which may include determining width, length, and height (BLH) and adjusting a bounding box around an irregularly shaped (or asymmetric) object accordingly. For example, the TOF sensor may be used to capture 3D images (e.g., TOF images) of an object from various positions around the object to determine the object dimensions. A pose may, for example, refer to the position of the TOF sensor relative to an object (e.g.,around the object), the angle of the TOF sensor relative to the object (e.g., a tilt of the TOF sensor), other alignment parameters, and / or the like, which may also be referred to more generally herein as position. The 3D images may be aligned in a common coordinate system (which may also be referred to as "stitching") by determining point correspondences between the 3D images. In one example, a larger 3D image may be generated that combines the views of the objects at the various poses. One or more objects may be located at and / or based on the points in the common coordinate system. In addition, for example, the size of the object (e.g., BLH) in the common coordinate system may be calculated. In some examples, a representation of the object may be displayed, which may also include one or more measurements of the object (e.g., BLH).

[0011] In this regard, for example, the system, method, and / or algorithm for performing object sizing as described herein may provide object dimensions based on a series of detections using the TOF sensor, thereby enabling efficient and substantially accurate measurement of objects regardless of shape. This allows for determining the size of objects to be shipped, stored, relocated, etc., for more accurate pricing or other logistical considerations.

[0012] As used in this application, the terms "component," "module," "system," and the like are intended to include a computer-related entity, such as, but not limited to, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component may be, among other things, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or a computer. For illustration, both an application running on a computing device and the computing device may be a component. One or more components may be in a process and / or thread of execution, and a component may be localized on one computer and / or distributed between two or more computers.Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. The components can communicate using local and / or non-local processes, e.g., according to a signal with one or more data packets, e.g., data from one component interacting with another component in a local system, a distributed system, and / or over a network such as the Internet with other systems using the signal.

[0013] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components, as appropriate. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, functions described with respect to some examples may be combined in other examples.

[0014] Various aspects or features are presented in the form of systems that may include a number of devices, components, modules, and the like. It is understood that the various systems may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in connection with the figures. A combination of these approaches may also be used.

[0015] In the description, reference may be made to the specific spatial relationships between various components and to the spatial orientation of various aspects of components, as illustrated in the accompanying drawings. However, those skilled in the art, after reading this disclosure in its entirety, will understand that the devices, components, elements, devices, etc. described herein may be arranged in any desired orientation. Accordingly, the use of terms such as "above," "below," "upper," "lower," "top," "bottom," or other similar terms to describe a spatial relationship between various components or to describe the spatial orientation of aspects of such components should be understood to imply a relative relationship between the components ora spatial orientation of aspects of such components is described, since the components described herein can be oriented in any desired direction. When used to describe a range of dimensions or other characteristics (e.g., time, pressure, temperature, length, width, etc.) of an element, operations, and / or conditions, the term "between X and Y" represents a range that includes X and Y.

[0016] Fig. 1 is a block diagram of an exemplary sensor system 100 according to aspects described herein. The sensor system 100 includes a TOF sensor 110, a processor 120, a camera 130, a display device 140, and a memory 150. In alternative configurations, other, fewer, and / or additional components may be included in the TOF sensor system than those described in Fig. 1. Furthermore, the in conjunction with one or more of the Fig. 1 may be distributed among the components in a manner different than described. In some embodiments, some or all of the components 110-150 may be integrated into a single unit, e.g., a handheld unit including a TOF sensor 110, a processor 120 for processing the TOF data, local storage 150, and a display device 140 for displaying an output of the processor 120 to a user. In some embodiments, some components may be located in different devices; e.g., a handheld TOF sensor 110 may transmit TOF data to an external processing system (e.g., a computer or tablet) that stores and processes the TOF data and provides one or more displays to a user. Different devices may communicate via wireless or wired connections.

[0017] The TOF sensor 110 captures distance data describing a distance between the TOF sensor 110 and various surfaces in the vicinity of the TOF sensor 110. The TOF sensor 110 may include a light source, such as a laser, and an image sensor for detecting light reflected from the surfaces. In some embodiments, the TOF sensor 110 emits a pulse of light and captures multiple image frames at different times to determine a time period required for the light pulse to travel to the surface and return to the image sensor. In other embodiments, the TOF sensor 110 detects phase shifts in the captured light, and the phase shifts indicate the distance between the TOF sensor 110 and various surfaces. In some embodiments, the TOF sensor 110 may generate and capture light at multiple different frequencies.When the TOF sensor 110 emits and detects light at multiple frequencies, this can help resolve ambiguous distances and assist the TOF sensor 110 in functioning at longer distance ranges. For example, if at a first frequency, a first observed phase may correspond to a surface at a distance of 0.5 meters, 1.5 meters, or 2.5 meters, and at a second frequency, a second observed phase may correspond to a surface at a distance of 0.75 meters, 1.5 meters, or 2.25 meters, the TOF sensor 110 may determine that the surface is 1.5 meters away by combining the two observations. Using multiple frequencies can also improve robustness to noise caused by certain frequencies of ambient light, regardless of whether phase shift or pulse return time is used to measure distance.In alternative embodiments, various types of sensors may be used instead of and / or in addition to the TOF sensor 110 to obtain distance data.

[0018] The processor 120 receives distance data from the TOF sensor 110 and processes the distance data to identify various features in the vicinity of the TOF sensor 110, such as, for example, in relation to Fig. 3-8. In some embodiments, the distance data includes the observed distances to various surfaces measured by the TOF sensor 110, for example, using the phase shift or pulse return time methods described above. In some embodiments, when the TOF sensor 110 measures phase shifts, the distance data that the processor 120 receives from the TOF sensor 110 is the phase shift data, and the processor 120 calculates the distances to the surfaces based on the phase shift data.

[0019] A camera 130 can capture image frames of the environment. The camera 130 can be a visible-light camera that captures images of the environment in the visible range. In other embodiments, the camera 130 is an infrared (IR) camera that captures IR intensities of the surfaces in the environment of the sensor system. The field of view of the camera 130 and the TOF sensor 110 partially or completely overlap; e.g., the field of view of the camera 130 can be slightly larger than the field of view of the TOF sensor 110. The camera 130 can forward captured images to the processor 120.

[0020] The display device 140 provides a visual output to a user of the sensor system 100. In some embodiments, the display device 140 displays an image obtained by the camera 130 and overlays visual images indicative of one or more features identified within the field of view of the camera 130 and the TOF sensor 110 based on the distance data. For example, the processor 120 may instruct the display device 140 to display an outline of a box over an image of the box obtained by the camera 130. A user may use this display to determine whether the sensor system 100 correctly identified the box and the edges of the box. The sensor system 100 may include additional or alternative input and / or output devices, e.g., buttons, speakers, a touchscreen, etc.

[0021] Memory 150 stores data for sensor system 100. For example, memory 150 stores processing instructions used by processor 120 to identify features in the vicinity of TOF sensor 110, e.g., instructions for identifying one or more Z-planes and / or calculating the dimensions of an observed object. Memory 150 may temporarily store data and images obtained by camera 130 and / or TOF sensor 110 and accessed by processor 120. Memory 150 may also store image data accessed by display device 140 to generate an output display.

[0022] Fig. Figure 2 illustrates beam directions of pixels of the TOF sensor 110 according to aspects described herein. The distance data obtained by the TOF sensor 110 may be arranged as a set of pixels, e.g., pixels 210a and 210b, in an image frame, e.g., image frame 220. Each pixel 210 has an associated beam direction 215, with beam direction 215 extending outward from the TOF sensor 110. The beam directions 215 are projected toward the image frame 220. While in Fig. 2 25 rays and pixels are shown, it is understood that the TOF sensor 110 can have many more pixels. While the image frame 220 in the Fig. While the image frame 220 has a rectangular shape in the example shown in Figure 2, the image frame 220 may have other shapes in other embodiments. In some examples, certain pixels, such as pixels near the edge of the image frame 220, may not be considered valid (e.g., not sufficiently reliable) and may be removed from the distance data.

[0023] For example, a first pixel 210a has a ray direction 215a that extends straight from the TOF sensor 110, the pixel 210a being located at the center of the image frame 220. A second pixel 210b at a corner of the image frame 220 is associated with a ray direction 215b that extends, for example, from the TOF sensor 110 at an angle of 30° in both an x-direction and a y-direction from the center of the image frame 220, where the image frame 220 is an xy-plane in a reference frame of the TOF sensor 110. The TOF sensor 110 returns distance data (e.g., a distance, one or more phase shifts) to a surface along the ray of each valid pixel.In one example, the first pixel 210a may have a measured distance of 1 meter, representing a distance to a particular point on a target object, and the second pixel 210b may have a measured distance of 2 meters, representing a distance to a particular point on a wall behind the target object.

[0024] With further reference to Fig. 1, the processor 120 may include, implement, or execute various components or processes for performing object measurement, as described herein. For example, the processor 120 may implement or execute components or processes based on corresponding instructions, associated parameters, etc., stored in the memory 150. In some examples, the processor 120 may include, implement, or execute the following: a detection verification component 158 ​​for verifying that images received from the TOF sensor 110 were captured in the desired poses for performing object measurement, a cloud generation component 160 for generating a 3D point cloud based on image capture from the TOF sensor 110, a plane identification component 162 for identifying large planes in the 3D point cloud, a feature extraction component 164 for determining features associated with points in the 3D point cloud,a multi-cloud merging component 166 for associating multiple 3D point clouds in a common coordinate system based on the detection of corresponding points in the 3D point clouds, a target isolation component 168 for isolating one or more target objects detected in the common coordinate system, and / or a BLH component 170 for calculating a bounding box around the target object and / or calculating or otherwise displaying a representation of BLH measurements or other measurements associated with the target object.

[0025] In some examples, the TOF sensor 110 may capture multiple images of an environment including one or more objects. For example, the TOF sensor 110 may be used to capture views of multiple images (e.g., image pairs) at each of multiple positions around the one or more objects, where each view pair may include two (or more) images captured by the TOF sensor 110 at a particular pose. Collecting image pairs in consecutive TOF frames may enable an improved signal-to-noise ratio (SNR) for the image pair to enable robust stitching of the corresponding point cloud, as further described herein. For example, if a captured object is dark or shiny, not enough photons transmitted by the TOF sensor 110 toward the object may return to the TOF sensor 110 for observation by the receiver of the TOF sensor 110.Thus, collecting multiple TOF frames at each pose can increase the number of returned photons to obtain a denser aggregated point cloud at that position. The images may also include, or be referred to as, range maps, depth maps, or depth images. In some examples, the TOF sensor 110 may additionally or alternatively capture an infrared (IR) image with a depth image, where the IR image may support further post-processing of the view to discard noisy data.

[0026] In some examples, the TOF sensor 110 may be used to acquire view pairs at a certain number of locations (e.g., positions of the TOF sensor 110 relative to an object, angles of view of the TOF sensor 110 at the locations, or other orientations, etc., of acquisition) that have substantially the same azimuth around the one or more objects to acquire a certain azimuth range around the one or more objects, e.g., a total of 90 degrees. In a specific example, the TOF sensor 110 may be used to acquire four view pairs at each of four locations substantially 30 degrees azimuth apart. The TOF sensor 110 may provide the view pairs and / or parameters corresponding to the TOF sensor 110 to the processor 120.In some examples, the capture verification component 158 ​​may validate or otherwise ensure that the view pairs are captured at desired locations for effective object sizing, as further described herein.

[0027] In some examples, the cloud generation component 160 may receive the view pairs from the TOF sensor 110 and generate 3D point clouds for each view pair (e.g., for each location). For example, the cloud generation component 160 may generate an aggregated 3D point cloud for a given location based on the multiple views in the view pair, as described, and / or generate a separate point cloud for each location. In some examples, the plane identification component 162 may identify large planes in each of the multiple 3D point clouds as collections of neighboring points sharing the same surface normal, where a large plane may be determined to have a minimum threshold size (e.g., a minimum threshold number of points sharing the same surface normal). In some examples, the feature extraction component 164 may determine or calculate features associated with each point in a given 3D point cloud.For example, the feature extraction component 164 may generate a point feature histogram (PFH) or fast PFH (FPFH) describing each point in the 3D point cloud. In some examples, the multi-cloud stitching component 166 may associate points in the multiple 3D point clouds as having a point correspondence (e.g., the same real-world point captured in different poses) based on the determined features for the points of multiple 3D point clouds. This allows the multi-cloud stitching component 166 to generate a more complete image (e.g., point cloud) of the environment in a common coordinate system, as described in further detail and in various examples herein, which may enable improved measurement of the BLH dimensions of one or more objects.

[0028] In some examples, the target isolation component 168 may detect one or more target objects in the shared coordinate system point cloud created by the multi-cloud stitching component 166, which may be based on detecting a size of the target object in the shared coordinate system as at least a threshold size. In some examples, the BLH component 170 may create a bounding box around the target object based on identifying the points in the shared coordinate system point cloud that correspond to the target object. In some examples, the BLH component 170 may calculate or otherwise display, for example, via the display device 140, an indication of the BLH of the target object, which may include displaying the actual measured BLH dimensions of the target object (e.g., the dimensions of the bounding box), displaying the bounding box, etc.In one example, the BLH component 170 may also display a representation of the environment or target object as observed at a current location, or a representation of the shared coordinate system point cloud, etc. In some examples, the BLH component 170 may display the BLH measurements, the bounding box, etc. on the representation of the environment or target object or the shared coordinate system point cloud, as described in various examples herein.

[0029] Fig. 3 illustrates an example of a multi-cloud stitching component 166 according to aspects described herein. The multi-cloud stitching component 166 may include a tiepoint matching component 302 that may receive features such as FPFH descriptors for multiple 3D point clouds (e.g., from the feature extraction component 164). The tiepoint matching component 302 may perform a first-pass tiepoint matching process to determine point pairs in at least two point clouds with a point correspondence. For example, the tiepoint matching component 302 may determine that the point pairs have descriptors that are within a threshold (e.g., have values ​​within a threshold of each other), which may indicate that the points correspond to the same real-world point captured in the images.In some examples further described herein, an outlier suppression component 304 may perform one or more processes to refine the points matched by the tiepoint matching component 302 to remove outliers and accelerate the associated processing of the point clouds for object dimensioning. Given the reduced set of corresponding points, a cloud pair registration component 306 may derive the rotation and translation of point clouds based on the detected tiepoints. In some examples, a motion synchronization component 308 may calculate an absolute TOF pose, which may be specified in a coordinate system associated with a first TOF image, and a bundle adjustment component 310 may vary the TOF rotations and translations, as well as reconstructed points, to reduce errors in the dimensioning process.The multi-cloud stitching component 166 may output an aggregated cloud and reconstructed frustums in the first TOF coordinate system, which may be used by the target isolation component 168 to isolate a target object in the images for performing a dimensioning thereof, as described in more detail herein.

[0030] Fig. 4 illustrates an example of a method 400 for validating TOF images for performing object sizing according to aspects described herein. The method 400 may be performed by a sensor system 100, one or more components thereof, e.g., a TOF sensor 110, a processor 120, etc.

[0031] The method 400 may include, at 402, displaying a prompt to capture an image of an object. For example, the display device 140, e.g., in conjunction with the processor 120, the memory 150, etc., may display the prompt to capture the image of the object. The display device 140 may display the prompt as part of activating, initializing, or powering on the sensor system 100, running an application for image capture and / or object sensing, etc. The processor 120 may cause the prompt to be displayed as a user interface on the display device 140 (e.g., a graphical user interface that may be part of the application running on the processor 120). The prompt may display a live camera view of the environment while the sensor system 100 is moved in the environment, and the live camera view may be a view of the TOF sensor 110, displayed as an infrared (IR) image, etc.The prompt may also include an interactive option to capture an image.

[0032] The method 400 may include, at 404, capturing an image of the object based on an interaction with the prompt. For example, the processor 120 or the corresponding application executing on the processor 120 may detect, e.g., via a display device (as a touchscreen display) or other input device, the interaction with the prompt to select the interactable option to capture the image from the TOF sensor 110 in conjunction with the memory 150, etc. Furthermore, as described in some examples, the processor 120 may capture multiple images from the TOF sensor 110 based on the interaction with the prompt, e.g., two (or more) images in a view pair. In one example, the number of images to be captured and / or the time between image captures upon detection of an interaction with the prompt may be configurable to optimize image quality for object sizing.

[0033] In this regard, multiple discrete images can be captured, unlike continuous video capture. Capturing multiple discrete images may be easier to capture accurately than continuous video. For example, with continuous video, it may be difficult to keep the system stable, perform continuous movement of the system to capture the correct views over a period of time, etc. Using discrete images may allow the user time to properly align the TOF sensor 110 before capturing the one or more images to be processed.

[0034] After the image is captured at 404, method 400 may include determining at 406 whether to retain the image. For example, capture verification component 158, in conjunction with processor 120, memory 150, etc., may determine whether to retain the image. For example, determining whether to retain the image may include one or more different actions performed by a user of sensor system 100 or automatically performed by the sensor system to ensure that images capable of supporting object sensing are captured and provided for further processing.

[0035] In one example, determining at 406 whether to retain the image may optionally include querying at 408 whether to retain the image. For example, the capture verification component 158, in communication with, for example, the processor 120, the memory 150, etc., may display, via the display device 140, a query asking whether to retain the image, which may be displayed after each image capture, after a series of images have been captured, etc. Based on the interaction with the query, the capture verification component 158 ​​may determine whether to retain the image.

[0036] In another example, determining at 406 whether to retain the image may optionally (or alternatively) also include determining at 410 whether the image conforms to parameters for effective object sizing. For example, the detection verification component 158, e.g., in conjunction with the processor 120, the memory 150, etc., may determine whether the image conforms to parameters for effective object sizing. In some examples, the detection verification component 158 ​​may ensure that a pair of views has sufficient discrimination to provide at least a threshold improvement in SNR for the pair of views over a single one of the views. In another example, the detection verification component 158 ​​may ensure that an object in the image is recognizable, e.g.,that an object has a sufficiently large number of associated points in a point cloud to enable the object to be identified and accurately measured from the image. For example, the size of the object in the image may be configurable, and the capture verification component 158 ​​may detect whether the image includes an object that has at least the configured size to determine whether to retain the image.

[0037] If it is determined at 406 that the image should not be retained, the method may proceed to 402 to display another prompt to capture an image. Method 400 may optionally include, at 412, providing device positioning feedback. For example, capture verification component 158, e.g., in conjunction with processor 120, memory 150, etc., may provide device positioning feedback, e.g., via display device 140 or another output device, e.g., haptic feedback via a haptic feedback device in sensor system 100, audible or visual feedback via display device 140, etc. In this regard, sensor system 100 may, for example, assist a user in capturing images at positions / orientations that may result in effective object sizing, as further described herein.In some examples, the sensor system 100 may include one or more sensors to enable positioning, such as an accelerometer, a compass, or other displacement or position sensors that can provide position information to the sensor system 100. In another example, the environment in which the sensor system 100 captures images may include external sensors, such as a TOF scanner, a camera that can detect the presence and position of the sensor system 100, etc. In either case, such internal or external sensors may be used to detect the position of the TOF sensor 110, to determine whether a captured image is in a desired position / orientation, and / or to provide real-time feedback to guide the user to properly position the TOF sensor 110 to capture the image.

[0038] For example, the detection verification component 158 ​​may obtain position information for the sensor system 100, or at least the TOF sensor 110 thereof, and determine whether the TOF sensor 110 is in a position to capture a next image for object sizing (e.g., determine whether the TOF sensor 110 is in a position that is within a particular azimuth of a position at which a previous image is captured). In this example, the detection verification component 158 ​​may provide feedback via the display device 140 regarding whether or not the sensor system 100, or at least the TOF sensor 110, is in the correct (or acceptable) position / orientation for capturing the next image.In one example, based on the currently detected position / orientation of the sensor system 100, or at least the TOF sensor 110, and a known desired position / orientation for the next image acquisition, the acquisition verification component 158 ​​may provide feedback to guide the user of the sensor system 100 to the desired or acceptable position. For example, the acquisition verification component 158 ​​may provide the feedback in the form of an arrow displayed on the display device 140 indicating a direction to move the sensor system 100 to the desired position, a box or other indicator displayed on the display device 140 showing where to align a center of the display device 140, etc. In one example, the acquisition verification component 158 ​​may indicate when the sensor system 100 is in a desired or acceptable position for acquiring the next image, e.g.,by displaying an icon or pop-up message, enabling a selectable option to capture the image, etc.

[0039] If at 406 it is determined to retain the image, the method 400 may include determining at 414 whether the images are sufficient for object sizing. For example, the capture verification component 158, in conjunction with the processor 120, the memory 150, etc., may determine whether the images are sufficient for object sizing. In some examples, the capture verification component 158 ​​may determine whether there are enough images for object sizing. In some examples, the capture verification component 158 ​​may determine whether the images are captured at the desired positions / orientations around the object to enable object sizing, as described below with reference to the method 500 in Fig. 5 (e.g., as part of the validation at 504 as to whether the multiple images are captured at the desired positions / orientations).

[0040] If it is determined at 414 that the images are insufficient for dimensioning, the method 400 may return to 402 to display the request to capture an image of the object. In some examples, as described above at 412, feedback about device positioning may be provided to assist in capturing an image useful for dimensioning. For example, if the capture verification component 158 ​​determines that a given image is insufficient for dimensioning, the capture verification component 158 ​​may provide feedback about recapturing the image, which may include an indication of the image to be recaptured. The indication may include, for example, displaying the image to be recaptured, indicating an index of the image in the set of multiple images to be recaptured, indicating a position of the device for capturing the image (e.g.,displaying arrows leading to the desired position for the image to be recaptured), etc. In another example, the capture verification component 158 ​​may discard all images and display the prompt to begin the initial capture.

[0041] If it is determined at 414 that the images are insufficient for dimensioning, the method 400 may include, at 416, providing a plurality of images for object dimensioning to calculate or display dimensions of the object. For example, the acquisition verification component 158, e.g., in connection with the processor 120, the memory 150, etc., may provide the plurality of images (e.g., to the cloud generation component 160 or one or more other components) for object dimensioning to calculate or display dimensions of the object. In one example, the acquisition verification component 158 ​​may display a prompt to perform object dimensioning on the display device 140, where the verification component 158 ​​may display the prompt based on validating that the object dimensioning can be performed using the images acquired for the object.As described, this may include a series of captured images, an SNR of the images, a detection angle relative to an object, etc.

[0042] Furthermore, in some examples, after acquiring an image at 418, preprocessing may be used to make the acquired image available for object sizing while waiting for additional images to be acquired. This may allow processing of the image and / or other images for object sizing to begin while waiting for other images to be acquired. For example, as further described herein, performing the object sizing actions may take some time, and preprocessing in this regard may allow some computational actions (e.g., point cloud generation, plane identification, feature extraction, multi-cloud stitching, etc.) to be performed while images are being acquired, which may reduce the overall processing time after all images have been acquired.In this regard, the processing time perceived by the user between the time of specifying the object dimension to be performed and receiving an output of the object dimension can be reduced by performing the preprocessing at 418.

[0043] The method 400 may optionally include, at 420, receiving BLH measurements of the object. For example, the BLH component 170, e.g., in conjunction with the processor 120, the memory 150, etc., may receive or otherwise determine BLH measurements of the object based on other components that perform object measurement, as further described herein. The method 400 may optionally include, at 422, displaying the BLH measurements of the object. For example, the display device 140, e.g., in conjunction with the processor 120, the memory 150, etc., may display the BLH measurements of the object. This may include, for example, as further described herein, displaying the measurement numbers, displaying a bounding box around a display of the object, etc.

[0044] In some examples, method 400 may optionally include, at 424, outputting a feedback score indicating the degree to which the plurality of images correspond to the desired poses. For example, acquisition verification component 158, in conjunction with processor 120, memory 150, etc., may output the feedback score indicating the degree to which the plurality of images correspond to the desired poses. Thus, as described above and further herein, feedback may be provided during image acquisition (e.g., at 412) or after acquisition (e.g., at 424). In this regard, feedback may be provided, for example, per image (or per currently acquired images) or for the image set, based on further processing of the images or attempts at association (or stitching), as further described herein.

[0045] In one example, the acquisition verification component 158 ​​may receive from one or more other components of the object measurement system (e.g., the multi-cloud stitching component 166) an indication of a feedback score for the images as used in object measurement. The feedback score may relate to how close the images are to the desired azimuth. In one example, this may be based on a distance to the center of certain points in the corresponding point clouds that have point correspondences with points in other point clouds. The feedback score may be based on other measurements of the images, e.g., SNR, etc. The feedback score may indicate a sufficiency or quality of the image set and / or be selected from a set of labels, or be numerically or alphabetically based.

[0046] Fig. 5 illustrates an example of a method 500 for obtaining TOF images for performing object sensing according to aspects described herein. The method 500 may be performed by a sensor system 100, one or more components thereof, e.g., a TOF sensor 110, a processor 120, etc.

[0047] The method 500 may include, at 502, obtaining, from a TOF sensor, a plurality of images including an object. For example, the detection verification component 158, e.g., in conjunction with the processor 120, the memory 150, etc., may obtain the plurality of images including the object from the TOF sensor 110. For example, the TOF sensor 110 may perform the plurality of detections as described above, which may include acquiring view pairs at each of a plurality of positions / orientations around the object. In some examples, the TOF sensor 110 may be operated by a user or other device to acquire views around an azimuthal range of the object, as described, which may include acquiring view pairs at a plurality of positions around the azimuthal range. In some examples, the positions may be within an azimuthal range of each other.In a specific example, the TOF sensor 110 may capture view pairs at each of four positions within an azimuthal range of substantially 30 degrees from each other to capture a total azimuthal range of substantially 90 degrees around the object.

[0048] The method 500 may include, at 504, validating whether the plurality of images are captured at the desired positions relative to the object. For example, the capture verification component 158, e.g., in conjunction with the processor 120, the memory 150, etc., may validate (e.g., before or as part of generating associated 3D point clouds) whether the plurality of images are captured at the desired positions relative to the object. In some examples, the capture verification component 158 ​​may validate whether the plurality of images are captured at the desired positions / orientations based on information provided in or along with the image received from the TOF sensor 110.For example, the TOF sensor 110 may indicate accelerometer, compass, or other displacement / position sensor measurements of the TOF sensor 110 at the time the views (or view pairs) are captured at each position, and the capture verification component 158 ​​may determine whether the measurements indicate that the views are at the desired position. For example, the capture verification component 158 ​​may compare the measurements to each other, compare a difference in the measurements to a threshold, compare the measurements to a threshold, etc., to validate whether the multiple images are captured at desired positions (e.g., substantially within a particular azimuth or azimuth range from each other, as described).

[0049] In some examples, other external sensors in the environment, such as a TOF scanner, a surveillance camera in a room, etc., may be used to track and / or observe the position of the TOF sensor 110 as it captures views of the environment. The external sensors may detect the information or measurements of the TOF sensor 110 as it acquires the images and provide the information or measurements that the capture verification component 158 ​​may use to validate whether the images are captured at desired positions. In some examples, the capture verification component 158 ​​may validate the images based on a model of an object detected in the images. For example, the capture verification component 158 ​​may infer the position of the capture for each of the image(s) based on the position of the object modeled in each image.In one example, the acquisition verification component 158 ​​may detect the object and an orientation of the object in each image. In this example, the acquisition verification component 158 ​​may compare the orientations in multiple acquisitions to derive the position information for the image (e.g., the azimuth or the difference in azimuth between acquisitions).

[0050] The method 500 may include determining at 506 whether the images are captured in desired poses. For example, the component 158, e.g., in conjunction with the processor 120, the memory 150, etc., may determine whether the images are captured in the desired poses (e.g., whether the images are validated at 504). If the result is yes, the plurality of images may be provided at 508 for object sizing to calculate or display dimensions of the object, as further described herein. If the result is no, an error or other feedback may optionally be triggered at 510. For example, the capture verification component 158, e.g., in conjunction with the processor 120, the memory 150, etc., may provide the images (e.g., to the cloud generation component 160 or one or more other components) for object sizing, or in those cases, trigger the error or feedback.

[0051] In one example, the acquisition verification component 158 ​​may trigger the error or feedback as an indication displayed on the display device 140, which may include an indication that the object measurement cannot be performed, or may include one or more of the feedback indications described in method 400 at 412 for acquiring additional images, etc. For example, if the acquisition verification component 158 ​​determines that one of the images is not captured in a desired pose, the acquisition verification component 158 ​​may display, via the display device 140, a prompt to recapture the image in the desired pose, which may include an indication of the image or pose to recapture, as described above, arrows, or other visual or audible (or haptic) feedback to guide the user to position the TOF sensor 110 for appropriate acquisition of the image, etc.to be positioned properly.

[0052] Fig. 6 illustrates an example of a method 600 for associating point clouds from different acquisition poses to perform object measurement according to aspects described herein. For example, the pose may refer to the angle and position of the acquisition (e.g., the rotation and translation, as further mentioned herein). The method 600 may be performed by a sensor system 100, one or more components thereof, e.g., a processor 120, etc.

[0053] The method 600 may include, at 602, associating a first subset of points in a first 3D point cloud with a second subset of points in a second 3D point cloud based on each point in the first subset of points having a threshold correspondence with a unique counterpart point in the second subset of points. For example, the tiepoint matching component 302, e.g., in conjunction with the processor 120, the memory 150, the multi-cloud stitching component 166, etc., may associate the first subset of points in the first 3D point cloud (e.g., a first TOF view, which may include multiple images) with the second subset of points in the second 3D point cloud (e.g., a second TOF view, which may include multiple images) based on each point in the first subset of points having a threshold correspondence with the unique counterpart point in the second subset of points.The point and its unique counterpart points may be referred to herein as tiepoints. In one example, for a set of images acquired using a TOF sensor 110, as described above, the cloud generation component 160 may generate the 3D point clouds for the images, including the first 3D point cloud generated for an image (or view pair, or other set of multiple images) acquired at a first location, the second 3D point cloud generated for an image (or view pair, or other set of multiple images) acquired at a second location, and / or other point clouds. For example, the cloud generation component 160 may generate the point clouds at least in part by multiplying pixels in a TOF range map generated by the TOF sensor 110 during image acquisition by its corresponding ray direction vector, as described above.The cloud generation component 160 may ignore pixels with invalid range or IR measurements, and the resulting point cloud may be located within the acquired coordinates of the TOF sensor 110.

[0054] Additionally, in some examples, plane identification component 162 may identify large planes in each of the multiple 3D point clouds. Plane identification component 162 may identify a ground plane and / or possibly wall planes in the 3D point clouds. In some examples, plane identification component 162 may calculate the surface normal for valid pixels in the TOF view and search for large connected components (e.g., clusters of pixels) with nearly constant normal angle coordinates. Random Sample Consensus (RANSAC) may be performed to fit 3D planes through the connected component regions. For a small TOF camera roll angle, the ground plane normal and the image plane vertical direction vector may overlap significantly. The dot product of these two can be used to distinguish the ground plane from the wall planes in the acquired images.

[0055] The method 600 may optionally include, at 604, generating a first FPH for points in the first subset of points and a second FPH for points in the second subset of points. For example, the feature extraction component 164, e.g., in conjunction with the processor 120, the memory 150, etc., may generate the first FPH for points in the first subset of points and the second FPH for points in the second subset of points. For example, the feature extraction component 164 may generate the PFH as FPFHs or other PFHs that include a geometric descriptor for each point or collection of points in the point cloud that describes the points. The descriptor may include multiple features or other parameters or values ​​that describe the point and surrounding points, e.g., how smooth or rough the point and surrounding parts are, an angle or angular measurement represented by the point and surrounding points, etc.For example, the feature extraction component 164 may generate the FPH by, for each of a number n of points, considering pairs of the k nearest neighboring points, where n and k are positive integers. In this example, the feature extraction component 164 may generate a local reference frame (e.g., a Darboux frame) between a neighbor pair, calculate an angular deviation between the normal vectors with respect to the local reference frame, and insert the angular deviation of pairs into the histogram.

[0056] In one example, the feature extraction component 164 may use FPFH to generate the histograms, which may have 33 values ​​(e.g., each value in the range of 0 and 255) in the descriptor for each point. In some examples, the tiepoint matching component 302 may associate the point sets based on the descriptors received from the feature extraction component 164. In one example, the tiepoint matching component 302 may attempt to identify, for a given point in the first subset of points, a unique counterpart point in the second subset of points based on comparing the descriptors or corresponding descriptor values ​​of the points. For example, the tiepoint matching component 302 may associate points with similar values ​​for the 33 values, or a similar aggregate value representing the 33 values, etc.

[0057] Fig. 7 illustrates an example of point clouds 700 and 702 and corresponding descriptor spaces, e.g., descriptor spaces 704 and 706, according to aspects described herein. Point cloud 700 may represent one point cloud, denoted in the figure as left-hand side (LHS), of a leaf captured by a TOF sensor 110 in a real 3D space, and point cloud 702 may represent a second point cloud, denoted in the figure as right-hand side (RHS), of the same leaf captured by the TOF sensor 110 in the real 3D space from a different position / orientation. Feature extraction component 164 may generate FPFH descriptors for the LHS point cloud 700 and the RHS point cloud 702. As described, in one example, the FPFH descriptor may be a 3D space describing each point. In one example, the tiepoint matching component 302 may detect point f in the LHS point cloud 700 and point f' in the RHS point cloud that have similar descriptors, e.g.have similar values ​​in the descriptor space, similar descriptor values, where the descriptor is a single value representing the different dimensions, etc. In one example, although the point clouds 700 and 702 may have unknown rotations and translations relative to each other, the feature extraction component 164 may detect that the descriptors for point f and point f' are within a threshold difference, and these may be considered as possible unique counterpart points in the point clouds 700 and 702. For example, the tiepoint matching component 302 may determine that point f 708 corresponds to point F in the 33D descriptor space 704 and that point f' corresponds to a similar point F' in the 33D descriptor space 706, and thus points f and f' are to be considered as possible unique counterpart points.

[0058] With further reference to Fig. 6, the method 600 may optionally include, at 606, downsampling a first acquired point cloud and a second acquired point cloud. For example, the cloud generation component 160, e.g., in conjunction with the processor 120, the memory 150, etc., may downsample the first acquired point cloud (e.g., to generate the first 3D point cloud) and the second acquired point cloud (e.g., to generate the second 3D point cloud). Downsampling the point clouds at least for the purpose of identifying tiepoints may reduce the number of points to be evaluated and / or for which a descriptor is to be calculated, which may reduce the processing resources required to perform tiepoint matching. For example, the cloud generation component 160 may downsample the acquired point clouds into a voxelized point cloud, where each voxel is an average of the pixels downsampled.In one specific example, the cloud generation component 160 may voxelize the point cloud based on a distance, e.g., four centimeters (cm), such that each voxel is an average of points in a 4 cm x 4 cm cuboid in which the voxel is the center point. In any case, in some examples, the feature extraction component 164 may calculate the FPFH descriptor at each voxel location in the point clouds (after downsampling or otherwise), and the tiepoint matching component 302 may identify descriptors and associate the points if they are close to each other in 33D descriptor space. In one example, the tiepoint matching component 302 may apply a Lowe ratio test to reduce false descriptor matches. In this regard, for example, the tiepoint matching component 302 may associate the first subset of points in the first 3D point cloud with the second subset of points in the second 3D point cloud.This association may yield many associations, which can be refined as described here to further reduce the processing required to consider each association at a later stage of object design.

[0059] In another example, the method 600 may optionally include, at 608, extracting the first subset of points and the second subset of points with a threshold descriptor entropy. For example, the feature extraction component 164, e.g., in conjunction with the processor 120, the memory 150, etc., may extract the first subset of points and the second subset of points with the threshold descriptor entropy. This may be a further optimization to perform the association of points at 602. Furthermore, in some examples, the feature extraction component 164 may extract the features from the downsampled point clouds for further optimization. In one example, entropy may be defined as a logarithm function of the descriptor values, which may provide an indication of how different the descriptor values ​​are from each other (e.g.,(e.g., how much variance exists between the 0-255 values ​​of each of the 33 different descriptors). A higher entropy value may indicate an FPFH with a richer internal structure, which may yield a more unique or interesting descriptor, so finding matching points with such descriptors may be a more likely actual real-world match. In this example, feature extraction component 164 may correlate the FPFH descriptor entropy with the geometric interest of the point's real-world 3D location. In some examples, feature extraction component 164 may ignore low-entropy points to increase the efficiency of subsequent actions in object sizing, described below.

[0060] The method 600 may include, at 610, refining the first subset of points to generate a first improved subset of points that includes fewer outlier point correspondences with the second subset of points than the first subset of points. For example, the outlier suppression component 304, e.g., in conjunction with the processor 120, the memory 150, the multi-cloud stitching component 166, etc., may refine the first subset of points to generate the first improved subset of points that includes fewer outlier point correspondences with the second subset of points than the first subset of points. Similarly, the method 600 may also include, at 612, refining the second subset of points to generate a second improved subset of points that includes fewer outlier point correspondences with the first subset of points than the second subset of points.For example, the outlier suppression component 304, e.g., in conjunction with the processor 120, the memory 150, the multi-cloud stitching component 166, etc., may refine the second subset of points to produce the second improved subset of points that includes fewer outlier point correspondences with the first subset of points than the second subset of points.

[0061] In other words, in some examples, given the point correspondences associated between the first subset of points in the first 3D point cloud and the second subset of points in the second 3D point cloud, the outlier suppression component 304 may break some of the point correspondences to improve the accuracy and / or efficiency of the remaining actions in object measurement, such as cloud pair registration or motion synchronization, as further described herein. In some examples, the outlier suppression component 304 may evaluate multiple points from the first subset with multiple points from the second subset to verify whether the multiple points should be retained as tiepoints between the two subsets of points.For example, the outlier suppression component 304 may calculate a distance between the plurality of points in each 3D point cloud to determine whether the distances are within a threshold correspondence. If so, the tiepoints between the first and second subsets may remain associated. If not, the outlier suppression component 304 may break the association between at least one of the plurality of points in the first subset and the corresponding one of the plurality of points in the second subset.

[0062] Fig. 8 illustrates an example of point clouds 800 and 802 and corresponding feature spaces, e.g., descriptor spaces 804 and 806, as well as point clouds 810 and 812 and a graph representation of tiepoint consistency 814, according to aspects described herein. Point cloud 800 may represent an LHS point cloud of a leaf captured by a TOF sensor 110 in real 3D space, and point cloud 802 may represent an RHS point cloud of the same leaf captured by the TOF sensor 110 in real 3D space from a different position / orientation. Feature extraction component 164 may generate FPFH descriptors for LHS point cloud 800 and RHS point cloud 802. In one example, the tiepoint matching component 302 may detect a point correspondence of point f in the LHS point cloud 800 and point f' in the RHS point cloud 802 based on them having similar descriptors F and F' in the 33D descriptor spaces 804 and 806.Similarly, in one example, the tiepoint matching component 302 may detect a point correspondence of point g in the LHS point cloud 800 and point g' in the RHS point cloud 802 based on them having similar descriptors G and G' in the 33D descriptor spaces 804 and 806. In one example, the outlier suppression component 304 may determine that a first distance between f and g and a second distance between f' and g' are within a threshold correspondence with respect to their corresponding real-world space, and may accordingly retain the associations between f and f' and g and g' in the enhanced subsets of points.

[0063] Point cloud 810 may represent a point cloud, denoted in the figure as LHS, of a leaf captured by a TOF sensor 110 in real 3D space, and point cloud 812 may represent a second point cloud, denoted in the figure as RHS, of the same leaf captured by the TOF sensor 110 in real 3D space from a different position. Feature extraction component 164 may generate FPFH descriptors for the LHS point cloud 810 and the RHS point cloud 812. In one example, the tiepoint matching component 302 may detect a point correspondence of points 820-a, 822-a, 824-a, and 826-a in the LHS point cloud 800 and points 820-b, 822-b, 824-b, and 826-b in the RHS point cloud 802, respectively, based on having similar descriptors in the 33D descriptor spaces.In one example, the outlier suppression component 304 may determine that distances between points 820-a, 822-a, 824-a and / or between 820-b, 822-b, 824-b, etc., are within a threshold correspondence, and may accordingly maintain the associations between these points in the respective enhanced subsets of points. In one example, the outlier suppression component 304 may determine that a first distance between point 826-a and one of points 820-a, 822-a, or 824-a, and a second distance between 826-b and a corresponding one of 820-b, 824-b, or 826-b are not within a threshold correspondence, and may accordingly break the association between points 826-a and 826-b in the respective enhanced subsets of points.This is also demonstrated in the tiepoint consistency diagram, where a correspondence exists at each of points 820 (820-a and 820-b), 822 (822-a and 822-b), and 824 (824-a and 824-b), and thus, these points can be said to form a group, but there is no correspondence for point 826 (826-a and 826-b). By disassociating points 826-a and 826-b as outliers, the processing performed in subsequent object sizing actions can be reduced, as described below.

[0064] For example, as described, outlier tiepoint suppression may be a maximum group problem that can be solved to find the tiepoints using threshold correspondences and distances to determine which points in the subsets are most likely the same point in a common coordinate system (even though their coordinates may be different in the corresponding 3D point clouds of the acquired TOF frames). In another example, the outlier suppression component 304 may further search for tiepoint triangles (e.g., instead of or after searching for pairs) that have a point correspondence and distance correspondence to determine which points to retain in the enhanced subsets of points.For example, the outlier suppression component 304 may search the first subset of points for tiepoint triangles (f, g, h) among the remaining features that match a triangle (f', g', h') in the second subset of points. In this example, the outlier suppression component 304 may retain the endpoints (a, b) and (a', b') from the longest sides of the triangles in the first and second enhanced subsets of points, respectively. Furthermore, in this example, the outlier suppression component 304 may retain the tiepoint (c, c') when |c - a| = |c' - a'| and |c - b| = |c' - b'| hold up to some noise threshold. This may result in the enhanced point sets having further geometric consistency.

[0065] The method 600 may include, at 614, determining, based on point correspondences between the first enhanced subset of points and the second enhanced subset of points, a relative rotation and translation between the first 3D point cloud and the second 3D point cloud. For example, the cloud pair registration component 306, in conjunction with, for example, the processor 120, the memory 150, the multi-cloud stitching component 166, etc., may determine the relative rotation and translation between the first 3D point cloud and the second 3D point cloud based on point correspondences between the first enhanced subset of points and the second enhanced subset of points.In one example, at least three valid tiepoint matches may be sufficient to properly determine the relative rotation and translation between the point clouds, and the outlier suppression component 304 may generate the improved subsets of points to attempt to include valid tiepoints using the processing described above. Given the improved tiepoint sets, the cloud pair registration component 306 may derive the rotation R and translation T of the second 3D point cloud with respect to the first 3D point cloud (or vice versa) by removing centers of mass from the sets of 3D inputs, calculating R using a singular value decomposition, and correcting T after R is known. In one example, the cloud pair registration component 306 may perform such cloud pair registration for TOF frame pairs that satisfy at least a threshold number (e.g.,10) have geometrically consistent tiepoint matches. For example, if the TOF frames do not have at least the threshold number of matches, the multi-cloud stitching component 166 may reject the TOF frames or otherwise send an error or other feedback to acquire new or additional replacement TOF frames, as described above. In another example, where the TOF frames have more than the threshold number of matches, the cloud pair registration component 306 may select a number of tiepoints equal to the threshold for determining rotation and translation (e.g., the first number of tiepoints, a random selection of the number of tiepoints, etc.).

[0066] The method 600 may include, at 616, aligning the first 3D point cloud and the second 3D point cloud in a common coordinate system based on the relative rotation and translation. For example, the cloud pair registration component 306, in conjunction with the processor 120, the memory 150, the multi-cloud stitching component 166, the motion synchronization component 308, the bundle matching component 310, etc., may align the first 3D point cloud and the second 3D point cloud in the common coordinate system based on the relative rotation and translation. In one example, the cloud pair registration may result in a redundant set of relative poses among TOF frames, and relative rotations and translations may have associated noise.Thus, for example, as part of determining relative rotation and translation at 614 and / or aligning the point clouds at 616, the motion synchronization component 308 may calculate absolute TOF images that optimally match the input set of relative transformations to synchronize the TOF images. The TOF poses may essentially be any common coordinate system, which may include the coordinate system of the first TOF image.

[0067] For example, the cloud pair registration component 306 may align the point clouds to the common coordinate system based on the determined rotation and translation, which may be the coordinate system of the first TOF image. In this regard, the aligned point clouds may create a combined point cloud in the common coordinate system. In some examples, aligning the point clouds may include the bundle adjustment component 310, which more accurately aligns the 3D point clouds. For example, the bundle adjustment component 310 may start with the 3D positions of the motion-synchronized tie points output by the motion synchronization component 308 and initialize reconstructed points based on averages of synchronized points.Bundle matching component 310 can form cost functions from discrepancies between reconstructed points and backprojected counterparts and minimize the costs by varying the TOF rotations and translations and the reconstructed point positions. The result can be aggregated point clouds and corresponding stumps in a common coordinate system, which can be the coordinate system of one of the TOF images (e.g., the first TOF image). Scene knowledge can propagate via 3D geometry between the merged point cloud and the TOF views.

[0068] In some examples, the multi-cloud stitching component 166 may perform the actions described above in method 600 for multiple additional point clouds corresponding to other TOF images, e.g., for four point clouds corresponding to four TOF images located substantially 30 degrees in azimuth around an object, and may construct the stitched point cloud of the four point clouds in the common coordinate system. In one example, the display device 140 may display a representation of the stitched point cloud. In another example, the stitched point cloud may be further used for target object isolation, object sizing, etc., as further described herein.

[0069] Fig. 9 illustrates an example of a method 900 for isolating a target object in multiple TOF images to perform its sizing in accordance with aspects described herein. The method 900 may be performed by a sensor system 100, one or more components thereof, e.g., a processor 120, etc.

[0070] The method 900 may include, at 902, transforming multiple images from a TOF sensor into multiple 3D point clouds. For example, the cloud generation component 160, in conjunction with the processor 120, memory 150, etc., may transform multiple images from a TOF sensor 110 into multiple 3D point clouds. As described above, the cloud generation component 160 may generate an aggregated 3D point cloud for a given location based on the multiple views in a view pair or other collection of multiple images at a given location. The cloud generation component 160 may also generate a separate 3D point cloud for each location.

[0071] The method 900 may include, at 904, identifying, in each of the plurality of 3D point clouds, possible large planes having a minimum threshold size. For example, the plane identification component 162, e.g., in conjunction with the processor 120, the memory 150, etc., may identify possible large planes having a minimum threshold size in each of the plurality of 3D point clouds. As described above, the plane identification component 162 may identify large planes in each of the plurality of 3D point clouds as collections of adjacent points having the same surface normal, where a large plane may be determined as having a minimum threshold size (e.g., a minimum threshold number of points having the same surface normal). The plane identification component 162 may identify at least one large plane as a floor and / or at least one other large plane as a wall, etc.In some examples, the plane identification component 162 may distinguish the floor plane from the wall plane based on a size of each plane, a proximity of the planes to the Z=0 plane (e.g., the floor may be closer to the Z=0 plane), etc. After stitching the 3D point clouds, as described above and further herein with reference to action 906, the floor plane and / or wall plane and / or other large planes in the common coordinate system of the common 3D point cloud may be identified by the TOF sensor 110 based on their coordinate positions in each corresponding 3D point cloud.

[0072] The method 900 may include, at 906, generating a common 3D point cloud by aligning rotated and translated point clouds corresponding to at least a portion of the multiple 3D point clouds. For example, the multi-cloud stitching component 166, e.g., in conjunction with the processor 120, the memory 150, etc., may generate the common 3D point cloud (in a common coordinate system) by aligning rotated and translated point clouds corresponding to at least the portion of the multiple 3D point clouds. As described above, the multi-cloud stitching component 166 may rotate and translate the multiple 3D point clouds and align the multiple 3D point clouds to generate the common 3D point cloud, as described in various examples above.

[0073] The method 900 may include, at 908, rotating and translating the shared 3D point cloud so that its ground is aligned with a Z-plane. For example, the target isolation component 168, e.g., in conjunction with the processor 120, the memory 150, etc., may rotate and translate the shared 3D point cloud so that its ground is aligned with the Z=0 plane. For example, the plane identification component 162 may identify coordinates in the shared 3D point cloud and / or in the multiple 3D point clouds that comprise the shared 3D point cloud that correspond to large planes. Furthermore, for example, the plane identification component 162 may identify a large plane as the ground, which may include identifying the plane with the largest size, a plane that is substantially at or closest to the ground plane (e.g., plane Z=0), etc.In any case, the target isolation component 168 may rotate and translate the shared 3D point cloud to place the identified ground plane on the ground plane (e.g., at Z=0) in the shared coordinate system.

[0074] The method 900 may include isolating the target object from the rotated and translated joint 3D point cloud at 910. For example, the target isolation component 168, in conjunction with the processor 120, the memory 150, etc., may isolate the target object from the rotated and translated joint 3D point cloud. Various actions may be performed to isolate the target object, as further described in Fig. 10 described.

[0075] Fig. 10 illustrates an example of a method for isolating the target object from the rotated and translated joint 3D point cloud at 910, according to aspects described herein. In some examples, isolating the target object may include deriving an average position of the TOF sensor 110 when capturing the images to infer that a position of the target object is within a threshold distance from a center of the detection position. For example, isolating the target object at 910 may optionally include, at 1002, projecting an average position of reconstructed TOF frustums in the aligned point clouds to the ground plane as the origin. For example, the target isolation component 168 may, for example, operate in conjunction with the processor 120, the memory 150, etc.Project the average position of the reconstructed TOF stumps in the aligned point clouds as the origin in the ground plane, where the origin may correspond to an (X, Y) coordinate on the plane Z=0 on which the TOF image was acquired. For example, the target isolation component 168 may reconstruct the TOF stumps from the 3D point cloud and use their average position, projected into the plane Z=0, as the origin.

[0076] For example, isolating the target object at 910 may optionally include averaging a pointing direction for the reconstructed stumps in the ground plane at 1004. For example, the target isolation component 168, e.g., in conjunction with the processor 120, the memory 150, etc., may average the pointing direction for the reconstructed stumps in the ground plane (e.g., the Z=0 plane), thereby defining a preferred axis from the origin to a region in which the target object can be detected and / or isolated.

[0077] For example, isolating the target object at 910 may optionally include, at 1006, modifying a shared 3D point cloud by removing points that lie outside a sub-cloud annulus of the shared 3D point cloud between two ground plane circles centered at the origin, and removing points outside a sub-cloud azimuthal wedge defined by a specified number of degrees from the pointing direction. For example, the target isolation component 168, e.g., in conjunction with the processor 120, the memory 150, etc., may isolate the shared 3D point cloud by removing points that lie outside a sub-cloud annulus of the shared 3D point cloud between two ground plane circles centered at the origin, and removing points outside the sub-cloud azimuthal wedge defined by a specified number of degrees from the pointing direction.

[0078] For example, by removing points that lie outside a sub-cloud annulus of the shared 3D point cloud between two ground plane circles centered at the origin, the target object distance from the origin can be limited to within a specific interval defined between the sub-cloud annuli (e.g., 0.75 meters to 4.0 meters). This can model an assumption that the user has not placed the TOF sensor 110 too close or too far from the target object captured for dimensioning.

[0079] Furthermore, for example, removing points outside the sub-cloud azimuthal wedge defined by a specified number of degrees from the pointing direction can constrain the target object to lie within the number of degrees from the pointing direction (e.g., 40 degrees in either direction of the preferred axis). This can model the assumption that the user intended to center the TOF sensor 110 largely based on the position of the target object. Removing the outlier points in this example can improve the accuracy of locating the target object to be isolated and / or improve the processing time for identifying and / or isolating the target object.

[0080] For example, isolating the target object at 910 may optionally include, at 1008, modifying the shared 3D point cloud to remove points associated with one or more large planes. For example, the target isolation component 168, e.g., in conjunction with the processor 120, the memory 150, etc., may modify the shared 3D point cloud to remove points associated with one or more large planes. As described above, the target isolation component 168 may detect points in the shared 3D point cloud associated with one or more large planes, such as a floor, a wall, or another large plane, based on the coordinates identified by the plane identification component 162. The target isolation component 168 may remove these points from the shared 3D point cloud to further isolate the target object.

[0081] For example, isolating the target object at 910 may optionally include, at 1010, converting the shared 3D point cloud into a height map and removing points associated with objects below a threshold size or objects that are not closest to the pointing direction from the origin. For example, the target isolation component 168, e.g., in conjunction with the processor 120, the memory 150, etc., may convert the shared 3D point cloud into the height map and remove points associated with an object below a threshold size or objects that are not closest to the pointing direction from the origin. In some examples, the target isolation component 168 may use the remaining points of the shared 3D point cloud after removing points associated with large planes and points outside the sub-cloud annulus and the sub-cloud azimuthal wedge and project them down to the ground plane.This may result in point clouds for objects in the X,Y coordinates. In this example, the target isolation component 168 may remove subclouds (also referred to herein as point clusters or connected components) associated with objects in the ground plane view that are smaller than a threshold size (e.g., 8 inches). Among the remaining subclouds, in some examples, the target isolation component 168 may retain the subcloud(s) closest to the pointing direction from the origin, as described above with reference to action 1004, and discard any remaining subclouds. Thus, the remaining subcloud of the target object may be considered to be the one that has at least a threshold size and is closest to the pointing direction line from the origin to the 3D point cloud space.

[0082] With further reference to Fig.9, the method 900 may optionally include, at 912, calculating dimensional measurements of the target object in the shared coordinate system. For example, the BLH component 170, in conjunction with, for example, the processor 120, the memory 150, etc., may calculate the dimensional measurements of the target object in the shared coordinate system. In some examples, the BLH component 170 may create a bounding box around the target object isolated by the target isolation component 168. For example, the BLH component 170 may search a seven-dimensional parameter space for a closest bounding box surrounding the subcloud of the target object in the shared coordinate system. The seven parameters of the seven-dimensional parameter space may include box dimensions (W, L, H) of the bounding box, a box center point (X, Y, Z), and a box orientation azimuth (Az). BLH component 170 can determine the envelope based on the seven parameters.

[0083] The method 900 may optionally include, at 914, displaying an indication of the dimensional measurements. For example, the display device 140, e.g., in conjunction with the processor 120, the memory 150, etc., may display an indication of the dimensional measurements, e.g., the BLH measurements of the bounding box determined by the BLH component 170. The displayed indication may include numbers corresponding to the width, length, and height measurements, a bounding box drawn around the target object in a view on the display device 140 (to ensure the correct object is being dimensioned), etc.

[0084] The method 900 may optionally include, at 916, displaying the target object as isolated from the shared 3D point cloud. For example, the display device 140, e.g., in conjunction with the processor 120, the memory 150, etc., may display the target object as within the shared 3D point cloud. In some examples, the display device 140 may display one or more projections of the shared 3D point cloud acquired in the TOF images. For example, the display device 140 may display an orthographic top view of the cloud, an orthographic front view of the cloud, and / or an orthographic side view of the cloud. Furthermore, the display device 140 may, for example, display the orthographic views with the bounding box drawn based on the BLH measurements.Furthermore, in one example, the display device 140 may display a generic perspective view of the scene captured in one of the TOF images along with the bounding box drawn in 3D around the target object in the common coordinate system. In these examples, the displays and / or point clouds used for object sizing may combine depth information from multiple images of the TOF sensor 110, IR information from the images, 3D geometries of the images, etc.

[0085] In some examples, the BLH component 170 may transmit the BLH measurements and / or the point cloud views described above to another device for storage or further processing, such as a system for determining packaging or shipping logistics. In another example, the BLH component 170 may store the BLH measurements and / or point cloud views described above in the memory 150 for retrieval of the measurements or displays on the display device 140.

[0086] The following aspects are merely illustrative, and aspects thereof may be combined without limitation with aspects of other embodiments or teachings described herein.

[0087] Aspect 1 is a method for acquiring images from a TOF sensor for determining dimensions of an object, including displaying, via a user interface, a prompt to capture at least one image of an object, acquiring, based on an interaction with the prompt, at least one image of the object, determining whether multiple images including the at least one image are sufficient to perform object dimensioning; and if the multiple images are sufficient to perform object dimensioning, providing the images for performing object dimensioning to calculate or display dimensions of the object.

[0088] In aspect 2, the method of aspect 1 includes displaying, via the user interface or other feedback, an indication of a position relative to the object for capturing a next image of the object.

[0089] In aspect 3, the method of aspect 2 includes wherein the position is substantially 30 degrees in azimuth from a previous position relative to the object at which the at least one image is captured.

[0090] In aspect 4, the method of any one of aspects 1 to 3 includes determining whether the plurality of images is sufficient to perform object sizing includes determining whether the plurality of images includes a certain number of images.

[0091] In aspect 5, the method of any one of aspects 1 to 4 includes determining whether the plurality of images is sufficient to perform object sizing includes determining whether the plurality of images includes two or more images for each of a plurality of positions relative to the object.

[0092] In aspect 6, the method of any one of aspects 1 to 5 includes, if the plurality of images are insufficient to perform object sizing, displaying, via the user interface, a request to recapture the at least one image of the object.

[0093] In aspect 7, the method of aspect 6 includes indicating, via the user interface or other feedback, an indication of a position relative to the object for recapturing the at least one image of the object.

[0094] In aspect 8, the method of any one of aspects 1 to 7 includes wherein the at least one image includes two images of the object captured at a similar position, and further comprising determining whether to retain the two images of the object to perform object sizing based at least in part on whether the two images are sufficiently different to achieve a threshold signal-to-noise ratio.

[0095] In aspect 9, the method of any one of aspects 1 to 8 includes determining whether to retain the at least one image of the object for performing object sizing based at least in part on whether the object in the at least one image has a threshold size.

[0096] In aspect 10, the method of any one of aspects 1 to 9 includes receiving, as part of performing object measurement, width, length and height measurements of the object and displaying, via the user interface, an indication of the width, length and height measurements of the object.

[0097] In aspect 11, the method of aspect 10 includes displaying one or more orthographic views of the object indicating the width, length, and height measurements of the object as a bounding box around the object.

[0098] In aspect 12, the method of any one of aspects 1 to 11 includes providing the at least one image for object measurement while waiting for the acquisition of a next image of the object from a next position.

[0099] In aspect 13, the method of any one of aspects 1 to 12 includes outputting a feedback score indicating the extent to which the plurality of images correspond to desired positions for capturing the object.

[0100] Aspect 14 is a method for obtaining images from a TOF sensor for determining dimensions of an object, including obtaining, from the TOF sensor, a plurality of images including the object, validating whether the plurality of images are captured in desired poses relative to the object, and if the plurality of images are validated as being captured in the desired poses, providing the plurality of images for object sizing to calculate or display dimensions of the object.

[0101] In aspect 15, the method of aspect 14 includes receiving, from a positioning sensor associated with the TOF camera, a position for each of the plurality of images, wherein validating whether the plurality of images are captured in desired poses is based on a comparison of at least a first position for one of the plurality of images and at least a second position for another of the plurality of images.

[0102] In aspect 16, the method of any one of aspects 14 or 15 includes validating whether the plurality of images are captured at desired positions includes determining that the plurality of images are captured at positions separated by substantially 30 degrees in azimuth around the object.

[0103] In aspect 17, the method of aspect 16, wherein validating whether the plurality of images are captured at desired positions includes determining that the plurality of images include at least four images captured at positions separated by substantially 30 degrees in azimuth around the object.

[0104] In aspect 18, the method of any one of aspects 14 to 17, wherein validating whether the plurality of images are captured in desired poses includes validating that the plurality of images include at least two images for each position of the desired poses.

[0105] In aspect 19, the method of any one of aspects 14 to 18 includes wherein validating whether the plurality of images are captured in the desired poses is based on a model of the object, wherein the desired poses are derived from the model of the object.

[0106] Aspect 20 is a method for aligning multiple 3D point clouds into a common 3D point cloud, including associating a first subset of points in a first 3D point cloud generated from one TOF view with a second subset of points in a second 3D point cloud generated from a different TOF view based on each point in the first subset of points having a threshold correspondence with a unique counterpart point in the second subset of points, refining the first subset of points and the second subset of points to generate a first improved subset of points that includes fewer outlier point correspondences with the second subset of points than the first subset of points, and a second improved subset of points that includes fewer outlier point correspondences with the first subset of points than the second subset of points, determiningbased on point correspondences between the first improved subset of points and the second improved subset of points, a relative rotation and translation between the first 3D point cloud and the second 3D point cloud, and aligning the first 3D point cloud and the second 3D point cloud in a common coordinate system based on the relative rotation and translation.

[0107] In aspect 21, the method of aspect 20 includes generating first FPFHs for points in the first subset of points and a second FPFH for points in the second subset of points, wherein associating a first point of the first subset of points with a second point of the second subset of points is based on determining whether a distance between a first FPFH descriptor for the first point in the first FPFHs and a second FPFH descriptor for the second point in the second FPFHs is within a threshold difference.

[0108] In aspect 22, the method of any of aspects 20 or 21 includes extracting the first subset of points from the first 3D point cloud having at least one threshold descriptor entropy and extracting the second subset of points from the second 3D point cloud having at least the same threshold descriptor entropy, wherein associating the first subset of points and the second subset of points includes evaluating the first subset of points and the second subset of points for descriptor correspondence.

[0109] In aspect 23, the method of any one of aspects 20 to 22 includes wherein refining the first subset of points and the second subset of points includes using a Lowe ratio test on 33-dimensional descriptors of the first subset of points and the second subset of points to compare points in the first subset of points and the second subset of points to point correspondences, and retaining, in the first improved subset of points and the second improved subset of points, points that pass the Lowe ratio test.

[0110] In aspect 24, the method of any one of aspects 20 to 23 includes wherein refining the first subset of points and the second subset of points includes comparing a first distance between a first portion of points from the first subset of points with a second distance between a second portion of corresponding points from the second subset of points, and retaining, in the first improved subset of points, the first portion of points and, in the second improved subset of points, the second portion of points if the difference between the first distance and the second distance is less than a threshold.

[0111] In aspect 25, the method of any one of aspects 20 to 25 includes refining the first subset of points and the second subset of points, for each first portion of points in the first subset of points and second portion of points in the second subset of points with separation distances of less than a threshold, comparing the separation distances between a third point in the first subset of points and the first portion of points with counterpart separation distances between the corresponding third point in the second subset of points and the second portion of points, and retaining, in the first improved subset of points, the first portion of points and the third point in the first subset of points and, in the second improved subset of points, the second portion of points and the corresponding third point in the second subset of points,if the separation distances and the counterpart separation distances are smaller than a threshold,

[0112] In aspect 26, the method of any one of aspects 20 to 25 includes downsampling a first acquired point cloud into the first 3D point cloud and downsampling a second acquired point cloud into the second 3D point cloud.

[0113] In aspect 27, the method of any one of aspects 20 to 26 includes aligning one or more 3D point clouds with the first 3D point cloud in a common coordinate system using relative rotations and translations calculated via motion synchronization and bundle matching.

[0114] In aspect 28, the method of any one of aspects 20 to 27 includes calculating dimensional measurements of an object in the common coordinate system.

[0115] In aspect 29, the method of aspect 28 includes displaying an indication of the dimensional measurements.

[0116] Aspect 30 is a method for isolating a target scene object from multiple images of a TOF sensor, including transforming the multiple images from a TOF sensor into multiple 3D point clouds, identifying, in each of the multiple 3D point clouds, one or more large planes having a minimum threshold size, generating a common 3D point cloud by aligning rotated and translated point clouds corresponding to at least a portion of the multiple 3D point clouds, rotating and translating the common 3D point cloud to identify one of one or more large planes as a ground plane in a common coordinate system of the common 3D point cloud, and isolating a target object from the rotated and translated common 3D point cloud.

[0117] In aspect 31, the method of aspect 30 includes isolating the target object, wherein isolating the target object includes projecting an average position of reconstructed TOF stumps in the shared 3D point cloud into the ground plane as the origin, averaging a pointing direction for the reconstructed TOF stumps in the ground plane, and modifying the shared 3D point cloud by removing points that lie outside a sub-cloud annulus of the shared 3D point cloud between two ground plane circles centered at the origin, and removing points outside a sub-cloud azimuthal wedge defined by a specified number of degrees from the pointing direction.

[0118] In aspect 32, the method of any one of aspects 30 or 31 includes isolating the target object further removing points from the shared 3D point cloud associated with the one or more large planes representing the floor or possible walls.

[0119] In aspect 33, the method of any one of aspects 30 to 32 includes isolating the target object including converting the shared 3D point cloud into a height map and removing points from the shared 3D point cloud associated with objects below a threshold size.

[0120] In aspect 34, the method of aspect 33 includes isolating the target object comprising selecting a sub-cloud having points whose position is closest to the preferred axis defined by the pointing direction from the origin.

[0121] In aspect 35, the method of any one of aspects 30 to 34 includes displaying, on a screen surface, the target object isolated from the common 3D point cloud.

[0122] In aspect 36, the method of any one of aspects 30 to 35 includes measuring a width, length and height dimension of the target object isolated from the common 3D point cloud.

[0123] In aspect 37, the method of aspect 36 includes searching a seven-dimensional parameter space for a bounding box surrounding the target object, wherein measuring the width, length, and height of the target object is based on the width, length, and height of the bounding box.

[0124] In aspect 38, the method of aspect 37 includes displaying, on a screen surface, the joint 3D point cloud and the bounding box surrounding the target object.

[0125] In aspect 39, the method of aspect 38 includes wherein displaying the target object includes jointly displaying the joint 3D point cloud and the bounding box surrounding the target object in three orthographic projections.

[0126] Aspect 40 is an imaging system including a TOF sensor or a processor, wherein the TOF sensor or the processor is configured to perform one or more of the operations of one or more of the methods of any of aspects 1 to 39.

[0127] Aspect 41 is an apparatus including means for performing the operations of one or more methods according to any one of aspects 1 to 39.

[0128] Aspect 42 is a computer-readable medium including code executable by one or more processors to perform the acts of one or more methods according to any one of aspects 1 to 39.

[0129] The above detailed description, set forth above in conjunction with the accompanying drawings, describes examples and does not represent the only examples that may be implemented or fall within the scope of the claims. The term "example," when used in this description, means "serving as an example, instance, or illustration" and not "preferred" or "advantageous over other examples." The detailed description includes specific details to provide an understanding of the techniques described. However, these techniques may be practiced without these specific details. For example, changes may be made in the function and arrangement of the elements discussed without departing from the scope of the disclosure. Also, various examples may omit, substitute, or add various procedures or components as appropriate.For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, functions described with respect to some examples may be combined in other examples. In some cases, well-known structures and devices are shown in block diagram form to avoid obscuring the concepts of the described examples.

[0130] Information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips referred to in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, computer-executable code or instructions stored on a computer-readable medium, or any combination thereof.

[0131] Further, for example, an element, any portion of an element, or any combination of elements may be implemented as a "processing system" that includes one or more processors. Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced command set computing (RISC) processors, systems on a chip (SoC), baseband processors, field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gate logic, discrete hardware circuits, and other suitable hardware configured to perform the functions described in this disclosure. One or more processors in the processing system may execute software.Software is generally understood to mean instructions, instruction sets, code, code segments, program code, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language or otherwise.

[0132] Accordingly, in one or more example aspects, the described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on a non-transitory computer-readable medium or encoded thereon as one or more instructions or code. Computer-readable media includes computer storage media. Storage media may be any available media that can be accessed by a computer.By way of example, and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), optical disk memory, magnetic disk memory, other magnetic storage devices, combinations of the above types of computer-readable media, or any other medium that can be used to store computer-executable code in the form of instructions or data structures accessible by a computer. Furthermore, features that implement functions may also be physically located in different locations, including being distributed such that portions of functions are implemented in different physical locations.

[0133] Furthermore, the various illustrative blocks and components described in connection with the present disclosure may be implemented or executed with a specially programmed device, such as, but not limited to, a processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, a discrete hardware component, or any combination thereof, configured to perform the functions described herein. A specially programmed processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or machine. A specially programmed processor may also be implemented as a combination of computing devices, e.g.,a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0134] In addition, as used herein, including in the claims, "or" when used in a list of items preceded by "at least one of" indicates a disjunctive list, so that, for example, a list of "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C).

[0135] The foregoing description of the disclosure is provided to enable those skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Furthermore, although elements of the described aspects may be described or claimed in the singular, the plural is contemplated unless expressly limited to the singular. Furthermore, all or a portion of one aspect may be used with all or a portion of any other aspect unless otherwise noted.Thus, the invention is not intended to be limited to the examples and embodiments shown herein, but is to be accorded the widest scope consistent with the principles of the novel features disclosed herein. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] WO 18 / 189,934

[0001] WO 63 / 324,042

[0001] WO 63 / 324,043

[0001]

Claims

[1] A method for isolating a target scene object from multiple images of a time-of-flight (TOF) sensor, comprising: Transforming the multiple images from a TOF sensor into multiple three-dimensional (3D) point clouds; Identifying, in each of the multiple 3D point clouds, one or more large planes with a minimum threshold size; Creating a common 3D point cloud by aligning rotated and translated point clouds corresponding to at least a part of the multiple 3D point clouds; Rotating and translating the shared 3D point cloud to identify one of one or more major planes as a ground plane in a common coordinate system of the shared 3D point cloud; and Isolating a target object from the rotated and translated joint 3D point cloud. [2] The method of claim 1, wherein isolating the target object includes: Projecting an average position of reconstructed TOF stumps in the joint 3D point cloud into the ground plane as the origin; Averaging a pointing direction for the reconstructed TOF stumps in the ground plane; and Modifying the shared 3D point cloud by removing points that lie outside a subcloud annulus of the shared 3D point cloud between two ground plane circles centered at the origin and removing points outside a subcloud azimuthal wedge defined by a specified number of degrees from the pointing direction. [3] The method of claim 2, wherein isolating the target object includes selecting a subcloud having points whose position is closest to a preferred axis defined by the pointing direction from the origin. [4] The method of claim 1, wherein isolating the target object includes further removing points from the common 3D point cloud associated with the one or more large planes representing the floor or possible walls. [5] The method of claim 1, wherein isolating the target object includes: Converting the shared 3D point cloud into a height map; and Removing points from the shared 3D point cloud that are associated with objects below a threshold size. [6] The method of claim 1, further comprising displaying, on a screen surface, the target object isolated from the common 3D point cloud. [7] The method of claim 1, further comprising measuring a width, length and height dimension of the target object isolated from the common 3D point cloud. [8] The method of claim 7, further comprising searching a seven-dimensional parameter space for a bounding box surrounding the target object, wherein measuring the width, length and height of the target object is based on the width, length and height of the bounding box. [9] The method of claim 8, further comprising displaying, on a screen surface, the joint 3D point cloud and the bounding box surrounding the target object. [10] The method of claim 9, wherein displaying the target object includes jointly displaying the joint 3D point cloud and the bounding box surrounding the target object in three orthographic projections. [11] System comprising: a memory; and a processor coupled to the memory, the processor being configured to: Transforming the multiple images from a time-of-flight (TOF) sensor into multiple three-dimensional (3D) point clouds; Identifying, in each of the multiple 3D point clouds, one or more large planes with a minimum threshold size; Creating a common 3D point cloud by aligning rotated and translated point clouds corresponding to at least a part of the multiple 3D point clouds; Rotating and translating the shared 3D point cloud to identify one of one or more major planes as a ground plane in a common coordinate system of the shared 3D point cloud; and Isolating a target object from the rotated and translated joint 3D point cloud. [12] The system of claim 11, wherein the processor is configured to isolate the target object at least in part by: Projecting an average position of reconstructed TOF stumps in the joint 3D point cloud into the ground plane as the origin; Averaging a pointing direction for the reconstructed TOF stumps in the ground plane; and Modifying the shared 3D point cloud by removing points that lie outside a subcloud annulus of the shared 3D point cloud between two ground plane circles centered at the origin and removing points outside a subcloud azimuthal wedge defined by a specified number of degrees from the pointing direction. [13] The system of claim 12, wherein the processor is configured to isolate the target object at least in part by selecting a subcloud including points whose position is closest to a preferred axis defined by the pointing direction from the origin. [14] The system of claim 11, wherein the processor is configured to isolate the target object at least in part by further removing points from the common 3D point cloud that are associated with the one or more large planes representing the floor or possible walls. [15] The system of claim 11, wherein the processor is configured to isolate the target object at least in part by: Converting the shared 3D point cloud into a height map; and Removing points from the shared 3D point cloud that are associated with objects below a threshold size. [16] The system of claim 11, wherein the processor is configured to display, on a screen surface, the target object isolated from the common 3D point cloud. [17] The system of claim 11, wherein the processor is configured to measure a width, length and height dimension of the target object isolated from the common 3D point cloud. [18] The system of claim 17, wherein the processor is configured to search a seven-dimensional parameter space for a bounding box surrounding the target object, wherein measuring the width, length, and height of the target object is based on the width, length, and height of the bounding box. [19] A computer-readable medium comprising code executable by one or more processors for isolating a target scene object from a plurality of images of a time-of-flight (TOF) sensor, the code comprising code for: Transforming the multiple images from a TOF sensor into multiple three-dimensional (3D) point clouds; Identifying, in each of the multiple 3D point clouds, one or more large planes with a minimum threshold size; Creating a common 3D point cloud by aligning rotated and translated point clouds corresponding to at least a part of the multiple 3D point clouds; Rotating and translating the shared 3D point cloud to identify one of one or more major planes as a ground plane in a common coordinate system of the shared 3D point cloud; and Isolating a target object from the rotated and translated joint 3D point cloud. [20] The computer-readable medium of claim 19, wherein the code for isolating the target object comprises code for: Projecting an average position of reconstructed TOF stumps in the joint 3D point cloud into the ground plane as the origin; Averaging a pointing direction for the reconstructed TOF stumps in the ground plane; and Modifying the shared 3D point cloud by removing points that lie outside a subcloud annulus of the shared 3D point cloud between two ground plane circles centered at the origin and removing points outside a subcloud azimuthal wedge defined by a specified number of degrees from the pointing direction.

Images