Method and system for performing object sizing

Abstract

Aspects of the disclosure include obtaining images from a time-of-flight (TOF) sensor to determine a size of an object, including obtaining a plurality of images including an object from the TOF sensor, verifying whether the plurality of images are captured at a desired pose relative to the object, and if the plurality of images are verified to be captured at the desired pose, determining the size of the object. The plurality of images are provided for object dimensioning to calculate or display a size of the object.

Description

[0001] Claiming priority

[0002] This patent application claims priority to non-provisional patent application No. 18 / 189916, entitled “Method and System for Performing Object Dimension Measurement,” filed on March 26, 2022, provisional patent application No. 63 / 324,042, entitled “Pallet Dimension Measurement System Using Time-of-Flight Camera,” filed on March 26, 2022, and provisional utility patent application No. 63 / 324,043, entitled “Object Dimension Measurement System and Algorithm,” filed on March 26, 2022, which are assigned to the assignee of this Agreement and are hereby expressly incorporated herein by reference. Background Art

[0003] In the shipping, storage or moving industry, companies often provide pricing based on the dimensions of one or more items to be shipped, stored and moved. Manually measuring the dimensions of each of multiple items, such as using a tape measure or other measuring mechanism, can be time consuming and / or inaccurate. Additionally, some items may not be box-shaped or symmetrical, which can add to the complexity of measuring these items. Summary of the invention

[0004] A simplified summary of one or more aspects is presented below in order to provide a basic understanding of these aspects. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify key or important elements of all aspects, nor to describe the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description presented later.

[0005] Aspects of the present disclosure include a method for obtaining images from a time-of-flight (TOF) sensor to determine the size of an object. The method includes obtaining a plurality of images including an object from the TOF sensor, verifying whether the plurality of images are taken at an expected pose relative to the object, and providing the plurality of images for object dimensioning to calculate or display the size of the object if the plurality of images are verified to be taken at the expected pose.

[0006] In another aspect, a system is provided that includes a memory and a processor coupled to the memory. The processor is configured to obtain a plurality of images including an object from the TOF sensor, verify whether the plurality of images are taken at an expected posture relative to the object, and if the plurality of images are verified to be taken at the expected posture, provide the plurality of images for object dimensioning to calculate or display the size of the object.

[0007] In another aspect, a computer readable medium is provided, comprising code executable by one or more processors for obtaining images from a TOF sensor to determine the size of an object. The code comprises code for obtaining a plurality of images including an object from the TOF sensor, verifying whether the plurality of images are taken at an expected pose relative to the object, and providing the plurality of images for object dimensioning to calculate or display the size of the object if the plurality of images are verified to be taken at an expected pose.

[0008] To accomplish the above and related ends, one or more aspects include the features hereinafter fully described and particularly pointed out in the claims. The following description and the accompanying drawings set forth in detail certain illustrative features of one or more aspects. However, these features are indicative of only a few of the various ways in which the principles of the various aspects may be employed, and this description is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are used to illustrate and not limit the disclosed aspects, wherein like reference numerals represent like elements, and wherein:

[0010] Figure 1 is a block diagram of an example sensor system according to aspects described herein.

[0011] Figure 2 Illustrated are light ray directions for a pixel of a time-of-flight (TOF) sensor according to aspects described herein.

[0012] Figure 3 An example of a multi-cloud stitching component according to aspects described herein is shown.

[0013] Figure 4 An example of a method for validating a TOF image to perform object dimensioning according to aspects described herein is shown.

[0014] Figure 5 An example of a method for obtaining a TOF image to perform object dimensioning according to aspects described herein is shown.

[0015] Figure 6 An example of a method for associating point clouds from different capture poses to perform object dimensioning according to aspects described herein is shown.

[0016] Figure 7 An example of a point cloud and a corresponding feature space according to aspects described herein is shown.

[0017] Figure 8 Examples of point clouds and corresponding feature spaces are shown, as well as graphical representations of point clouds and layered consistency according to aspects described herein.

[0018] Fig. 9 An example of a method for isolating a target object in a plurality of TOF images to perform dimensioning thereof according to aspects described herein is shown.

[0019] Fig.10 An example of a method for isolating a target object from a rotated and translated common three-dimensional (3D) point cloud according to aspects described herein is shown. DETAILED DESCRIPTION

[0020] Various aspects are now described with reference to the accompanying drawings. In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it is apparent that these aspects can be practiced without these specific details.

[0021] The described features generally relate to object size measurement systems, methods, and / or algorithms for measuring the size of an object observed using a time-of-flight (TOF) sensor or other camera (such as a depth camera or sensor). In one example, the described features can enable measurement of irregularly shaped objects, which can include determining width, length, height (WLH), and fitting a bounding box around the irregularly shaped (or asymmetric) object accordingly. For example, a TOF sensor can be used to capture a 3D image (e.g., a TOF image) of an object from various poses around the object to determine the size of the object. For example, the pose can be related to the position of the TOF sensor relative to the object (e.g., around the object), the angle of the TOF sensor relative to the object (e.g., the tilt of the TOF sensor), other orientation parameters, etc., which can also be more generally referred to as position in this article. By finding point correspondences between the 3D images, the 3D images can be aligned in a common coordinate system (also known as "stitching"). In one example, a larger 3D image can be generated that combines views of the object in different poses. One or more objects can be located in points in a common coordinate system and / or based on these points. In addition, for example, the size of the object (e.g., WLH) can be calculated in the common coordinate system. In some examples, a representation of the object may be displayed, which may also include one or more measurements of the object (eg, WLH).

[0022] In this regard, for example, as described herein, systems, methods, and / or algorithms for performing object dimensioning can use a TOF sensor to provide object dimensions based on multiple captures, which can allow efficient and substantially accurate measurement of objects regardless of shape. This can allow the size of objects to be transported, stored, moved, etc. to be determined to enable more accurate pricing or other logistical considerations.

[0023] As used in this application, the terms "component", "module", "system", etc. are intended to include computer-related entities, such as but not limited to hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As an illustration, both an application running on a computing device and a computing device can be a component. One or more components can reside in a process and / or an execution thread, and a component can be localized on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media on which various data structures are stored. These components can communicate through local and / or remote processes, such as according to a signal with one or more data packets, such as data from a component, which interacts with another component in a local system, a distributed system, and / or interacts with other systems through a network (such as the Internet).

[0024] The following description provides examples and does not limit the scope, applicability or examples set forth in the claims. The functions and arrangements of the elements discussed may be changed without departing from the scope of the present disclosure. Various examples may appropriately omit, replace or add various programs or components. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted or combined. In addition, the features described with respect to some examples may be combined in other examples.

[0025] Various aspects or features will be presented in terms of systems that may include multiple devices, components, modules, etc. It should be understood and appreciated that the various systems may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Combinations of these methods may also be used.

[0026] In this specification, reference may be made to the spatial relationship between the various components shown in the accompanying drawings and the spatial orientation of the various aspects of the components. However, as will be appreciated by those skilled in the art after a complete reading of this disclosure, the devices, components, members, devices, etc. described herein may be positioned in any desired direction. Therefore, the use of terms such as "above", "below", "upper", "lower", "top", "bottom" or other similar terms to describe the spatial relationship between the various components or the spatial orientation of the aspects of these components should be understood to describe the relative relationship between the components or the spatial orientation of the various aspects of these components, respectively, because the components described herein may be oriented in any desired direction. When used to describe a series of dimensions or other features (such as time, pressure, temperature, length, width, etc.) of an element, operation and / or condition, the phrase "between X and Y" represents a range including X and Y.

[0027] Figure 1 1 is a block diagram of an example sensor system 100 according to aspects described herein. The sensor system 100 includes a TOF sensor 110, a processor 120, a camera 130, a display device 140, and a memory 150. In an alternative configuration, the TOF sensor system may include Figure 1 Different, fewer and / or additional components are shown. Figure 1 The functions described by one or more components shown in the figure may be distributed between components in a different manner than described. In some embodiments, some or all of the components 110-150 may be integrated into a single unit, for example, a handheld unit having a TOF sensor 110, a processor 120 for processing TOF data, a local memory 150, and a display device 140 for displaying the output of the processor 120 to a user. In some embodiments, some components may be located in different devices, for example, a handheld TOF sensor 110 may transmit TOF data to an external processing system (e.g., a computer or tablet), which stores and processes the TOF data and provides one or more displays to the user. Different devices may communicate via wireless or wired connections.

[0028] The TOF sensor 110 collects distance data describing the distance between the TOF sensor 110 and various surfaces in the environment of the TOF sensor 110. The TOF sensor 110 may include a light source (e.g., a laser) and an image sensor for capturing light reflected from a surface. In some embodiments, the TOF sensor 110 emits a pulse of light and captures multiple image frames at different times to determine the amount of time it takes for the light pulse to propagate to the surface and return to the image sensor. In other embodiments, the TOF sensor 110 detects a phase shift in the captured light, and the phase shift indicates the distance between the TOF sensor 110 and various surfaces. In some embodiments, the TOF sensor 110 may generate and capture multiple different frequencies of light. If the TOF sensor 110 emits and captures multiple frequencies of light, this can help resolve ambiguous distance issues and help the TOF sensor 110 operate over a larger range of distances. For example, if for a first frequency, a first observed phase may correspond to a surface that is 0.5 meters, 1.5 meters, or 2.5 meters away, and for a second frequency, a second observed phase may correspond to a surface area that is 0.75 meters, 1.5 meters, or 2.25 meters away, by combining these two observations, the TOF sensor 110 may determine that the surface is 1.5 meters away. Using multiple frequencies may also improve robustness to noise caused by specific frequencies of ambient light, whether using phase shift or pulse return time to measure distance. In alternative embodiments, different types of sensors may be used in place of and / or in addition to the TOF sensor 110 to obtain distance data.

[0029] Processor 120 receives distance data from TOF sensor 110 and processes the distance data to identify various features in the environment of TOF sensor 110, as described in detail herein, for example with reference to Figure 3-8 In some embodiments, the distance data includes observed distances to various surfaces measured by the TOF sensor 110 using, for example, the phase shift or pulse return time methods described above. In some embodiments, if the TOF sensor 110 measures phase shift, the distance data received by the processor 120 from the TOF sensor 110 is phase shift data, and the processor 120 calculates the distance to the surface based on the phase shift data.

[0030] The camera 130 may capture image frames of the environment. The camera 130 may be a visible light camera that captures images of the environment within the visible range. In other embodiments, the camera 130 is an infrared (IR) camera that captures the IR intensity of surfaces in the environment of the sensor system. The field of view of the camera 130 and the TOF sensor 110 partially or completely overlap, for example, the field of view of the camera 130 may be slightly larger than the TOF sensor 110. The camera 130 may pass the captured image to the processor 120.

[0031] The display device 140 provides a visual output for a user of the sensor system 100. In some embodiments, the display device 140 displays the image obtained by the camera 130 and overlays a visual image indicating one or more features identified in the field of view of the camera 130 and the TOF sensor 110 based on the distance data. For example, the processor 120 can instruct the display device 140 to display the outline of the box on the image of the box obtained by the camera 130. The user can use the display to determine whether the sensor system 100 correctly identified the box and the edges of the box. The sensor system 100 may include additional or alternative input and / or output devices, such as buttons, speakers, touch screens, etc.

[0032] The memory 150 stores data for the sensor system 100. For example, the memory 150 stores processing instructions used by the processor 120 to identify features in the environment of the TOF sensor 110, for example, instructions for identifying one or more Z planes and / or calculating the size of an observed object. The memory 150 can temporarily store data and images obtained by the camera 130 and / or the TOF sensor 110 and accessed by the processor 120. The memory 150 can also store image data accessed by the display device 140 to generate an output display.

[0033] Figure 21 shows the ray directions of pixels of the TOF sensor 110 according to aspects described herein. The distance data obtained by the TOF sensor 110 can be arranged as a group of pixels, such as pixels 210a and 210b, within an image frame (e.g., image frame 220). Each pixel 210 has an associated ray direction 215, where the ray direction 215 points outward from the TOF sensor 110. The ray direction 215 is projected toward the image frame 220. Although Figure 2 25 rays and pixels are shown in FIG. 1 , but it should be understood that the TOF sensor 110 may have more pixels. Figure 2 In the example shown, the image frame 220 has a square shape, but in other embodiments, the image frame 220 can have other shapes. In some examples, certain pixels, such as pixels near the edge of the image frame 220, may be considered invalid (e.g., not reliable enough) and removed from the distance data.

[0034] For example, a first pixel 210a has a ray direction 215a extending directly outward from the TOF sensor 110; the pixel 210a is located at the center of the image frame 220. A second pixel 210b at the corner of the image frame 220 is associated with a ray direction 215b that extends from the TOF sensor 110, for example, at a 30° angle in the x-direction and the y-direction from the center of the image frame 220, where the image frame 220 is the xy plane in the reference frame of the TOF sensor 110. The TOF sensor 110 returns distance data (e.g., distance, one or more phase shifts) to the surface along the ray for each effective pixel. In one example, the first pixel 210a may have a measured distance of 1 meter, representing the distance to a specific point on a target object, and the second pixel 210b may have a measured length of 2 meters, representing the distance to a specific point on a wall behind the target object.

[0035] Back to Figure 1, the processor 120 may include, implement or execute various components or processes for performing object annotation, as described herein. For example, the processor 120 may implement or execute components or processes based on corresponding instructions, related parameters, etc. stored in the memory 150. In some examples, the processor 120 may include, implement, or execute a capture verification component 158, a cloud generation component 160, a plane recognition component 162, a feature extraction component 164, a multi-cloud stitching component 166, a target isolation component 168, and / or a WLH component 170, wherein the capture verification component 158 ​​is used to verify that the image received from the TOF sensor 110 is taken at an expected pose for performing object size measurement, the cloud generation component 160 is used to generate a 3D point cloud based on the image captured from the TOF sensor 110, the plane recognition component 162 is used to identify large planes in the 3D point cloud, the feature extraction component 164 is used to determine features associated with points in the 3D point cloud, the multi-cloud stitching component 166 is used to associate multiple 3D point clouds with each other in a common coordinate system based on detecting corresponding points in the 3D point cloud, the target isolation component 168 is used to isolate one or more target objects detected in the common coordinate system, and the WLH component 170 is used to calculate a bounding box around the target object and / or calculate or otherwise display a WLH measurement value or other measurement value associated with the target object.

[0036] In some examples, the TOF sensor 110 may capture multiple images of an environment with one or more objects. For example, the TOF sensor 110 may be used to capture a view of multiple images (e.g., image pairs) at each of a plurality of locations around one or more objects, wherein each view pair may include two (or more) images captured by the TOF sensor 110 at a given pose. Collecting pairs of images in back-to-back TOF frames may provide an improved signal-to-noise ratio (SNR) for the pair of images to allow robust stitching of corresponding point clouds, as further described herein. For example, if the captured object is dark or shiny, the photons transmitted by the TOF sensor 110 toward the object may not be sufficient to return to the TOF sensor 110 to be observed by the receiver of the TOF sensor 110. Therefore, collecting multiple TOF frames at each pose may increase the number of returned photons, thereby producing a denser clustered point cloud at the location. The image may also include or be referred to as a distance map, a depth map, or a depth image. In some examples, the TOF sensor 110 may additionally or alternatively capture an infrared (IR) image with a depth image, wherein the IR image may help further post-process the view to discard noise data.

[0037] In some examples, the TOF sensor 110 can be used to capture view pairs at a specific number of poses (e.g., the position of the TOF sensor 100 relative to the object, the viewing angle or other capture direction of the TOF sensor 110 at these positions, etc.) around one or more objects at substantially the same azimuth to capture a specific azimuth range around one or more objects, such as a total of 90 degrees. In a specific example, the TOF sensor 110 can be used to capture four view pairs at each of four positions that are substantially spaced apart from each other by 30 degrees in azimuth. The TOF sensor 110 can provide the view pairs and / or parameters corresponding to the TOF sensor 110 to the processor 120. In some examples, the capture verification component 158 ​​can verify or otherwise ensure that the view pairs are captured at the desired positions for valid object dimensioning, as further described herein.

[0038] In some examples, cloud generation component 160 can receive view pairs from TOF sensor 110, and can generate a 3D point cloud for each view pair (e.g., for each location). For example, cloud generation component 160 can generate an aggregated 3D point cloud for a given location based on multiple views in the view pair, as described, and / or can generate a separate point cloud for each location. In some examples, plane identification component 162 can identify a large plane in each of multiple 3D point clouds as a set of adjacent points with the same surface normal, where the large plane can be determined as having a minimum threshold size (e.g., a minimum threshold number of points with the same surface normal). In some examples, feature extraction component 164 can determine or calculate features associated with each point in a given 3D point cloud. For example, feature extraction component 164 can generate a point feature histogram (PFH) or a fast PFH (FPFH) describing each point in the 3D point cloud. In some examples, multi-cloud stitching component 166 can associate points in multiple 3D point clouds as having point correspondences (e.g., as the same real-world point captured at different poses) based on the determined features of the points in the multiple 3D point clouds. This can allow multi-cloud stitching component 166 to create a more complete image (e.g., point cloud) of the environment in a common coordinate system, which can facilitate improved measurement of WLH dimensions of one or more objects, as described in further detail and various examples herein.

[0039] In some examples, the target isolation component 168 can detect one or more target objects in the common coordinate system point cloud created by the multi-cloud stitching component 166, which can be based on detecting the size of the target object in the common coordinate system as at least a threshold size. In some examples, the WLH component 170 can fit a bounding box around the target object based on identifying points corresponding to the target object in the common coordinate system point cloud. In some examples, the WLH component 170 can calculate or otherwise display an indication of the WLH of the target object, for example, via the display device 140, which can include displaying the actual WLH size measured by the target object (e.g., the size of the bounding box), displaying the bounding box, etc. In one example, the WLH component 170 can also display a representation of the environment or target object observed at the current location, or a representation of the common coordinate system point cloud, etc. In some examples, the WLH component 170 can display WLH measurements, bounding boxes, etc. on a representation of the environment or target object or the common coordinate system point cloud, as described in various examples herein.

[0040] Figure 3 An example of a multi-cloud stitching component 166 according to aspects described herein is shown. The multi-cloud stitching component 166 may include a hierarchical point matching component 302 that may receive features of a plurality of 3D point clouds, such as FPFH descriptors (e.g., from the feature extraction component 164). The hierarchical point matching component 302 may perform a first pass hierarchical point matching process to determine point pairs having point correspondences in at least two point clouds. For example, the hierarchical point matching component 302 may determine the point pairs as having point descriptors within a threshold (e.g., having values ​​within a threshold of each other), which may indicate that the points correspond to the same real-world point captured in the image. In some examples further described herein, the outlier rejection component 304 may perform one or more processes to refine the points matched by the hierarchical point matching component 302 to remove outliers and speed up the relevant processing of the point cloud for object dimensioning. Given a reduced set of corresponding points, the cloud pair registration component 306 may derive a rotation and translation of the point cloud based on the detected hierarchical points. In some examples, the motion synchronization component 308 can calculate an absolute TOF pose, which can be specified in a coordinate system associated with the first TOF image, and the bundle adjustment component 310 can change the TOF rotation and translation and the reconstruction points to reduce errors in the sizing process. The multi-cloud stitching component 166 can output the aggregated cloud and the reconstructed frustum in the first TOF coordinate system, which can be used by the target isolation component 168 to isolate the target object in the image to perform its sizing, as described in further detail herein.

[0041] Figure 4An example of a method 400 for verifying a TOF image to perform object dimensioning according to aspects described herein is shown. The method 400 may be performed by the sensor system 100, one or more components thereof (eg, the TOF sensor 110, the processor 120, etc.).

[0042] The method 400 may include, at 402, displaying a prompt to capture an image of an object. For example, the display device 140, e.g., in conjunction with the processor 120, the memory 150, etc., may display a prompt to capture an image of an object. The display device 140 may display the prompt as part of activating, initializing, or opening the sensor system 100, executing an application for image capture and / or object dimensioning, etc. The processor 120 may cause the prompt to be displayed on the display device 140 as an interface (e.g., a graphical user interface, which may be part of an application executed on the processor 120). As the sensor system 100 moves through the environment, the prompt may display a live camera view of the environment, and the live camera view may be a TOF sensor 110 view, which may be represented as an infrared (IR) image, etc. The prompt may also include an interactive option to capture an image.

[0043] The method 400 may include capturing an image of the object based on the interaction with the prompt at 404. For example, the processor 120 or a corresponding application executing on the processor 120 may detect the interaction with the prompt through a display device (such as a touch screen display) or other input device, in conjunction with the memory 150, etc., to select an interactive option to capture an image from the TOF sensor 110. In addition, as described in some examples, the processor 120 may capture multiple images from the TOF sensor 110 based on the interaction with the prompt, such as two (or more) images in a view pair. In one example, when the interaction with the prompt is detected, the number of images to be captured and / or the time between image captures can be configured to optimize the image quality of the object size.

[0044] In this regard, multiple discrete images may be captured rather than a continuous video capture. Capturing multiple discrete images may be easier to capture accurately than a continuous video. For example, a continuous video may present challenges such as keeping the system stable, performing continuous motion of the system to capture the correct view within a specific time period, etc. Using discrete images may allow the user time to properly align the TOF sensor 110 before capturing the image to be processed.

[0045] After capturing the image at 404, method 400 may include determining whether to retain the image at 406. For example, capture verification component 158, e.g., in conjunction with processor 120, memory 150, etc., may determine whether to retain the image. For example, determining whether to retain the image may include one or more various actions performed by a user of sensor system 100 or automatically performed by the sensor system to ensure that an image useful for object size determination is captured and provided for further processing.

[0046] In one example, determining whether to retain the image at 406 may optionally include prompting whether to retain the image at 408. For example, the capture verification component 158, for example, in conjunction with the processor 120, the memory 150, etc., may display a prompt via the display device 140 as to whether to retain the image, which may be displayed after each image capture, after capturing multiple images, etc. Based on the interaction with the prompt, the capture verification component 158 ​​may determine whether to retain the image.

[0047] In another example, determining whether to retain the image at 406 may also optionally (or alternatively) include determining whether the image conforms to parameters of a valid object size at 410. For example, the capture verification component 158, for example in combination with the processor 120, the memory 150, etc., may determine whether the image conforms to parameters of a valid object size. In some examples, the capture verification component 158 ​​may ensure that the view pair has sufficient distinction to provide at least a threshold improvement in SNR for the view pair over a single view. In another example, the capture verification component 158 ​​may ensure that the object is discernible in the image, for example, the object has a sufficient number of associated points in the point cloud to facilitate identification and accurate measurement of the object from the image. For example, the size of the object in the picture is configurable, and the capture verification component 158 ​​may detect whether the image includes an object of at least the configured size when determining whether to retain the image.

[0048] If it is determined at 406 that the image is not to be retained, the method can continue to 402 to display another prompt for capturing an image. The method 400 can optionally include feedback indicating device positioning at 412. For example, the capture verification component 158, for example in combination with the processor 120, the memory 150, etc., can indicate feedback for device positioning, for example, via the display device 140 or other output device, such as tactile feedback via a tactile feedback device on the sensor system 100, audio or visual feedback via the display device 140, etc. In this regard, for example, the sensor system 100 can help the user capture an image at a position / orientation that can produce an effective object size, as further described herein. In some examples, the sensor system 100 may include one or more sensors to enable position determination, such as an accelerometer, a compass, or other displacement or position sensor, which can provide position information for the sensor system 100. In another example, the environment in which the sensor system 100 captures the image can include external sensors, such as a TOF scanner, a camera that can detect the presence and position of the sensor system 100, etc. In either case, such internal or external sensors may be used to detect the position of the TOF sensor 110 to determine whether the captured image is in the desired position / orientation and / or to guide the user through real-time feedback to properly position the TOF sensor 110 to capture the image.

[0049] For example, the capture verification component 158 ​​can obtain position information of the sensor system 100 or at least its TOF sensor 110, and can determine whether the TOF sensor 110 is in a position to capture the next image for object dimensioning (e.g., determine whether the TOF sensor 110 is in a position within a certain azimuth of the position where the previous image was captured). In this example, the capture verification component 158 ​​can provide feedback via the display device 140 whether the sensor system 100 or at least the TOF sensor 110 is in the correct (or acceptable) position / orientation to capture the next image. In one example, based on the current detected position / orientation of the sensor system 100 or at least the TOF sensor 110 and the known expected position / orientation for the next image capture, the capture verification component 158 ​​can provide feedback to guide the user of the sensor system 100 to the expected or acceptable position. For example, the capture verification component 158 ​​can provide feedback, such as an arrow displayed on the display device 140 indicating the direction to move the sensor system 100 to the desired position, a box or other indicator displayed on the display device 140 showing where to align the center of the display device 140, etc. In one example, the capture verification component 158 ​​can indicate when the sensor system 100 is in a desired or acceptable position to capture the next image, such as by displaying an icon or pop-up message, activating a selectable option to capture an image, etc.

[0050] If it is determined at 406 that the image is retained, the method 400 may include determining at 414 whether the image is sufficient to determine the size of the object. For example, the capture verification component 158, such as in conjunction with the processor 120, the memory 150, etc., may determine whether the image is sufficient to determine the size of the object. In some examples, the capture verification component 158 ​​may determine whether there is enough image for object dimensioning. In some examples, the capture verification component 158 ​​may determine whether the image is taken at an expected location / orientation around the object to allow object dimensioning, as described below with reference to Figure 5 As described in method 500 (e.g., as part of verifying at 504 whether the multiple images were taken in the expected position / orientation).

[0051] If it is determined at 414 that the image is insufficient for dimensioning, the method 400 can process back to 402 to display a prompt to capture an image of the object. In some examples, as described above at 412, feedback for device positioning can be indicated to assist in capturing an image useful for dimensioning. For example, in the event that the capture verification component 158 ​​determines that a given image is insufficient for determining dimensions, the capture verification component 158 ​​can indicate feedback to retake the image, which feedback can include an indication of the image to be retaken. For example, the indication can include displaying the image to be retaken, indicating the index of an image in a plurality of image sets to be retaken, indicating the location of a device used to capture the image (e.g., displaying an arrow to guide to the desired location of the image to be retaken), etc. In another example, the capture verification component 158 ​​can discard all images and display a prompt to start from the initial capture.

[0052] If it is determined at 414 that the images are insufficient for dimensioning, the method 400 may include providing a plurality of images for object dimensioning to calculate or display the size of the object at 416. For example, the capture verification component 158, e.g., in combination with the processor 120, the memory 150, etc., may provide a plurality of images (e.g., to the cloud generation component 160 or one or more other components) for object dimensioning to calculate or display the size of the object. In one example, the capture verification component 158 ​​may display a prompt on the display device 140 to perform object annotation, wherein the capture verification component 158 ​​may display the prompt based on verifying that the image captured for the object can be used to perform object annotation. As described above, this may include the number of images captured, the signal-to-noise ratio of the images, the capture angle relative to the object, etc.

[0053] Additionally, in some examples, after an image is captured, preprocessing may be used at 418 to provide the captured image for object dimension measurement while waiting for other images to be captured. This may allow processing of the image and / or other images for object dimensioning to begin while waiting for other images to be captured. For example, as further described herein, an object annotation action may take some time to execute, and preprocessing in this regard may allow some computational actions (e.g., point cloud generation, plane identification, feature extraction, multi-cloud stitching, etc.) to be performed while the image is captured, which may reduce the overall processing time after all images are captured. In this regard, by performing preprocessing at 418, the processing time perceived by a user between indicating an object dimensioning to be performed and receiving the output of the object dimensioning may be reduced.

[0054] The method 400 may optionally include receiving a WLH measurement of the object at 420. For example, the WLH component 170, e.g., in combination with the processor 120, the memory 150, etc., may receive or otherwise determine the WLH measurement of the object based on other components that perform object dimensioning, as further described herein. The method 400 may optionally include displaying the WLH measurement of the object at 422. For example, the display device 140, e.g., in combination with the processor 120, the memory 150, etc., may display the WLH measurement of the object. For example, as further described herein, this may include displaying the measurement, displaying a bounding box around the display of the object, etc.

[0055] In some examples, method 400 may optionally include outputting a feedback score at 424 that indicates the extent to which the multiple images correspond to the desired pose. For example, capture verification component 158, e.g., in conjunction with processor 120, memory 150, etc., may output a feedback score that indicates the extent to which the multiple images correspond to the desired pose. Thus, as described above and further described herein, feedback may be provided during image capture (e.g., at 412) or after capture (e.g., at 424). For example, in this regard, the feedback may be per image (or per currently captured image) or may be a set of images based on further processing of the images or an attempt to perform an association (or stitching), as further described herein.

[0056] In one example, the capture verification component 158 ​​can receive an indication of a feedback score for an image from one or more other components of the object annotation system, such as the multi-cloud stitching component 166, as used in object annotation. The feedback score can be related to how close the image is to a desired azimuth. In one example, this can be based on the distance to the center of certain points in the corresponding point cloud that have point correspondences with points in other point clouds. The feedback score can be based on other measurements of the image, such as signal-to-noise ratio, etc. The feedback score can indicate the adequacy or quality of the image set, and / or can be selected from a set of labels, or can be numeric or alphabetic based.

[0057] Figure 5 An example of a method 500 for obtaining a TOF image to perform object dimensioning according to aspects described herein is shown. The method 500 may be performed by the sensor system 100, one or more components thereof (eg, the TOF sensor 110, the processor 120, etc.).

[0058] The method 500 may include obtaining a plurality of images including an object from a TOF sensor at 502. For example, the capture verification component 158, for example in combination with the processor 120, the memory 150, etc., may obtain a plurality of images including an object from the TOF sensor 110. For example, as described above, the TOF sensor 110 may perform multiple captures, which may include capturing a view pair at each of a plurality of positions / directions around the object. In some examples, the TOF sensor 110 may be operated by a user or another device to capture views around an azimuth range of the object, as described, which may include capturing a view pair at a plurality of positions near the azimuth range. Further, in some examples, the positions may be within an azimuth range of each other. In a specific example, the TOF sensor 110 may capture a view pair at each of four positions within a substantially 30 degree azimuth range of each other to capture a total azimuth range of substantially 90 degrees around the object.

[0059] The method 500 may include verifying at 504 whether the plurality of images are taken at a desired position relative to the object. For example, the capture verification component 158, for example, in combination with the processor 120, the memory 150, etc., may verify (e.g., before or as part of generating the associated 3D point cloud) whether the plurality of images are taken at a desired position relative to the object. In some examples, the capture verification component 158 ​​may verify whether the plurality of images are taken at a desired position / orientation based on information indicated in or with the images received from the TOF sensor 110. For example, the TOF sensor 110 may indicate an accelerometer, compass, or other displacement / position sensor measurement of the TOF sensor 110 when capturing a view (or pair of views) at each position, and the capture verification component 158 ​​may determine whether the measurement indicates that the view is at a desired position. For example, the capture verification component 158 ​​may compare the measurements to each other, compare the measurement difference to a threshold, compare the measurement to a threshold, etc. when verifying whether the plurality of images are taken at a desired position (e.g., as described, taken at a certain azimuth or azimuth range substantially within each other).

[0060] In some examples, other external sensors in the environment, such as a TOF scanner, a surveillance camera in a room, and the like, can be used to track and / or observe the position of the TOF sensor 110 when capturing a view of the environment. The external sensors can detect information or measurements of the TOF sensor 110 when acquiring an image, and can provide information or measurements that the capture verification component 158 ​​can use to verify whether the image was taken at the desired location. In some examples, the capture verification component 158 ​​can verify the image based on a model of an object detected in the image. For example, the capture verification component 158 ​​can infer the capture location of each image based on the location of the modeled object in each image. In one example, the capture verification component 158 ​​can detect objects and the direction of the objects in each image. In this example, the capture verification component 158 ​​can compare the directions in multiple captures to infer the location information of the image (e.g., the azimuth or azimuth difference between captures).

[0061] The method 500 may include determining, at 506, whether the image was taken in an expected pose. For example, the capture verification component 158, e.g., in combination with the processor 120, the memory 150, etc., may determine whether the image was taken in an expected pose (e.g., whether the image was verified at 504). If the result is yes, then at 508, multiple images may be provided for object dimensioning to calculate or display the size of the object, as further described herein. If the result is no, optionally at 510, an error or other feedback may be triggered. For example, the capture verification component 158, e.g., in combination with the processor 120, the memory 150, etc., may provide images (e.g., to the cloud generation component 160 or one or more other components) for object dimensioning, or an error or feedback may be triggered in these cases.

[0062] In one example, the capture verification component 158 ​​may trigger an error or feedback as an indication displayed on the display device 140, which may include an indication that the object dimensioning cannot be performed, or may include one or more feedback indications for capturing additional images as described above at 412 of the method 400, etc. For example, in the event that the capture verification component 158 ​​determines that one of the images is not captured in the desired pose, the capture verification component 158 ​​may display a prompt to retake the image in the desired pose through the display device 140, which may include an indication of the image or pose to be recaptured as described above, an arrow, or other visual or audio (or tactile) feedback to guide the user to correctly position the TOF sensor 110 to properly capture the image, etc.

[0063] Figure 6 An example of a method 600 for associating point clouds from different capture poses to perform object dimensioning according to aspects described herein is shown. For example, a pose may refer to an angle and position of capture (e.g., rotation and translation as further mentioned herein). The method 600 may be performed by the sensor system 100, one or more components thereof (e.g., the processor 120, etc.).

[0064] Method 600 may include, at 602, associating a first subset of points in a first 3D point cloud with a second subset of points in a second 3D point cloud based on each point in the first subset of points having a threshold correspondence with a unique corresponding point in the second subset of points. For example, hierarchical point matching component 302, for example in combination with processor 120, memory 150, multi-cloud stitching component 166, etc., may associate a first subset of points in a first 3D point cloud (e.g., a first TOF view, which may include multiple images) with a second subset of points in a second 3D point cloud based on each point in the first subset of points having a threshold correspondence with a unique corresponding point in the second subset of points. The point and its unique corresponding point may be referred to herein as a connection point. In one example, given a set of images captured using TOF sensor 110, as described above, cloud generation component 160 may generate a 3D point cloud for the images, which may include a first 3D point cloud generated for an image (or view pair or other set of multiple images) captured at a first location, a second 3D point cloud generated for an image (or view pair or other set of multiple images) captured at a second location, and / or other point clouds. For example, the cloud generation component 160 can generate a point cloud at least in part by multiplying pixels in a TOF range map generated by the TOF sensor 110 during image capture by their corresponding ray direction vectors, as described above. The cloud generation component 160 can ignore pixels with invalid range or IR readings, and the resulting point cloud can be in the captured TOF sensor 110 coordinates.

[0065] In addition, in some examples, the plane recognition component 162 can identify large planes in each of the multiple 3D point clouds. The plane recognition component 162 can identify the ground plane and / or possible wall planes in the 3D point cloud. In some examples, the plane recognition component 162 can calculate the surface normals of the valid pixels in the TOF view, and can search for large connected components (e.g., sets of pixels) with almost constant normal angle coordinates. Random sample consistency (RANSAC) can be performed to fit the 3D plane through the connected component area. For smaller TOF camera roll angles, the normal of the ground plane and the vertical direction vector of the image plane may have a large overlap. The dot product of the two can be used to distinguish the floor plan from the wall plane in the captured image.

[0066] The method 600 may optionally include, at 604, generating a first FPH for points in the first subset of points and generating a second FPH for points in the second subset of points. For example, the feature extraction component 164, for example, in combination with the processor 120, the memory 150, etc., may generate a first FPH for points in the first subset of points and a second FPH for points in the second subset of points. For example, the feature extraction component 164 may generate the PFH as an FPFH or other PFH that includes a geometric descriptor for each point or set of points in the point cloud that describes the point. The descriptor may have multiple features or other parameters or values ​​that describe the point and surrounding points, such as the smoothness or roughness of the point and surrounding parts, the angle or angle measurement represented by the point and surrounding points, etc. For example, the feature extraction component 164 may generate an FPH for each of the n points by looking at the k nearest neighbor point pairs, where n and k are positive integers. In this example, the feature extraction component 164 can create a local reference frame (eg, a Darboux frame) between a pair of neighbors, calculate the angular change between the normal vectors relative to the local reference frame, and place the pair-wise angular changes into a histogram.

[0067] In one example, the feature extraction component 164 can use the FPFH to generate a histogram that can have 33 values ​​for each point in the descriptor (e.g., each value is in the range of 0 to 255). In some examples, the hierarchical point matching component 302 can associate the point sets based on the descriptors received from the feature extraction component 164. In one example, the hierarchical point matching component 302 can attempt to identify a unique corresponding point in the second point subset for a given point in the first point subset based on comparing the descriptors or corresponding descriptor values ​​of the points. For example, the hierarchical point matching component 302 can associate points that have similar values ​​of 33 values ​​or similar aggregate values ​​representing 33 values, etc.

[0068] Figure 7Examples of point clouds 700 and 702 and corresponding descriptor spaces (such as descriptor spaces 704 and 706) according to aspects described herein are shown. Point cloud 700 can represent a point cloud of a leaf captured by the TOF sensor 110 in 3D world space, referred to as the left side (LHS) in the figure, and point cloud 702 can represent a second point cloud of the same leaf captured by the TOF sensor 110 in 3D world space from a different position / direction (also referred to as the right side (RHS) in the figure). The feature extraction component 164 can generate FPFH descriptors for the LHS point cloud 700 and the RHS point cloud 702. As described above, in one example, the FPFH descriptor can be a 33D space describing each point. In one example, the node matching component 302 can detect points f in the LHS point cloud and points f' in the RHS point cloud that have similar descriptors, such as similar values ​​within the descriptor space, similar descriptor values ​​where the descriptors are single values ​​representing various dimensions, etc. In one example, although the point clouds 700 and 702 may have unknown rotations and translations relative to each other, the feature extraction component 164 may detect that the descriptors of point f and point f' are within a threshold difference, which may be considered as possible unique corresponding points in the point clouds 700 and 702. For example, the node matching component 302 may determine that point f 708 corresponds to point F in the 3D descriptor space 704, and point f' corresponds to a similar point F' in the 3D descriptor space 706, so points f and f' will be considered as possible unique corresponding points.

[0069] Back to Figure 6, the method 600 may optionally include downsampling the first captured point cloud and the second captured point cloud at 606. For example, the cloud generation component 160, for example, in combination with the processor 120, the memory 150, etc., may downsample the first captured point cloud (e.g., to generate a first 3D point cloud) and the second captured point cloud (e.g., to generate a second 3D point cloud). Downsampling the point cloud at least for the purpose of identifying hierarchical points can reduce the number of points to be evaluated and / or for which descriptors to be calculated, which can reduce the processing resources required to perform hierarchical point matching. For example, the cloud generation component 160 downsamples the captured point cloud into a voxelized point cloud, where each voxel is the average of the downsampled pixels. In one specific example, the cloud generation component 160 may voxelize the point cloud based on a distance (e.g., four centimeters (cm)) such that each voxel is the average of the points in a 4 cm×4 cm box centered at the voxel. In any case, in some examples, the feature extraction component 164 can calculate the FPFH descriptor at each voxel location in the point cloud (e.g., downsampled or otherwise), and if the points are close together in the 3D descriptor space, the node matching component 302 can identify the descriptors and associate them. In one example, the hierarchical point matching component 302 can apply a Lowe ratio test to reduce false descriptor matches. In this regard, for example, the hierarchical point matching component 302 can associate a first subset of points in a first 3D point cloud with a second subset of points within a second 3D point cloud. This association can produce many associations, which can be refined as described herein to further reduce the processing required to consider each association at a later stage of object annotation.

[0070] In another example, the method 600 may optionally include extracting a first subset of points and a second subset of points having a threshold descriptor entropy at 608. For example, the feature extraction component 164, for example, in combination with the processor 120, the memory 150, etc., may extract a first subset of points and a second subset of points having a threshold descriptor entropy. This may be another optimization of performing point association at 602. In addition, in some examples, the feature extraction component 164 may extract features from the downsampled point cloud for further optimization. In one example, entropy may be defined as a logarithmic function of the value of the descriptor, which may provide an indication of how different the values ​​of the descriptors are relative to each other (e.g., how much variance there is between the 0-255 values ​​of each of the 33 different descriptors). A higher entropy value may indicate that the FPFH has a richer internal structure, which may produce more unique or interesting descriptors, so finding matching points with such descriptors may be a more likely actual real-world match. In this example, the feature extraction component 164 may associate the FPFH descriptor entropy with the geometric interest of the 3D world space position of the point. In some examples, feature extraction component 164 can ignore low entropy points to improve the efficiency of subsequent actions in object annotation described below.

[0071] The method 600 may include refining the first subset of points at 610 to generate a first improved subset of points, the first improved subset of points corresponding to fewer outliers in the second subset of points than in the first subset of points. For example, the outlier rejection component 304, for example in combination with the processor 120, the memory 150, the multi-cloud stitching component 166, etc., may refine the points of the first subset to generate a first improved subset of points, the subset corresponding to fewer outliers in the second subset than in the first subset. Similarly, the method 600 may also include refining the second subset of points at 612 to generate a second improved subset of points, the second improved subset of points corresponding to fewer outliers in the first subset of points than in the second subset of points. For example, the outlier rejection component 304, e.g., in combination with the processor 120, the memory 150, the multi-cloud stitching component 166, etc., can refine the points of the second subset to generate a second improved subset of points that corresponds to fewer outlier points of the first subset than the second subset.

[0072] In other words, in some examples, given point correspondences between a first subset of points in a first 3D point cloud and a second subset of points in a second 3D point cloud, the outlier rejection component 304 can delete some of the point correspondences to improve the accuracy and / or efficiency of the remaining actions in object annotation, such as cloud pair registration or motion synchronization, as further described herein. In some examples, the outlier rejection component 304 can evaluate multiple points from the first subset with multiple points from the second subset to verify whether these multiple points will still serve as connecting points between the two subsets. For example, the outlier rejection component 304 can calculate the distances between multiple points in each 3D point cloud to determine whether these distances are within a threshold correspondence range, and if so, the association can be maintained between the first and second subsets. If not, the outlier rejection component 304 can delete the association between at least one of the multiple points in the first subset and the corresponding one of the points in the second subset.

[0073] Figure 8An example of a graphical representation of point clouds 800 and 802 and corresponding feature spaces (such as descriptor spaces 804 and 806), point clouds 810 and 812, and hierarchical consistency 814 according to aspects described herein is shown. Point cloud 800 can represent the LHS point cloud of a leaf captured by TOF sensor 110 in 3D world space, and point cloud 802 can represent the RHS point cloud of the same leaf captured by TOF sensor 110 in 3D world space from different positions / directions. Feature extraction component 164 can generate FPFH descriptors for LHS point cloud 800 and RHS point cloud 802. In one example, based on having similar descriptors F and F' in 3D descriptor spaces 804 and 806, node matching component 302 can detect point f in LHS point cloud 800 and point f' in RHS point cloud 802 to have a point correspondence. Similarly, in one example, the node matching component 302 can detect that a point g in the LHS point cloud 800 and a point g' in the RHS point cloud 802 have a point correspondence relationship based on having similar descriptors G and G' in the 3D descriptor spaces 804 and 806. In one example, the outlier rejection component 304 can determine that a first distance between f and g and a second distance between f' and g' are within a threshold correspondence range relative to their corresponding real-world spaces, and can accordingly retain the association between f and f' and g and g' in the improved point subset.

[0074] Point cloud 810 may represent a point cloud of a leaf captured by TOF sensor 110 in a 3D world space, referred to as LHS in the figure, and point cloud 812 may represent a point cloud of the same leaf captured by TOF sensor 110 from different positions in the 3D world space, referred to as RHS in the figure. Feature extraction component 164 may generate FPFH descriptors for LHS point cloud 810 and RHS point cloud 812. In one example, hierarchical point matching component 302 may respectively detect points 820-a, 822-a, 824-a, and 826-a in LHS point cloud 800 and points 820-b, 822-b, 824-b, and 826-b in RHS point cloud 802 to have point correspondences based on having similar descriptors in 3D descriptor space. In one example, the outlier rejection component 304 can determine that the distances between points 820-a, 822-a, 824-a and / or between 820-b, 822-b, 824-b, etc. are within a threshold correspondence range, and can accordingly retain the association between these points in the corresponding improved point subset. In one example, the outlier rejection component 304 can determine that a first distance between point 826-a and one of points 820-a, 822-a, or 824-a, and a second distance between 826-b and a corresponding one of 820-b, 824-b, or 826-b are not within a threshold correspondence relationship, and can accordingly delete the association between points 826-a and 826-b in the corresponding improved point subset. This is also shown in the graphical representation of hierarchical consistency, where each of points 820 (820-a and 820-b), 822 (822-a and 822-b), and 824 (824-a and 824-b) have a correspondence, so these points can be said to form a cluster, but point 826 (826-a and 826-b) does not have a correspondence. As described below, removing the association of points 826-a and 826-b as outliers can reduce the processing performed in a later object annotation action.

[0075] As previously described, for example, outlier rejection may be a maximum clique problem that can be solved to find layer points with threshold correspondences and distances to determine which points in the subset are most likely to be the same point in a common coordinate system (although their coordinates may be different in the corresponding 3D point cloud of the captured TOF frame). In another example, the outlier rejection component 304 may further search for node triangles with point correspondences and distance correspondences (e.g., instead of searching for pairs or after searching for pairs) to determine which points to retain in the improved point subset. For example, the outlier rejection component 304 may search for node triangles (f, g, h) in the surviving features that are consistent with the triangles (f', g', h') in the second subset of points. In this example, the outlier rejection component 304 may retain the endpoints (a, b) and (a', b') of the longest sides of the triangles in the first and second improved point subsets, respectively. Furthermore, in this example, if |c-a|=|c'-a'| and |c-b|=|c'-b'| reaches a certain noise threshold, the outlier rejection component 304 can retain the (c, c') connection point. This can result in an improved point set with further geometric consistency.

[0076] The method 600 may include, at 614, determining a relative rotation and translation between the first 3D point cloud and the second 3D point cloud based on the point correspondence between the first refined point subset and the second refined point subset. For example, the cloud pair registration component 306, for example in combination with the processor 120, the memory 150, the multi-cloud stitching component 166, etc., may determine the relative rotation and translation between the first 3D point cloud and the second 3D point cloud based on the point correspondence between the first refined point subset and the second refined point subset. In one example, at least three valid hierarchical point matches are sufficient to correctly determine the relative rotation and translation between the point clouds, and the outlier rejection component 304 may generate a refined point subset using the above processing to attempt to include a valid hierarchy. Given the refined connected point set, the cloud pair registration component 306 may derive the rotation R and translation T of the second 3D point cloud relative to the first 3D point cloud (or vice versa) by removing the centroid from the 3D input set, calculating R from the singular value decomposition, and fixing T after R is known. In one example, the cloud pair registration component 306 can perform such cloud pair registration for TOF frame pairs that have at least a threshold number (e.g., 10) of geometrically consistent connection point matches. For example, if a TOF frame does not have at least a threshold number of matches, the multi-cloud stitching component 166 can reject the TOF frame, or otherwise send an error or other feedback to capture a new or additional replacement TOF frame, as described above. In another example, in the event that a TOF frame has more than a threshold number of matches, the cloud pair registration component 306 can select a number of layer points equal to the threshold to determine the rotation and translation (e.g., the first number of layer points, a randomly selected number of layer points, etc.).

[0077] The method 600 may include, at 616, aligning the first 3D point cloud and the second 3D point cloud in a common coordinate system based on relative rotation and translation. For example, the cloud pair registration component 306, for example in combination with the processor 120, the memory 150, the multi-cloud stitching component 166, the motion synchronization component 308, the bundle adjustment component 310, etc., may align the first 3D point cloud and the second 3D point cloud in a common coordinate system based on relative rotation and translation. In one example, the cloud pair registration may produce a set of redundant relative poses between TOF frames, and the relative rotation and translation may produce correlated noise. Thus, for example, as part of determining the relative rotation and translation at 614 and / or aligning the point clouds at 616, the motion synchronization component 308 may calculate an absolute TOF image that best agrees with the input set of relative transformations to synchronize the TOF images. The TOF poses may be in substantially any common coordinate system, which may include the coordinate system of the first TOF image.

[0078] For example, the cloud pair registration component 306 can align the point cloud to a common coordinate system based on the determined rotation and translation, which can be the coordinate system of the first TOF image. In this regard, the aligned point cloud can generate a combined point cloud in the common coordinate system. In addition, in some examples, the aligned point cloud can include a bundle adjustment component 310 that more accurately aligns the 3D point cloud. For example, the bundle adjustment component 310 can start from the 3D position of the motion synchronization layered point output by the motion synchronization component 308, and the reconstruction point can be initialized from the synchronization point average. The bundle adjustment component 310 can form a cost function based on the difference between the reconstructed point and the back-projected corresponding point, and minimize the cost by changing the TOF rotation and translation and the reconstructed point position. The result can be a clustered point cloud and a corresponding frustum returned in a common coordinate system, which can be the coordinate system of one of the TOF images (e.g., the first TOF image). Scene knowledge can be propagated between the stitched cloud and the TOF view through 3D geometry.

[0079] In some examples, the multi-cloud stitching component 166 can perform the above actions described in method 600 on multiple additional point clouds corresponding to other TOF images, such as performing the above actions on four point clouds corresponding to four TOF images with an azimuth angle of substantially 30 degrees around the object, and can construct a stitched point cloud of the four point clouds in a common coordinate system. In one example, the display device 140 can display a representation of the stitched point cloud. In another example, the stitched point cloud can also be used for target object isolation, object annotation, etc., as further described herein.

[0080] Fig. 9An example of a method 900 for isolating a target object in a plurality of TOF images to perform dimensioning thereof according to aspects described herein is shown. The method 900 may be performed by the sensor system 100, one or more components thereof (eg, the processor 120, etc.).

[0081] The method 900 may include converting a plurality of images from a TOF sensor into a plurality of 3D point clouds at 902. For example, the cloud generation component 160, for example in conjunction with the processor 120, the memory 150, etc., may convert a plurality of images from the TOF sensor 110 into a plurality of 3D point clouds. As described above, the cloud generation component 160 may generate an aggregated 3D point cloud for a given location based on a plurality of views in a view pair or other collection of a plurality of images at a given location. The cloud generation component 160 may also generate a separate 3D point cloud for each location.

[0082] Method 900 may include, at 904, identifying a possible large plane having a minimum threshold size in each of the plurality of 3D point clouds. For example, plane identification component 162, e.g., in combination with processor 120, memory 150, etc., may identify a possible large plane having a minimum threshold size in each of the plurality of 3D point clouds. As described above, plane identification component 162 may identify a large plane in each of the plurality of 3D point clouds as a set of adjacent points having the same surface normal, where the large plane may be determined to have a minimum threshold size (e.g., a minimum threshold number of points having the same surface normal). Plane identification component 162 may identify at least one large surface as a floor and / or may identify at least one other large surface as a wall, etc. In some examples, plane identification component 162 may distinguish between a floor plane and a wall plane based on the size of each plane, the proximity of the plane to the Z=0 plane (e.g., the floor may be closer to the Z=0 plane), etc. Once the 3D point clouds are stitched, as described above, and with further reference to action 906 , floor planes and / or wall planes and / or other large planes may be identified in the common coordinate system of the common 3D point cloud based on their coordinate positions in each corresponding 3D point cloud of the TOF sensor 110 .

[0083] The method 900 may include, at 906, generating a common 3D point cloud by aligning rotated and translated point clouds corresponding to at least a portion of the plurality of 3D point clouds. For example, the multi-cloud stitching component 166, e.g., in combination with the processor 120, the memory 150, etc., may generate a common 3D point cloud (in a common coordinate system) by aligning rotated and translated point clouds corresponding to at least a portion of the plurality of 3D point clouds. As described above, the multi-cloud stitching component 166 may rotate and translate the plurality of 3D point clouds, and may align the plurality of 3D point clouds to generate a common 3D point cloud.

[0084] The method 900 may include rotating and translating the common 3D point cloud at 908 to align the floor with the Z plane. For example, the target isolation component 168, for example in combination with the processor 120, the memory 150, etc., may rotate and translate the common 3D point cloud to align the floor with the Z=0 plane. For example, the plane identification component 162 may identify coordinates belonging to a large plane in the common 3D point cloud and / or in multiple 3D point clouds that form the common 3D point cloud. In addition, for example, the plane identification component 162 may identify a large plane as the floor, which may include identifying a plane with a maximum size, a plane that is substantially at or closest to the ground plane (e.g., the Z=0 plane), etc. In any case, the target isolation component 168 may rotate and translate the common 3D point cloud to place the identified floor plane on the ground in the common coordinate system (e.g., at Z=0).

[0085] The method 900 may include isolating the target object from the common 3D point cloud during rotation and translation at 910. For example, the target isolation component 168, such as in combination with the processor 120, the memory 150, etc., may isolate the target object from the common 3D point cloud during rotation and translation. Various actions may be performed to isolate the target object, such as Fig.10 Further described in .

[0086] Fig.10 An example of a method for isolating a target object from a common 3D point cloud while rotating and translating at 910 according to aspects described herein is shown. In some examples, isolating the target object may include inferring an average position of the TOF sensor 110 when capturing the image to infer that the position of the target object is within a threshold distance of the center of the capture position. For example, isolating the target object at 910 may optionally include projecting the average position of the TOF frustum reconstructed in the aligned point cloud to a ground plane as an origin at 1002. For example, the target isolation component 168, for example in combination with the processor 120, the memory 150, etc., may project the average position of the TOF frustum reconstructed in the aligned point cloud into a ground plane as an origin, where the origin may correspond to an (X, Y) coordinate on the Z=0 plane of the captured TOF image. For example, the target isolation component 168 may reconstruct a TOF frustum from a 3D point cloud and may project it to an average position in the Z=0 plane as an origin.

[0087] For example, isolating the target object at 910 may optionally include averaging the pointing directions of the frustums reconstructed within the ground plane at 1004. For example, the target isolation component 168, e.g., in conjunction with the processor 120, the memory 150, etc., may average the pointing directions of the frustums reconstructed within the ground plane (e.g., the Z=0 plane), which may define a preferred axis from the origin to the region where the target object may be detected and / or isolated.

[0088] For example, isolating the target object at 910 may optionally include modifying points in the common 3D point cloud by deleting points outside a sub-cloud ring of the common 3D point cloud between two ground plane circles centered at the origin, and deleting points outside a sub-cloud azimuth wedge defined by a specified number of degrees from the pointing direction, at 1006. For example, the target isolation component 168, e.g., in combination with the processor 120, the memory 150, etc., may modify points in the common 3D point cloud by deleting points outside a sub-cloud ring of the common 3D point cloud between two ground plane circles centered at the origin, and deleting points outside a sub-cloud azimuth wedge defined by a specified number of degrees from the pointing direction.

[0089] For example, by removing points outside the sub-cloud ring of the common 3D point cloud between two ground plane circles centered at the origin, the distance of the target object from the origin can be limited to a specific interval defined between the sub-cloud rings (e.g., 0.75 meters to 4.0 meters). This can simulate the assumption that the user does not place the TOF sensor 110 too close or too far from the imaged target object for dimensioning.

[0090] Additionally, for example, deleting points outside of the sub-cloud azimuth wedge defined by a specified number of degrees of the pointing direction can constrain the target object to be within a range of degrees of the pointing direction (e.g., 40 degrees in either direction of the preferred axis). This can simulate the assumption that the user centers the TOF sensor 110 primarily based on the location of the target object. In this example, deleting outliers can improve the accuracy of locating the target object to be isolated, and / or can reduce the processing time of identifying and / or isolating the target object.

[0091] For example, isolating the target object at 910 may optionally include modifying the common 3D point cloud at 1008 to delete points associated with one or more large planes. For example, the target isolation component 168, for example, in combination with the processor 120, the memory 150, etc., may modify the common 3D point cloud to delete points associated with one or more large planes. As described above, the target isolation component 168 may detect points in the common 3D point cloud associated with one or more large planes (such as a floor or wall or other large plane) based on the coordinates identified by the plane identification component 162. The target isolation component 168 may delete these points from the common 3D point cloud to further isolate the target object.

[0092] For example, isolating the target object at 910 may optionally include, at 1010, converting the common 3D point cloud into a height map and deleting points associated with objects less than a threshold size or objects not closest to the pointing direction from the origin. For example, the target isolation component 168, for example, in combination with the processor 120, the memory 150, etc., may convert the common 3D point cloud into a height map and delete points associated with objects less than a threshold size or objects not closest to the pointing direction of the origin. In some examples, the target isolation component 168 may obtain the remaining points of the common 3D point cloud after deleting the points associated with the large plane and the points outside the sub-cloud ring and the sub-cloud azimuth wedge, and project them down into the ground plane. This can generate a point cloud for an object in X, Y coordinates. In this example, the target isolation component 168 may delete sub-clouds (also referred to herein as point sets or connected components) associated with objects less than a threshold size (e.g., 8 inches) in the ground plane view. Of the remaining sub-clouds, in some examples, the target isolation component 168 can retain the sub-cloud closest to the pointing direction of the origin, as described above with reference to action 1004, and can discard any remaining sub-clouds. Therefore, it can be assumed that the remaining sub-clouds of the target object are sub-clouds that have at least a threshold size and are closest to the pointing line from the origin to the 3D point cloud space.

[0093] Back to Fig. 9 , the method 900 may optionally include calculating a size measurement of the target object in the common coordinate system at 912. For example, the WLH component 170, for example, in combination with the processor 120, the memory 150, etc., may calculate a size measurement of the target object in the common coordinate system. In some examples, the WLH component 170 may fit a bounding box around the target object isolated by the target isolation component 168. For example, the WLH component 170 may search in a seven-dimensional parameter space for the tightest bounding box of a sub-cloud surrounding the target object in the common coordinate system. The seven parameters of the seven-dimensional parameter space may include a box size (W, L, H), a box center (X, Y, Z), and a box direction azimuth (Az) of the bounding box. The WLH component 170 may determine the bounding box based on the seven parameters.

[0094] The method 900 may optionally include displaying an indication of the size measurement at 914. For example, the display device 140, e.g., in conjunction with the processor 120, the memory 150, etc., may display an indication of the size measurement, such as the WLH measurement of the bounding box determined by the WLH component 170. The displayed indication may include a number corresponding to each of the width, length, and height measurements, a bounding box drawn around the target object in the view on the display device 140 (to ensure that the correct object is sized), etc.

[0095] The method 900 may optionally include displaying the target object isolated from the common 3D point cloud at 916. For example, the display device 140, for example in combination with the processor 120, the memory 150, etc., may display the target object in the common 3D point cloud. In some examples, the display device 140 may display one or more projections of the common 3D point cloud captured in the TOF image. For example, the display device 140 may display a top orthogonal view of the cloud, a front orthogonal view of the cloud, and / or a side orthogonal view of the cloud. In addition, for example, the display device 140 may display an orthogonal view with a bounding box drawn based on the WLH measurement. In addition, in one example, the display device 140 may display a general perspective view of the scene captured in one of the TOF images, and a bounding box drawn in 3D around the target object in a common coordinate system. In these examples, the display and / or point cloud for object dimensioning may bring together depth information from multiple TOF sensor 110 images, IR information from the images, 3D geometry of the images, etc.

[0096] In some examples, WLH component 170 may transmit the WLH measurements and / or point cloud views to another device for storage or further processing, such as a system for determining packaging or shipping logistics. In another example, WLH component 170 may store the WLH measurements and / or point cloud views in memory 150 for use in calling measurements or displaying on display device 140.

[0097] The following aspects are merely illustrative and aspects thereof may be combined with other embodiments or aspects of the teachings described herein without limitation.

[0098] Aspect 1 is a method for obtaining an image from a TOF sensor to determine the size of an object, comprising displaying a prompt for capturing at least one image of an object through an interface, capturing at least one image of the object based on interaction with the prompt, determining whether multiple images including the at least one image are sufficient to perform object dimensioning, and if the multiple images are sufficient to perform object dimensioning, providing images for performing object dimensioning to calculate or display the size of the object.

[0099] In aspect 2, the method of aspect 1 includes displaying an indication of a position relative to the object through an interface or other feedback to capture a next image of the object.

[0100] In aspect 3, the method of aspect 2, includes wherein the position is substantially 30 degrees in azimuth relative to a previous position of the object at which the at least one image was captured.

[0101] In aspect 4, the method of any one of aspects 1 to 3, includes wherein determining whether the plurality of images is sufficient to perform object dimensioning includes determining whether the plurality of images includes a certain number of images.

[0102] In aspect 5, the method of any of aspects 1 to 4, includes wherein determining whether the plurality of images is sufficient to perform object dimensioning includes determining whether the plurality of images includes two or more images for each of a plurality of positions relative to the object.

[0103] In aspect 6, the method of any one of aspects 1 to 5 includes wherein in a case where the plurality of images are insufficient to perform object dimensioning, a prompt to recapture at least one image of the object is displayed through the interface.

[0104] In aspect 7, the method of aspect 6 includes indicating the position relative to the object through an interface or other feedback to recapture at least one image of the object.

[0105] In aspect 8, the method of any of aspects 1 to 7 includes wherein at least one image comprises two images of an object captured at similar locations, and further includes determining whether to retain the two images of the object to perform object dimensioning based at least in part on whether the two images have sufficient difference to achieve a threshold signal-to-noise ratio.

[0106] In aspect 9, the method of any of aspects 1 to 8 includes determining whether to retain at least one image of the object to perform object dimensioning based at least in part on whether the object has a threshold size in the at least one image.

[0107] In aspect 10, the method of any of aspects 1 to 9 includes receiving width, length, and height measurements of the object as part of performing object sizing, and displaying an indication of the width, length, and height measurements of the object through an interface.

[0108] In aspect 11, the method of aspect 10 includes displaying one or more orthogonal views of the object with indications of width, length, and height measurements of the object as a bounding box around the object.

[0109] In aspect 12, the method of any of aspects 1 to 11, comprises providing at least one image for object dimensioning while awaiting capture of a next image of the object from a next position.

[0110] In aspect 13, the method of any one of aspects 1 to 12 includes outputting a feedback score indicating the extent to which the plurality of images correspond to the desired location of the captured object.

[0111] Aspect 14 is a method for obtaining images from a TOF sensor to determine the size of an object, comprising acquiring multiple images including an object from the TOF sensor, verifying whether the multiple images are taken at an expected posture relative to the object, and if the multiple images are verified to be taken at the expected posture, providing the multiple images for object dimensioning to calculate or display the size of the object.

[0112] In aspect 15, the method of aspect 14 includes receiving a position of each of a plurality of images from a positioning sensor associated with a TOF camera, wherein verifying whether the plurality of images are captured in an expected posture is based on comparing at least a first position of one of the plurality of images and at least a second position of a different image in the plurality of images.

[0113] In aspect 16, the method of any of aspects 14 or 15, includes wherein verifying whether the plurality of images are taken at a desired pose includes determining that the plurality of images are taken at positions approximately 30 degrees apart in azimuth about the object.

[0114] In aspect 17, according to the method of aspect 16, verifying whether the multiple images are taken in the expected posture includes determining that the multiple images include at least four images, and the multiple images are taken at positions approximately 30 degrees apart in azimuth around the object.

[0115] In aspect 18, the method according to any one of aspects 14 to 17, wherein verifying whether the plurality of images are taken at the desired posture comprises verifying that for each position of the desired posture, the plurality of images comprises at least two images.

[0116] In aspect 19, the method of any one of aspects 14 to 18 includes, wherein verifying whether the plurality of images are taken in the expected pose is based on a model of the object, wherein the expected pose is inferred from the model of the object.

[0117] Aspect 20 is a method for aligning multiple 3D point clouds into a common 3D point cloud, comprising: associating a first point subset in a first 3D point cloud generated from one TOF view with a second point subset in a second 3D point cloud generated from another TOF view based on each point in the first point subset having a threshold correspondence relationship with a unique corresponding point in the second point subset; refining the first point subset and the second point subset to generate a first improved point subset and a first improved point subset, the first improved point subset having fewer correspondence relationships with outliers in the second subset than in the first point subset, and refining the first point subset and the second improved point subset. A second point subset is used to generate a first improved point subset and a second improved point subset, wherein the first improved point subset includes fewer outlier point correspondences with the second point subset than the first point subset, and the second improved point subset includes fewer outlier point correspondences with the first point subset than the second point subset; based on the point correspondence relationship between the first improved point subset and the second improved point subset, determine the relative rotation and translation between the first 3D point cloud and the second 3D point cloud; and align the first 3D point cloud and the second 3D point cloud in a common coordinate system based on the relative rotation and translation.

[0118] In aspect 21, the method of aspect 20 includes generating a first FPFH for a point in a first point subset, and generating a second FPFH for a point in a second point subset, wherein associating a first point of the first point subset with a second point of the second point subset is based on determining whether a distance between a first FPFH descriptor of the first point in the first FPFH and a second FPFF descriptor of the second point in the second FPFH is within a threshold difference.

[0119] In aspect 22, the method of either aspect 20 or 21 comprises extracting a first subset of points from a first 3D point cloud having at least a threshold descriptor entropy, and extracting a second subset of points from a second 3D point cloud having at least the same threshold descriptor entropy, wherein associating the first subset of points with the second subset of points comprises evaluating descriptor correspondences of the first subset of points and the second subset of points.

[0120] In aspect 23, the method of any one of aspects 20 to 22 includes wherein refining the first point subset and the second point subset comprises using a Lowes ratio test on the 33-dimensional descriptors of the first point subset and the second point subset to compare points having a point correspondence relationship between the first point subset and the second point subset, and retaining points that pass the Lowes ratio test in the first improved point subset and the second improved point subset.

[0121] In aspect 24, the method of any one of aspects 20 to 23, includes wherein refining the first point subset and the second point subset includes comparing a first distance between a first portion of points from the first point subset with a second distance between a second portion of corresponding points from the second point subset, and if the difference between the first distance and the second distance is less than a threshold, retaining the first portion of the points in the first refined point subset and retaining the second portion of the points in the second refined point subset.

[0122] In aspect 25, the method of any one of aspects 20 to 25, includes wherein refining the first point subset and the second point subset comprises: for each first part of points of the first point subset and each second part of points of the second subset whose separation distance is less than a threshold, comparing the separation distance between a third point in the first point subset and the first part of the points with the corresponding separation distance between a corresponding third point in the second point subset and the second part of the points, and if the separation distance and the corresponding separation distance are less than the threshold, retaining the first part and the third point of the points of the first point subset in the first improved point subset, and retaining the second part and the corresponding third point of the points of the second point subset in the second improved point subset.

[0123] In aspect 26, the method of any one of aspects 20 to 25 comprises downsampling the first captured point cloud to a first 3D point cloud, and downsampling the second captured point cloud to a second 3D point cloud.

[0124] In aspect 27, the method of any one of aspects 20 to 26 comprises aligning the one or more 3D point clouds with the first 3D point cloud in a common coordinate system using relative rotations and translations calculated by motion synchronization and bundle adjustment.

[0125] In aspect 28, the method of any one of aspects 20 to 27 includes calculating a size measurement of the object within the common coordinate system.

[0126] In aspect 29, the method of aspect 28 includes displaying an indication of the dimensional measurement.

[0127] Aspect 30 is a method for separating a target scene object from multiple images of a TOF sensor, comprising converting multiple images from a TOF sensor into multiple 3D point clouds, identifying one or more large planes having a minimum threshold size in each of the multiple 3D point clouds, generating a common 3D point cloud by aligning rotated and translated point clouds corresponding to at least a portion of the multiple 3D point clouds, rotating and translating the common 3D point cloud so that one of the one or more large planes is identified as a ground plane within a common coordinate system of the common 3D point cloud, and separating the target object from the common 3D point cloud while rotating and translating.

[0128] In aspect 31, the method of aspect 30 includes: separating the target object includes projecting the average position of the reconstructed TOF frustum in the common 3D point cloud into the ground plane as the origin, averaging the pointing directions of the reconstructed TOF frustum in the ground plane, and modifying the common 3D point cloud by deleting points outside the sub-cloud ring of the common 3D point cloud located between two ground plane circles centered on the origin and deleting points outside the sub-cloud azimuth wedge defined by a specified degree from the pointing direction.

[0129] In aspect 32, the method of any of aspects 30 or 31 includes separating the target object including further removing points associated with one or more large planes representing floors or potential walls from the common 3D point cloud.

[0130] In aspect 33, the method of any one of aspects 30 to 32 comprises: separating the target object comprises converting the common 3D point cloud into a height map; and removing points associated with objects smaller than a threshold size from the common 3D point cloud.

[0131] In aspect 34, the method of aspect 33 includes: separating the target object includes selecting a sub-cloud having a point located closest to the preferred axis defined by the direction in which the origin points.

[0132] In aspect 35, the method of any one of aspects 30 to 34 includes displaying the target object separated from the common 3D point cloud on a screen interface.

[0133] In aspect 36, the method of any one of aspects 30 to 35 comprises measuring the width, length and height dimensions of the target object separated from the common 3D point cloud.

[0134] In aspect 37, the method of aspect 36 includes searching for a bounding box around the target object in a seven-dimensional parameter space, wherein measuring the width, length, and height of the target object is based on the width, height, and length of the bounding box.

[0135] In aspect 38, the method of aspect 37 includes displaying the common 3D point cloud and a bounding box surrounding the target object on a screen interface.

[0136] In aspect 39, the method of aspect 38 includes: displaying the target object includes jointly displaying the common 3D point cloud and a bounding box surrounding the target object in three orthogonal projections.

[0137] Aspect 40 is an imaging system comprising a TOF sensor or a processor, wherein one of the TOF sensor or the processor is configured to perform one or more operations of one or more methods in any one of aspects 1 to 39.

[0138] Aspect 41 is an apparatus comprising means for performing the operations of one or more methods in any one of aspects 1 to 39.

[0139] Aspect 42 is a computer-readable medium comprising code executable by one or more processors to perform the operations of one or more methods of any one of Aspects 1 to 39.

[0140] The detailed description described above in conjunction with the accompanying drawings describes examples and does not represent the only example that can be implemented or within the scope of the claims. The term "example" used in this specification means "as an example, instance, or illustration", rather than "preferred" or "superior to other examples". The detailed description includes specific details for providing an understanding of the described technology. However, these technologies can be practiced without these specific details. For example, the functions and arrangements of the elements discussed can be changed without departing from the scope of this disclosure. In addition, various examples can appropriately omit, replace, or add various programs or components. For example, the described method can be performed in an order different from the order described, and various steps can be added, omitted, or combined. In addition, the features described in some examples can be combined in other examples. In some cases, well-known structures and devices are shown in block diagram form to avoid blurring the concepts of the described examples.

[0141] Information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, computer executable codes or instructions stored on a computer-readable medium, or any combination thereof.

[0142] In addition, for example, an element, any part of an element, or any combination of elements may be implemented as a "processing system" including one or more processors. Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on chip (SoCs), baseband processors, field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described in this disclosure. One or more processors in a processing system may execute software. Software should be broadly interpreted as instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, processes, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description languages, or other names.

[0143] Therefore, in one or more example aspects, the functions described can be implemented with hardware, software, or any combination thereof. If implemented with software, these functions can be stored on a non-transient computer-readable medium or encoded as one or more instructions or codes. Computer-readable media include computer storage media. Storage media can be any available media that a computer can access. As an example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, a combination of computer-readable media of the above types, or any other medium that can be used to store computer executable code in the form of instructions or data structures accessible by a computer. In addition, the features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented in different physical locations.

[0144] In addition, the various illustrative blocks and components described in conjunction with the disclosure herein may be implemented or executed with a specially programmed device, such as, but not limited to, a processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, which are designed to perform the functions described herein. The specially programmed processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The specially programmed processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0145] Furthermore, as used herein, including in the claims, “or”, when used in a list of items beginning with “at least one of,” means a disjunctive list, for example, a list of “at least one of A, B, or C” means A or B or C or AB or AC or BC or ABC (i.e., A and B and C).

[0146] The previous description of the disclosure is provided to enable those skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the common principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. In addition, although the elements of the described aspects may be described or claimed in the singular, the plural form may be considered unless a limitation to the singular is explicitly stated. In addition, unless otherwise stated, all or a portion of any aspect may be used together with all or a portion of any other aspect. Therefore, the disclosure is not limited to the examples and designs described herein, but should conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for obtaining an image from a time-of-flight (TOF) sensor to determine the size of an object, comprising: obtaining a plurality of images including an object from the TOF sensor; verifying that the plurality of images are captured at an expected pose relative to the object; and In case the plurality of images are verified to be taken in the expected posture, the plurality of images are provided for object dimensioning to calculate or display the size of the object.

2. The method according to claim 1 further includes receiving a position of each of the multiple images from a positioning sensor associated with the TOF sensor, wherein verifying whether the multiple images are captured in an expected posture is based on comparing at least a first position of one of the multiple images and at least a second position of a different one of the multiple images. 3 . The method of claim 1 , wherein verifying whether the plurality of images are captured at an expected pose comprises determining that the plurality of images are captured at positions approximately 30 degrees apart in azimuth about the object.

4. The method of claim 3, wherein verifying whether the plurality of images are captured in an expected pose comprises determining that the plurality of images include at least four images, the plurality of images being captured at positions approximately 30 degrees apart in azimuth around the object. 5 . The method of claim 1 , wherein verifying whether the plurality of images are taken at an expected posture comprises verifying that for each position of the expected posture, the plurality of images includes at least two images. 6 . The method of claim 5 , wherein verifying whether the plurality of images are captured at a desired pose comprises verifying whether the at least two images produce a threshold signal-to-noise ratio.

7. The method of claim 1, wherein verifying whether the plurality of images are taken at the expected pose is based on a model of the object, wherein the expected pose is inferred from the object model.

8. A system comprising: Memory; and a processor coupled to the memory, wherein the processor is configured to: obtaining a plurality of images including an object from the TOF sensor; verifying that the plurality of images are captured at an expected pose relative to the object; and In case the plurality of images are verified to be taken in the expected posture, the plurality of images are provided for object dimensioning to calculate or display the size of the object.

9. The system of claim 8, wherein the processor is further configured to receive a position of each of the plurality of images from a positioning sensor associated with the TOF sensor, wherein verifying whether the plurality of images are captured in an expected posture is based on comparing at least a first position of one of the plurality of images and at least a second position of a different one of the plurality of images.

10. The system of claim 8, wherein the processor is configured to verify that the plurality of images are captured at an expected pose comprises determining that the plurality of images are captured at positions approximately 30 degrees apart in azimuth about the object.

11. The system of claim 10, wherein the processor is configured to verify that the plurality of images are captured in an expected pose comprises determining that the plurality of images comprises at least four images captured at positions approximately 30 degrees apart in azimuth around the object.

12. The system of claim 8, wherein the processor is configured to verify that the plurality of images are taken at an expected posture comprises verifying that for each position of the expected posture, the plurality of images includes at least two images.

13. The system of claim 12, wherein the processor is configured to verify whether the plurality of images are captured at a desired pose comprises verifying whether the at least two images produce a threshold signal-to-noise ratio.

14. The system of claim 8, wherein the processor is configured to verify whether the plurality of images are taken at the expected pose based on a model of the object, wherein the expected pose is inferred from the object model.

15. A computer readable medium comprising code executable by one or more processors for obtaining an image from a time-of-flight (TOF) sensor to determine the size of an object, the code comprising code for: obtaining a plurality of images including an object from the TOF sensor; verifying that the plurality of images are captured at an expected pose relative to the object; and In case the plurality of images are verified to be taken in the expected posture, the plurality of images are provided for object dimensioning to calculate or display the size of the object.

16. The computer-readable medium of claim 15, further comprising code for receiving a position of each of the plurality of images from a positioning sensor associated with the TOF sensor, wherein the code is for verifying whether the plurality of images are captured in an expected posture based on comparing at least a first position of one of the plurality of images and at least a second position of a different one of the plurality of images.

17. The computer-readable medium of claim 15, wherein the code for verifying that the plurality of images were taken at an expected pose comprises determining that the plurality of images were taken at positions approximately 30 degrees apart in azimuth about the object.

18. The computer-readable medium of claim 17, wherein the code for verifying that the plurality of images are captured in an expected pose comprises determining that the plurality of images comprises at least four images captured at positions approximately 30 degrees apart in azimuth about the object.

19. The computer-readable medium of claim 15, wherein the code for verifying that the plurality of images were taken at an expected pose comprises verifying that for each position of the expected pose, the plurality of images includes at least two images.

20. The computer-readable medium of claim 15, wherein said code for verifying that the plurality of images are taken at the expected pose is based on a model of the object, wherein the expected pose is inferred from the object model.

Images