Physical object boundary detection techniques and systems

Abstract

Physical object boundary detection techniques and systems are described. In one example, a method implemented by at least one computing device is presented, the method comprising: receiving, by the at least one computing device, motion sensing data and a digital image corresponding to a physical environment comprising a physical object; generating, by the at least one computing device, point cloud data based on the motion sensing data and a digital image corresponding to a physical environment including the physical object; generating, by the at least one computing device, a three-dimensional boundary of the physical object from point cloud data generated based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating, by the at least one computing device, digital content based on the three-dimensional boundary, the digital content indicating availability of the physical object; and displaying, by the at least one computing device, the digital content in a user interface.

Description

[0001] This application is a divisional application of invention patent application No. 201980032813.X, whose application date is May 10, 2019, international application number is PCT / US2019 / 031842, and which entered the Chinese national phase on November 16, 2020, and whose invention name is "Physical Object Boundary Detection Technology and System".

[0002] Related Applications

[0003] This application claims priority to U.S. Application No. 15 / 984,140, ​​filed on May 18, 2018, entitled “Physical Object Boundary Detection Techniques and Systems,” the entire disclosure of which is incorporated herein by reference. Technical Field

[0004] The present disclosure relates to physical object boundary detection techniques. Background Art

[0005] Conventional techniques for estimating delivery costs and packaging materials are modal and require multiple different devices that may not be readily available to the user. For example, a user may wish to have a physical object delivered. To do this, the user first uses a tape measure to determine the size of the physical object. The user then determines the size of the available packaging (e.g., envelope or box size) that corresponds to the size of the physical object, and also makes a best guess at the amount of filler to be used in the packaging. To do this, the user may visit a website from which these materials are available, go to a store in person to purchase these materials based on memorized sizes, etc. Therefore, these conventional techniques cannot determine the appropriate size of the packaging materials (e.g., box size), but are limited to at best using various tools that may not be readily available to get a rough size of the object. Therefore, conventional techniques rely on a "best guess" on the part of the user, which is inaccurate, frustrating, and costly due to inaccuracies in obtaining excess packaging for "safety reasons." Similar frustrations may also be encountered in determining an estimate of the delivery cost of a physical object. Therefore, these conventional techniques are inefficient and cause user frustration. Summary of the invention

[0006] Physical object boundary detection techniques and systems are described that overcome challenges of conventional techniques by achieving improved accuracy in calculating three-dimensional boundaries of physical objects. For example, this can be performed using a device that is readily available to a typical user (e.g., a mobile phone). The three-dimensional boundaries can then be used to automatically and without user intervention to obtain shipping data, such as shipping costs, order packaging, etc.

[0007] In one example, an augmented reality module generates three-dimensional point cloud data. This data describes the depth at various points within a physical environment that includes physical objects. A physical object boundary detection module is then employed to filter the point cloud data by removing points corresponding to a base plane. In this example, a user input is received to select a location corresponding to a physical object. The physical object boundary detection module then performs a nearest neighbor search to find a subset of points within the filtered point cloud data that correspond to the physical object.

[0008] Based on the subset, the physical object boundary detection module projects the subset of points onto a base plane to generate a two-dimensional boundary, such as a two-dimensional bounding box. The two-dimensional boundary is then extruded to form a three-dimensional boundary based on a height determined from a point in the filtered point cloud data having a maximum distance from the base plane.

[0009] In another example, point cloud data is also generated by the AR module using a three-dimensional sensing device and a point cloud generation module. For each viewing angle (e.g., on a surface where a physical object is arranged), a boundary (e.g., a rectangular boundary) is calculated in two dimensions. The module selects at least one of these two-dimensional boundaries for calculating a three-dimensional boundary, e.g., a three-dimensional boundary. To this end, a three-dimensional boundary is generated using the selected two-dimensional boundary, the size of which is based on the selected two-dimensional boundary and the height of the physical object detected using the point cloud data.

[0010] In another example, a method implemented by at least one computing device is provided, the method comprising: receiving, by the at least one computing device, motion sensing data and a digital image corresponding to a physical environment including a physical object; generating, by the at least one computing device, point cloud data based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating, by the at least one computing device, a three-dimensional boundary of the physical object based on the point cloud data generated based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating, by the at least one computing device, digital content based on the three-dimensional boundary, the digital content indicating availability of the physical object; and displaying, by the at least one computing device, the digital content in a user interface.

[0011] In another example, a system is provided, comprising: at least one hardware processor; and

[0012] A non-transitory computer-readable storage medium storing executable instructions, which, when executed, cause the at least one hardware processor to perform operations, the operations comprising: receiving motion sensing data and a digital image corresponding to a physical environment including a physical object; generating point cloud data based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating a three-dimensional boundary of the physical object based on the point cloud data generated based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating digital content based on the three-dimensional boundary, the digital content indicating availability of the physical object; and displaying the digital content in a user interface.

[0013] This summary introduces a series of concepts in a simplified form that will be further described in the following "Detailed Description". Therefore, this summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The detailed description is described with reference to the accompanying drawings. The entities shown in the drawings may refer to one or more entities, and thus singular or plural entities may be referred to interchangeably in the discussion.

[0015] Figure 1 is an illustration of an environment in which the techniques described herein may be employed during operation in an example implementation.

[0016] Figure 2 Depicts a more detailed view of an example implementation Figure 1 The AR module and the physical object boundary detection module operate in the system.

[0017] Figure 3 Depicted is an example of point cloud data.

[0018] Figure 4 Describes from Figure 3 Example of filtered point cloud data generated by point cloud data.

[0019] Figure 5 Describes the use of proximity search from Figure 4 An example of a subset of points generated from the filtered point cloud data.

[0020] Figure 6 Describes the generation Figure 5 Example of a 2D projection of a subset of the points.

[0021] Figure 7 Describes from Figure 6 An example of generating a 2D boundary from the 2D projection of .

[0022] Figure 8 Depicts the use of extrusion from Figure 7 Example of generating a 3D boundary from a 2D boundary.

[0023] Fig. 9 An example implementation is depicted in which: Figure 8 The three-dimensional boundary of the Figure 1 A portion of the computing device's live camera feed.

[0024] Fig.10 is a flow chart depicting a process of generating a three-dimensional boundary based on point cloud data obtained from an AR module in an example implementation.

[0025] Fig.11 is a flow chart depicting a process for generating a two-dimensional boundary based on point cloud data generated from multiple perspectives of a physical object in an example implementation.

[0026] Fig.12 is a diagram depicting an example implementation in which Fig.11 A flowchart of a process of selecting a two-dimensional boundary from two-dimensional boundaries and using the two-dimensional boundary to generate a three-dimensional boundary.

[0027] Fig.13 Depicted are example implementations of generating point cloud data from multiple views and generating a two-dimensional boundary for each of the views.

[0028] Fig.14 An example system is shown that includes various components of an example device that can be implemented as described with reference to Figures 1 to 13 Any type of computing device described and / or utilized to implement implementations of the techniques described herein. DETAILED DESCRIPTION

[0029] Overview

[0030] Physical object boundary detection techniques and systems are described. These techniques support the calculation of the three-dimensional boundaries of physical objects, for example, as a three-dimensional bounding box including height, width, and depth dimensions. To this end, point cloud data is generated by an AR module of a computing device, which can be included as part of a mobile phone and is therefore easily accessible to a user, and the point cloud data describes the depth of a point within a physical environment. The point cloud data is then used by the computing device to determine the three-dimensional boundaries (e.g., bounding boxes) of the physical objects in the physical environment. The three-dimensional boundaries can then be used to support a variety of functions, such as generating shipping data for packaging materials (e.g., boxes, tubes, envelopes) recommended for packaging physical objects, estimated shipping costs for physical objects, and can include the formation of digital content that is configured to provide a physical object for sale including shipping data. In this way, the techniques described herein overcome the challenges of conventional techniques, and therefore, a suitable packaging size can be determined for a physical object without being limited to the conventional techniques of predicting a predefined box based on predefined measurements and "best guesses."

[0031] For example, a user may launch an application associated with a service provider system to offer a product for sale. In this example, the application is configured to generate digital content as part of the offering for sale, such as a web page, screen, etc. in a mobile application. As part of this, the application may include a text description that can be used to detail the physical object, a user interface that supports the capture of a digital image of the physical object, and may also include shipping data for the object.

[0032] In this example, in response to user input received via the user interface to initiate generation of the shipping data, the shipping data is generated automatically and without user intervention.The shipping data may be based on the dimensions of the three-dimensional boundary calculated by execution of the application by the computing device.

[0033] In a first example, an augmented reality module of a computing device includes a three-dimensional sensing device and a point cloud generation module configured to generate three-dimensional point cloud data. The data describes the depth at various points within a physical environment including physical objects, for example, a depth map. These depths can be determined using various functions, examples of which include using digital images and motion detection sensors to determine parallax, using radio waves (e.g., Wi-Fi waves as part of Project Tango), time-of-flight cameras, structured grating arrays, etc. In some cases, the point cloud data may have a resolution (i.e., number of points) that may result in inefficient processing. Therefore, in such cases, the point cloud data is downsampled to improve the operation of the computing device.

[0034] A physical object boundary detection module is then employed to filter the point cloud data by removing points corresponding to the base plane. For example, a user input may be received relative to a digital image (e.g., a real-time camera feed) displayed in a user interface that corresponds to the point cloud data being generated. The user may then "tap" a portion of the image corresponding to the base plane, e.g., a surface on which a physical object is arranged. The physical object boundary detection module then uses a nearest neighbor search to remove points corresponding to the base plane. This serves to isolate points above the base plane and therefore that may correspond to a physical object. In another example, the module detects the base plane automatically and without user intervention.

[0035] In this example, another user input is received to select a location corresponding to a physical object. To do this, the user then "taps" on that location in the digital image (e.g., a live camera feed), and the physical object boundary detection module performs a nearest neighbor search to find a subset of points within the filtered point cloud data that correspond to the physical object.

[0036] Based on the subset, the physical object boundary detection module projects the subset of points onto the base plane to generate a two-dimensional boundary, such as a bounding box. Then, based on the height determined from the point with the maximum distance from the base plane in the filtered point cloud data, the two-dimensional boundary is extruded to form a three-dimensional boundary. Therefore, the three-dimensional boundary includes dimensions that define the maximum external dimensions of the physical object, such as width, depth, and height.

[0037] The three-dimensional boundary can support a variety of functions, such as generating shipping data that recommends packaging materials for packaging the physical object, estimating the shipping cost of the physical object, etc. Continuing with the aforementioned example, the shipping data can be included by the module automatically and without user intervention as part of providing the digital content of the physical object for sale. In this way, the techniques described herein increase the efficiency of user interaction with the computing device in a non-modal manner in which the user can stay within the context of the application to effectively generate shipping costs to include as part of the digital content.

[0038] In another example, point cloud data is also generated by the AR module using a three-dimensional sensing device and a point cloud generation module. In this example, point cloud data is generated from multiple different perspectives. A two-dimensional boundary is calculated in two dimensions for each perspective, such as a two-dimensional bounding box on a surface where a physical object is arranged.

[0039] At least one of these two-dimensional boundaries is selected by the module for calculating the three-dimensional boundary. For example, the selection can be based on which of the two-dimensional boundaries is the smallest, for example, by area. The selected two-dimensional boundary is then used to generate a bounding box or other shape (e.g., selected based on the size of the available packaging material) to generate a three-dimensional boundary, the size of the above-mentioned bounding box or other shape being based on the selected two-dimensional boundary and the height of the physical object detected using the point cloud data. The three-dimensional boundary can then be used to support various functions, such as outputting the size of the physical object, generating augmented reality content showing the size and box, generating delivery data, etc. Further discussion of these and other examples is included in the following sections and is illustrated using corresponding figures.

[0040] In the following discussion, an example environment in which the techniques described herein may be employed is first described. Example procedures that may be performed in the example environment as well as other environments are then described. Thus, the execution of the example procedures is not limited to the example environment, and the example environment is not limited to the execution of the example procedures.

[0041] Example Environment

[0042] Figure 1 1 is an illustration of a digital media environment 100 in an example implementation that, when in operation, can employ the physical object boundary detection techniques described herein. The illustrated environment 100 includes a computing device 102, which is shown being held by a user 104. The computing device 102 can be configured in a variety of ways.

[0043] For example, computing device 102 may be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet or mobile phone as shown), etc. Thus, computing device 102 may range from a full-resource device with large amounts of memory and processor resources (e.g., a personal computer, a game console) to a low-resource device with limited memory and / or processor resources (e.g., a mobile device). Furthermore, although a single computing device 102 is shown, computing device 102 may represent a plurality of different devices, such as those used by an enterprise to perform tasks such as Fig.14 Multiple servers operating “on the cloud” as described.

[0044] In the example shown, the computing device 102 is configured to capture data of a physical environment in which the physical object 106 is arranged to generate the delivery data 108. To this end, the computing device 102 utilizes the AR module 110. The AR module 110 includes a three-dimensional (3D) sensing device 112 and a point cloud generation module 114, which are configured to generate point cloud data 116 describing the depth at various points within the physical environment. For example, following the right-hand convention, the point cloud data 116 can describe points, where, as part of a coordinate system, the y-axis is vertical and the x-axis is horizontal and the z-axis defines the depth.

[0045] In one example, the 3D sensing device 112 and the point cloud generation module 114 employ a technique known as visual inertial odometry, wherein the 3D sensing device 112 includes motion detection sensors (e.g., gyroscopes, accelerometers) and cameras to capture digital images of the physical environment. The point cloud generation module 114 is configured to perform computer vision analysis of the digital images to identify features in the digital images and track differences in the positions of these features.

[0046] The point cloud generation module 114 compares the motion sensing data from the motion detection sensor with the differences in the tracked feature positions to generate point cloud data 116, which models the depth of points in the physical environment as, for example, a depth map. In this way, the AR module 110 can track the position of the computing device 102 as it moves to build an understanding of the physical environment based on features recognized from digital images. This functionality can also support basic plane detection, for example, a flat surface such as a floor or a table. Therefore, in this example, the point cloud data 116 can be generated by a variety of mobile computing devices such as smart phones and tablets without dedicated hardware.

[0047] In another example, dedicated hardware is utilized to transmit radio waves (e.g., Wi-Fi waves), the return of which is sensed by the 3D sensing device 112 (e.g., using radar technology). This includes using motion tracking as described above (e.g., using accelerometers, barometers, GPS, and / or gyroscopes) so that the point cloud generation module 114 can gain an understanding of the depth of points within the physical environment to generate point cloud data 116. Other examples are also contemplated, such as time-of-flight cameras, structured grating array devices, etc.

[0048] The AR module 110 then exposes the point cloud data 116 to a physical object boundary detection module 118 via an application programming interface (API). This module represents functionality for generating boundary data 120 based on the point cloud data 116, which describes the outer dimensions of the physical object 106 as a boundary (e.g., a three-dimensional polygon, a tube, etc.).

[0049] The boundary data 120 may support various functions, such as representing the dimensions of the physical object 106 as augmented reality digital content as part of a real-time camera feed output by a computing device. In the illustrated example, the boundary data 120 is received as input by a delivery module 122 to generate the delivery data 108. For example, the delivery module 122 may transmit the boundary data 120 to a service provider system 126 via a network 124. The service provider system 126 includes a service manager module 128 configured to generate the delivery data 108. For example, the service manager module 128 may employ a packaging material service 130 configured to determine packaging materials (e.g., box, tube, envelope, and filler size) to be used to package the physical object 106. In another example, the delivery service 132 is configured to estimate the delivery cost of the physical object 106. In another example, the digital commerce service 134 uses the boundary data 120 as part of the generation of digital content (e.g., providing a web page for selling the physical object 106 by the user 104).

[0050] For example, the physical object boundary detection module 118 and the delivery module 122 may be incorporated as part of an application corresponding to the service provider system 126. In this example, the service provider system 126 is configured to provide physical objects for sale by publishing digital content (e.g., web pages, screens in the user interface of the application, etc.).

[0051] Thus, a user 104 may interact with an application to generate digital content (e.g., a web page, screen of the application) that provides a physical object 106 for sale. As part of this, a delivery module 122 is configured to generate delivery data 108 within the context of the application, and is therefore non-modal. This may include automatic ordering of packaging materials, listing of estimated delivery costs, etc., which may be included as part of the digital content. In this manner, generation of boundary data 120 may improve the efficiency of user interaction with computing device 102, and may do so in real time.

[0052] Typically, the functions, features, and concepts described in the examples above and below can be adopted in the context of the example processes described in this section. In addition, the functions, features, and concepts described in the different figures and examples in this document can be interchangeable with each other and are not limited to being implemented in the context of a specific figure or process. In addition, the frames associated with the different representative processes and corresponding figures in this article can be applied together and / or combined in different ways. Therefore, the various functions, features, and concepts described in the different example environments, devices, parts, figures, and processes in this article can be used in any suitable combination and are not limited to the specific combination represented by the examples listed in this specification.

[0053] Physical object boundary detection

[0054] Figure 2 Depicting system 200 in an example implementation, showing in more detail Figure 1 The AR module 110 and the physical object boundary detection module 118 operate. Fig.10 Depicted is a process 1000 of generating a three-dimensional boundary based on point cloud data obtained from an AR module in an example implementation. Although a three-dimensional bounding box is described in this example, other shapes are also contemplated, such as tubes, polygons, triangular prisms, envelopes, etc.

[0055] The following discussion describes techniques that can be implemented using the described systems and devices. Aspects of the process can be implemented in hardware, firmware, software, or a combination thereof. The process is shown as a series of blocks detailing operations performed by one or more devices and is not necessarily limited to the order shown in which the operations are performed by the various blocks. In portions of the following discussion, references may be made interchangeably to Figure 2 and Fig.10 .

[0056] In this example, the augmented reality functionality available on the computing device 102 (e.g., a mobile phone) is utilized to obtain the measurements of the physical object 106 as a three-dimensional boundary. In this case, the three-dimensional boundary does not reconstruct the geometry of the physical object, but rather creates a minimum bounding box to facilitate packaging selection and delivery cost estimation.

[0057] To this end, point cloud data 116 defining depths at various points within a physical environment including physical objects 106 is first obtained (block 1002). Thus, the point cloud data 116 is configured to represent the three-dimensional physical environment through the AR module 110. This can be performed in various ways, such as by using radar technology, visual inertial odometry, time-of-flight cameras, structured grating technology, etc.

[0058] Figure 3 An example 300 of point cloud data 116 is shown. Figure 3 , a physical object 116 (lamp) is included on a table in a physical environment. In some instances, point cloud data 116 may include a large number of points (e.g., 250,000), which may limit the operation of the computing device. Therefore, in such instances, point cloud data 116 may be downsampled (e.g., sampling one point out of twenty points per frame, thereby processing 15,000 points) to improve the efficiency of operation of computing device 102, such as to support real-time output.

[0059] As previously described, the computing device 102 may utilize the AR module 110 to generate the point cloud data 116. The AR module 110 includes a three-dimensional (3D) sensing device 112 and a point cloud generation module 114, which are configured to generate the point cloud data 116 describing the depth at various points within the physical environment. For example, in a right-handed habit, the point cloud data 116 may describe points where, as part of a coordinate system, the y-axis is vertical and the x-axis is horizontal and the z-axis defines the depth.

[0060] In one example, the 3D sensing device 112 and the point cloud generation module 114 employ a technique known as visual inertial odometry, wherein the 3D sensing device 112 includes motion detection sensors (e.g., gyroscopes, accelerometers) and cameras to capture digital images of the physical environment. The point cloud generation module 114 is configured to perform computer vision analysis of the digital images to identify features in the digital images and track differences in the positions of these features.

[0061] The point cloud generation module 114 compares the motion sensing data from the motion detection sensor with the differences in the tracked feature positions to generate point cloud data 116, which models the depth of points in the physical environment as, for example, a depth map. In this way, the AR module 110 can track the position of the computing device 102 as it moves to build an understanding of the physical environment based on features recognized from digital images. This functionality can also support basic plane detection (e.g., a flat surface such as a floor or table). Therefore, in this example, the point cloud data 116 can be generated by a variety of mobile computing devices such as smart phones and tablets without dedicated hardware.

[0062] In another example, dedicated hardware is utilized to transmit radio waves (e.g., Wi-Fi waves), the return of which is sensed by the 3D sensing device 112 (e.g., using radar technology). This includes using motion tracking as described above (e.g., using accelerometers, barometers, GPS, and / or gyroscopes) so that the point cloud generation module 114 can gain an understanding of the depth of points within the physical environment to generate point cloud data 116. Other examples are also contemplated, such as time-of-flight cameras, structured grating array devices, etc.

[0063] The base point filtering module 202 then filters the point cloud data 116 by removing points corresponding to the base plane (block 1004). For example, the user input 204 may be received via a user interface that displays a digital image corresponding to the point cloud data 116, e.g., as a real-time camera feed from a digital camera of the computing device 102. The user input 204 specifies the location of the base plane in the digital image and, therefore, may be associated with the base plane in the point cloud data 116 by the base point filtering module 202.

[0064] Points corresponding to or located below the base plane are removed from the point cloud data 116 to form filtered point cloud data 206. This may be performed such that any points in the point cloud data 116 that have a negative dot product with the base plane within an error tolerance "e" as indicated by the user input 204 are removed. This serves to reduce the point cloud search space to improve the efficiency and operation of the computing device 102. Other instances are also contemplated in which the functionality of the base point filtering module 202 is included as part of the AR module 110 itself. Figure 4 An example 400 of the filtered point cloud data 206 is shown in FIG.

[0065] Then, user input 208 specifying a location of a physical object in the user interface is received (block 1006). Continuing with the previous example, computing device 102 may output in the user interface a real-time camera feed of a captured digital image of the physical environment corresponding to point cloud data 116. Then, as Figure 5 As shown, a user may “tap” a location representing a physical object 106 in the user interface.

[0066] Then, a subset 212 of points within the filtered point cloud data 206 corresponding to the physical object 106 is found by the proximity search module 210, for example, via a nearest neighbor search (block 1008). For example, the proximity search module 210 may iteratively find each point in the filtered point cloud data 206 that is close to the initial selection point "p0" of the user input 208. This is accomplished using a KD tree by building a spatial index of the filtered point cloud data 206, and then querying these points within a proximity "d" of the initial object selection point "p0", and then adding these points to the point set "ps". The proximity "d" of the newly added points within the set "ps" is further searched in the next iteration.

[0067] As more points within proximity "d" are discovered, other points close to those newly discovered points are searched until no such points are found and added to the set "ps", thereby forming a subset of points 212. As part of this, the point farthest from the base plane is found in the subset of points 212, i.e., the highest point. Figure 5 An example 500 of a subset of points 212 is shown.

[0068] Then, the subset of points 212 is received as input by the two-dimensional projection module 214, which is configured to generate a two-dimensional projection 216 (block 1010). For example, the two-dimensional projection module 214 can reset the Y coordinate to 0 for each point in the subset of points 212, example 600 of which is shown in FIG. Figure 6 This forms a two-dimensional projection 216 of the subset 212 as points arranged on the base plane.

[0069] Then, a two-dimensional boundary module 218 is used to generate a two-dimensional boundary 220 based on the two-dimensional projection 216. The two-dimensional boundary 220 may have a shape based on the projection. To this end, Figure 7 As shown in example 700 of FIG. 1 , a two-dimensional convex hull 702 (i.e., a boundary shape) is calculated, which surrounds the points in the two-dimensional projection 216 projected on the base plane by the two-dimensional boundary module 218. Then, the two-dimensional boundary module 218 generates the minimum two-dimensional boundary 220 based on the two-dimensional convex hull using a rotating caliper technique.

[0070] Then, in this example, the two-dimensional boundary 220 is extruded into a three-dimensional boundary 224 as the boundary data 120 by the three-dimensional extrusion module 222 (block 1012). Figure 8 As shown in example 800 of , the two-dimensional boundary is extended into a rectangular cuboid that shares a face with the two-dimensional boundary. The height of the three-dimensional boundary 224 is based on the point farthest from the base plane, i.e., the highest point, in the subset of points 212 that are found as part of the filtering. Therefore, the three-dimensional boundary 224 has a shape based on the two-dimensional boundary.

[0071] The dimensions of the three-dimensional boundary are displayed in the user interface (block 1014). Fig. 9 As shown in example 900 of FIG. 1 , a three-dimensional boundary 224 is displayed as augmented reality digital content within a user interface 902 that includes a real-time camera feed. The three-dimensional boundary 224 in this example surrounds the representation of the physical object in the digital image (the surrounding display).

[0072] The three-dimensional boundary 224 can be used to support various functions. For example, Figure 1 , the boundary data 120 including the three-dimensional boundary 224 can be transmitted to the service provider system 126 via the network 124. The service manager module 128 of the service provider system 126 then generates the delivery data 108 based on the bounding box. For example, the service manager module 128 can employ a packaging material service 140 that is configured to determine the packaging material (e.g., box, shape, envelope, and filler size) used to package the physical object 106. For example, the packaging material service 140 can select the size and shape of the packaging material based on the size of the three-dimensional boundary 224 and the filler (e.g., bubble wrap) to be used as the filler within the box volume. In this way, the correct size of the packaging material and the amount of filler material can be accurately and automatically determined by the computing device 102 that is easy for the user 104 to use.

[0073] In another example, the delivery service 142 is configured to estimate the delivery cost of the physical object 106. For example, the delivery service 142 may make the estimate based on the dimensions of the three-dimensional boundary. In another example, the digital commerce service 144 employs the boundary data 120 as part of the generation of digital content (e.g., providing a web page for selling the physical object 106 by the user 104). Continuing with the foregoing example, a web page may be generated to provide the physical object 106 for sale. The three-dimensional boundary 224 is generated and used to generate the delivery data 108 as part of the generation of an application executed by the computing device 102, for example, as part of capturing digital images from multiple perspectives, which will also be included as part of the digital content. In this way, the techniques may effectively utilize the capture of these digital images for a dual purpose, thereby improving the efficiency of the operation of the computing device 102.

[0074] Fig.11 and Fig.12 Other examples 1100, 1200 of physical object boundary detection techniques and systems are depicted. In this example, user input is received at a computing device 102 via a user interface to initiate generation of digital content (block 1102) that is provided by a service provider via a network for sale of a physical object 106. For example, an application may be executed by a computing device 102 associated with a service provider system 126. The service provider system 126 includes a digital commerce service that can be used to provide products for sale. To this end, digital content (e.g., a web page for output in a mobile application, a user interface screen, etc.) is generated to indicate the availability of the physical object 106.

[0075] As part of this example, user input is received at computing device 102 via a user interface of an application to initiate estimating a shipping cost for a physical object as part of the generation of digital content (block 1104). Accordingly, point cloud data 116 is generated from AR module 110 as previously described.

[0076] As previously described, the computing device 102 may utilize the AR module 110 to generate the point cloud data 116. The AR module 110 includes a three-dimensional (3D) sensing device 112 and a point cloud generation module 114, which are configured to generate the point cloud data 116 describing the depth at various points within the physical environment. For example, in a right-handed habit, the point cloud data 116 may describe points where, as part of a coordinate system, the y-axis is vertical and the x-axis is horizontal and the z-axis defines the depth.

[0077] In one example, the 3D sensing device 112 and the point cloud generation module 114 employ a technique known as visual inertial odometry, in which the 3D sensing device 112 includes motion detection sensors (e.g., gyroscopes, accelerometers) and cameras to capture digital images of the physical environment. The point cloud generation module 114 is configured to perform computer vision analysis of the digital images to identify features in the digital images and track differences in the positions of these features.

[0078] The point cloud generation module 114 compares the motion sensing data from the motion detection sensor with the difference of the tracked feature positions to generate point cloud data 116, which models the depth of points in the physical environment as, for example, a depth map. In this way, the AR module 110 can track the position of the computing device 102 as it moves to build an understanding of the physical environment based on features recognized from digital images. This functionality can also support basic plane detection, for example, a flat surface such as a floor or a table. Therefore, in this example, the point cloud data 116 can be generated by a variety of mobile computing devices such as smart phones and tablets without dedicated hardware.

[0079] In another example, dedicated hardware is utilized to transmit radio waves (e.g., Wi-Fi waves), the return of which is sensed by the 3D sensing device 112 (e.g., using radar technology). This includes using motion tracking as described above (e.g., using accelerometers, barometers, GPS, and / or gyroscopes) so that the point cloud generation module 114 can gain an understanding of the depth of points within the physical environment to generate point cloud data 116. Other examples are also contemplated, such as time-of-flight cameras, structured grating array devices, etc.

[0080] Fig.13 An example implementation of generating point cloud data from multiple perspectives and generating a two-dimensional boundary for each of these perspectives is depicted. In this example, point cloud data 116 is captured from multiple different perspectives 1302, 1304, 1306 (e.g., top, side, front, back, etc.) relative to a physical object 106. The point cloud data defines depth at various points within the physical environment (block 1106).

[0081] For example, the AR module 110 may include functions for hit test points to indicate the distance from the camera to the nearest object along a vector in the camera frustum. Using these functions, the AR module 110 builds a 1 cm grid of points and hit tests multiple points per frame to build a depth map of the physical environment in real time. This produces point cloud data 116 of points within the object, where points within a predetermined distance from the base plane are removed.

[0082] The physical object boundary detection module 118 is configured to determine when a sufficient number of points have been collected from a sufficient number of viewing angles, and in response, stop the AR module 110 from generating the point cloud data 116. This serves to conserve computing resources of the computing device 102. This also supports user feedback indicating the point in time at which a three-dimensional bounding box can be generated.

[0083] The physical object boundary detection module 118 then calculates a plurality of two-dimensional boundaries 1308, 1310, 1312, wherein each two-dimensional boundary corresponds to a respective view in the plurality of views 1302, 1304, 1306 (block 1110). To this end, the physical object boundary detection module 118 determines the best fit by rotating through a defined arc 1314 (e.g., ninety degrees) in increments (e.g., five degrees) and generates a two-dimensional boundary of the point cloud at each angle. In an implementation, the two-dimensional boundary has a shape corresponding to the minimum area of ​​the points at that view.

[0084] The physical object boundary detection module 118 then selects a two-dimensional boundary from the plurality of two-dimensional boundaries for estimating a bounding box (block 1202). For example, the physical object boundary detection module 118 may select the two-dimensional boundary with the smallest (ie, smallest amount) area.

[0085] The physical object boundary detection module 118 then uses the two-dimensional boundary to generate a three-dimensional boundary (block 1204) whose size (e.g., and shape) is based on the selected boundary and the height of the physical object obtained from the point cloud data. For example, the points within the bounding box can be compared along the Y axis to determine the point 1316 or point group in the point cloud data that has the largest distance to the base plane 1318. This defines the Y axis, which together with the size of the selected two-dimensional boundary with the smallest size defines the three-dimensional boundary.

[0086] The three-dimensional boundary is then used by the delivery module 122 to obtain the delivery data 108 (box 1206). For example, the dimensions of the three-dimensional boundary can be compared to the dimensions of the packaging material to determine a box with a minimum size and shape and the amount of packaging (e.g., "filler") used to support the physical object 106 within the box. This can also be used to estimate the delivery cost. The delivery data 108 can be displayed in the user interface as augmented reality digital content within the real-time camera feed along with the three-dimensional boundary (box 1208). The delivery data 108 can also be included as part of the digital content automatically and without user intervention, for example, in response to a user input "okaying" to include the data as part of the offer to sell the physical object 106. In this way, the application supports efficient non-modal interaction to include the delivery data as part of the digital content, and thus improves user and operational efficiency.

[0087] Example Systems and Apparatus

[0088] Fig.14 An example system is shown generally at 1400 that includes an example computing device 1402, which represents one or more computing systems and / or devices that may implement the various techniques described herein. This is illustrated by the inclusion of a physical object boundary detection module 118. For example, the computing device 1402 may be a server of a service provider, a device associated with a client (e.g., a client device), a system on a chip, and / or any other suitable computing device or computing system.

[0089] The example computing device 1402 shown in the figure includes a processing system 1404, one or more computer-readable media 1406, and one or more I / O interfaces 1408 that are communicatively coupled to each other. Although not shown, the computing device 1402 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any of the following different bus structures or a combination of different bus structures: for example, a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus that utilizes any of a variety of bus architectures. Various other examples, such as control lines and data lines, are also contemplated.

[0090] Processing system 1404 represents the function of performing one or more operations using hardware. Therefore, processing system 1404 is shown as including hardware element 1410, which can be configured as a processor, a functional block, etc. This can include an implementation in the form of hardware as a dedicated integrated circuit or other logic device formed using one or more semiconductors. Hardware element 1410 is not limited by the materials from which they are formed or the processing mechanisms used herein. For example, a processor may include semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions.

[0091] Computer-readable storage media 1406 is shown as including memory / storage 1412. Memory / storage 1412 represents the capacity of memory / storage associated with one or more computer-readable media. Memory / storage component 1412 may include volatile media (e.g., random access memory (RAM)) and / or non-volatile media (e.g., read-only memory (ROM), flash memory, optical disk, magnetic disk, etc.). Memory / storage component 1412 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Computer-readable media 1406 may be configured in various other ways as further described below.

[0092] Input / output interface 1408 represents functionality that allows a user to input commands and information to computing device 1402 and also allows information to be presented to the user and / or other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone, a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., employing visible or invisible wavelengths (e.g., infrared frequencies) to recognize motion as gestures that do not involve touch), etc. Examples of output devices include a display device (e.g., a monitor or projector), a speaker, a printer, a network card, a tactile response device, etc. Thus, computing device 1402 may be configured in various ways as further described below to support user interaction.

[0093] Various techniques may be described herein in the general context of software, hardware elements, or program modules. Typically, such modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module," "function," and "component" as used herein generally represent software, firmware, hardware, or a combination thereof. Features of the techniques described herein are platform-independent, meaning that the techniques may be implemented on a variety of commercial computing platforms with a variety of processors.

[0094] An implementation of the described modules and techniques may be stored on or transmitted over some form of computer-readable media. Computer-readable media may include various media accessible by computing device 1402. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media.”

[0095] "Computer-readable storage media" may refer to media and / or devices that enable persistent and / or non-transitory storage of information as compared to signal transmission alone, carrier waves, or signals themselves. Therefore, computer-readable storage media refers to non-signal bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information such as computer-readable instructions, data structures, program modules, logical elements / circuits, or other data. Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage, hard disk, cassette, magnetic tape, disk storage device or other magnetic storage device, or other storage device, tangible media, or an article of manufacture suitable for storing the desired information and accessible by a computer.

[0096] "Computer readable signal media" may refer to signal bearing media that is configured to send instructions to the hardware of the computing device 1402, for example, via a network. Signal media may typically embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also include any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in a manner that encodes the information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0097] As previously described, hardware elements 1410 and computer-readable media 1406 represent modules, programmable device logic, and / or fixed device logic implemented in hardware form, which can be adopted in some embodiments to implement at least some aspects of the technology described herein, such as executing one or more instructions. Hardware can include components of integrated circuits or systems on chips, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or other hardware. In this context, hardware can be operated as a logical processing device that performs program tasks defined by instructions and / or embodied by hardware, as well as hardware for storing instructions for execution (e.g., the computer-readable storage medium described previously).

[0098] The aforementioned combination can also be adopted to implement the various technologies described herein. Therefore, software, hardware or executable modules can be implemented as one or more instructions and / or logic implemented on some form of computer-readable storage medium and / or by one or more hardware elements 1410. Computing device 1402 can be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Therefore, the implementation of the module executable as software by computing device 1402 can be implemented at least partially in hardware, for example, by using the computer-readable storage medium and / or hardware elements 1410 of processing system 1404. Instructions and / or functions can be executable / operable by one or more manufacturing products (e.g., one or more computing devices 1402 and / or processing system 1404) to implement the technology, modules and examples described herein.

[0099] Various configurations of computing device 1402 can support the technology described herein, and are not limited to the specific examples of the technology described herein. As described below, the functionality can also be implemented in whole or in part by using a distributed system, such as on a "cloud" 1414 via a platform 1416.

[0100] The cloud 1414 includes and / or represents a platform 1416 for resources 1418. The platform 1416 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of the cloud 1414. The resources 1418 may include applications and / or data that may be utilized when performing computer processing on a server remote from the computing device 1402. The resources 1418 may also include services provided over the Internet and / or over a subscriber network such as a cellular or Wi-Fi network.

[0101] The platform 1416 can abstract resources and functionality to connect the computing device 1402 to other computing devices. The platform 1416 can also be used to abstract the scale of resources to provide a corresponding level of scale to the demand encountered by the resources 1418 implemented via the platform 1416. Therefore, in an interconnected device embodiment, the implementation of the functionality described herein can be distributed throughout the system 1400. For example, the functionality can be implemented in part on the computing device 1402 and via the platform 1416 that abstracts the functionality of the cloud 1414.

[0102] in conclusion

[0103] Although the invention has been described in language specific to structural features and / or methodological acts, it should be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed invention.

[0104] The solution of the present invention also includes:

[0105] (1) A method implemented by at least one computing device, comprising:

[0106] obtaining, by the at least one computing device, point cloud data defining depths at various points within a physical environment including a physical object;

[0107] filtering, by the at least one computing device, the point cloud data by removing points from the point cloud data that correspond to a base plane;

[0108] receiving, by the at least one computing device via a user interface, user input specifying a location of the physical object;

[0109] searching, by the at least one computing device, a subset of points within the filtered point cloud data that correspond to the physical object, the searching being performed using a proximity search based on a location of the user input;

[0110] generating, by the at least one computing device, a two-dimensional boundary based on the found subset;

[0111] extruding, by the at least one computing device, the two-dimensional boundary into a three-dimensional boundary; and

[0112] The three-dimensional boundary is displayed in the user interface by the at least one computing device.

[0113] (2) The method of (1), wherein the user interface comprises a display of a digital image of the physical environment as part of a real-time camera feed, the digital image comprising the physical object and corresponding to the point cloud data.

[0114] (3) The method of (2), wherein the filtering is based on another user input received via the user interface indicating a location of the base plane in the point cloud data.

[0115] (4) The method according to (1), wherein the search is based on a nearest neighbor search.

[0116] (5) The method according to (1), wherein the generation of the two-dimensional boundary comprises:

[0117] Projecting the subset of searched points onto the base plane;

[0118] Compute the convex hull of the points surrounding the projection; and

[0119] The two-dimensional boundary is generated from the convex hull using a rotating caliper technique.

[0120] (6) The method according to (1), wherein:

[0121] The filtering includes determining the height of the point in the filtered point cloud data that is farthest from the base plane; and

[0122] The extrusion is based on the height such that the three-dimensional boundary has the height.

[0123] (7) The method according to (1), wherein the displaying includes displaying the size of the physical object as augmented reality digital content in the user interface.

[0124] (8) The method according to (1), wherein the obtained point cloud data is a downsampled version of the data generated by the point cloud generation module.

[0125] (9) The method of (1), further comprising obtaining data describing an estimated delivery cost based on the size of the three-dimensional boundary, and wherein the display includes the estimated delivery cost.

[0126] (10) The method of (1), wherein the user interface comprises a digital image as a real-time camera feed, and the display comprises displaying the three-dimensional boundary as augmented reality digital content arranged together with the physical object.

[0127] (11) A method comprising:

[0128] displaying a real-time camera feed of a digital image in a user interface by at least one computing device;

[0129] generating, by the at least one computing device, point cloud data generated from a plurality of perspectives about a physical object, the point cloud data defining depths at respective points within the physical environment;

[0130] calculating, by at least one computing device, a plurality of two-dimensional boundaries, wherein each of the boundaries corresponds to a respective perspective of the plurality of perspectives;

[0131] selecting, by the at least one computing device, a two-dimensional boundary from among the plurality of two-dimensional boundaries;

[0132] generating, by the at least one computing device, a three-dimensional boundary having a size based on the selected two-dimensional boundary and a height of the physical object from the point cloud data; and

[0133] The three-dimensional boundary is displayed by the at least one computing device as augmented reality digital content within a real-time camera feed.

[0134] (12) The method of (11), further comprising obtaining, by the at least one computing device, delivery data based at least in part on the three-dimensional boundary.

[0135] (13) The method of (12), wherein the delivery data is an estimated cost of shipping the physical object.

[0136] (14) The method according to (12), wherein the shipping data is a packaging size or shape used when packaging the physical object.

[0137] (15) The method according to (11), further comprising:

[0138] receiving, by the at least one computing device via the user interface, user input to initiate generation of digital content as part of the physical object offered for sale by a service provider via a network;

[0139] receiving, by the at least one computing device via the user interface, another user input to initiate generation of the shipping data for the physical object as part of generation of the digital content, and wherein displaying the real-time camera feed is performed in response to the another user input; and

[0140] The digital content is generated by the at least one computing device to include the delivery data.

[0141] (16) A computing device comprising:

[0142] Display device;

[0143] an augmented reality module including a three-dimensional sensing device and a point cloud generation module implemented at least in part in hardware to generate point cloud data defining depths at various points of a physical object within a physical environment; and

[0144] A physical object boundary detection module, the physical object boundary detection module is at least partially implemented in hardware to generate a three-dimensional boundary based on the point cloud data and generate delivery data based on the three-dimensional boundary, the three-dimensional boundary and the delivery data are output as augmented reality digital content in a real-time camera feed of the physical object in a user interface of the display device.

[0145] (17) A computing device according to claim (16), wherein the shipping data is an estimated cost of shipping the physical object.

[0146] (18) A computing device according to (16), wherein the shipping data is a packaging size or shape used when packaging the physical object.

[0147] (19) The computing device according to (16), wherein the physical object boundary detection module is configured to generate the three-dimensional boundary by performing the following operations:

[0148] calculating a plurality of two-dimensional boundaries, wherein each of the two-dimensional boundaries corresponds to a corresponding perspective among a plurality of perspectives captured by the point cloud data;

[0149] selecting a two-dimensional boundary from the plurality of two-dimensional boundaries; and

[0150] A three-dimensional boundary is generated, the dimensions of the three-dimensional boundary being based on the selected two-dimensional boundary and a height of the physical object from the point cloud data.

[0151] (20) The computing device according to (16), wherein the physical object boundary detection module is configured to generate the three-dimensional boundary by:

[0152] filtering the point cloud data by removing points corresponding to a base plane from the point cloud data;

[0153] receiving, via the user interface, user input specifying a location of the physical object;

[0154] finding a subset of points within the filtered point cloud data that correspond to the physical object, the finding being performed using a proximity search based on the location of the user input;

[0155] generating a two-dimensional boundary based on the found subset; and

[0156] The two-dimensional boundary is extruded into the three-dimensional boundary.

Claims

1. A method implemented by at least one computing device, the method comprising: receiving, by the at least one computing device, motion sensing data and a digital image corresponding to a physical environment including a physical object; generating, by the at least one computing device, point cloud data based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating, by the at least one computing device, a three-dimensional boundary of the physical object based on the point cloud data generated based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating, by the at least one computing device, digital content based on the three-dimensional boundary, the digital content indicating availability of the physical object; as well as The digital content is displayed in a user interface by the at least one computing device.

2. The method according to claim 1, wherein: The user interface includes a display of the digital image of the physical environment as part of a real-time camera feed, the display of the digital image including the physical object and corresponding to the motion sensing data and the digital image corresponding to the physical environment including the physical object.

3. The method according to claim 1, wherein: The digital content is a web page.

4. The method according to claim 1, wherein: The digital content includes one or more of the digital images.

5. The method according to claim 1, wherein: The digital content is a user interface screen for output in a mobile phone application.

6. The method according to claim 1, further comprising: As part of the generation of the digital content, shipping data for the physical object is generated by the at least one computing device, and wherein the digital content includes the shipping data.

7. The method according to claim 1, wherein: The shipping data includes a packaging quantity to be used when packaging the physical object.

8. The method according to claim 1, wherein: The shipping data is an estimated cost to ship the physical object.

9. The method according to claim 1, wherein: The shipping data is the packaging size or shape to be used when packaging the physical object.

10. A system comprising: at least one hardware processor; as well as A non-transitory computer-readable storage medium storing executable instructions that, when executed, cause the at least one hardware processor to perform operations comprising: receiving motion sensing data and a digital image corresponding to a physical environment including a physical object; generating point cloud data based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating a three-dimensional boundary of the physical object based on the point cloud data generated based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating digital content based on the three-dimensional boundary, the digital content indicating availability of the physical object; and The generated digital content is displayed in a user interface.

11. The system according to claim 10, wherein: The digital content includes one or more of the digital images.

12. The system of claim 10, wherein the operations further comprise: As part of the generation of the digital content, shipping data for the physical object is generated, and wherein the digital content includes the shipping data.

13. A non-transitory computer-readable storage medium storing a set of instructions, which, when executed by at least one processor, causes the at least one processor to perform operations comprising: receiving motion sensing data and a digital image corresponding to a physical environment including a physical object; generating point cloud data based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating a three-dimensional boundary of the physical object based on the point cloud data generated based on the motion sensing data and the digital image corresponding to the physical environment including the physical object; generating digital content based on the three-dimensional boundary, the digital content indicating availability of the physical object; and The generated digital content is displayed in a user interface.

14. The non-transitory computer readable storage medium of claim 13, the operations further comprising: As part of the generation of the digital content, shipping data for the physical object is generated based on the three-dimensional boundary, and wherein the digital content includes the shipping data.

15. The non-transitory computer readable storage medium of claim 13, wherein the generating of the three-dimensional boundary further comprises: calculating a plurality of two-dimensional boundaries, wherein each two-dimensional boundary of the plurality of two-dimensional boundaries corresponds to a respective perspective of a plurality of perspectives captured by the point cloud data; selecting a two-dimensional boundary from the plurality of two-dimensional boundaries; and A three-dimensional boundary is generated having dimensions based on the selected two-dimensional boundary and a height of the physical object from the point cloud data.

Images