Object-centric scanning

Abstract

The invention relates to object-centric scanning. Various implementations disclosed herein include devices, systems, and methods of generating a three-dimensional (3D) model of an object based on images and a tracked position of the device during acquisition of the images. For example, an exemplary process may include acquiring sensor data during movement of the device in a physical environment including an object, the sensor data including an image of the physical environment acquired via a camera on the device; identifying the object in at least some of the images; tracking a position of the device during acquisition of the images based on identifying the object in at least some of the images, the position identifying a location of the device relative to a coordinate system defined based on a position and orientation of the object; and generating a 3D model of the object based on the image and the position of the device during acquisition of the image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of the invention patent application with Chinese national application number 202110226395.7, application date March 1, 2021, and invention name “Object-centered scanning”.

[0003] This application claims the benefit of U.S. Provisional Application Serial No. 62 / 986076, filed on March 6, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates generally to generating three-dimensional geometric representations of physical environments, and in particular to systems, methods, and devices for generating geometric representations based on depth information detected in the physical environment. Background Art

[0005] The physical environment and objects therein have been modeled (e.g., reconstructed) by generating three-dimensional (3D) meshes, utilizing 3D point clouds, and by other means. These reconstructed meshes represent 3D surface points and other surface features of the floor, walls, and other objects of the physical environment. Such reconstructions can be generated based on images and depth measurements of the physical environment (e.g., using an RGB camera and a depth sensor).

[0006] Existing techniques for using mobile devices to generate 3D models based on images of the physical environment and depth information detected in the physical environment may be inaccurate and inefficient, for example, based on capturing photos or videos or other sensor data while the user walks around the room. In addition, existing techniques may not provide sufficiently accurate and efficient object detection in a real-time environment. Summary of the invention

[0007] Various embodiments disclosed herein include devices, systems, and methods for generating a three-dimensional (3D) model of an object based on images and the position of the device during the acquisition of these images. The 3D model is generated based on images of the physical environment, depth information detected in the physical environment, and other information for tracking the specific position and orientation (e.g., camera pose) of the device / depth camera. It may be desirable to utilize, for example, an object-centered camera pose defined according to an object-based coordinate system. In other words, the constrained pose is associated with a target object (e.g., a single object of interest), and the position is relative to the target object. Doing so can provide a more useful, more realistic, or physically meaningful model of the object.

[0008] Some specific implementations of the present disclosure relate to an exemplary method for generating a 3D model of an object based on an image and the position of a device during the image acquisition. The exemplary method initially involves: collecting sensor data during the movement of a device in a physical environment including an object, the sensor data including an image of the physical environment acquired via a camera on the device. For example, a user moves a device (e.g., a mobile device) around an object (e.g., a shoe above a table) in a physical environment to acquire images of the object from different sides. In some specific implementations, the sensor data may include depth data and motion sensor data. In some specific implementations, during the movement of the device, a user interface may display the acquired environment including the object and provide user interface elements. For example, a user interface element (e.g., an extended reality image, such as a 3D arrow overlaid on a real-time video stream) may display additional angles and / or perspectives of the acquired object to the user. In some specific implementations, the user interface may display a preliminary 3D model of the object (e.g., a 3D mesh, a 3D bounding box, etc.). For example, when a user is capturing real-time video and when the real-time video is streaming on a main viewing screen, a picture-in-picture display of another window may display a 3D model reconstruction to the user in a real-time preview screen.

[0009] The exemplary method also involves identifying objects in at least some of the images. For example, identifying an object may involve identifying an object using a preliminary object model based on information from multiple images and depth information. Using the preliminary object model to identify an object may involve masking by creating a 3D bounding box corresponding to the object. In addition or alternatively, a two-dimensional (2D) mask corresponding to the object may also be created. During image acquisition, the 3D bounding box is adjusted to better fit the object so as to define the object of interest and separate it from the background. In some specific implementations, object identification may be based on cropping the preliminary object model and the 3D key points associated with the preliminary object model / adjusting the size of the preliminary object model and the 3D key points associated with the preliminary object model. For example, a 3D bounding box constraint is used to remove background pixels outside the 3D bounding box, and / or a 2D mask constraint associated with the object is used to remove background pixels outside the 2D mask. Masking (e.g., cropping / sizing preliminary object models) can be performed during scanning (e.g., during image acquisition) to detect objects in a first stage, and then when performing image scanning, objects are isolated by removing background pixels using those masks (e.g., 3D bounding box constraints, 2D masks, etc.). In some implementations, masking can use a covering algorithm and bypass objects and calculate dynamic positions during image acquisition.

[0010] In some implementations, object identification can involve densification of a sparse 3D depth cloud. For example, a dense 3D depth cloud is created from a sparse data set using a densification algorithm. Additionally or alternatively, in some implementations, object identification can involve keypoint interpolation and / or exclusion of keypoints near depth edges. For example, using a keypoint object algorithm, keypoints of an object are identified (e.g., semantically labeled via a neural network for an RGB image), and additional keypoints (e.g., additional pixels associated with the object) can be interpolated (e.g., added) to a preliminary object model or excluded (e.g., removed) from the preliminary object model.

[0011] The exemplary method also involves tracking the position of the device during the acquisition of images based on identification of objects in at least some of the images, the positions identifying the location and / or orientation of the device relative to a coordinate system defined based on the position and orientation of the objects (e.g., pose information). In an exemplary implementation, tracking the position of the device during the acquisition of images based on identification of the objects tracks the objects by updating an object-centric pose of the device (e.g., the position of the camera relative to the object in object-based coordinates as each image in the images is acquired). In some implementations, tracking may use a pose graph defined in object-based coordinates (e.g., a history of relative RGB-D camera movements).

[0012] In some implementations, tracking may involve repositioning when an object leaves view or when a previously unseen portion of an object is acquired. For example, when an object leaves the camera view and flips, tracking may require repositioning and further identifying the object that is the same as the object of interest, and recalibrating the object-centered coordinates to determine that the flipped object is the same as the object of interest. In some implementations, the method may iteratively update both the camera's object-centered pose and the preliminary object model.

[0013] In some implementations, the pose can be verified based on cropping / sizing the preliminary object model and its 3D keypoints (e.g., using 3D bounding box constraints to remove background image information and / or using 2D masks to remove background image pixels (e.g., similar to the process described above for masks during the object identification stage)). In some implementations, pose verification can also involve densification of sparse depth clouds, keypoint interpolation, and / or exclusion of keypoints close to depth edges.

[0014] The exemplary method also involves generating a 3D model of an object based on an image and the position of the device during the acquisition of the image. The 3D model may be based on a refined / cropped image, associated depth information, and a relative position of a camera associated with such image and depth information. The 3D model may be a 3D mesh representation or a 3D point cloud. In some implementations, the 3D model data may be a 3D representation of a surface in a 3D environment using a 3D point cloud with associated semantic tags. In some implementations, the 3D model data is a 3D reconstructed mesh using a meshing algorithm based on depth information detected in the generated environment, which is integrated (e.g., fused) to reconstruct the physical environment. A meshing algorithm (e.g., a double marching cube meshing algorithm, a Poisson meshing algorithm, a tetrahedron meshing algorithm, etc.) may be used to generate a mesh representing a room and / or objects in the room (e.g., furniture, a dog statue on a table, a chair, etc.).

[0015] In some implementations, collecting sensor data during movement of the device includes collecting images from different perspectives of the object as the device moves around the object.

[0016] In some implementations, the user interface may display an extended reality image with a plus sign or other virtual indicator (e.g., a pointer or arrow) positioned in 3D space to guide the user to position the device at the appropriate position and orientation to capture the appropriate image of the object. Since the tracking is object-centric, if the object moves, the indicator will also move relative to the position and orientation of the object.

[0017] In some implementations, identifying an object includes generating a preliminary object model based on depth information from an image of a physical environment, wherein the preliminary object model includes 3D key points corresponding to the object. In some implementations, the preliminary object model is a 3D bounding box, and generating the 3D bounding box includes obtaining a 3D representation of the physical environment generated based on the depth data (e.g., a 3D semantic point cloud) and generating an exemplary method of a 3D bounding box corresponding to the object in the physical environment based on the 3D representation. In some implementations, generating the 3D bounding box includes an exemplary method having the following operations: obtaining a 3D representation of the physical environment generated based on the depth data, determining a ground plane corresponding to the object in the physical environment based on the 3D representation, and generating a 3D bounding box corresponding to the object in the physical environment based on the ground plane and the 3D representation.

[0018] In some implementations, identifying the object further comprises adjusting the preliminary object model based on the 3D key points corresponding to the object (e.g., cropping the preliminary object model and / or adjusting the size of the preliminary object model). In some implementations, adjusting the preliminary object model is based on a 3D bounding box constraint, the 3D bounding box constraint being used to remove background information included in the 3D bounding box to generate an updated 3D bounding box. In some implementations, the depth information comprises a sparse 3D point cloud, wherein identifying the object further comprises densification of the sparse 3D point cloud, the densification of the sparse 3D point cloud being based on the 3D key points corresponding to the object.

[0019] In some implementations, identifying the object further comprises performing keypoint interpolation on 3D keypoints corresponding to the object. In some implementations, the keypoint interpolation comprises excluding 3D keypoints within a proximity of a depth edge of the object.

[0020] In some implementations, the tracked position of the device identifies the position and orientation of the device relative to a coordinate system defined based on the position and orientation of the object. In some implementations, tracking the position of the device includes determining a pose graph based on the coordinate system. For example, the history can be made of relative RGB-D camera movements, which is called a pose graph. For example, keyframes can be assigned or positioned along the pose graph, and the current camera position can be shown relative to the pose graph.

[0021] In some implementations, the device position criteria are determined based on the geometry of the target object and the data collected for the target object. For example, the system identifies the object from the initial camera position, and the device position criteria can determine or estimate (e.g., driven by back-end reconstruction requirements) the number of additional views and device positions that may be required to collect enough image data to optimally fully and accurately reconstruct the target object, thereby minimizing the number of viewing positions or stands required to make it easier for the user and less data to be processed by the local device and / or sent to the server for processing.

[0022] In some implementations, the device includes a user interface, and tracking the position of the device during image acquisition includes displaying a guidance indicator (e.g., an extended reality image, such as a plus sign positioned in 3D space) on the user interface to guide the user to move the device to a new position (and / or orientation) to capture additional images of the object at the new position. In some implementations, the guidance indicator guides the user to move the device to the new position and orientation. In some implementations, the guidance indicator is positioned in 3D space in a real-time camera view of the device. In some implementations, when the object moves within the field of view of the device's camera, the guidance indicator moves relative to the object based on an adjusted coordinate system, the adjusted coordinate system being defined based on the object's adjusted position and adjusted orientation, wherein the object's adjusted position and adjusted orientation are based on the movement of the object.

[0023] In some implementations, tracking the position of the device includes adjusting an image of the physical environment using a 2D mask to remove background image pixels of the image, wherein the 2D mask is determined based on a coordinate system of the object. For example, determining the 2D mask of the object may include accumulating a 3D point cloud, detecting a plane corresponding to the 3D point cloud of the object, removing detected plane points, clustering the point cloud using a 2D mask algorithm, obtaining points belonging to the object, and then performing a convex hull process to obtain the 2D mask.

[0024] In some implementations, tracking the position of the device includes adjusting an image of the physical environment using a 3D bounding box constraint to remove background image pixels of the image, wherein the 3D bounding box constraint is determined based on a coordinate system of the object.

[0025] In some implementations, the sensor data includes depth information comprising a sparse 3D point cloud for each image, wherein tracking the position of the device includes adjusting the image of the physical environment based on densification of the sparse 3D point cloud based on 3D key points corresponding to the object.

[0026] In some implementations, the exemplary method further involves flipping the object and switching between a conventional pose estimation mode (e.g., used when the object is flipped within the field of view of the camera) and a repositioning mode (e.g., used when the object is flipped outside the field of view of the camera). For example, in some implementations, the exemplary method further involves: when the object is reoriented or repositioned within the field of view of the camera, the method also includes a repositioning process, the repositioning process including: comparing a first image of the physical environment with a plurality of keyframe images of the object, the first image including the object; identifying a first keyframe from the plurality of keyframes based on the comparison, the keyframe being associated with a first keyframe position in a coordinate system; and determining a repositioned position of the device relative to the coordinate system of the object during acquisition of the first image based on the first keyframe position based on the identification of the first keyframe. For example, identifying a 3D feature of the object in the first image and matching the identified keyframe. In some implementations, the object is reoriented or repositioned within the field of view of the camera after a period of time when the object is not within the field of view of the camera.

[0027] In some implementations, the generated 3D model of the object is determined based on the refined images, wherein the refined images are determined based on at least one of: 3D keypoint interpolation, densification of a 3D sparse point cloud associated with the image, a 2D mask corresponding to the object for removing background image pixels of the image, and a 3D bounding box constraint corresponding to the object for removing background image pixels of the image. In some implementations, the 3D keypoint interpolation, the densification of the 3D sparse point cloud, the 2D mask, and the 3D bounding box constraint are based on a coordinate system of the object.

[0028] In some implementations, the generated 3D model of the object is based on generating a mesh, 3D point cloud, or voxel representation of the object. In some implementations, the sensor data includes depth data (e.g., RGB-D) and light intensity image data (e.g., RGB) of the physical environment.

[0029] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0031] Figure 1 is a block diagram of an exemplary operating environment according to some implementations.

[0032] Figure 2 is a block diagram of an exemplary server according to some specific implementations.

[0033] Figure 3 is a block diagram of an exemplary device according to some implementations.

[0034] Figure 4 is a flowchart representation of an exemplary method for generating a three-dimensional (3D) model of an object based on images and a tracked position of a device during acquisition of the images, according to some implementations.

[0035] Figure 5A is a block diagram illustrating exemplary positions of cameras relative to objects in a physical environment according to some implementations.

[0036] Figure 5B is a block diagram illustrating an exemplary user interface with respect to different positions of a camera relative to objects in a physical environment according to some implementations.

[0037] FIG. 6A to FIG. 6D is a block diagram illustrating an exemplary preliminary object model associated with objects in a physical environment according to some implementations.

[0038] FIG. 7A to FIG. 7C is a block diagram illustrating an exemplary preliminary object model associated with objects in a physical environment according to some implementations.

[0039] Figure 8 is a system flow diagram for exemplary generation of a 3D model of an object based on images and a tracked position of a device during acquisition of the images, according to some implementations.

[0040] Fig. 9 is a system flow diagram for exemplary generation of a real-time preview of a 3D model of an object based on images and a tracked position of a device during acquisition of the images, according to some implementations.

[0041] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0042] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0043] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some implementations. In this example, the exemplary operating environment 100 shows an exemplary physical environment 105 including an object 130 (e.g., a dog figurine), a table 140, and a chair 142. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a server 110 and a device 120. In the exemplary implementation, the operating environment 100 does not include the server 110, and the methods described herein are performed on the device 120.

[0044] In some implementations, the server 110 is configured to manage and coordinate the user experience. In some implementations, the server 110 includes a suitable combination of software, firmware, and / or hardware. Figure 2The server 110 is described in more detail. In some implementations, the server 110 is a computing device located locally or remotely relative to the physical environment 105. In one example, the server 110 is a local server located within the physical environment 105. In another example, the server 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the physical environment 105. In some implementations, the server 110 is communicatively coupled to the device 120 via one or more wired or wireless communication channels (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0045] In some implementations, the device 120 is configured to present an environment to a user. In some implementations, the device 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 Device 120 is described in more detail. In some implementations, the functionality of server 110 is provided by and / or combined with device 120.

[0046] In some implementations, device 120 is a handheld electronic device (e.g., a smartphone or tablet) configured to present content to a user. In some implementations, user 102 wears device 120 on his / her head. Device 120 may include one or more displays provided for displaying content. For example, device 120 may surround the field of view of user 102. In some implementations, device 120 is replaced with a chamber, enclosure, or room configured to present content, in which user 102 does not wear or hold device 120.

[0047] Figure 2is a block diagram of an example of a server 110 according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the server 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0048] In some implementations, the one or more communication buses 204 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0049] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 230 and one or more application programs 240.

[0050] The operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the application 240 is configured to manage and coordinate one or more experiences of one or more users (e.g., a single experience of one or more users, or multiple experiences of corresponding groups of one or more users).

[0051] Application 240 includes an image evaluation unit 242, a 3D representation unit 244, and a real-time preview unit 246. Image evaluation unit 242, 3D representation unit 244, and real-time preview unit 246 may be combined into a single application or unit, or separated into one or more additional applications or units.

[0052] The image evaluation unit 242 is configured with instructions that can be executed by the processor to perform the following operations: use one or more of the techniques disclosed in this document to obtain sensor data (e.g., image data such as light intensity data, depth data, camera position information, etc.), and determine and select object-centric data based on evaluating the image relative to the object and based on the image and the tracking position of the device during the acquisition of the image. For example, the image evaluation unit 242 analyzes an RGB image from a light intensity camera and a sparse depth map from a depth camera (e.g., a time-of-flight sensor) and other sources of physical environment information (e.g., camera positioning information from a camera's SLAM system, VIO, etc.) to select a subset of sensor information for 3D reconstruction. In some specific implementations, the image evaluation unit 242 includes independent units, such as an object detection unit, a pose tracking unit, a mask unit, a preliminary object model unit, and a cropping unit 3D, as described in reference to this document. Figure 8 Further described.

[0053] The 3D representation unit 244 is configured with instructions that can be executed by the processor to perform the following operations: using one or more techniques disclosed herein to obtain object-centric data from the image evaluation unit 242 and generate a 3D model. For example, the 3D representation unit 244 obtains the image evaluation unit 242 from the image evaluation unit 242, obtains segmentation data (e.g., RGB-S data), other sources of physical environment information (e.g., camera positioning information), and generates a 3D model (e.g., a 3D mesh representation, a 3D point cloud with associated semantic labels, etc.).

[0054] The real-time preview unit 246 is configured with instructions that can be executed by the processor to perform the following operations: generate and display a real-time preview of a preliminary 3D object model based on the image and the position of the device during the acquisition of the image of the object in the physical environment using one or more of the techniques disclosed herein. The preliminary 3D object model is then overlaid on the real-time camera feed for picture-in-picture display on the device. For example, the real-time preview unit 246 obtains a sequence of light intensity images from a light intensity camera (e.g., a real-time camera feed), tracking data generated by a pose tracking unit (e.g., from the image evaluation unit 242) (e.g., camera positioning information from a camera simultaneous localization and mapping (SLAM) system), and segmentation data generated by a cropping unit (e.g., from the image evaluation unit 242) to output a preliminary 3D object model that is iteratively updated with the sequence of light intensity images. In some specific implementations, the real-time preview unit 246 is further configured with instructions that can be executed by the processor to perform the following operations: generate a user interface guidance element to guide the user to acquire additional images of different perspectives. The guidance element can be an extended reality image overlaid in the real-time preview picture-in-picture view or the real-time camera feed. The guidance element can guide the user to stand in a position to capture additional images. In some embodiments, the guidance element can provide feedback to the user to guide the user to capture images with higher image quality. For example, if it is determined that the camera is moving too fast, the guidance element can indicate to the user that the camera is moving too fast as a specific symbol, via text, or both. In addition, the guidance element can indicate that the lighting is too dark, another angle is needed, etc. In some embodiments, the real-time preview unit 246 includes independent units such as a 3D preview model unit and a user interface guidance unit to generate the guidance element, as described in reference to this article. Fig. 9 Further described.

[0055] Although these elements are shown as residing on a single device (e.g., server 110), it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Figure 2 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0056] Figure 3is a block diagram of an example of a device 120 according to some implementations. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more AR / VR displays 312, one or more internal-facing and / or external-facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0057] In some implementations, the one or more communication buses 304 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices and sensors 306 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0058] In some implementations, one or more displays 312 are configured to present an experience to a user. In some implementations, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more displays 312 correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, device 120 includes a single display. As another example, device 120 includes a display for each eye of the user.

[0059] In some implementations, the one or more image sensor systems 314 are configured to acquire image data corresponding to at least a portion of the physical environment 105. For example, the one or more image sensor systems 314 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, infrared cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 314 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 314 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data, the multiple processing operations including at least a portion of the processes and techniques described herein.

[0060] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures, or a subset thereof, including an optional operating system 330 and one or more application programs 340.

[0061] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the application 340 is configured to manage and coordinate one or more experiences of one or more users (e.g., a single experience of one or more users, or multiple experiences of corresponding groups of one or more users).

[0062] Application 340 includes an image evaluation unit 342, a 3D representation unit 344, and a real-time preview unit 346. Image evaluation unit 342, 3D representation unit 344, and real-time preview unit 346 may be combined into a single application or unit, or separated into one or more additional applications or units.

[0063] The image evaluation unit 342 is configured with instructions that can be executed by the processor to perform the following operations: using one or more of the techniques disclosed in this document to obtain sensor data (e.g., image data such as light intensity data, depth data, camera position information, etc.), and determine and select object-centric data based on evaluating the image relative to the object and based on the image and the tracking position of the device during the acquisition of the image. For example, the image evaluation unit 342 analyzes an RGB image from a light intensity camera and a sparse depth map from a depth camera (e.g., a time-of-flight sensor) and other sources of physical environment information (e.g., camera positioning information from a camera's SLAM system, VIO, etc.) to select a subset of sensor information for 3D reconstruction. In some specific implementations, the image evaluation unit 342 includes independent units, such as an object detection unit, a pose tracking unit, a mask unit, a preliminary object model unit, and a cropping unit 3D, as described in reference to this document. Figure 8 Further described.

[0064] The 3D representation unit 344 is configured with instructions that can be executed by the processor to perform the following operations: using one or more techniques disclosed herein to obtain object-centric data from the image evaluation unit 342 and generate a 3D model. For example, the 3D representation unit 344 obtains the image evaluation unit 342 from the image evaluation unit 342, obtains segmentation data (e.g., RGB-S data), other sources of physical environment information (e.g., camera positioning information), and generates a 3D model (e.g., a 3D mesh representation, a 3D point cloud with associated semantic labels, etc.).

[0065] The real-time preview unit 346 is configured with instructions that can be executed by the processor to perform the following operations: generate and display a real-time preview of a preliminary 3D object model based on the image and the position of the device during the acquisition of the image of the object in the physical environment using one or more of the techniques disclosed herein. The preliminary 3D object model is then overlaid on the real-time camera feed for picture-in-picture display on the device. For example, the real-time preview unit 346 obtains a sequence of light intensity images from a light intensity camera (e.g., a real-time camera feed), tracking data generated by a pose tracking unit (e.g., from the image evaluation unit 342) (e.g., camera positioning information from a camera simultaneous localization and mapping (SLAM) system), and segmentation data generated by a cropping unit (e.g., from the image evaluation unit 342) to output a preliminary 3D object model that is iteratively updated with the sequence of light intensity images. In some specific implementations, the real-time preview unit 346 is further configured with instructions that can be executed by the processor to perform the following operations: generate a user interface guidance element to guide the user to collect additional images of different perspectives. The guidance element can be an extended reality image overlaid in the real-time preview picture-in-picture view or the real-time camera feed. The guidance element can guide the user to stand in a position to capture additional images. The guidance element can guide the user to stand in a position to capture additional images. In some embodiments, the guidance element can provide feedback to the user to guide the user to capture images with higher image quality. For example, if it is determined that the camera is moving too fast, the guidance element can indicate to the user that the camera is moving too fast as a specific symbol, via text, or both. In addition, the guidance element can indicate that the lighting is too dim, another angle is needed, etc. In some embodiments, the real-time preview unit 346 includes independent units such as a 3D preview model unit and a user interface guidance unit to generate the guidance element, as described in reference to this article. Fig. 9 Further described.

[0066] Although these elements are shown as residing on a single device (e.g., device 120), it should be understood that in other implementations, any combination of the elements may be located in separate computing devices. Figure 3 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules (e.g., application 340) shown separately in the figure can be implemented in a single module, and the various functions of a single functional block can be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0067] Figure 4 4 is a flowchart representation of an exemplary method 400 for generating a 3D model of an object based on images and a tracked position of a device during acquisition of the images according to some implementations. In some implementations, the method 400 is performed by a device (e.g., Figures 1 to 3 The method 400 may be performed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., a server 110 or device 120). Figure 1 and Figure 3 The method 400 is performed on a device 120 such as a head mounted display (HMD). In some implementations, the method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 400 is performed by a processor executing code stored in a non-transitory computer readable medium (e.g., a memory). Fig. 9 The 3D model creation process of method 400 is shown.

[0068] At block 402, method 400 collects sensor data during movement of a device in a physical environment including an object, wherein the sensor data includes images of the physical environment collected via a camera on the device (e.g., image sensor system 314 of device 120). For example, a user moves a device (e.g., a mobile device) around an object (e.g., a statue above a table, such as object 130) in the physical environment to collect images of the object from different sides. In some implementations, the sensor data may include depth data and motion sensor data. In some implementations, during movement of the device, a user interface may display the collected environment including the object and provide user interface elements. For example, a user interface element (e.g., an extended reality image, such as a 3D arrow overlaid on a real-time video stream) may show the user additional angles and / or perspectives of the collected object. In some implementations, the user interface may display a preliminary 3D model of the object (e.g., a 3D mesh, a 3D bounding box, etc.). For example, while the user is capturing real-time video and while the real-time video is streaming on the main viewing screen, a picture-in-picture display of another window may display the 3D model reconstruction to the user in a real-time preview screen.

[0069] The sensor data may include depth data. The depth data may include pixel depth values ​​from a viewpoint and sensor position and orientation data. In some implementations, the depth data is acquired using one or more depth cameras. For example, the one or more depth cameras may acquire depth based on structured light (SL), passive stereo (PS), active stereo (AS), time of flight (ToF), etc. Various techniques may be applied to acquire depth image data to allocate each part of the image (e.g., at the pixel level). For example, voxel data (e.g., a raster graphic on a 3D grid with values ​​of length, width, and depth) may also include multiple scalar values, such as opacity, color, and density. In some implementations, depth data is acquired from a sensor or a 3D model of the image content. Some or all of the contents of the image may be based on a real environment, such as depicting the physical environment 105 around the device 120. The image sensor may acquire an image of the physical environment 105 to be included in the image and depth information about the physical environment 105. In some implementations, the depth sensor on the device 120 determines the depth value of the voxel determined based on the image acquired by the image sensor on the device 120.

[0070] At block 404, method 400 identifies objects in at least some of the images. For example, an image evaluation unit (e.g., Figure 2 The image evaluation unit 242 and / or Figure 3 The image evaluation unit 342 evaluates the sensor data to identify objects (e.g., Figure 1 130). In some implementations, identifying an object may involve identifying an object using a preliminary object model based on information from multiple images and depth information. Identifying an object using the preliminary object model may involve masking by creating a 3D bounding box corresponding to the object. In addition or alternatively, a two-dimensional (2D) mask corresponding to the object may also be created. During image acquisition, the 3D bounding box is adjusted to better fit the object so as to define the object of interest and separate it from the background. In some implementations, object identification may be based on cropping the preliminary object model and the 3D key points associated with the preliminary object model / adjusting the size of the preliminary object model and the 3D key points associated with the preliminary object model. For example, a 3D bounding box constraint is used to remove background pixels outside the 3D bounding box, and / or a 2D mask constraint associated with the object is used to remove background pixels outside the 2D mask. Masking (e.g., cropping / sizing preliminary object models) can be performed during scanning (e.g., during image acquisition) to detect objects in a first stage, and then when performing image scanning, objects are isolated by removing background pixels using those masks (e.g., 3D bounding box constraints, 2D masks, etc.). In some implementations, masking can use a covering algorithm and bypass objects and calculate dynamic positions during image acquisition.

[0071] In some implementations, object identification can involve densification of a sparse 3D depth cloud. For example, a dense 3D depth cloud is created from a sparse data set using a densification algorithm. Additionally or alternatively, in some implementations, object identification can involve keypoint interpolation and / or exclusion of keypoints near depth edges. For example, using a keypoint object algorithm, keypoints of an object are identified (e.g., semantically labeled via a neural network for an RGB image), and additional keypoints (e.g., additional pixels associated with the object) can be interpolated (e.g., added) to a preliminary object model or excluded (e.g., removed) from the preliminary object model.

[0072] At block 406, method 400 tracks the position of the device during acquisition of the images based on identification of the objects in at least some of the images, the positions identifying the location and / or orientation of the device relative to a coordinate system defined based on the position and orientation of the objects (e.g., pose information). In an exemplary implementation, tracking the position of the device during acquisition of the images based on identification of the objects tracks the objects by updating an object-centric pose of the device (e.g., a position of the camera relative to the object in object-based coordinates as each of the images is acquired). In some implementations, tracking can use a pose graph defined in object-based coordinates (e.g., a history made of relative RGB-D camera movements).

[0073] In some implementations, tracking may involve repositioning when an object leaves view or when a previously unseen portion of an object is acquired. For example, when an object leaves the camera view and flips, tracking may require repositioning and further identifying the object that is the same as the object of interest, and recalibrating the object-centered coordinates to determine that the flipped object is the same as the object of interest. In some implementations, the method may iteratively update both the camera's object-centered pose and the preliminary object model.

[0074] In some implementations, the pose can be verified based on cropping / sizing the preliminary object model and its 3D keypoints (e.g., using 3D bounding box constraints to remove background image information and / or using 2D masks to remove background image pixels (e.g., similar to the process described above for masks during the object identification stage)). In some implementations, pose verification can also involve densification of sparse depth clouds, keypoint interpolation, and / or exclusion of keypoints close to depth edges.

[0075] At block 408, the method 400 generates a 3D model of the object based on the image and the position of the device during the acquisition of the image. For example, the 3D model may be a 3D mesh representation or a 3D point cloud. In some implementations, the 3D model data may be a 3D representation of a surface in a 3D environment using a 3D point cloud with associated semantic tags. In some implementations, the 3D model data is a 3D reconstruction mesh generated using a meshing algorithm based on depth information detected in the physical environment, which is integrated (e.g., fused) to reconstruct the object in the physical environment. The meshing algorithm (e.g., a double marching cube meshing algorithm, a Poisson meshing algorithm, a tetrahedron meshing algorithm, etc.) may be used to generate a mesh representing a room (e.g., the physical environment 105) and / or an object in the room (e.g., the object 130, the table 140, the chair 142, etc.). In some implementations, for 3D reconstruction using a mesh, in order to effectively reduce the amount of memory used during the reconstruction process, a voxel hashing method is used, in which the 3D space is divided into voxel blocks, which are referenced by a hash table using their 3D positions as keys. Voxel blocks are constructed only around the surface of the object, freeing up memory that would otherwise be used to store empty space. The voxel hashing method is also faster than competing methods at the time, such as octree-based methods. In addition, it supports data flow between GPUs, where memory is usually limited, and CPUs, where memory is more abundant.

[0076] In use, for process 400, a user may desire to create a 3D reconstruction of a dog figurine on a table (e.g., object 130), and the user may scan a room for objects with a device (e.g., a smart phone such as device 120), and the process described herein will: acquire sensor data (e.g., image data such as light intensity data, depth data, camera position information, etc.), evaluate the image relative to object-centric position tracking of the device relative to the object, select a subset of the sensor data based on the evaluation, and provide a 3D representation of the object while the object is being scanned by the user. In some embodiments, the 3D representation may be automatically displayed and updated on the user device overlaid during a real-time camera feed. In some embodiments, the 3D representation may be provided after some type of user interaction after scanning the physical environment that identifies more than one object. For example, the user may be presented with options for the identified objects, and the user may select or click on a particular object that the user wants to include in the 3D representation, and the 3D representation will then be displayed with the selected object. Thus, as described below with reference to Figure 8 As shown and described, the image evaluation unit (e.g. Figure 2 The image evaluation unit 242 and / or Figure 3 The image evaluation unit 342 of FIG. 1 identifies the image to be represented by the 3D representation unit (eg, Figure 2 3D representation unit 244 and / or Figure 3The 3D representation unit 344) utilizes a subset of the object-centric data of the image.

[0077] Figure 5A is a block diagram of an exemplary operating environment 500A illustrating exemplary camera positions of a camera on a device (e.g., image sensor system 314 of device 120) relative to an object (e.g., object 130) in a physical environment according to some implementations. In this example, the exemplary operating environment 500A illustrates an exemplary operating environment including a Figure 1 105, an environment of a device 510, an object 130, and a table 140 of a physical environment 105. Device 510 is shown in four different camera views, with each camera of devices 510a to 510d having a different field of view and perspective relative to object 130, a target object (e.g., a dog figurine on a table). Although only four different camera views are shown on pose graph path 502, there may be several more images captured at any point on pose graph path 502 as the user walks around object 130 and scans the target object. Pose graph path 502 shows an example path created by the user's movement around the object formed by the device. For example, a pose graph is a history of relative RGB-D camera movements relative to an object. For example, keyframes may be assigned or positioned along the pose graph, and the current camera position may be shown relative to the pose graph. Although relevant features are shown, one of ordinary skill in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and to not obscure more relevant aspects of the exemplary embodiments disclosed herein. To this end, operating environment 500A includes device 510 as a non-limiting example.

[0078] Figure 5B 1 is a block diagram of an exemplary operating environment 500B illustrating an exemplary user interface for different camera positions relative to a camera on a device (e.g., image sensor system 314 of device 120) relative to an object (e.g., object 130) in a physical environment (as shown in exemplary operating environment 500A). In this example, exemplary operating environment 500B illustrates an exemplary user interface from a device including device 510, from Figure 11. A user interface of an exemplary operating environment 500A for an object 130 and a table 140 of a physical environment 105. For example, when the device 510a is in a position to collect sensor data for the object 130 as shown, a user interface 562 is shown on the device display 560 with a picture-in-picture display 520a. The picture-in-picture display 520a includes a preliminary object model 522a (e.g., a 3D bounding box, a 3D rendering of the object such as a 3D mesh, a 3D reconstructed model, etc.) and a user interface guide element 524a. The guide element 524a is shown as a 3D extended reality image that is shown to the user to guide the user to move the device to a viewing angle for collecting additional sensor data. As the user and the device move, the guide element will also move to continuously guide the user to a position to collect objects at different viewing angles so that a 3D reconstruction (e.g., a high-quality rendering of the object) can be appropriately generated as needed. Guidance element 524a is associated with object 130 based on object-centric positioning data that is collected as the device moves around and relative to the object while the user is essentially creating a path 502 of the pose graph. When device 510b is in a position to collect sensor data for object 130 as shown, user interface 562 is shown on device display 560 with picture-in-picture display 520b. Picture-in-picture display 520b includes preliminary object model 522b (e.g., 3D bounding box, 3D rendering of the object such as 3D mesh, 3D reconstruction model, etc.) and user interface guidance element 524b that guides the user toward the position shown by device 510c to collect additional images. When device 510c is in a position to collect sensor data for object 130 as shown, user interface 562 is shown on device display 560 with picture-in-picture display 520c. The picture-in-picture display 520c includes a preliminary object model 522c (e.g., a 3D bounding box, a 3D rendering of the object such as a 3D mesh, a 3D reconstructed model, etc.) and a user interface guidance element 524c that guides the user toward the location shown by the device 510d to capture additional images. In some embodiments, the guidance elements 524a-524c follow a preset path around the object and relative to the center of the object. Alternatively, the guidance elements 524a-524c are iteratively updated when the system described herein (e.g., the image evaluation unit) determines the specific location where the device should capture image data, which may not require full 360-degree coverage, such as Figure 5A 502. For example, the system may determine that only 200 degrees of coverage around the object is needed to collect enough sensor data to generate a 3D model of the object.

[0079] In some implementations, the device position criteria are determined based on the geometry of the target object and the data collected for the target object. For example, the system identifies the object 130 from the initial camera position of the device 510a, and the device position criteria can determine or estimate (e.g., driven by back-end reconstruction requirements) the number of additional views and device positions that may be required to collect enough image data to optimally fully and accurately reconstruct the target object, thereby minimizing the number of viewing positions or stands required to make it easier for the user and to minimize the number of viewing positions or stands required for the local device (e.g., Figure 1 and Figure 3 device 120) to be processed and / or sent to a server (e.g., Figure 1 and Figure 2 The server 110) has less data to process.

[0080] In some implementations, device 510 may include one or more depth sensors (e.g., structured light, time of flight, etc.). Figure 5A and Figure 5B As shown, device 510 is angled toward object 130. As shown, the user is capturing sensor data around object 130 at different camera perspectives. In some embodiments, the user is continuously capturing image data as real-time video, so that as the user moves device 510a along path 502 of the pose graph to a position at device 510d, multiple images from the sensor data (e.g., a real-time video feed) can be captured. Given the current motion of the camera and the distance from the camera to the object, position data centered on the object can be determined by capturing the device position relative to the object between image frames. Refer to FIG. 6A to FIG. 6D Further illustrated are exemplary views of devices 510a through 510d and their corresponding generated 3D bounding boxes (eg, preliminary object models).

[0081] FIG. 6A to FIG. 6D It is shown that according to some specific implementations Figure 5A 5. Specifically, Fig. 6A is an exemplary camera view of device 510a with an object perspective view 610a and a preliminary object model 620a (eg, a 3D bounding box, a 3D rendering of the object such as a 3D mesh, a 3D reconstructed model, etc.) that is generated in real time as images of object 130 are acquired. Figure 6B is an exemplary camera view of device 510b with object perspective view 610b and preliminary object model 620b. Figure 6C is an exemplary camera view of device 510c with object perspective view 610c and preliminary object model 620c. Fig.6Dis an exemplary camera view of device 510d with object perspective view 610d and preliminary object model 620d.

[0082] Fig. 7A 7 is a block diagram 710 illustrating a preliminary object model (e.g., 3D bounding box 712) associated with an object (e.g., object 130) in a physical environment according to some implementations. For example, a 3D bounding box constraint algorithm may be used to exclude image data regions 714 outside of the 3D bounding box 712 in order to remove "background" data from the 3D model reconstruction. In other words, the 3D bounding box constraint limits will be sent to the 3D representation unit (e.g., Figure 2 3D representation unit 244 and / or Figure 3 The 3D representation unit 344) generates the amount of sensor data of the 3D model of the object.

[0083] Figure 7B 7 is a block diagram 720 illustrating a preliminary object model (e.g., 2D mask 722) associated with an object (e.g., object 130) in a physical environment according to some implementations. For example, a 2D mask constraint algorithm may be used to exclude image data regions 724 outside of the 2D mask 722 in order to remove "background" data from the RGB image sent to the 3D model reconstruction unit. In other words, the 2D mask constraint limits the image data that will be sent to the 3D representation unit (e.g., Figure 2 3D representation unit 244 and / or Figure 3 The 3D representation unit 344) generates the amount of sensor data of the 3D model of the object.

[0084] Figure 7C 7 is a block diagram of an exemplary environment for densifying a 3D sparse depth cloud for an object. For example, 3D sparse depth cloud 730 for object 130 is sent to densification unit 750, which generates 3D dense data 752. 3D dense data 752 may include exemplary dense 3D depth cloud 740. In some implementations, densification unit 750 acquires depth data from a camera of a device (e.g., device 120), and may generate 3D dense data 752 and send it to an image evaluation unit (e.g., Figure 2 The image evaluation unit 242 and / or Figure 3 Alternatively, the image evaluation unit may include a densification unit 750.

[0085] Figure 8 is a system flow diagram of an exemplary environment 800 according to some implementations, wherein the system can generate a 3D model of an object based on an image and a position of a device during acquisition of the image. In some implementations, the system flow of the exemplary environment 800 is performed on a device (e.g., Figures 1 to 3The system flow of the exemplary environment 800 may be displayed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., Figure 1 and Figure 3 The system flow of the exemplary environment 800 is executed on a device 120 such as a head mounted display (HMD). In some implementations, the system flow of the exemplary environment 800 is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 800 is executed on a processor that executes code stored in a non-transitory computer readable medium (e.g., a memory).

[0086] The system flow of the exemplary environment 800 collects light intensity image data 803 (e.g., a real-time camera feed such as RGB from a light intensity camera 802), depth image data 805 (e.g., depth image data such as RGB-D from a depth camera 804), and physical environment (e.g., Figure 1 The device may also include a device that uses other sources of physical environment information 807 (e.g., camera positioning information 807 such as position and orientation data from a position sensor 806) to evaluate the images and determine and select object-centric data relative to the object based on the images and the tracked position of the device during acquisition of the images (e.g., an image evaluation unit 820), and generate 3D model data 842 of the object from the object-centric data (e.g., a 3D representation unit 880).

[0087] In one exemplary implementation, environment 800 includes an image synthesis pipeline that collects or acquires data of a physical environment (e.g., image data from an image source such as sensor 810). Exemplary environment 800 is an example of image sensor data (e.g., light intensity data, depth data, and position information) that collects multiple image frames. The image source may include a depth camera 804 that collects depth data 805 of the physical environment, a light intensity camera 802 (e.g., an RGB camera) that collects light intensity image data 803 (e.g., a sequence of RGB image frames), and a position sensor 806 for collecting positioning information. For positioning information 807, some implementations include a visual inertial odometry (VIO) system that uses sequential camera images (e.g., light intensity data 803) to determine equivalent odometry information to estimate travel distance. Alternatively, some implementations of the present disclosure may include a SLAM system (e.g., position sensor 806). The SLAM system may include a multi-dimensional (e.g., 3D) laser scanning and range measurement system that is independent of GPS and provides real-time simultaneous positioning and mapping. The SLAM system can generate and manage very accurate point cloud data generated by reflections from laser scans of objects in the environment. The movement of any point in the point cloud is accurately tracked over time, allowing the SLAM system to use points in the point cloud as reference points for location, maintaining an accurate understanding of its position and orientation as it travels through the environment.

[0088] In one exemplary implementation, environment 800 includes an image evaluation unit 820 configured with instructions executable by a processor to acquire sensor data (e.g., image data such as light intensity data, depth data, camera position information, etc.) using one or more of the techniques disclosed herein and select an object-centric subset of the sensor data (e.g., object-centric data 822). In some implementations, image evaluation unit 820 includes object detection unit 830 configured with instructions executable by a processor to analyze image information and identify objects within the image data. For example, image evaluation unit 820 (e.g., Figure 2 The image evaluation unit 242 and / or Figure 3An object detection unit 830 (e.g., an image evaluation unit 342) analyzes the RGB image from the light intensity camera 802 and the sparse depth map from the depth camera 804 (e.g., a time-of-flight sensor) and other physical environment information sources (e.g., camera positioning information 807 from the camera's SLAM system, VIO, etc. (such as, position sensor 806)) to identify objects (e.g., furniture, appliances, statues, etc.) in the light intensity image sequence. In some embodiments, the object detection unit 830 uses machine learning for object identification. In some embodiments, the machine learning model is a neural network (e.g., an artificial neural network), a decision tree, a support vector machine, a Bayesian network, etc. For example, the object detection unit 830 uses an object detection neural network unit to identify objects and / or an object classification neural network to classify each type of object.

[0089] In some implementations, the image evaluation unit 820 includes a pose tracking unit 840 configured with instructions executable by the processor to analyze image information relative to position and orientation information (e.g., position data 807) of device motion during image acquisition. Figure 2 The image evaluation unit 242 and / or Figure 3 The pose tracking unit 840 of the image evaluation unit 342 analyzes the camera positioning information 807 from the camera's SLAM system, VIO, etc. (e.g., position sensor 806) to track the device position relative to the object (e.g., object-centered data) in the light intensity image sequence. In some specific implementations, the pose tracking unit 840 generates a pose graph (e.g., Figure 5A The pose graph path 502 of FIG. 502 is a graph of a pose graph as a history of relative RGB-D camera movements (e.g., devices 510a to 510d) relative to an object (e.g., object 130). For example, keyframes may be assigned or positioned along the pose graph, and the current camera position may be shown relative to the pose graph.

[0090] In some implementations, the image evaluation unit 820 includes a mask unit 850 for generating a 2D mask, a preliminary object model unit 860 for generating a preliminary object model (e.g., a 3D bounding box, a 3D rendering of an object such as a 3D mesh, a 3D reconstructed model, etc.), and a cropping unit 870 for removing background information from the image data based on the 2D mask and the 3D bounding box information. For example, a 2D mask constraint algorithm can be used by the cropping unit 870 to exclude image data regions (e.g., image data region 724) outside of the 2D mask (e.g., 2D mask 722) in order to remove "background" data from the RGB image sent to the 3D model reconstruction unit (e.g., 3D representation unit 880). In other words, the 2D mask constraint limits the image data to be sent to the 3D representation unit 880 (e.g., 3D representation unit 880). Figure 2 3D representation unit 244 and / or Figure 3 The amount of sensor data of the 3D representation unit 344 of the target object (e.g., object 130) is generated. In some specific implementations, object identification can be based on cropping / sizing the preliminary object model and the 3D key points associated with the preliminary object model. For example, the cropping unit 870 can use a 3D bounding box constraint to remove background pixels outside the 3D bounding box generated by the preliminary object model unit 860 to remove background pixels associated with the object outside the 3D bounding box. Masking (e.g., cropping / sizing the preliminary object model) can be performed during scanning (e.g., during image acquisition) to detect objects in a first stage, and then when performing image scanning, those masks (e.g., 3D bounding box constraints, 2D masks, etc.) are used to separate the objects by removing background pixels. In some specific implementations, the masking can use a covering algorithm and bypass the object and calculate the dynamic position during image acquisition.

[0091] The image evaluation unit 820 selects a subset of sensor data from the sensor 810 (e.g., selects object-centric data) based on analysis from the sub-units (e.g., the object detection unit 830, the pose tracking unit 840, the mask unit 850, the preliminary object model unit 860, and the cropping unit 870) - evaluating the images based on the images and the tracked position of the device (e.g., the device 120) during the acquisition of the images and determining and selecting object-centric data relative to the objects. The image evaluation unit 820 focuses on image acquisition relative to an object-centric camera pose defined, for example, according to an object-based coordinate system. Therefore, the pose tracking unit 840 of the image acquisition system 820 tracks the object by updating the object-centric pose of the device (e.g., the position of the camera relative to the object in object-based coordinates when each image in the image is acquired).

[0092] Additionally or alternatively, in some implementations, tracking may involve repositioning when an object leaves view or when a previously unseen portion of an object is acquired. For example, when an object leaves the camera view and flips, tracking may require repositioning and further identifying an object that is identical to the object of interest, and recalibrating object-centric coordinates to determine that the flipped object is identical to the object of interest. In some implementations, the method may iteratively update both the camera's object-centric pose and the preliminary object model.

[0093] Additionally or alternatively, in some implementations, the image information can be selected to ensure that the image includes the object or specific object features (e.g., the edge of the object, the closest point of the object to the camera, recognized indicia such as a brand name and / or symbol, etc.). For example, the image evaluation unit 820 can analyze the image information, determine specific features of the object, and limit the subset of images that include those features to ensure that only images in the subset that are included in the subset data (e.g., the object-centric data 822) are sent to the 3D reconstruction unit 880. Thus, the data set in the object-centric data 822 is limited to only images that include specific objects. In some implementations, a picture-in-picture preview can be displayed during movement of the device based on the selected subset of images, as shown in reference to Fig. 9 shown.

[0094] In one exemplary implementation, the environment 800 further includes a 3D representation unit 880 configured with instructions executable by the processor to: obtain an object-centric subset of sensor data (e.g., object-centric data 822) from the image evaluation unit 820 using one or more techniques and generate 3D model data 882. For example, the 3D representation unit 880 (e.g., Figure 2 3D representation unit 244 and / or Figure 3 The 3D representation unit 344) generates a 3D model 884a to 884d for each detected object (e.g., a 3D bounding box 844a for object 130, a 3D reconstructed model 884b, a 3D bounding box 844b for table 140, and a 3D bounding box 844c for chair 142).

[0095] The 3D model data may be 3D representations 844a to 844d that represent surfaces in a 3D environment using 3D point clouds with associated semantic labels. The 3D representations 844a, 884c, and 844d are shown as 3D bounding boxes for the object 130, the table 140, and the chair 142, respectively. In some implementations, the 3D model data 842 is a 3D reconstruction mesh generated using a meshing algorithm based on depth information detected in the physical environment, which is integrated (e.g., fused) to reconstruct the physical environment. The meshing algorithm (e.g., a double marching cube meshing algorithm, a Poisson meshing algorithm, a tetrahedron meshing algorithm, etc.) may be used to generate a mesh representing a room (e.g., the physical environment 105) and / or objects within the room (e.g., the object 130, the table 140, the chair 142, etc.). In some implementations, for 3D reconstruction using meshes, in order to effectively reduce the amount of memory used during the reconstruction process, a voxel hashing method is used, in which the 3D space is divided into voxel blocks, which are referenced by a hash table using their 3D position as a key. Voxel blocks are constructed only around the surface of the object, freeing up memory that would otherwise be used to store empty space. The voxel hashing method is also faster than competing methods at the time, such as octree-based methods. In addition, it supports data flow between GPUs and CPUs, where memory is usually limited and CPUs, where memory is more abundant.

[0096] In some implementations, the generated 3D model data 882 of the object is determined based on the refined images, wherein the refined images are determined based on at least one of: 3D keypoint interpolation, densification of a 3D sparse point cloud associated with the image, a 2D mask corresponding to the object for removing background image pixels of the image, and / or a 3D bounding box constraint corresponding to the object for removing background image pixels of the image. In some implementations, the 3D keypoint interpolation, the densification of the 3D sparse point cloud, the 2D mask, and the 3D bounding box constraint are based on a coordinate system (e.g., pose tracking data of the pose tracking unit 840) of the object (e.g., the object-centric data 822).

[0097] In some embodiments, the 3D representation unit 880 includes an integration unit configured with instructions executable by a processor to perform the following operations: using one or more known techniques to obtain a subset of image data (e.g., light intensity data 803, depth data 805, etc.) and positioning information (e.g., camera pose information 807 from a position sensor 806) and integrate (e.g., fuse) the subset of image data. For example, the image integration unit receives a subset of depth image data 805 (e.g., sparse depth data) and a subset of intensity image data 803 (e.g., RGB) from an image source (e.g., a light intensity camera 802 and a depth camera 804), and integrates the subset of image data and generates 3D data. The 3D data may include a dense 3D point cloud (e.g., an imperfect depth map and camera pose of multiple image frames surrounding an object) that is sent to the 3D representation unit 880. The 3D data may also be voxelized. In some implementations, the integration unit is within the image evaluation unit 820 , and the object-centric data 822 is integrated by the integration unit before being processed by the 3D representation unit 880 .

[0098] In some embodiments, the 3D representation unit includes a semantic segmentation unit configured with instructions that can be executed by a processor to perform the following operations: use one or more known techniques to obtain light intensity image data (e.g., light intensity data 803) and identify and segment wall structures (walls, doors, windows, etc.) and objects (e.g., people, tables, teapots, chairs, vases, etc.). For example, the segmentation unit receives a subset of the intensity image data 803 from an image source (e.g., a light intensity camera 802) and generates segmentation data (e.g., semantic segmentation data such as RGB-S data). In some embodiments, the segmentation unit uses a machine learning model, wherein the semantic segmentation model can be configured to identify semantic labels of pixels or voxels of the image data. In some embodiments, the machine learning model is a neural network (e.g., an artificial neural network), a decision tree, a support vector machine, a Bayesian network, etc. In some embodiments, the semantic segmentation unit is within the image evaluation unit 820, and the object-centric data 822 is semantically labeled by the semantic segmentation unit before being processed by the 3D representation unit 880.

[0099] Fig. 9 is a system flow diagram of an exemplary environment 900 according to some implementations, in which the system can generate and display a real-time preview of a 3D model of an object (e.g., a 3D point cloud, a 3D mesh reconstruction, a 3D bounding box associated with the object, etc.) based on an image and a position of the device during acquisition of the image. In some implementations, the system flow of the exemplary environment 900 is performed on a device (e.g., Figures 1 to 3The system flow of the exemplary environment 900 may be displayed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., Figure 1 and Figure 3 The system flow of the exemplary environment 900 is executed on a device 120 such as a head mounted display (HMD). In some implementations, the system flow of the exemplary environment 900 is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 900 is executed on a processor that executes code stored in a non-transitory computer readable medium (e.g., a memory).

[0100] The system flow of the exemplary environment 900 collects light intensity image data 903 (e.g., a real-time camera feed such as RGB from a light intensity camera 902), depth image data 905 (e.g., depth image data such as RGB-D from a depth camera 904), and physical environment (e.g., Figure 1 The system process of the exemplary environment 900 includes generating and displaying guidance elements to help users capture images at specific locations to correctly capture objects (e.g., capture additional images with higher image quality, different angles, etc.).

[0101] In one exemplary implementation, environment 900 includes an image synthesis pipeline that collects or acquires data of a physical environment (e.g., image data from an image source). Exemplary environment 900 is an example of image sensor data (e.g., light intensity data, depth data, and position information) that collects multiple image frames. The image source may include a depth camera 904 that collects depth data 905 of the physical environment, a light intensity camera 902 (e.g., an RGB camera) that collects light intensity image data 903 (e.g., a sequence of RGB image frames), and a position sensor for collecting positioning information. For positioning information 907, some implementations include a visual inertial odometry (VIO) system that uses sequential camera images (e.g., light intensity data 903) to determine equivalent odometry information to estimate travel distance. Alternatively, some implementations of the present disclosure may include a SLAM system (e.g., a position sensor 906). The SLAM system may include a multi-dimensional (e.g., 3D) laser scanning and range measurement system that is independent of GPS and provides real-time simultaneous positioning and mapping. The SLAM system can generate and manage very accurate point cloud data generated by reflections from laser scans of objects in the environment. The movement of any point in the point cloud is accurately tracked over time, allowing the SLAM system to use points in the point cloud as reference points for location, maintaining an accurate understanding of its position and orientation as it travels through the environment.

[0102] In one exemplary implementation, environment 900 includes an image evaluation unit 930 configured with instructions executable by a processor to acquire sensor data (e.g., image data such as light intensity data, depth data, camera position information, etc.) using one or more of the techniques disclosed herein and select an object-centric subset of the sensor data (e.g., object-centric data 922). In some implementations, image evaluation unit 930 includes object detection unit 932 configured with instructions executable by a processor to analyze image information and identify objects within the image data. For example, image evaluation unit 930 (e.g., Figure 2 The image evaluation unit 242 and / or Figure 3An object detection unit 932 of the image evaluation unit 342 (e.g., the image evaluation unit 342) analyzes the RGB image from the light intensity camera 902 and the sparse depth map from the depth camera 904 (e.g., a time-of-flight sensor) and other physical environment information sources (e.g., camera positioning information 907 from the camera's SLAM system, VIO, etc. (such as, position sensor 906)) to identify objects (e.g., furniture, appliances, statues, etc.) in the light intensity image sequence. In some embodiments, the object detection unit 932 uses machine learning to perform object identification. In some embodiments, the machine learning model is a neural network (e.g., an artificial neural network), a decision tree, a support vector machine, a Bayesian network, etc. For example, the object detection unit 932 uses an object detection neural network unit to identify objects and / or an object classification neural network to classify each type of object. In some embodiments, the object detection unit 932 generates a 3D bounding box (e.g., 3D bounding box 948) associated with the identified object.

[0103] In some implementations, the image evaluation unit 930 includes a pose tracking unit 934 configured with instructions executable by the processor to analyze image information relative to position and orientation information (e.g., position data 907) of device motion during image acquisition. Figure 2 The image evaluation unit 242 and / or Figure 3 The pose tracking unit 934 of the image evaluation unit 342 of the camera analyzes the camera positioning information 907 from the camera's SLAM system, VIO, etc. (e.g., position sensor 906) to track the device position relative to the object (e.g., object-centered data) in the light intensity image sequence. In some specific implementations, the pose tracking unit 934 generates a pose graph (e.g., Figure 5A The pose graph path 502 of FIG. 502 is a graph of a pose graph as a history of relative RGB-D camera movements (e.g., devices 510a to 510d) relative to an object (e.g., object 130). For example, keyframes may be assigned or positioned along the pose graph, and the current camera position may be shown relative to the pose graph.

[0104] In some implementations, the mask unit 920 generates a 2D mask based on the received image data (e.g., RGB data 903 and / or depth data 905), and sends the 2D mask data 922 to the image evaluation unit 930. In some implementations, the image evaluation unit 930 includes the mask unit 920 to generate the 2D mask data (e.g., as described herein for the image evaluation unit 820). The image evaluation unit 930 also includes an object detection unit 932 for generating a 3D bounding box associated with the identified object, and a cropping unit 940 for removing background information from the image data based on the 2D mask and the 3D bounding box information to generate segmentation data 943. For example, as shown in 2D mask data 942, the 2D mask constraint algorithm can be used by the cropping unit 940 to exclude the image data area (e.g., image data area 943) outside the 2D mask (e.g., 2D mask 944) in order to remove the "background" data from the RGB image sent to the 3D model reconstruction unit (e.g., 3D preview model unit 952). In other words, the 2D mask constraint limits the image data to be sent to the 3D representation unit (e.g., Figure 2 3D representation unit 244 and / or Figure 3 3D representation unit 344) to generate a preview 3D model of the target object (e.g., object 130). In some implementations, object identification can be based on cropping / sizing the preliminary object model and the 3D key points associated with the preliminary object model. For example, as shown in 3D bounding box data 946, cropping unit 940 can use 3D bounding box constraints to remove objects located within the preview 3D model generated by object detection unit 932 (e.g., Figure 8 The background pixels (e.g., image data area 947) outside the 3D bounding box (e.g., 3D bounding box 948) generated by the preliminary object model unit 860) are removed to remove the background pixels outside the 3D bounding box 948 associated with the object (e.g., object 130). Masking (e.g., cropping / adjusting the size of the preliminary object model) can be performed during scanning (e.g., during image acquisition) to detect the object in the first stage, and then when performing image scanning, the object is separated by removing the background pixels using those masks (e.g., 3D bounding box constraints, 2D masks, etc.). In some specific implementations, the masking can use a covering algorithm and bypass the object and calculate the dynamic position during image acquisition.

[0105] The image evaluation unit 930 selects a subset of sensor data from the sensor 910 (e.g., selects object-centric data) based on analysis from the sub-units (e.g., the object detection unit 932, the pose tracking unit 934, and the cropping unit 940) - evaluating the images based on the images and the tracked position of the device (e.g., the device 120) during the acquisition of the images and determining and selecting object-centric data relative to the object (e.g., object 3D data 933, tracking data 935, and segmentation data 943). The image evaluation unit 930 focuses on image acquisition relative to an object-centric camera pose defined, for example, according to an object-based coordinate system. Therefore, the pose tracking unit 934 of the image evaluation unit 930 tracks the object by updating the object-centric pose of the device (e.g., the position of the camera relative to the object in object-based coordinates when each image in the image is acquired).

[0106] Additionally or alternatively, in some implementations, tracking may involve repositioning when an object leaves view or when a previously unseen portion of an object is acquired. For example, when an object leaves the camera view and flips, tracking may require repositioning and further identifying an object that is identical to the object of interest, and recalibrating object-centric coordinates to determine that the flipped object is identical to the object of interest. In some implementations, the method may iteratively update both the camera's object-centric pose and the preliminary object model.

[0107] Additionally or alternatively, in some implementations, the image information can be selected to ensure that the image includes the object or specific object features (e.g., the edge of the object, the closest point of the object to the camera, recognized indicia such as a brand name and / or symbol, etc.). For example, the image evaluation unit 930 can analyze the image information, determine specific features of the object, and limit the subset of images that include those features to ensure that only images in the subset that are included in the subset data (e.g., object-centric data 922) are sent to the 3D reconstruction unit 980. Thus, the data set in the object-centric data 922 is limited to only images that include specific objects. In some implementations, a picture-in-picture preview (e.g., PIP display 970) can be displayed during movement of the device based on the selected subset of object-centric image data.

[0108] In an exemplary implementation, the environment 900 also includes a real-time preview unit 950 configured with instructions executable by a processor to acquire object-centric data (e.g., object 3D data 933, tracking data 935, and segmentation data 943) and the real-time camera feed 903 from the image evaluation unit 930. The real-time preview unit 950 includes a 3D preview model unit 952 configured with instructions executable by a processor to acquire object-centric data (e.g., object 3D data 933, tracking data 935, and segmentation data 943) using one or more techniques and generate 3D model preview data 953. For example, the 3D preview model unit 952 generates a 3D model preview for each detected object (e.g., object 130, table 140, and chair 142).

[0109] The 3D model preview data 953 may be a 3D representation of a surface in a 3D environment using a 3D point cloud with associated semantic labels. The 3D representation may be similar to Figure 8 3D representations 884a, 884c, and 844d are shown as 3D bounding boxes for object 130, table 140, and chair 142, respectively. In some implementations, the 3D model preview data is a 3D reconstruction mesh generated using a meshing algorithm based on depth information detected in the physical environment, which is integrated (e.g., fused) to reconstruct the physical environment. A meshing algorithm (e.g., a two-marching cube meshing algorithm, a Poisson meshing algorithm, a tetrahedron meshing algorithm, etc.) can be used to generate a mesh representing a room (e.g., physical environment 105) and / or objects within the room (e.g., object 130, table 140, chair 142, etc.).

[0110] In addition, the real-time preview unit 950 includes a user interface guidance unit 954 that obtains object-centric data (e.g., object 3D data 933, tracking data 935, and segmentation data 943) and the real-time camera feed 903 from the image evaluation unit 930 and generates a user interface guidance element 955 (e.g., guidance elements 524a to 524c as shown in Figure 5). For example, a guidance element 974 in a picture-in-picture display 972 and a guidance element 976 in a device display 960 are shown as exemplary implementations. The guidance element data 955 can be a 3D extended reality image, etc., which is displayed to the user to guide the user to move the device to a viewing angle for collecting additional sensor data. When the user and the device move, the guidance element will also move to continuously guide the user to a position to collect objects at different viewing angles so as to appropriately generate a 3D reconstruction (e.g., a high-quality rendering of the object) as needed. In addition or alternatively, the guidance element can guide a guidance element that can provide feedback to the user to guide the user to collect an image with higher image quality. For example, if it is determined that the camera is moving too fast, the guidance element may indicate to the user as a specific symbol, via text, or both that the camera is moving too fast. Additionally, the guidance element may indicate that the lighting is too dim, that another angle is needed, etc. The guidance element is associated with the target object based on object-centric positioning data that is collected as the device moves around and relative to the object.

[0111] In an exemplary implementation, environment 900 also includes a device display 960 (e.g., Figure 3The device display 960 may be configured to acquire guidance element data 955, a real-time camera feed 903, and a 3D model preview data 953 from a real-time preview unit 950, and generate a real-time view and a picture-in-picture (PIP) display of the 3D model when the 3D model is generated using one or more techniques. For example, the device display 960 may display a real-time view 962 (e.g., light intensity image data 903), and the PIP display module 970 may generate and display a PIP preview 972. The PIP preview 972 may be iteratively updated as the 3D preview model unit 952 continuously updates the 3D preview model while the subset image data is acquired from the image evaluation unit 930. In addition, the PIP display module 970 may acquire user interface guidance element data 955 from the user interface guidance unit 954, and display a user interface guidance element 974 in the PIP preview 972. For example, the PIP preview 972 shows a guidance element 974 (e.g., a 3D arrow pointing to an area of ​​an object to position the device to acquire additional image data). As the user acquires images of the object, the system determines which parts of the object are effectively acquired, and continuously moves the guide element 974 around the object until sufficient image data is collected. The user interface guide element helps the user acquire the images needed to generate a 3D model of the object. In addition or alternatively, the guide element data 955 can generate a guide element (e.g., real-time view 962) overlaid on the real-time camera feed. For example, the guide element 976 is shown on the real-time view of the device display 960, thereby guiding the user to move the device to the back of the object (e.g., a rear view of the object 130 (dog statue)).

[0112] In some implementations, the image synthesis pipeline may include virtual content generated for an extended reality (XR) environment (e.g., placed in Figure 1, a virtual frame on the table 140 in the figure). In some embodiments, the operating system 230, 330 includes built-in XR functionality, such as including an XR environment application or a viewer configured to be called from one or more applications 240, 340 to display the XR environment within a user interface. For example, the system described herein may include an XR unit configured with instructions executable by a processor to provide an XR environment, the XR environment including a description of a physical environment including real physical objects and virtual content. The XR unit may generate virtual depth data (e.g., a depth image of the virtual content) and virtual intensity data (e.g., a light intensity image (e.g., RGB) of the virtual content). For example, one of the applications 240 for the server 110 or the applications 340 for the device 120 may include an XR unit configured with instructions executable by a processor to provide an XR environment, the XR environment including a description of a physical environment including real objects or virtual objects. For example, virtual objects may be positioned based on detection, tracking, and representation of objects relative to each other in 3D space based on stored 3D models of real objects and virtual objects using one or more of the techniques disclosed herein.

[0113] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0114] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0115] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0116] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the blocks presented in the above examples can be changed, for example, the blocks can be reordered, combined and / or divided into sub-blocks. Some blocks or processes can be executed in parallel.

[0117] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0118] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0119] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or groups thereof.

[0120] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0121] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method for generating a three-dimensional 3D model, comprising: At a device with a processor: When acquiring a first set of images of an object in the physical environment: identifying one or more surfaces of the object; as well as tracking a position of the device in an object-based coordinate system, wherein the object-based coordinate system has a first set of positions relative to a coordinate system of the physical environment; In response to detecting movement of the object in the physical environment: acquiring a second set of images of the object; identifying the one or more surfaces of the object in the second set of images; as well as tracking a position of the device in the object-based coordinate system based on identifying the one or more surfaces of the object in the second set of images, wherein the object-based coordinate system has a second set of positions relative to the coordinate system of the physical environment, the second set of positions of the device relative to the coordinate system of the physical environment being different than the first set of positions of the device; as well as A 3D model of the object is generated based on the first and second sets of images and the tracked position of the device in a coordinate system based on the object.

2. The method of claim 1 , wherein acquiring the second set of one or more images of the object after the movement of the object comprises: As the device moves around the object, images are acquired from different perspectives of the object.

3. The method of claim 1, wherein the device comprises a user interface, and wherein the method further comprises: During the movement of the object, the second set of one or more acquired images including the physical environment of the object are displayed within the user interface.

4. The method of claim 1 , wherein identifying the 3D key points on the one or more surfaces of the object in the first set of one or more images comprises: A preliminary object model is generated based on depth information from the first set of one or more images of the physical environment.

5. The method of claim 4, wherein the preliminary object model is a 3D bounding box, wherein generating the 3D bounding box comprises: Obtaining a 3D representation of the physical environment generated based on the depth information; determining a ground plane corresponding to the object in the physical environment based on the 3D representation; as well as The 3D bounding box corresponding to the object in the physical environment is generated based on the ground plane and the 3D representation. 6 . The method of claim 5 , wherein generating the 3D model of the object further comprises adjusting the preliminary object model based on the 3D key points on the one or more surfaces corresponding to the object. 7 . The method of claim 6 , wherein adjusting the preliminary object model is based on a 3D bounding box constraint, the 3D bounding box constraint being used to remove background information included in the 3D bounding box to generate an updated 3D bounding box.

8. The method of claim 4, wherein the depth information comprises a sparse 3D point cloud, wherein generating the 3D model of the object further comprises densification of the sparse 3D point cloud, the sparse 3D point cloud being based on the 3D keypoints corresponding to the object.

9. The method of claim 1, wherein generating the 3D model of the object further comprises keypoint interpolation based on the 3D keypoints corresponding to the object, wherein keypoint interpolation comprises excluding 3D keypoints within a proximity of a depth edge of the object.

10. The method of claim 1, wherein the tracked position of the device identifies a position and orientation of the device relative to the object-based coordinate system.

11. The method of claim 1 , wherein the device comprises a user interface, wherein tracking a position of the device in the object-based coordinate system during acquisition of the second set of images comprises: A guide indicator is displayed on the user interface to guide moving the device to a new position to capture additional images of the object at the new position, wherein the guide indicator guides moving the device to the new position and new orientation, wherein the guide indicator is positioned in 3D space in a real-time camera view of the device.

12. A method according to claim 11, wherein when the object moves within the field of view of the camera of the device, the guidance indicator moves relative to the object based on an adjusted object-based coordinate system, the adjusted coordinate system being defined based on an adjusted position and an adjusted orientation of the object, wherein the adjusted position and the adjusted orientation of the object are based on the movement of the object.

13. The method of claim 1 , wherein tracking the position of the device in the object-based coordinate system comprises adjusting the image of the physical environment using: a two-dimensional (2D) mask for removing background image pixels of the image, wherein the 2D mask is determined based on the coordinate system of the object; or A 3D bounding box constraint for removing background image pixels of the image, wherein the 3D bounding box constraint is determined based on the object-based coordinate system of the object.

14. The method of claim 1 , wherein acquiring the first set of one or more images or the second set of one or more images of the object comprises acquiring depth information, the depth information comprising a sparse 3D point cloud for each image, wherein tracking the position of the device in the object-based coordinate system comprises adjusting the image of the physical environment based on densification of the sparse 3D point cloud based on the 3D key points corresponding to the object.

15. The method of claim 1, wherein when the object is reoriented or repositioned within the field of view of the camera, the method further comprises a repositioning process, the repositioning process comprising: comparing a first image in the first set of images of the physical environment to a plurality of keyframe images of the object, the first image including the object; Based on the comparing, identifying a first key frame from the plurality of key frame images, the first key frame being associated with a first key frame position in the coordinate system; as well as Based on identifying the first keyframe, a repositioned position of the device relative to the object-based coordinate system is determined based on the first keyframe position.

16. The method of claim 15, wherein the object is re-oriented or re-positioned within the field of view of the camera after a period of time during which the object is not within the field of view of the camera.

17. The method of claim 1 , wherein the generated 3D model of the object is determined based on a refined image, wherein the refined image is determined based on at least one of: 3D key point interpolation, densification of a 3D sparse point cloud associated with the image, a two-dimensional (2D) mask corresponding to the object for removing background image pixels of the image, a 3D bounding box constraint corresponding to the object for removing background image pixels of the image.

18. The method of claim 17, wherein the 3D keypoint interpolation, the densification of the 3D sparse point cloud, the 2D mask, and the 3D bounding box constraints are based on the coordinate system of the object.

19. A device for generating a three-dimensional 3D model, comprising: a non-transitory computer-readable storage medium; and One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations including: When acquiring a first set of images of an object in the physical environment: identifying one or more surfaces of the object; as well as tracking a position of the device in an object-based coordinate system, wherein the object-based coordinate system has a first set of positions relative to a coordinate system of the physical environment; In response to detecting movement of the object in the physical environment: acquiring a second set of images of the object; identifying the one or more surfaces of the objects in the second group; and tracking a position of the device in the object-based coordinate system based on identifying the one or more surfaces of the object in the second set of images, wherein the object-based coordinate system has a second set of positions relative to the coordinate system of the physical environment, the second set of positions of the device relative to the coordinate system of the physical environment being different than the first set of positions of the device; and A 3D model of the object is generated based on the first and second sets of images and the tracked position of the device in a coordinate system based on the object.

20. A non-transitory computer-readable storage medium storing computer-executable program instructions on a computer to perform operations comprising: When acquiring a first set of images of an object in the physical environment: identifying one or more surfaces of the object; as well as Tracking the position of the device in an object-based coordinate system, wherein The object-based coordinate system has a first set of positions relative to a coordinate system of the physical environment; In response to detecting movement of the object in the physical environment: acquiring a second set of images of the object; identifying the one or more surfaces of the object in the second set of images; as well as tracking a position of the device in the object-based coordinate system based on identifying the one or more surfaces of the object in the second set of images, wherein the object-based coordinate system has a second set of positions relative to the coordinate system of the physical environment, the second set of positions of the device relative to the coordinate system of the physical environment being different than the first set of positions of the device; as well as A 3D model of the object is generated based on the first and second sets of images and the tracked position of the device in a coordinate system based on the object.

Images