Method, apparatus and system for generating a three-dimensional model of a scene - Patents.com

Abstract

A method for generating a 3D model of a scene is disclosed. An imaging means in a 3D modeling system acquires first depth data. The first depth data includes pixels of depth images of a plurality of frames. A scanning means in the 3D modeling system acquires second depth data. The second depth data includes depth data points of the plurality of image frames. The 3D modeling system acquires color data. The color data includes pixels of the plurality of color images. The 3D modeling system generates a 3D model based on the color data, the first depth data, and the second depth data. The 3D modeling system displays the 3D model.

Description

[Technical field]

[0001] The present disclosure relates to the field of three-dimensional (3D) reconstruction techniques, and more particularly, to image processing for generating VR data. [Background technology]

[0002] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to Chinese Patent Application No. 202011180650.0, filed on October 29, 2020, the entire contents of which are incorporated herein by reference.

[0003] [Background technology] 3D reconstruction is a method of establishing a mathematical model suitable for computer representation and processing of 3D objects. In computers, the established 3D model can realize processing, manipulation, and analysis of 3D objects. 3D reconstruction technology is an important method for constructing VR environments that represent the objective world in computers. In general, 3D reconstruction may include steps such as image acquisition, camera calibration, feature extraction, stereo matching, and 3D reconstruction.

[0004] Conventional 3D reconstruction techniques usually use either a depth camera or a LIDAR (light detection and ranging) system for 3D modeling. A depth camera can provide high-resolution depth data. However, the accuracy of the depth data acquired by a depth camera is highly dependent on the distance between the depth camera and objects in the scene, so the high-precision depth data range of the depth camera is limited. In addition, the field distance of a depth camera is usually limited by its power and resolution. On the other hand, the accuracy of the depth data acquired by a LIDAR system is high and relatively stable in a wide depth range, so the LIDAR system provides high-precision depth data in a wide depth range. However, the resolution of the depth data acquired by a LIDAR system is low. Therefore, the point cloud generated for 3D modeling is relatively sparse, and it is difficult to model small objects (such as water pipes, pens, etc.) with sufficient details.

[0005] Therefore, there is a need to provide a reliable 3D reconstruction solution that takes into account the wide detection range of a wide scene, the accuracy of the generated 3D model, and the resolution. Summary of the Invention

[0006] A method, computer readable medium, system, and apparatus are disclosed for generating a 3D model of a scene. Depth data acquired by an imaging means and a scanning means are combined such that a high resolution depth image with consistent accuracy can be generated for the 3D model.

[0007] In some embodiments, a method of generating a 3D model of a scene is provided, the method comprising: an imaging means in a 3D modeling system acquiring first depth data; a scanning means in the 3D modeling system acquiring second depth data; a 3D modeling system receiving color data; generating a 3D model of the scene based on the color data, the first depth data, and the second depth data; and displaying the 3D model of the scene in the 3D modeling system. The first depth data comprises pixels of a plurality of frames of a depth image. The second depth data comprises depth data points of a plurality of image frames.

[0008] In some embodiments, each of the depth images in the first depth data is a first data, and each of the image frames in the second depth data is a second data. The method further includes determining a plurality of data pairs and determining a positional relationship between the first depth data and the second depth data based on the plurality of data pairs. Each of the plurality of data pairs includes the first data and the corresponding second data. The first data and the corresponding second data in the data pairs include similar target objects.

[0009] In some embodiments, each of the first data is tagged with a first external parameter indicating attitude information of the imaging means. Each of the second data is tagged with a second external parameter indicating attitude information of the scanning means. The method further includes determining a first attitude associated with the first data based on the first external parameter, determining a second attitude associated with the second data based on the second external parameter, and determining the data pair including the first data and the second data in response to the first attitude and the second attitude being similar.

[0010] In some embodiments, the first extrinsic parameter and the second extrinsic parameter are output from the same attitude sensor in the 3D modeling system.

[0011] In some embodiments, each of the first data is provided with a first time stamp indicating a fixed time when the first data was acquired by the imaging means, each of the second data is provided with a second time stamp indicating a fixed time when the second data was acquired by the scanning means, and the first data and the corresponding second data in the data pair have a time interval that is less than a threshold value.

[0012] In some embodiments, the first data and the corresponding second data in the data pair are identified with one or more similar objects.

[0013] In some embodiments, the method further includes determining a depth threshold and removing the one or more data points in the first depth data in response to a depth value of the one or more data points being greater than the depth threshold.

[0014] In some embodiments, the method further includes identifying one or more image regions in the generated 3D model of the scene that are below a resolution threshold; the imaging means in the 3D modeling system acquiring first fill data comprising a plurality of depth images; and supplementing the generated 3D model of the scene with the first fill data to generate a new 3D model of the scene.

[0015] In some embodiments, the method further comprises identifying one or more image regions in the generated 3D model of the scene that do not have sufficient depth data points, the scanning means in the 3D modeling system acquiring second fill data comprising depth data points of the plurality of image frames, and supplementing the generated 3D model of the scene with the second fill data to generate a new 3D model of the scene.

[0016] In some embodiments, the imaging means acquires the plurality of depth images at a first frame rate and the scanning means acquires the plurality of image frames at a second frame rate.

[0017] In some embodiments, the 3D modeling system includes a display for displaying a 3D model of the scene in real time based on the acquired depth data.

[0018] In some embodiments, a system for generating a 3D model of a scene is provided. The system comprises an imaging means, a scanning means, and one or more processors. The imaging means is configured to acquire first depth data. The first depth data comprises pixels of depth images of a plurality of frames. The scanning means is configured to acquire second depth data. The second depth data comprises depth data points of a plurality of image frames. The one or more processors are configured to acquire the first depth data from the imaging means, the second depth data from the scanning means, and color data, generate a 3D model of the scene based on the first depth data, the second depth data, and the color data, and output the generated 3D model of the scene. The color data comprises pixels of a plurality of color images.

[0019] In some embodiments, each of the depth images in the first depth data is first data. Each of the image frames in the second depth data is second data. The one or more processors in the 3D modeling system are further configured to determine a plurality of data pairs and determine a positional relationship between the first depth data and the second depth data based on the plurality of data pairs. Each of the plurality of data pairs includes the first data and the corresponding second data. The first data and the corresponding second data include similar target objects.

[0020] In some embodiments, the 3D modeling system further comprises one or more attitude sensors. The one or more attitude sensors are configured to output external parameters indicative of attitude information of the imaging means and the scanning means in the 3D modeling system. Each of the first data acquired by the imaging means is tagged with a first external parameter indicative of attitude information of the imaging means. Each of the second data acquired by the scanning means is tagged with a second external parameter indicative of attitude information of the scanning means. The one or more processors in the 3D modeling system are further configured to determine a first attitude associated with the first data based on the first external parameter, determine a second attitude associated with the second data based on the second external parameter, and determine a data pair including the first data and the second data in response to the first attitude and the second attitude being similar.

[0021] In some embodiments, each of the first data is provided with a first time stamp indicating a fixed time when the first data was acquired by the imaging means, each of the second data is provided with a second time stamp indicating a fixed time when the second data was acquired by the scanning means, and the first data and the corresponding second data in the data pair have a time interval that is less than a threshold value.

[0022] In some embodiments, the first data and the corresponding second data in the data pair are identified with one or more similar objects.

[0023] In some embodiments, the one or more processors in the 3D modeling system are further configured to determine a depth threshold and to remove the one or more data points in the first depth data in response to a depth value of the one or more data points being greater than the depth threshold.

[0024] In some embodiments, the one or more processors in the 3D modeling system are further configured to: identify one or more image regions in the generated 3D model of the scene that are below a resolution threshold; receive first fill data from the imaging means; and fill the generated 3D model of the scene with the first fill data to generate a new 3D model of the scene, the first fill data comprising pixels of a plurality of depth images.

[0025] In some embodiments, the one or more processors in the 3D modeling system are further configured to identify one or more image regions in the generated 3D model of the scene that do not have sufficient depth data points, receive second fill data from the scanning means, and supplement the generated 3D model of the scene with the second fill data to generate a new 3D model of the scene, wherein the second fill data comprises depth data points of a plurality of image frames.

[0026] In some embodiments, a non-volatile computer readable medium is provided that stores computer executable instructions that, when executed by one or more processors, facilitate causing an imaging means in a 3D modeling system to acquire first depth data, a scanning means in the 3D modeling system to acquire second depth data, the 3D modeling system to receive color data, and generate a 3D model of a scene based on the first set of depth data, the second depth data, and the color data, and display the 3D model of the scene. The first depth data includes pixels of depth images of a plurality of frames. The second depth data includes depth data points of a plurality of image frames. [Brief description of the drawings]

[0027] The technology of the present invention will be described in more detail below based on the exemplary figures, but not limited to the examples. All features described and / or illustrated in this specification can be used alone or combined in different combinations. The features and advantages of the various examples will become clear by reading the following detailed description with reference to the attached drawings, which show: [Figure 1] FIG. 1 illustrates an exemplary 3D virtual reality (VR) environment in accordance with one or more embodiments. [Diagram 2] FIG. 2 illustrates a block diagram of an exemplary computer system in accordance with one or more embodiments. [Diagram 3] FIG. 3 illustrates a process for generating a 3D model of a scene according to one or more embodiments. [Figure 4] FIG. 4 illustrates an application scenario for generating a 3D model of a scene in accordance with one or more embodiments. [Diagram 5] FIG. 3 illustrates a process for generating a 3D model of a scene according to one or more embodiments. [Figure 6] FIG. 3 illustrates a process for generating a 3D model of a scene according to one or more embodiments. [Figure 7] FIG. 7 illustrates a system for generating a 3D model of a scene in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] The disclosure described herein provides techniques for generating a 3D model of a scene. Depth information is acquired using an imaging means and a scanning means. The imaging means, such as a depth camera, is configured to acquire depth data below a threshold. The depth data acquired by the imaging means includes high resolution depth data, thereby providing details of the 3D model. The scanning means, such as a LIDAR system, is configured to acquire depth data with consistent accuracy over a wide range of shooting distances, such that distant objects can be generated with high accuracy in the 3D model. The depth data points acquired by the imaging means and the scanning means are combined based on a positional relationship between the depth data points. In some examples, the positional relationship is determined based on attitude information of a data acquisition device, such as the imaging means or the scanning means. In other examples, the positional relationship is determined by identifying multiple target objects present in multiple images and / or frames of depth data points. Additionally and / or alternatively, the data points in the 3D model are rendered according to color information provided by a color imaging means. In this manner, a 3D model is generated that includes both depth and color information.

[0029] By applying the techniques provided herein, high resolution and accuracy 3D models can be generated to enhance the user experience of simulated VR applications.

[0030] FIG. 1 illustrates an exemplary 3DVR environment 100 according to some embodiments. As illustrated in FIG. 1, the 3DVR environment 100 may simulate or represent a residential unit, such as an apartment or a floor of a house. It is noted that the 3DVR environment 100 may include a VR representation of any indoor space or environment. With reference to FIG. 1, the 3DVR environment 100 may include one or more functional spaces, such as 110, 120, 130, 140, 150, and 160. As used herein, a functional space refers to an enclosed or partially enclosed space associated with a given function. In some cases, a functional space may correspond to a room. For example, the functional space 110 may correspond to a first bedroom, and the functional space 130 may correspond to a second bedroom. In some examples, a functional space may correspond to an enclosed or partially enclosed space within or adjacent to a room. For example, the functional space 140 may correspond to a closet. In other examples, a functional space may correspond to an area commonly used for a given purpose. For example, functional space 120 may correspond to a kitchen area, functional space 150 may correspond to a dining area, and functional space 160 may correspond to a living room. Functional spaces 120, 150, 160 may share the same room (e.g., an enclosed area) but may be considered different functional spaces because their functions are different.

[0031] FIG. 2 is a block diagram of an exemplary computer system 200 configured to implement various functions disclosed herein. For example, the computer system 200 may be configured as a server for creating or reconstructing the VR environment 100. In another example, the computer system 200 may be configured as a terminal device for displaying or enhancing the VR environment 100. As shown in FIG. 2, the computer system 200 may include a processor 210, a communication interface 220, a memory / storage 230, and a display 240. The memory / storage 230 may be configured to store computer-readable instructions that, when executed by the processor 210, cause the processor 210 to perform various operations disclosed herein. The memory 230 may be any non-volatile type of mass storage, such as a volatile or non-volatile, magnetic, semiconductor-based, tape-based, optical, removable, non-removable, or other type of storage device or tangible computer-readable medium, including, but not limited to, read-only memory (ROM), flash memory, dynamic random access memory (RAM), and / or static RAM.

[0032] The processor 210 may be configured to perform operations according to instructions stored in the memory 230. The processor 210 may include any suitable type of general purpose or special purpose microprocessor, digital signal processor, microcontroller, etc. In some examples, the processor 210 may be configured as a separate processor module dedicated to performing one or more predetermined operations disclosed herein. In other examples, the processor 210 may be configured as a shared processor module to be able to perform other operations unrelated to one or more predetermined operations disclosed herein.

[0033] The communication interface 220 may be configured to communicate information between the computer system 200 and other devices or systems. For example, the communication interface 220 may include an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection. As another example, the communication interface 220 may include a local area network (LAN) card to provide a data communication connection to a compatible LAN. As another example, the communication interface 220 may include a high-speed network adapter, such as a fiber optic network adapter, a 10G Ethernet adapter (Ethernet is a registered trademark), or the like. A wireless link may also be implemented by the communication interface 220. In such an implementation, the communication interface 220 may send and receive electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information over a network. The network may typically include a cellular communication network, a wireless local area network (WLAN), a wide area network (WAN), or the like.

[0034] The communication interface 220 may include various I / O devices such as a keyboard, a mouse, a touchpad, a touchscreen, a microphone, a camera, a biosensor, etc. A user can input data to the terminal device via the communication interface 220.

[0035] The display 240 may be integrated as part of the computer system 200 or may be provided as a separate device communicatively coupled to the computer system 200. The display 240 may include a display device such as a liquid crystal display (LCD), a light emitting diode display (LED), a plasma display, or any other type of display, and may provide a graphical user interface (GUI) presented on the display for user input and data representation. In some embodiments, the display device 240 may include VR goggles, VR glasses, or other similar devices that provide an immersive VR experience. For example, the VR environment 100 may be displayed on the display 240. In some embodiments, the display 240 may be integrated as part of the communication interface 220.

[0036] FIG. 3 is an exemplary process 300 for generating a 3D model of a scene according to one or more embodiments. Process 300 may be performed by the 3D modeling system and / or device 200 according to computer-executable instructions stored in memory 230 shown in FIG. 2. The 3D modeling system may include an imaging means, a scanning means, and / or one or more orientation sensors. Additionally and / or alternatively, the 3D modeling system may further include one or more platforms, motors, actuators for positioning and repositioning one or more means, such as the imaging means and / or the scanning means. For example, the 3D modeling system may include one or more platforms, motors, and / or actuators for rotating and / or moving the imaging means separately from the scanning means. In some examples, the 3D modeling system may include a color imaging means (e.g., a color camera) configured to acquire color data associated with the image. In other examples, the 3D modeling system may acquire color data from an external color imaging means separate from the 3D modeling system.

[0037] The imaging means may be a depth camera and may be configured to capture depth images of the scene. Each depth image of the scene consists of a number of pixels, each pixel including a depth value. The depth value may represent a distance between a representative object depicted by the one or more pixels and a position of the imaging means. The scanning means (e.g., a LIDAR device) may be configured to scan the scene to collect a number of depth data points indicative of a number of depth values. The one or more orientation sensors may be configured to output a number of extrinsic parameters. The extrinsic parameters include position and rotation information related to the imaging means and the scanning means. For example, the extrinsic parameters may include an orientation matrix including a 3×3 rotation matrix and a 3×1 translation vector. In some variations, the imaging means may capture a number of images (e.g., five images), and the orientation sensor may determine one or more orientation matrices for the images. For example, the imaging means may be stationary (e.g., at the same position) when capturing the multiple images, and the orientation sensor may obtain / determine a single orientation matrix for the multiple images. The orientation matrix may indicate rotational (e.g., pitch, yaw, roll) and / or translational (e.g., x, y, z positional values) values ​​associated with the imaging means at this position. In some examples, the imaging means may be at different positions when acquiring multiple images. The orientation sensor may obtain / determine an orientation matrix indicating rotational and / or translational values ​​of the imaging means when acquiring each of these images.

[0038] The extrinsic parameters may be used for localization and / or to calculate the position information of the images from the imaging means and the scanning means to be in the same 3D coordinate plane. In other words, the imaging means captures a first image and the scanning means acquires a second image. The extrinsic parameters from the attitude sensor may be used to align the first and second images from these two means to be in the same coordinate plane. The 3D modeling system may further include a device 200 for processing the acquired data and / or an interface for transmitting the acquired data to the device 200 for data processing. However, it will be appreciated that the process 300 may be performed in any suitable environment and any of the following blocks may be performed in any suitable order.

[0039] In block 310, the 3D modeling system acquires first depth data. Specifically, an imaging means of the 3D modeling system captures a plurality of depth images of a scene. The imaging means may be and / or include one or more cameras, including but not limited to stereo cameras, structured light cameras, time-of-flight (TOF) cameras, and / or other types of depth cameras. Each depth image captured by the imaging means includes a plurality of pixels. Each pixel is associated with a depth data point that includes a depth value indicating a distance between the imaging means at a capture position (e.g., the position of the imaging means when the image is captured) and a spot in the scene. The first depth data includes pixels of the depth image captured by the imaging means.

[0040] A scene may be an image of one or more objects and / or imaging areas, and said scene may be any type of scene including any type / number of objects and / or imaging areas. For example, the scene may be an indoor scene of a house or an outdoor scene of a house. In some examples, at least some of the objects in the scene (e.g., a faucet and / or a door handle) may be imaged twice. For example, the imaged scene may include partial scene A, partial scene B, partial scene C. The size of each partial scene is the field of view of the imaging means. The imaging means may capture two depth images of each partial scene. Additionally and / or alternatively, if partial scene A and partial scene B include a common object, the imaging means may capture one depth image for partial scene A and one depth image for partial scene B. In this case, the common object is imaged twice in two depth images.

[0041] In some examples, the imaging means may be arranged in different attitudes (e.g., positions and / or shooting angles) to capture multiple depth images of a scene, such that multiple depth images captured for a partial scene may cover the entire scene. The imaging means may be moved or rotated to be arranged in different attitudes. For example, the imaging means may be arranged on a platform. The platform may be configured to move according to a preset motion trajectory. The imaging means may capture a depth image every time the imaging means moves a preset distance (e.g., 0.2 m). In some variations, the platform may be configured to rotate around a preset object. The imaging means may capture a depth image every time the imaging means moves a preset angle (e.g., 10 degrees). Furthermore, the platform may be configured to rotate around a preset object while moving according to a preset motion trajectory. In other words, the imaging means may capture a depth image when the imaging means rotates / moves a preset distance and / or a preset angle.

[0042] In some examples, the 3D modeling system can perform keypoint detection during or after the process of capturing multiple depth images of the scene. For example, the 3D modeling system can perform keypoint detection on one or more depth images captured by the imaging means of a partial scene to identify one or more target objects and / or one or more image regions in the captured depth images and determine whether the identified target objects require additional depth data due to one or more causes (e.g., low resolution and / or missing / incomplete data points). For example, the scene may include a target object (e.g., a door handle), and the 3D modeling system can capture one or more depth images of the scene. Each depth image may be a partial scene (e.g., may include the target object and show a part of the entire scene). In other words, the 3D modeling system will instruct the imaging means to capture a first depth image of the target object, and then instruct the imaging means to capture a next depth image including the target object. Additionally and / or alternatively, the 3D modeling system determines that a target object in the scene requires additional depth data based on the keypoint detection results (e.g., based on low resolution and / or missing data points). For example, the 3D modeling system determines that additional depth data is required because the resolution of the target object is below a predetermined threshold. In this case, the processor of the 3D modeling system instructs the imaging means to move to a predetermined location where an image of the target object can be captured. The processor then instructs the imaging means to capture depth images of the partial scene to provide additional depth data for the target object. The target object may be a predetermined object (e.g., a user may provide a user input specifying the target object). Additionally and / or alternatively, the target object may be an object that satisfies a pre-set condition (e.g., the target object is smaller than a predetermined volume or area in the image).

[0043] In block 320, the 3D modeling system acquires second depth data of the scene. Specifically, a scanning means of the 3D modeling system scans the scene to collect a plurality of depth data points. Each depth data point includes a depth value indicating a distance between the scanning means and a spot in the scene. In some examples, the scanning means may be a LIDAR system that utilizes an energy source, such as a laser, to scan an area and detect reflected energy resulting from bouncing off objects in the scanned area.

[0044] Similar to block 310, the 3D modeling system may perform keypoint detection during or after scanning the scene. For example, the 3D modeling system performs keypoint detection on one or more frames of depth data points acquired by the scanning means for the sub-scene to identify one or more target objects and / or one or more image regions in the acquired frames and determine whether the identified target objects require supplemental depth data due to one or more causes (e.g., low resolution and / or missing / incomplete data points). Based on the keypoint detection results (e.g., based on low resolution and / or missing data points), the processor of the 3D modeling system instructs the scanning means to scan a predetermined region to provide supplemental depth data for the target objects.

[0045] In some examples, the relative position between the imaging means and the scanning means is fixed. In other cases, the relative position is not fixed. The relative position between the imaging means and the scanning means may be dynamically determined during the depth data acquisition process if the relative position is not fixed. In this case, the 3D modeling system may include one or more attitude sensors that output external parameters indicative of position and rotation information of the imaging means and / or the scanning means. In some variations, the 3D modeling system may include a single attitude sensor for both the imaging means and the scanning means. In other variations, the 3D modeling system may include an attitude sensor for the imaging means and a separate attitude sensor for the scanning means.

[0046] In block 330, the 3D modeling system generates a 3D model of the scene based on the color data, the first depth data, and the second depth data. In some variations, the color data is obtained based on an image of the scene captured using a color imaging means (e.g., a color camera). In some examples, the 3D modeling system includes a color imaging means for obtaining the color data. In other examples, a set of color data captured by the color imaging means is transmitted to the 3D modeling system. The image of the scene captured by the color imaging means may be and / or may include a red-green-blue (RGB) image, a grayscale image, and / or a black / white image. The color data is formed based on pixels of the color image (e.g., the color data may include RGB values, grayscale values, and / or luminance values). The 3D modeling system may include a processor configured to process the received color data and depth data to generate a 3D model of the scene.

[0047] In some examples, the 3D modeling system can utilize the orientation information corresponding to the imaging means and the scanning means to combine a first set of depth data from the imaging means and a second set of depth data from the scanning means. The processor of the 3D modeling system can obtain external parameters including position and rotation information of the imaging means and the scanning means from the orientation sensor. The processor can determine and / or calculate the position, rotation, and / or shooting angle of the imaging means and / or the scanning means in the same 3D coordinate plane based on the external parameters. For example, the imaging means obtains multiple depth images of a scene. Each depth image is associated with an external parameter indicating the orientation of the imaging means at the time when the depth image was obtained. Based on the calculated orientation information of the imaging means, a positional relationship between the depth images obtained by the imaging means is determined. In this way, the depth images obtained by the imaging means can be combined based on the determined positional relationship. Additionally and / or alternatively, the depth data points obtained by the scanning means can be combined based on the positional relationship determined from the orientation information of the scanning means. Additionally and / or alternatively, the depth images acquired by the imaging means and the depth data points acquired by the scanning means may be combined based on a positional relationship determined from the orientation information of the imaging means and the scanning means. In this way, the first depth data and the second depth data can be combined to generate a complete depth image of the scene. Additionally and / or alternatively, each color image acquired by the color imaging means may be associated with an external parameter indicative of the orientation information of the color imaging means. Similarly, the color images may be combined based on a positional relationship between the color images. Additionally and / or alternatively, the color data formed by the color images may be aligned with the depth data points in the scene based on a positional relationship determined from the orientation information of the data acquisition means. Thus, a 3D model including both depth and color information can be generated.

[0048] In some variations, the 3D modeling system can combine the color data, the first depth data, and the second depth data based on the results of the keypoint detection (e.g., based on one or more identified target objects). For example, the 3D modeling system can perform keypoint detection on each of the depth images in the first depth data to identify multiple target objects. The target objects are captured by at least two depth images from the imaging means such that the depth images can be combined by aligning the identified target objects in the depth images. The second depth data from the scanning means includes multiple frames. The 3D modeling system can perform keypoint detection on each frame acquired by the scanning means to identify the number of target objects. Similarly, the depth data points acquired by the scanning means can be combined by aligning the identified target objects in the frames. Additionally and / or alternatively, the first depth data and the second depth data can be combined by aligning the identified target objects present in both the first depth data and the second depth data. Additionally and / or alternatively, the 3D modeling system may perform keypoint detection on the color image in the color data to identify a number of target objects. In this manner, the color data may be aligned with the depth data based on the identified target objects present in both the depth data and the color data. Thus, the 3D modeling system may combine the color data, the first depth data, and the second depth data by aligning the number of target objects in the scene to generate a 3D model of the scene.

[0049] After generating the 3D model, the 3D model is output to a display, where it can be used for VR applications.

[0050] 4 is an exemplary application for generating a 3D model of a scene 400 according to one or more embodiments. The 3D modeling system 402 includes an imaging means 404 and a scanning means 406. The imaging means 404 of the 3D modeling system 402 captures a plurality of depth images of the scene 400 to form a first depth data. The depth images may include a partial scene 408 including at least one object. For example, the partial scene 408 includes a desk lamp. In some examples, at least two depth images captured by the imaging means 404 include the partial scene 408. The scanning means 406 scans the scene 400 to collect a plurality of depth data points to form a second depth data. The 3D modeling system 402 then generates a 3D model of the scene based on the color data, the first depth data, and the second depth data.

[0051] The methods, systems, and apparatus for generating a 3D model of a scene described herein can generate a high-resolution 3D model of a scene with consistent accuracy for a wide range of distances of objects in the scene. A high-quality 3D model of a scene is generated by combining depth data obtained from an imaging means, such as a depth camera, and depth data obtained from a scanning means, such as a LIDAR system.

[0052] Depth cameras have the advantages of high resolution and high frame rate. Thus, depth cameras can capture details of small objects in a scene. However, depth cameras are usually used at relatively close shooting distances because they may be limited by their output or resolution. Most depth cameras have a shooting distance of 0.2-8m, and the maximum shooting distance is usually within 10m. Also, the accuracy of the depth data acquired by a depth camera depends heavily on the shooting distance. For example, in an example where the depth camera is a structured light camera, the highest accuracy of the structured light camera can reach the level of mm within a shooting range of 0.5-3m. However, within a shooting range of 3-5m, the accuracy of the structured light camera drops to tens of mm. When the shooting distance exceeds 5m, the accuracy of the structured light camera may fall below 0.5m. On the other hand, scanning means such as LIDAR systems usually have a significantly longer detection distance than depth cameras. For example, commercially available LIDAR systems have ranging distances of 10m, 30m, 100m, 300m, or even more. Furthermore, the accuracy of LIDAR systems within the detection range is known to be consistent and highly accurate. However, LIDAR systems typically cannot provide focused data points like depth cameras, and cannot provide details of small objects in a scene.

[0053] The present disclosure provides methods, systems, and apparatus for combining depth data from imaging and scanning means to ensure modeling of small and distant objects with consistent high accuracy and fine detail within a scene.

[0054] Referring back to block 310 of FIG. 3, the 3D modeling system is configured to acquire first depth data according to a predetermined resolution threshold. The imaging means of the 3D modeling system first captures multiple depth images of the scene while moving and rotating to different positions. The processor of the 3D modeling system processes the depth images captured by the imaging means and acquires a total depth image of the scene by combining pixels of the depth images. The processor of the 3D modeling system then identifies one or more image regions in the scene having a resolution below the predetermined resolution threshold and instructs the imaging means to capture supplemental depth images for the one or more image regions. Each image region may be included in a partial scene captured in a depth image. For example, in some variations, the imaging means may be instructed to capture a target number of depth images. The number of depth images captured in a partial scene may be determined based on the complexity of the partial scene. The more complex the partial scene, the more depth images are captured in the partial scene. By supplementing the initial depth image captured by the imaging means with a supplementary depth image for one or more image regions, the resolution of the one or more image regions can be improved above a predefined resolution threshold, so that the resulting first depth data including pixels of the depth image captured by the imaging means can provide sufficient depth data points for the image regions to meet the resolution requirement.

[0055] In some examples, the 3D modeling system may obtain the first depth data based on a predefined depth threshold according to a target accuracy. For example, the imaging means is used to accurately image data points / objects between a distance of 0.3 m and 2.3 m, and the maximum error of depth values ​​between a distance of 0.3 m and 2.3 m may be 0.5 cm. Thus, the depth threshold may be 2.3 m, and depth data values ​​indicating values ​​greater than 2.3 m are flagged by the 3D modeling system as possibly not meeting the target accuracy. Additionally and / or alternatively, a user may set the depth threshold based on how accurate he or she wants the image to be. In other words, the first depth data may be adjusted according to the predefined depth threshold. For example, each data point having a depth value greater than the depth threshold may be removed from the first depth data. The depth threshold may be determined based on the accuracy of the imaging means in the 3D modeling system. The accuracy of the imaging means is obtained by calibrating the imaging means or from related parameters provided by the manufacturer. Calibration may be performed by placing the imaging means at multiple shooting distances away from the target object and acquiring multiple depth images at each shooting distance. For example, the imaging means may have a shooting range of 0.3 m to 4 m, and the shooting distances may be set between 0.3 m and 4 m in predetermined increments (e.g., 0.1 m). The depth values ​​contained in the pixels of the depth images may be compared with the actual distance between the imaging means and the target object so that the accuracy of the imaging means at different shooting distances can be calibrated. For example, when the shooting distance is 0.3 m, the error of the depth value is 0.3 cm, when the shooting distance is 0.8 m, the error of the depth value is 0.2 cm, when the shooting distance is 1.3 m, the error of the depth value is 0.1 cm, when the shooting distance is 1.8 m, the error of the depth value is 0.3 cm, when the shooting distance is 2.3 m, the error of the depth value is 0.5 cm, when the shooting distance is 2.8 m, the error of the depth value is 1 cm, when the shooting distance is 3.3 m, the error of the depth value is 5 cm, and when the shooting distance is 3.8 m, the error of the depth value may be 10 cm. In this example, when the target accuracy is 0.5 cm, the depth threshold may be defined as 2.3 m.

[0056] The depth threshold may be changed during data processing depending on the accuracy and / or resolution requirements for generating the 3D model. For example, for higher accuracy requirements, a smaller depth threshold may be determined, resulting in fewer data points in the first depth data and sacrificing the resolution of the depth image. In another example, the larger the depth threshold, the more data points are included in the first depth data and the higher the resolution of the depth image. However, in this case, the accuracy requirement needs to be relaxed. Additionally and / or alternatively, the depth threshold may be applied to adjust the second depth data. For example, a depth data point in the second depth data may be removed if the depth data point has a depth value smaller than the depth threshold.

[0057] In some examples, the processor of the 3D modeling system may process the first depth data to reduce the number of depth data points in the first depth data by performing the data processing described in block 330 of FIG. 3, so as to improve the processing speed and reduce the computational complexity. The data processing includes at least one of downsampling and filtering. In some examples, the downsampling step may be performed by merging overlapping pixels of the depth image. For example, overlapping pixels related to the same shooting spot in the scene are merged into one depth data point by averaging the depth values ​​of the overlapping pixels. Such a merging step of multiple data points into one data point corresponds to averaging the results of repeated sampling, and can reduce errors caused by jittering of the data points according to a Gaussian distribution. Furthermore, a filtering step may be performed to remove unexpected depth data points such as tracks by applying a filtering algorithm. Similar data processing to reduce the number of data points may be applied to the second depth data acquired by the scanning means in the 3D modeling system, as described in block 320 of FIG. 3.

[0058] In some examples, the 3D modeling system may determine after block 330 that the generated 3D model includes one or more image regions that cannot reach a predetermined resolution threshold. For example, the one or more image regions may include objects that are not generated with enough data points in the scene. The processor of the 3D modeling system instructs the imaging means to capture supplemental depth images for the one or more image regions. In some examples, the imaging means moves and / or rotates to capture depth images from multiple shooting angles for each of the one or more image regions so that richer depth data can be obtained for the one or more image regions. The pixels of the supplemental depth images captured by the imaging means may form first filling data. The processor of the 3D modeling system may supplement the generated 3D model with the first filling data to improve the degree of refinement of local details in order to generate a 3D model of a new scene.

[0059] In some examples, the 3D modeling system can display the 3D model in real time based on the acquired depth data. The display of the 3D model is available during the aforementioned data acquisition steps such as blocks 310 and 320, data processing steps such as block 330, and / or the step of generating a new 3D model by capturing more depth images. Based on the real-time display of the 3D model, the user can determine whether the presented 3D model meets the accuracy, resolution, and / or completeness requirements. For example, the user can determine whether supplemental depth data is required and whether an imaging means or a scanning means in the 3D modeling system should be used to capture the supplemental depth data.

[0060] 5 is an exemplary process 500 for generating a 3D model of a scene according to one or more embodiments. Process 500 may be performed by the aforementioned 3D modeling system and / or device 200 according to computer-executable instructions stored in memory 230 shown in FIG. 2. However, it will be appreciated that process 500 may be performed in any suitable environment and any of the following blocks may be performed in any suitable order. Compared to process 300, process 500 may include an additional block that determines a number of data pairs and uses the data pairs to combine the first and second depth data from the imaging means and the scanning means.

[0061] In block 510, the 3D modeling system acquires first depth data. Block 510 may be similar to block 310 described above. Additionally and / or alternatively, an imaging means of the 3D modeling system may capture depth images at a first frame rate (e.g., the number of depth images per second captured by the imaging means). Each frame includes a number of pixels. The first depth data includes a number of frames acquired by the imaging means. Each frame of a depth image in the first set of depth data is defined as first data.

[0062] In block 520, the 3D modeling system acquires second depth data. Block 520 may be similar to block 320 described above. Additionally, the scanning means of the 3D modeling system may collect depth data points at a second frame rate. Each frame includes a plurality of depth data points collected by the scanning means, and the second frame rate indicates the number of data points collected by the scanning means per second. The second depth data includes a plurality of frames of depth data points collected by the scanning means. Each frame in the second depth data may be defined as second data.

[0063] In block 530, the 3D modeling system determines a plurality of data pairs, each of the plurality of data pairs including a first data and a corresponding second data. For example, the first data, which may be a frame of a depth image, is captured for an image region included in the sub-scene. The image region may be an object in the scene. The corresponding second data may be determined as a frame acquired by scanning the sub-scene including the same image region. Note that the sub-scene captured by the first data and the sub-scene captured by the second data may be similar or different.

[0064] In some examples, the imaging means and the scanning means in the 3D modeling system may be integrated on a platform, and the relative position of the imaging means to the scanning means may be fixed. In that example, the platform may further be integrated with an orientation sensor that provides extrinsic parameters indicative of orientation information such as the position and rotation information of the platform. A communication connection may be established between the platform, the imaging means, the scanning means, and the orientation sensor such that each frame of the depth image acquired by the imaging means, which is the first data, is tagged with the extrinsic parameters provided by the orientation sensor at the fixed time when the frame of the depth image was captured. The fixed time when the frame of the depth image was captured may be tagged by the imaging means as a timestamp associated with the frame of the depth image. Furthermore, the extrinsic parameters output from the orientation sensor may be tagged with a timestamp generated by the orientation sensor, the timestamp indicating the fixed time when the extrinsic parameters were generated by the orientation sensor. Based on the timestamp information, the 3D modeling system may determine the extrinsic parameters to be tagged to the frame of the depth image by a time interval between the timestamp of the extrinsic parameters and the timestamp of the captured image being smaller than a predetermined threshold. Similarly, each frame of depth data points acquired by the scanning means may be annotated with extrinsic parameters provided by the orientation sensor at the exact time the frame of depth data points was scanned. The frames of depth data points acquired by the scanning means are annotated with a timestamp indicating the exact time generated by the scanning means to generate the frame of depth data points. The 3D modeling system may determine, based on the timestamp information, the extrinsic parameters to be annotated with the frames of depth data points scanned by the scanning means when the time interval is less than a predefined threshold.

[0065] In some examples, the platform may be moved and rotated to different positions for data acquisition. Once the platform is moved and rotated to one of the positions for data acquisition, the imaging means may be instructed to capture one or more frames of depth images, the scanning means may be instructed to scan one or more frames of depth data points, and the orientation sensor may be instructed to output extrinsic parameters indicative of the current position and rotation of the platform. The one or more frames of depth images captured by the imaging means at this position may form first data. The first data is annotated with the extrinsic parameters output from the orientation sensor at this position. Additionally and / or alternatively, the one or more frames of depth data points scanned by the scanning means at this position may form second data. The second data is annotated with the extrinsic parameters output from the orientation sensor at this position. The 3D modeling system may determine that the first data acquired at this position corresponds to the second data acquired at the same position. By moving and rotating the platform and repeating the data acquisition process described above at different capture positions, multiple data pairs including the first data and the corresponding second data may be acquired. In some examples, the platform may be fixed at a position in the scene and rotated 360 degrees to perform data acquisition of the entire scene. In some examples, the platform may be hand-held or placed by a user, and data acquisition may be performed according to the user's movement. In some examples, the platform may be placed on a moving body such as a robot or an autonomous vehicle. It will be understood that the motion trajectory of the platform should not be limited in this disclosure. In some examples, the platform may be further integrated with a color imager. Similarly, the image captured by the color imager may be annotated with extrinsic parameters output from the orientation sensor.

[0066] The attitude sensor may be at least one of an inertial measurement unit (IMU), a simultaneous localization and mapping (SLAM) unit in a LIDAR system, and a visual inertial odometry (VIO) unit in a color camera. The platform in the 3D modeling system may include one or more attitude sensors. Each of the imaging means, the scanning means, and optionally the color imaging means may include a separate attitude sensor. The extrinsic parameters attached to the first data, the second data, and the image captured by the color imaging means may be a combination of extrinsic parameters output from multiple attitude sensors. In some examples, the extrinsic parameters are output from both the IMU and the SLAM unit. In some examples, the extrinsic parameters are output from both the IMU unit and the VIO unit. In some examples, the extrinsic parameters are output from the IMU, the SLAM unit, and the VIO unit. Alternatively, each of the extrinsic parameters of the imaging means and the scanning means may be obtained through calibration of the imaging means and the scanning means. Using the extrinsic parameters attached to the first data, the second data and the image captured by the color imaging means, the orientation of the data acquisition device on the same 3D coordinate plane can be calculated and determined.

[0067] In some examples, the 3D modeling system may determine the data pairs according to a preset time interval threshold. For example, the first data and the corresponding second data included in the data pairs may be acquired within a time interval below the preset time interval threshold. Based on the time interval between the first data and the second data being less than the threshold, the first data and the corresponding second data may include depth data points of the same partial scene. In this manner, the 3D modeling system may determine a plurality of data pairs depending on the current time interval threshold, each of the data pairs including the first data and the corresponding second data.

[0068] In some examples, the 3D modeling system may determine the data pairs based on keypoint detection performed on the first data and the second data. The 3D modeling system may identify the same image regions in the first data and the corresponding second data by performing keypoint detection on the first data and the second data. The identification of the image regions may be based on a number of feature points. The feature points may be predefined by a user. For example, in some variations, the feature points may be pixels identified in the image having a dramatic change in grayscale, i.e., intersections at edges, and / or pixels identified at the target object. The 3D modeling system may determine that the first data includes image regions that are also included in the second data to determine the data pairs depending on the keypoint detection results. Thus, the 3D modeling system may determine multiple data pairs.

[0069] At block 540, the 3D modeling system combines the first depth data and the second depth data to generate combined depth data based on the plurality of data pairs, specifically, for each data pair, combining the first data and the corresponding second data.

[0070] For example, the 3D modeling system may combine the first and second depth data using feature points / target objects, pose information from a pose sensor, and / or external parameters. For example, the 3D modeling system may determine first and second depth data that have the same or substantially the same pose information (e.g., translation and rotation vectors are substantially the same). The 3D modeling system may combine the first and second depth data with similar pose information to generate combined depth data. Additionally and / or alternatively, the 3D modeling system may determine one or more feature points and / or target objects shown through multiple images / frames. The 3D modeling system may determine first and second depth data associated with the feature points and / or target objects, and then combine the first and second depth data associated with the feature points and / or target objects to generate combined depth data. Additionally and / or alternatively, the 3D modeling system may use external parameters to generate combined depth data having the same coordinate plane (e.g., transposing the first and second depth data into the same coordinate plane and then combining the first and second depth data based on coordinates in the same coordinate plane).

[0071] In block 550, the 3D modeling system generates a 3D model of the scene based on the color data and the combined depth data. In the generated 3D model, each of the data points comprises a depth value and a color value. The combined depth data includes frames of depth images acquired by the imaging means or frames of depth data points scanned by the scanning means. Each frame includes a partial scene. In some examples, the partial scenes included in the frame can be combined according to a positional relationship between the partial scenes. The positional relationship between the partial scenes may be designed in advance. For example, a motion trajectory of a platform carrying the imaging means and the scanning means may be preset. The positional relationship between the partial scenes may be determined based on a preset motion trajectory of the platform. Similarly, the partial scenes in the color data can be combined based on a predetermined motion trajectory of the color imaging means used to acquire the color images forming the color data. Alternatively, the positional relationship between the partial scenes may be dynamically determined during the data acquisition process based on either external parameters to which the frames of acquired data are attached or on identified image areas in different frames.

[0072] In some examples, the depth values ​​of the data points in the 3D model are determined based on the combined depth data that is transposed to the same coordinate plane during the aforementioned data processing steps. Similarly, the color data may be transposed to the same coordinate plane based on external parameters that indicate the position and rotation of the color imaging means while capturing the color image. Thus, each of the data points in the 3D model is rendered according to the color value contained in the color data point that is determined to be at the same coordinate as the data point in the 3D model.

[0073] In some examples, the 3D modeling system determines depth and color values ​​of data points in the generated 3D model by aligning one or more identified image regions that are present in both the combined depth data and color data.

[0074] 6 is an exemplary process 600 for generating a 3D model of a scene according to one or more embodiments. The process 600 may be performed by the aforementioned 3D modeling system and / or device 200 according to computer-executable instructions stored in the memory 230 shown in FIG. 2. However, it will be appreciated that the process 600 may be performed in any suitable environment and any of the following blocks may be performed in any suitable order. Compared to the process 300, the process 600 may include an additional block of determining lack of resolution and / or incomplete data in the original data of the acquired first / second depth data, and then supplementing the original data of the acquired depth data with new depth data.

[0075] In block 610, the 3D modeling system obtains first depth data. Block 610 may be similar to block 310 described above.

[0076] In block 620, the 3D modeling system obtains second depth data. Block 620 may be similar to block 320 described above.

[0077] In block 630, the 3D modeling system generates a 3D model of the scene based on the color data, the first depth data, and the second depth data. Block 630 may be similar to block 330 described above.

[0078] In block 640, in response to determining that one or more image regions do not have enough data points in the 3D model of the scene (e.g., based on a portion of the scene that was not modeled and / or low resolution and / or missing / incomplete data), the processor of the 3D modeling system instructs the scanning device to acquire second fill data. The processor of the 3D modeling system instructs the scanning means to scan the one or more image regions for fill depth data points. In some examples, the scanning means is moved and / or rotated to scan the sub-scene that includes the one or more image regions. The depth data points acquired by scanning the one or more image regions may form the second fill data.

[0079] In block 650, the processor of the 3D modeling system supplements the generated 3D model of the scene with second fill data to generate a new 3D model of the scene to improve the completeness of the generated 3D model of the scene.

[0080] The above disclosed steps may be performed by a 3D modeling system 700, as shown in Figure 7, according to one or more embodiments. The 3D modeling system 700 includes multiple components, such as an imaging means 710, a scanning means 720, one or more orientation sensors 730, one or more 3D modeling processors 740, a memory 750, and a display 760.

[0081] The imager 710 may be configured to capture a number of depth images of the scene. The imager 710 may be set to capture the depth images of the scene at a first frame rate. Each depth image includes a number of pixels, the pixels including depth values ​​indicative of distances between the imager and a number of capture spots in the scene. Additionally, each depth image may be time-stamped to indicate the exact time the depth image was captured. The imager 710 outputs the captured depth images of the scene to the 3D modeling processor 740 as first depth data or first fill data.

[0082] The scanning means 720 may be configured to scan a plurality of depth data points of the scene. The scanning means 720 may be set to acquire depth data points of the scene at a second frame rate. Each frame includes a plurality of depth data points, the depth data points including depth values ​​indicating distances between the scanning means 720 and a plurality of spots in the scene. Additionally, each frame of depth data points may be time-stamped to indicate the exact time the frame was acquired. The scanning means 720 outputs the acquired depth data points to the 3D modeling processor 740 as second depth data or second fill data.

[0083] The orientation sensor 730 may be configured to determine and / or output extrinsic parameters for the acquired depth data. The extrinsic parameters include orientation information such as position and rotation of the means in which the orientation sensor 730 is embedded. The extrinsic parameters generated by the orientation sensor 730 may be time-stamped indicating the time at which the extrinsic parameters were generated. The data acquisition means such as the imaging means 710 and the scanning means 720 may each be incorporated with an orientation sensor 730. Alternatively, one orientation sensor 730 may output extrinsic parameters that are subsequently determined for the acquired frames of depth data. The orientation sensor 730 may output the time-stamped extrinsic parameters to the 3D modeling processor 740 so that the 3D modeling processor 740 can determine the extrinsic parameters attached to the acquired frames of data based on the time interval indicated by the time stamp.

[0084] The 3D modeling processor 740 may be configured to perform the aforementioned steps based on computer-executable instructions stored in the memory 750. In some examples, the 3D modeling processor 740 receives color data from an external color imager. In some examples, the 3D modeling processor 740 receives color data from a color imaging means in the 3D modeling system 700.

[0085] The display 760 may be configured to display the 3D model generated by the 3D modeling process 740. In some examples, the display 760 may be further configured to display a real-time 3D model of the scene during the data acquisition process.

[0086] The techniques described herein may be embodied in executable instructions stored on a computer-readable medium for use by or in connection with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will appreciate that various types of computer-readable media may be included for storing data in some embodiments. As used herein, a "computer-readable medium" includes one or more of any suitable medium for storing executable instructions of a computer program such that an instruction execution machine, system, apparatus, or device can read (or fetch) the instructions from the computer-readable medium and execute the instructions to implement the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional example computer-readable media includes portable computer diskettes, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory devices, and optical storage devices such as portable compact disks (CDs), portable digital video disks (DVDs), and the like.

[0087] It should be understood that the arrangement of components depicted in the accompanying figures is illustrative, and that other arrangements are possible. For example, one or more of the elements described herein may be realized in whole or in part as electronic hardware components. Other elements may be implemented in software, hardware, or a combination of software and hardware. Furthermore, some or all of these other elements may be combined, some may be omitted entirely, and additional components may be added while still achieving the functionality described herein. Thus, the subject matter described herein may be embodied in many different variations, and all such variations are believed to be within the scope of the claims.

[0088] To facilitate understanding of the subject matter described herein, many aspects are described in terms of a sequence of operations. Those skilled in the art will recognize that various operations may be performed by specialized circuits or circuits, by program instructions executed by one or more processors, or by a combination of both. The description of a sequence of operations herein is not intended to imply that the particular order described must be followed to perform the sequence of operations. All methods described herein may be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context.

[0089] The use of the terms "a," "an," and "said," and similar references in the context of describing the subject matter (particularly in the context of the claims below) shall be interpreted to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context. The use of the term "at least one" followed by a list of one or more items (e.g., "at least one of A and B") shall be interpreted to mean one item selected from the listed items (A or B), or any combination of two or more of the listed items (A and B), unless otherwise indicated herein or clearly contradicted by the context. Moreover, the foregoing description is for illustrative purposes only, and not for limiting purposes, since the scope of protection sought is defined by the claims set forth below, together with their equivalents. The use of any and all examples or example phrases (e.g., "such as") provided herein is intended merely to better describe the subject matter, and does not pose any limitation on the scope of the subject matter, unless otherwise claimed. In both the claims and the written description, the use of the term "based on" and other similar phrases indicating conditions for producing a certain result is not intended to exclude other conditions for producing that result. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the claimed invention.

Claims

1. 1. A method for generating a 3D model of a scene, comprising: a 3D modeling system including an imaging means acquiring first depth data including pixels of a plurality of frames of depth images; said 3D modeling system including a scanning means acquiring second depth data including depth data points of a plurality of frames of images; determining a depth threshold; removing one or more data points in the first depth data in response to a depth value of the one or more data points in the first depth data being greater than the depth threshold; receiving, by the 3D modeling system, color data for the scene including a plurality of color image pixels; generating a 3D model of the scene based on the color data, the first depth data, and the second depth data, displaying a 3D model of the scene.

2. each of the depth images in the first depth data is first data and each of the image frames in the second depth data is second data; The method further comprises: determining a plurality of data pairs, each of the plurality of data pairs including the first data and the corresponding second data, the first data and the corresponding second data including similar target objects; determining a positional relationship between the first depth data and the second depth data based on the plurality of data pairs. The method of claim 1.

3. Each of the first data is associated with a first external parameter indicating posture information of the imaging means, and each of the second data is associated with a second external parameter indicating posture information of the scanning means, the posture information indicating position and rotation information, and determining the plurality of data pairs includes: determining a first pose of the 3D modeling system associated with the first datum based on the first extrinsic parameters; determining a second pose of the 3D modeling system relative to the second data based on the second extrinsic parameters; and determining a data pair including the first data and the second data in response to the first orientation and the second orientation being the same; The method of claim 2.

4. the first external parameter and the second external parameter are output from a same attitude sensor in the 3D modeling system; The method according to claim 3.

5. a first time stamp indicating a fixed time when the first data was acquired by the imaging means is associated with each of the first data, a second time stamp indicating a fixed time when the second data was acquired by the scanning means is associated with each of the second data, and the first data and the corresponding second data in the data pair have a time interval smaller than a threshold value. The method of claim 2.

6. the first data and the corresponding second data in the data pair are identified with one or more similar objects; The method of claim 2.

7. identifying one or more image regions in the generated 3D model of the scene that are below a resolution threshold; said 3D modeling system including said imaging means acquiring first fill data including pixels of a plurality of depth images; supplementing the generated 3D model of the scene with the first fill data to generate a new 3D model of the scene. The method of claim 1.

8. identifying one or more image regions in the generated 3D model of the scene that do not have enough of the depth data points; said 3D modeling system including said scanning means acquiring second fill data including depth data points of a plurality of image frames; supplementing the generated 3D model of the scene with second fill data to generate a new 3D model of the scene. The method of claim 1.

9. the imaging means acquiring the plurality of depth images at a first frame rate, and the scanning means acquiring the plurality of image frames at a second frame rate. The method of claim 1.

10. the 3D modeling system including a display for displaying in real time a 3D model of the scene based on the acquired depth data; The method of claim 1.

11. 1. A system for generating a 3D model of a scene, comprising: an imaging means configured to acquire first depth data comprising pixels of a depth image of a plurality of frames; a scanning means configured to acquire second depth data comprising depth data points of a plurality of image frames; one or more processors; The one or more processors: acquiring first depth data from said imaging means, second depth data from said scanning means, and color data including a plurality of color image pixels; determining a depth threshold; removing one or more data points in the first depth data in response to a depth value of the one or more data points in the first depth data being greater than the depth threshold; generating a 3D model of the scene based on the color data, the first depth data and the second depth data; outputting the generated 3D model of the scene. system.

12. each of the depth images in the first depth data is first data, and each of the image frames in the second depth data is second data; The one or more processors further include: determining a plurality of data pairs, each of the plurality of data pairs including the first data and the corresponding second data, the first data and the corresponding second data including similar target objects; and configured to determine a positional relationship between the first depth data and the second depth data based on the plurality of data pairs. The system of claim 11.

13. The system further comprises one or more attitude sensors configured to output external parameters indicative of attitude information of the imaging means and the scanning means in the system; a first external parameter indicating posture information of the imaging means is associated with each of the first data, and a second external parameter indicating posture information of the scanning means is associated with each of the second data, The attitude information indicates position and rotation information, The one or more processors further include: determining a first attitude of the system associated with the first data based on the first external parameters; determining a second attitude of the system associated with the second data based on the second external parameters; and determining a data pair including the first data and the second data in response to the first orientation and the second orientation being the same. The system of claim 12.

14. A first time stamp indicating a fixed time when the first data was acquired by the imaging means is associated with each of the first data, and a second time stamp indicating a fixed time when the second data was acquired by the scanning means is associated with each of the second data, the first data and the corresponding second data in the data pair have a time interval that is less than a threshold value; The system of claim 12.

15. the first data and the corresponding second data in the data pair are identified with one or more similar objects; The system of claim 12.

16. The one or more processors further include: identifying one or more image regions in the generated 3D model of the scene that are below a resolution threshold; receiving first fill data comprising pixels of a plurality of depth images from said imaging means; - configured to fill the generated 3D model of the scene with the first filling data to generate a new 3D model of the scene. The system of claim 15.

17. The one or more processors further include: identifying one or more image regions in the generated 3D model of the scene that do not have sufficient depth data points; receiving second fill data from said scanning means, said fill data including depth data points for a plurality of image frames; - configured to fill the generated 3D model of the scene with the second filling data to generate a new 3D model of the scene. The system of claim 12.

18. A non-volatile computer readable medium having computer executable instructions stored thereon, comprising: The computer-executable instructions, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 10. Computer-readable medium.

19. 1. A computer program for execution by a 3D modeling system having an imaging device, a scanning device, and one or more processors, comprising: The computer program, when executed by the one or more processors, causes the 3D modeling system to perform the method of any one of claims 1 to 10. Computer program.

Images