System and method for capturing and generating panoramic three-dimensional images

Abstract

The invention discloses a system and a method for capturing and generating panoramic three-dimensional images. An apparatus includes a housing, a mount configured to be coupled to a motor to horizontally move the apparatus, a wide-angle lens coupled to the housing, the wide-angle lens positioned above the mount along an axis of rotation, an image capture device coupled to the wide-angle lens, and a LiDAR device coupled to the mount. The apparatus includes a housing having an axis of rotation, the axis of rotation being an axis along which the apparatus rotates, an image capture device within the housing, the image capture device configured to receive a two-dimensional image of an environment through the wide-angle lens, and a LiDAR device within the housing, the LiDAR device configured to generate depth data based on the environment.

Description

[0001] This application is a divisional application of Chinese patent application 202080084506.9 (PCT / US2020 / 067474) filed on December 30, 2020 and entitled “System and method for capturing and generating panoramic three-dimensional images”. Technical Field

[0002] Embodiments of the present invention generally relate to capturing and stitching panoramic images of a scene in a physical environment. Background Art

[0003] The popularity of providing three-dimensional (3D) panoramic images of the physical world has given rise to many solutions with the ability to capture multiple two-dimensional (2D) images and generate 3D images based on the captured 2D images. There are hardware solutions and software applications (or "apps") that are able to capture multiple 2D images and stitch them into a panoramic image.

[0004] Technology exists for capturing and generating 3D data from buildings. However, existing technology is often unable to capture and generate 3D renderings of brightly lit areas. Windows where sunlight shines through or areas of floors or walls in bright light often appear as holes in the 3D rendering, which may require additional post-production work to fill. This increases the turnaround time and realism of the 3D rendering. In addition, outdoor environments also present challenges for many existing 3D capture devices because structured light may not be used to capture 3D images.

[0005] Other limitations of existing techniques for capturing and generating 3D data include the amount of time required to capture and process the digital images needed to produce a 3D panoramic image. Summary of the invention

[0006] An example device includes a housing, a mount, a wide angle lens, an image capture device, and a LiDAR device, wherein the mount is configured to be coupled to a motor to horizontally move the device, the wide angle lens is coupled to the housing, the wide angle lens is positioned above the mount so as to be along a rotation axis, the rotation axis being the axis along which the device rotates when coupled to the motor, the image capture device is within the housing, the image capture device is configured to receive a two-dimensional image of an environment through the wide angle lens, and the LiDAR device is within the housing, the LiDAR device is configured to generate depth data based on the environment.

[0007] An image capture device may include a housing, a first motor, a wide angle lens, an image sensor, a mount, a LiDAR, a second motor, and a reflector. The housing may have a front side and a rear side. The first motor may be coupled to the housing at a first position between the front side and the rear side of the housing, the first motor being configured to rotate the image capture device horizontally substantially 270 degrees around a vertical axis. The wide angle lens may be coupled to the housing at a second position between the front side and the rear side of the housing along the vertical axis, the second position being a parallax-free point, the wide angle lens having a field of view away from the front side of the housing. The image sensor may be coupled to the housing and configured to generate an image signal from light received by the wide angle lens. The mount may be coupled to the first motor. The LiDAR may be coupled to the housing at a third position, the LiDAR being configured to generate laser pulses and generate a depth signal. The second motor may be coupled to the housing. The reflector may be coupled to the second motor, which may be configured to rotate the reflector about a horizontal axis, the reflector comprising an angled surface configured to receive the laser pulses from the LiDAR and direct the laser pulses about the horizontal axis.

[0008] In some embodiments, the image sensor is configured to generate a first plurality of images at different exposures when the image capture device is stationary and pointing in a first direction. The first motor may be configured to rotate the image capture device around the vertical axis after generating the first plurality of images. In various embodiments, the image sensor does not generate an image when the first motor rotates the image capture device, and wherein the LiDAR generates a depth signal based on the laser pulses when the first motor rotates the image capture device. The image sensor may be configured to generate a second plurality of images at the different exposures when the image capture device is stationary and pointing in a second direction, and the first motor is configured to rotate the image capture device 90 degrees around the vertical axis after generating the second plurality of images. The image sensor may be configured to generate a third plurality of images at the different exposures when the image capture device is stationary and pointing in a third direction, and the first motor is configured to rotate the image capture device 90 degrees around the vertical axis after generating the third plurality of images. The image sensor may be configured to generate a fourth plurality of images at the different exposures when the image capture device is stationary and pointed in a fourth direction, and the first motor may be configured to rotate the image capture device 90 degrees about the vertical axis after generating the fourth plurality of images.

[0009] In some embodiments, the system may further include a processor configured to blend frames of the first plurality of images before the image sensor generates the second plurality of images. A remote digital device may communicate with the image capture device and be configured to generate a 3D visualization based on the first, second, third, and fourth plurality of images and the depth signal, the remote digital device being configured to generate the 3D visualization using no more than the first, second, third, and fourth plurality of images. In some embodiments, the first, second, third, and fourth plurality of images are generated between rotations that combine a rotation that rotates the image capture device 270 degrees around the vertical axis. The speed or rotation of the reflector around the horizontal axis increases as the first motor rotates the image capture device. The angled surface of the reflector may be 90 degrees. In some embodiments, the LiDAR emits the laser pulses in a direction opposite to the front side of the housing.

[0010] An example method includes receiving light from a wide angle lens of an image capture device, the wide angle lens coupled to a housing of the image capture device, the light being received at a field of view of the wide angle lens, the field of view extending away from a front side of the housing, generating a first plurality of images by an image sensor of the image capture device using the light from the wide angle lens, the image sensor coupled to the housing, the first plurality of images being at different exposures, rotating the image capture device horizontally about a vertical axis substantially 270 degrees by a first motor, the first motor coupled to the housing in a first position between the front side and the back side of the housing, the wide angle lens being at a second position along the vertical axis, the second position being a zero-parallax point, rotating a mirror having an angled surface about a horizontal axis by a second motor, the second motor coupled to the housing, generating laser pulses by a LiDAR, the LiDAR coupled to the housing at a third position, the laser pulses being directed to the rotating mirror when the image capture device is rotated horizontally, and generating a depth signal by the LiDAR based on the laser pulses.

[0011] Generating the first plurality of images by the image sensor may occur before the image capture device is rotated horizontally. In some embodiments, the image sensor does not generate images when the first motor rotates the image capture device, and wherein the LiDAR generates the depth signal based on the laser pulses when the first motor rotates the image capture device.

[0012] The method may further include generating, by the image sensor, a second plurality of images with the different exposures while the image capture device is stationary and pointing in a second direction, and rotating, by the first motor, the image capture device 90 degrees about the vertical axis after generating the second plurality of images.

[0013] In some embodiments, the method may further include generating, by the image sensor, a third plurality of images at the different exposures when the image capture device is stationary and pointing in a third direction, and rotating the image capture device 90 degrees around the vertical axis by the first motor after generating the third plurality of images. The method may further include generating, by the image sensor, a fourth plurality of images at the different exposures when the image capture device is stationary and pointing in a fourth direction. The method may include generating a 3D visualization using the first, second, third, and fourth plurality of images and based on the depth signal, the generating the 3D visualization not using any other images.

[0014] In some embodiments, the method may further include blending frames of the first plurality of images before the image sensor generates the second plurality of images. The first, second, third, and fourth pluralities of images may be generated between rotations that combine a rotation of the image capture device 270 degrees around the vertical axis. In some embodiments, the speed or rotation of the mirror around the horizontal axis increases as the first motor rotates the image capture device. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG. 1 a depicts a dollhouse view of an example environment, such as a house, in accordance with some embodiments.

[0016] FIG. 1 b depicts a plan view of a first floor of a house, according to some embodiments.

[0017] Figure 2 Depicted is an example eye-level view of a living room that may be part of a virtual tour.

[0018] Figure 3 An example of an environment capture system in accordance with some embodiments is depicted.

[0019] Figure 4 Depicted is a perspective view of an environmental capture system in some embodiments.

[0020] Figure 5 is a depiction of laser pulses from a LiDAR surrounding an environment capture system in some embodiments.

[0021] Fig. 6A A side view of the environmental capture system is depicted.

[0022] Figure 6B Depicted is a view of capturing a system from above in some embodiments.

[0023] Figure 7 Depicted is a perspective view of components of one example of an environment capture system in accordance with some embodiments.

[0024] Fig. 8A Example lens sizes in some embodiments are depicted.

[0025] Figure 8B Depicted are example lens design specifications in some embodiments.

[0026] 9A depicts a block diagram of an example of an environment capture system in accordance with some embodiments.

[0027] 9B depicts a block diagram of an example SOM PCBA of an environment capture system in accordance with some embodiments.

[0028] Figure 10a-10c Depicted is a process for an environment capture system in some embodiments for capturing images.

[0029] Fig.11 Depicted is a block diagram of an example environment capable of capturing and stitching images to form a 3D visualization in accordance with some embodiments.

[0030] Fig.12 is a block diagram of an example of an alignment and stitching system according to some embodiments.

[0031] Fig.13 Depicted is a flow diagram of a 3D panoramic image capture and generation process in accordance with some embodiments.

[0032] Fig.14 Depicted is a flow diagram of a 3D and panoramic capture and stitching process in accordance with some embodiments.

[0033] Fig.15 Depicted showing Fig.14 A flow chart of further details of one step of the 3D and panoramic capture and stitching process.

[0034] Fig.16 Depicted is a block diagram of an example digital device in accordance with some embodiments. DETAILED DESCRIPTION

[0035] Many of the innovations described herein are made with reference to the accompanying drawings. The same reference numerals are used to refer to the same elements. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding. However, it is apparent that different innovations can be practiced without these specific details. In other examples, well-known structures and components are shown in block diagram form to facilitate the description of the innovation.

[0036] Various embodiments of the device provide users with 3D panoramic images of indoor and outdoor environments. In some embodiments, the device can effectively and quickly provide users with 3D panoramic images of indoor and outdoor environments using a single wide field of view (FOV) lens and a single light detection and ranging sensor (LiDAR sensor).

[0037] The following is an example use case of the example device described herein. The following use case is one of the embodiments. Different embodiments of the device as discussed herein may include one or more features and capabilities similar to those of this use case.

[0038] FIG. 1a depicts a toy house view 100 of an example environment (e.g., a house) according to some embodiments. The toy house view 100 gives an overall view of the example environment captured by the environment capture system (discussed herein). A user can interact with the toy house view 100 on a user system by switching between different views of the example environment. For example, a user can interact with area 110 to trigger a floor plan view of the first floor of a house, as shown in FIG. 1b. In some embodiments, a user can interact with icons (such as icons 120, 130, and 140) in the toy house view 100 to provide a walkthrough view (e.g., for 3D walkthrough), a floor plan view, or a measurement view, respectively.

[0039] FIG. 1b depicts a floor plan view of a first floor of a house according to some embodiments. The floor plan view is a top view of the first floor of the house. A user can interact with an area of ​​the floor plan view, such as area 150, to trigger an eye level view of a particular portion of the floor plan, such as a living room. An example of an eye level view of a living room can be seen in FIG. Figure 2 which can be part of a virtual tour.

[0040] The user can interact with a portion of floor plan 200 corresponding to area 150 of FIG. 1b. The user can move the view around the room as if the user were actually in the living room. In addition to the horizontal 360° view of the living room, the user can also view or navigate the floor or ceiling of the living room. In addition, the user can pass through the living room to other parts of the house by interacting with specific areas of that portion of floor plan 200, such as areas 210 and 220. When the user interacts with area 220, the environment capture system can provide a walkable transition between an area of ​​the house that substantially corresponds to the block of the house depicted by area 150 to an area of ​​the house that substantially corresponds to the block of the house depicted by area 220.

[0041] Figure 3 An example of an environment capture system 300 is depicted in accordance with some embodiments. The environment capture system 300 includes a lens 310, a housing 320, a mounting attachment 330, and a removable cover 340.

[0042] When in use, the environment capture system 300 can be positioned in an environment such as a room. The environment capture system 300 can be positioned on a support (e.g., a tripod). The removable cover 340 can be moved to reveal the LiDAR and the rotatable reflector. Once activated, the environment capture system 300 can take a burst of images and then rotate using a motor. The environment capture system 300 can open the mounting attachment 330. While rotating, the LiDAR can take measurements (while rotating, the environment capture system may not take images). Once pointed in a new direction, the environment capture system can take another burst of images before rotating to the next direction.

[0043] For example, once positioned, the user can command the environment capture system 300 to start scanning. The scanning can be as follows:

[0044] (1) Exposure estimation and then capturing HDR RGB images

[0045] Rotate 90 degrees to capture depth data

[0046] (2) Exposure estimation and then capturing HDR RGB images

[0047] Rotate 90 degrees to capture depth data

[0048] (3) Exposure estimation, and then capturing HDR RGB images

[0049] Rotate 90 degrees to capture depth data

[0050] (4) Exposure estimation, and then capturing HDR RGB images

[0051] Capture depth data by rotating 90 degrees (360 degrees total)

[0052] For each burst, there can be any number of images at different exposures. The environment capture system can blend together any number of images from a burst while waiting for another frame and / or waiting for the next burst.

[0053] The housing 320 can protect the electronic components of the environment capture system 300 and can provide an interface for the user to interact with a power button, a scan button, etc. For example, the housing 320 can include a removable cover 340, which can be removable to expose the LiDAR. In addition, the housing 320 can include electronic interfaces such as a power adapter and an indicator light. In some embodiments, the housing 320 is a molded plastic housing. In various embodiments, the housing 320 is a combination of one or more of plastic, metal, and polymer.

[0054] Lens 310 may be part of a lens assembly. Further details of the lens assembly may be found at Figure 7 . The lens 310 is strategically placed at the center of the rotation axis 305 of the environment capture system 300. In this example, the rotation axis 305 is in the xy plane. By placing the lens 310 at the center of the rotation axis 305, the parallax effect can be eliminated or reduced. Parallax is an error caused by the rotation of the image capture device around a point that is not a non-parallax point (NPP). In this example, the NPP can be found at the center of the entrance pupil of the lens.

[0055] For example, assume that a panoramic image of a physical environment is generated based on four images captured by the environment capture system 300, where the images of the panoramic image have a 25% overlap. If there is no parallax, then 25% of one image may completely overlap with another image of the same area of ​​the physical environment. Eliminating or reducing the parallax effect of multiple images captured by the image sensor through the lens 310 can help stitch the multiple images into a 2D panoramic image.

[0056] The lens 310 may include a large field of view (eg, the lens 310 may be a fisheye lens). In some embodiments, the lens may have a horizontal FOV (HFOV) of at least 148 degrees and a vertical FOV (VFOV) of at least 94 degrees.

[0057] Mounting attachment 330 may allow environment capture system 300 to be attached to a mount. The mount may allow environment capture system 300 to be coupled to a tripod, a flat surface, or a motorized mount (e.g., to move environment capture system 300). In some embodiments, the mount may allow environment capture system 300 to rotate along a horizontal axis.

[0058] In some embodiments, the environment capture system 300 may include a motor for horizontally rotating the environment capture system 300 about the mounting attachment 330 .

[0059] In some embodiments, the motorized mount can move the environment capture system 300 along a horizontal axis, a vertical axis, or both. In some embodiments, the motorized mount can rotate or move in the xy plane. The use of the mounting attachment 330 can allow the environment capture system 300 to be coupled to a motorized mount, a tripod, etc. to stabilize the environment capture system 300 to reduce or minimize shaking. In another example, the mounting attachment 330 can be coupled to a motorized mount that allows the 3D and environment capture system 300 to rotate at a stable known speed, which helps the LiDAR determine the (x, y, z) coordinates of each laser pulse of the LiDAR.

[0060] Figure 4 A rendering of an environment capture system 400 in some embodiments is depicted. The rendering shows the environment capture system 400 (which may be a front view 410, a top view 420, a side view 430, and a rear view 440) from various views. Figure 3 4.) In these renderings, environment capture system 400 may include an optional hollow portion depicted in side view 430.

[0061] In some embodiments, the environment capture system 400 has a width of 75 mm, a height of 180 mm, and a depth of 189 mm. It should be understood that the environment capture system 400 can have any width, height, or depth. In various embodiments, regardless of the specific measurements, the ratio of width to height to depth in the first example is maintained.

[0062] The housing of the 3D and environment capture system 400 can protect the electronic components of the environment capture system 400 and can provide an interface / interface for user interaction (e.g., a screen on the rear view 440). In addition, the housing can include electronic interfaces such as a power adapter and indicator lights. In some embodiments, the housing is a molded plastic housing. In various embodiments, the housing is a combination of one or more of plastic, metal, and polymer. The environment capture system 400 may include a removable cover that can be removable to expose the LiDAR and protect the LiDAR from the elements when not in use.

[0063] The lens depicted on the front view 410 can be part of a lens assembly. Similar to the environmental capture system 300, the lens of the environmental capture system 400 is strategically placed at the center of the rotation axis. The lens can include a large field of view. In various embodiments, the lens depicted on the front view 410 is concave and the housing is flared so that the wide angle lens is directly at the zero parallax point (e.g., directly above the midpoint of the mount and / or motor), but the image can still be captured without interference from the housing.

[0064] A mounting attachment at the base of environment capture system 400 may allow the environment capture system to be attached to a mount. The mount may allow environment capture system 400 to be coupled to a tripod, a flat surface, or a motorized mount (e.g., to move environment capture system 400). In some embodiments, the mount may be coupled to an internal motor for rotating environment capture system 400 about the mount.

[0065] In some embodiments, the mount may allow the environment capture system 400 to rotate along a horizontal axis. In various embodiments, the motorized mount may move the environment capture system 400 along a horizontal axis, a vertical axis, or both. The use of a mounting attachment may allow the environment capture system 400 to be coupled to a motorized mount, a tripod, or the like to stabilize the environment capture system 400 to reduce or minimize shaking. In another example, the mounting attachment may be coupled to a motorized mount that allows the environment capture system 400 to rotate at a stable known speed, which aids the LiDAR in determining the (x, y, z) coordinates of each laser pulse of the LiDAR.

[0066] In view 430, a reflector 450 is revealed. The LiDAR can transmit laser pulses to the reflector (in the direction opposite to the lens view). The laser pulses can hit the reflector 450, which can be angled (e.g., at a 90 degree angle). The reflector 450 can be coupled to an internal motor that rotates the reflector so that the LiDAR's laser pulses can be transmitted and / or received at many different angles around the environment capture system 400.

[0067] Figure 5 is a depiction of laser pulses from a LiDAR around an environment capture system 400 in some embodiments. In this example, the laser pulses are emitted at a rotating mirror 450. The laser pulses may be emitted and received perpendicular to a horizontal axis 602 (see FIG6 ) of the environment capture system 400. The mirror 450 may be angled so that the laser pulses from the LiDAR are directed away from the environment capture system 400. In some examples, the angle of the angled surface of the mirror may be 90 degrees, or at or between 60 and 120 degrees.

[0068] In some embodiments, when the environment capture system 400 is stationary and in operation, the environment capture system 400 can take a burst of images through the lens. The environment capture system 400 can turn on the horizontal motor between the bursts of images. When rotating along the mount, the LiDAR of the environment capture system 400 can emit and / or receive laser pulses that strike the rotating mirror 450. The LiDAR can generate a depth signal and / or generate depth data from the reflections of the received laser pulses.

[0069] In some embodiments, the depth data may be associated with coordinates with respect to the environment capture system 400. Similarly, pixels or portions of an image may be associated with coordinates with respect to the environment capture system 400 to enable the generation of 3D visualizations (e.g., images from different directions, 3D walkthroughs, etc.) to be generated using the image and depth data.

[0070] like Figure 5 As shown, the LiDAR pulses may be blocked by the bottom of the environment capture system 400. It should be appreciated that the mirror 450 may rotate in unison while the environment capture system 400 moves about the mount, or the mirror 450 may rotate more slowly when the environment capture system 400 begins to move and further when the environment capture system 400 slows to a stop (e.g., maintaining a constant speed between starting and stopping of the mount motor).

[0071] The LiDAR may receive depth data from the pulses. Due to movement of the environment capture system 400 and / or increase or decrease in speed of the mirror 450, the density of the depth data about the environment capture system 400 may be inconsistent (e.g., denser in some areas and less dense in other areas).

[0072] FIG6a depicts a side view of the environment capture system 400. In this view, the mirror 450 is depicted, and the mirror 450 can rotate about a horizontal axis. Pulses 604 can be emitted by the LiDAR at the rotating mirror 450, and can be emitted perpendicular to the horizontal axis 602. Similarly, pulses 604 can be received by the LiDAR in a similar manner.

[0073] Although the LiDAR pulses are discussed as being perpendicular to the horizontal axis 602, it is understood that the LiDAR pulses may be at any angle relative to the horizontal axis 602 (e.g., the reflector angle may be at any angle between 60 degrees and 120 degrees, inclusive). In various embodiments, the LiDAR transmits pulses relative to the front side (e.g., the front side 604) of the environment capture system 400 (e.g., in a direction opposite to the center of the field of view of the lens or toward the back side 606).

[0074] As discussed herein, the environment capture system 400 can rotate about the vertical axis 608. In various embodiments, the environment capture system 400 captures images and then rotates 90 degrees, such that a fourth set of images is captured when the environment capture system 400 completes a 270 degree rotation from the original starting position where the first set of images was captured. Thus, the environment capture system 400 can generate four sets of images between a total of 270 degrees of rotation (e.g., assuming the first set of images were captured prior to the initial rotation of the environment capture system 400). In various embodiments, images (e.g., four sets of images) from a single sweep of the environment capture system 400 (e.g., captured in a single full rotation or 270 degree rotation around the vertical axis) together with depth data acquired during the same sweep are sufficient to generate a 3D visualization without any additional sweeps or rotations of the environment capture system 400.

[0075] It should be understood that in this example, the LiDAR pulses are emitted and directed by a rotating mirror in a position away from the rotation point of the environment capture system 400. In this example, the distance from the rotation point of the mount is 608 (e.g., the lens can be at a zero parallax point, while the lens can be in a position behind the lens relative to the front of the environment capture system 400). Because the LiDAR pulses are directed by the mirror 450 at a position offset from the rotation point, the LiDAR cannot receive depth data from a cylinder extending from above the environment capture system 400 to below the environment capture system 400. In this example, the radius of the cylinder (e.g., a cylinder with no depth information) can be measured from the center of the rotation point of the motor mount to the point where the mirror 450 directs the LiDAR pulses.

[0076] Additionally, in FIG6b , a cavity 610 is depicted. In this example, the environment capture system 400 includes a rotating mirror within the body of the housing of the environment capture system 400. There is a cutout section from the housing. The laser pulses may be reflected out of the housing by the mirror, and then the reflections may be received by the mirror and directed back to the LiDAR to enable the LiDAR to generate a depth signal and / or depth data. The base of the body of the environment capture system 400 below the cavity 610 may block some of the laser pulses. The cavity 610 may be defined by the base of the environment capture system 400 and the rotating mirror. As shown in FIG6b , the environment capture system 400 may be formed by a rotating mirror. Figure 6B As shown, there may still be space between the edge of the angled reflector and the housing of the environment capture system 400 containing the LiDAR.

[0077] In various embodiments, the LiDAR is configured to stop emitting laser pulses if the rotation speed of the mirror drops below a rotation safety threshold (e.g., if there is a failure of the motor that rotates the mirror or the mirror is held in place.) In this way, the LiDAR can be configured for safety and reduce the likelihood that laser pulses will continue to be emitted in the same direction (e.g., at the user's eyes).

[0078] FIG6b depicts a view from above the environment capture system 400 in some embodiments. In this example, the front of the environment capture system 400 is depicted, where the lens is concave and directly above the center of the rotation point (e.g., above the center of the mount). The front of the camera is concave for the lens, and the front of the housing is flared to allow the field of view of the image sensor to not be blocked by the housing. The reflector 450 is depicted as pointing upward.

[0079] Figure 7 A perspective view of components of one example of an environment capture system 300 is depicted in accordance with some embodiments. Environment capture system 700 includes front cover 702, lens assembly 704, structural frame 706, LiDAR 708, front housing 710, mirror assembly 712, GPS antenna 714, rear housing 716, vertical motor 718, display 720, battery pack 722, mount 724, and horizontal motor 726.

[0080] In various embodiments, the environment capture system 700 can be configured to scan, align, and generate 3D meshes outdoors as well as indoors in full sunlight. This eliminates the barriers of other systems that are only indoor tools. The environment capture system 700 can be able to scan large spaces faster than other devices. In some embodiments, the environment capture system 700 can provide improved depth accuracy by improving the single scan depth accuracy at 90m.

[0081] In some embodiments, the environment capture system 700 may weigh at or about 1 kg. In one example, the environment capture system 700 may weigh 1-3 kg.

[0082] The front cover 702, the front housing 710 and the rear housing 716 form part of a housing. In one example, the front cover may have a width w of 75 mm.

[0083] The lens assembly 704 may include a camera lens that focuses light onto an image capture device. The image capture device may capture an image of a physical environment. A user may place the environment capture system 700 to capture a portion of a floor of a building (such as the second building 422 of FIG. 1 ) to obtain a panoramic image of the portion of the floor. The environment capture system 700 may be moved to another portion of the floor of the building to obtain a panoramic image of another portion of the floor. In one example, the depth of field of the image capture device is 0.5 meters to infinity. Fig. 8A Example lens sizes in some embodiments are depicted.

[0084] In some embodiments, the image capture device is a complementary metal oxide semiconductor (CMOS) image sensor (e.g., a Sony IMX283-20 ​​Megapixel CMOS MIPI sensor with an NVidia Jetson Nano SOM). In various embodiments, the image capture device is a charge coupled device (CCD). In one example, the image capture device is a red-green-blue (RGB) sensor. In one embodiment, the image capture device is an infrared (IR) sensor. The lens assembly 704 can give the image capture device a wide field of view.

[0085] Image sensors can have many different specifications. In one example, the image sensor includes the following:

[0086] Number of pixels per column Pixel 5496 Number of pixels per line Pixel 3694 Resolution MP >20 Image circle diameter mm 15.86mm Pixel pitch um 2.4um Pixels per degree (PPD) PPD >37 Chief ray angle at full height Spend 3.0° Output Interface - MIPI Green Sensitivity V / lux*s >1.7 SNR (100 lux, 1x gain) dB >65 Dynamic Range dB >70

[0087] An example specification could be as follows:

[0088] Image circle diameter mm 15.86 Minimum object distance mm 500 Maximum object distance mm Infinity Chief ray angle at full sensor height Spend 3.0 L1 diameter mm <60 Total lens length (TTL) mm <＝80 Back focal length (BFL) mm - Effective focal length (EFL) mm - Relative Illuminance ％ >50 Maximum Distortion ％ <5 52lp / mm(on axis) ％ >85 104lp / mm(on axis) ％ >66 208lp / mm(on axis) ％ >45 52lp / mm (83% field) ％ >75 104lp / mm (83% field) ％ >41 208lp / mm (83% field) ％ >25

[0089] In various embodiments, when observing the MTF at F0 relative field (ie, center), the focus shift may vary from +28 microns at 0.5 m to -25 microns at infinity for a total through focus shift of 53 microns.

[0090] Figure 8B Depicted are example lens design specifications in some embodiments.

[0091] In some examples, the lens assembly 704 has a HFOV of at least 148 degrees and a VFOV of at least 94 degrees. In one example, the lens assembly 704 has a field of view of 150°, 180°, or in the range of 145° to 180°. In one example, three or four separate image captures from the image capture device of the environment capture system 700 can be used to obtain image capture of a 360° view around the environment capture system 700. In various embodiments, the image capture device can have a resolution of at least 37 pixels per degree. In some embodiments, the environment capture system 700 includes a lens cap (not shown) to protect the lens assembly 704 when not in use. The output of the lens assembly 704 can be a digital image of an area of ​​the physical environment. The images captured by the lens assembly 704 can be stitched together to form a 2D panoramic image of the physical environment. A 3D panorama can be generated by combining the depth data captured by the LiDAR 708 with a 2D panoramic image generated by stitching together multiple images from the lens assembly 704. In some embodiments, the images captured by the environment capture system 402 are stitched together by the image processing system 406. In various embodiments, the environment capture system 402 generates a "preview" or "thumbnail" version of the 2D panoramic image. The preview or thumbnail version of the 2D panoramic image can be presented on a user system 1110 such as an iPad, a personal computer, a smartphone, etc. In some embodiments, the environment capture system 402 can generate a mini map of the physical environment representing an area of ​​the physical environment. In various embodiments, the image processing system 406 generates a mini map representing the area of ​​the physical environment.

[0092] The image captured by the lens assembly 704 may include capture device location data identifying or indicating the capture location of the 2D image. For example, in some embodiments, the capture device location data can include global positioning system (GPS) coordinates associated with the 2D image. In other embodiments, the capture device location data can include location information indicating the relative position of the capture device (e.g., a camera and / or a 3D sensor) relative to its environment, such as the relative or calibrated position of the capture device relative to an object in the environment, another camera in the environment, another device in the environment, etc. In some embodiments, this type of location data can be determined by a capture device associated with the capture of the image (e.g., a camera and / or a device including positioning hardware and / or software that is operably coupled to the camera) and received with the image. The placement of the lens assembly 704 is not just by design. By placing the lens assembly 704 at or substantially at the center of the rotation axis, the parallax effect can be reduced.

[0093] In some embodiments, the structural frame 706 holds the lens assembly 704 and the LiDAR 708 in a specific position and can help protect the components of the example of the environment capture system. The structural frame 706 can be used to help rigidly mount the LiDAR 708 and place the LiDAR 708 in a fixed position. In addition, the fixed position of the lens assembly 704 and the LiDAR 708 enables a fixed relationship to align the depth data with the image information to help generate a 3D image. The 2D image data and depth data captured in the physical environment can be aligned relative to a common 3D coordinate space to generate a 3D model of the physical environment.

[0094] In various embodiments, the LiDAR 708 captures depth information of the physical environment. When the user places the environment capture system 700 in a portion of the floor of the second building, the LiDAR 708 can obtain depth information of the object. The LiDAR 708 may include an optical sensing module that is capable of measuring the distance to a target or object in the scene by illuminating the target or scene with pulses from a laser and measuring the time it takes for photons to travel to the target and return to the LiDAR 708. The measurement results can then be transformed into a grid coordinate system by using information derived from the horizontal drive train of the environment capture system 700.

[0095] In some embodiments, the LiDAR 708 may return a depth data point with a timestamp (of an internal clock) every 10 microseconds. The LiDAR 708 may sample a portion of the sphere (the small holes at the top and bottom) every 0.25 degrees. In some embodiments, with one data point every 10 microseconds and 0.25 degrees, each point "disk" may have 14.40 milliseconds and 1440 disks to form a nominal 20.7 second sphere. Because each disk captures front to back, the sphere may be captured in a 180° scan.

[0096] In one example, the LiDAR 708 specifications may be as follows:

[0097]

[0098]

[0099] One advantage of utilizing LiDAR is that in the case of lower wavelength (e.g., 905nm, 900-940nm, etc.) LiDAR, it can allow the environment capture system 700 to determine depth information for outdoor environments or indoor environments with bright light.

[0100] The placement of lens assembly 704 and LiDAR 708 can allow environment capture system 700 or a digital device in communication with environment capture system 700 to generate 3D panoramic images using depth data from LiDAR 708 and lens assembly 704. In some embodiments, 2D and 3D panoramic images are not generated on environment capture system 402.

[0101] The output of LiDAR 708 may include attributes associated with each laser pulse sent by LiDAR 708. The attributes include the intensity of the laser pulse, the number of returns, the current number of returns, the classification point, the RGC value, the GPS time, the scanning angle, the scanning direction, or any combination thereof. The depth of field may be (0.5m; infinity), (1m; infinity), etc. In some embodiments, the depth of field is 0.2m to 1m and infinity.

[0102] In some embodiments, while the environment capture system 700 is stationary, the environment capture system 700 uses the lens assembly 704 to capture four separate RBG images. In various embodiments, while the environment capture system 700 is in motion moving from one RBG image capture position to another RBG image capture position, the LiDAR 708 captures depth data in four different situations. In one example, a 360° rotation of the environment capture system 700 (which may be referred to as a sweep) is used to capture a 3D panoramic image. In various embodiments, a rotation of less than 360° of the environment capture system 700 is used to capture a 3D panoramic image. The output of the sweep may be a sweep list (SWL), which includes image data from the lens assembly 704 and depth data from the LiDAR 708 and the nature of the sweep, including GPS location and a timestamp when the sweep occurred. In various embodiments, a single sweep (e.g., a single 360 ​​degree rotation of the environment capture system 700) captures sufficient image and depth information to generate a 3D visualization (e.g., by a digital device in communication with the environment capture system 700, which receives image and depth data from the environment capture system 700 and uses only the image and depth data from the environment capture system 700 captured in the single sweep to produce the 3D visualization).

[0103] In some embodiments, images captured by the environment capture system 402 may be blended, stitched together, and combined with depth data from the LiDAR 708 via the image stitching and processing systems discussed herein.

[0104] In various embodiments, the environment capture system 402 and / or an application on the user system 1110 may generate a preview or thumbnail version of the 3D panoramic image. The preview or thumbnail version of the 3D panoramic image may be presented on the user system 1110 and may have a lower image resolution than the 3D panoramic image generated by the image processing system 406. After the lens assembly 704 and the LiDAR 708 capture images and depth data of the physical environment, the environment capture system 402 may generate a mini map representing an area of ​​the physical environment that has been captured by the environment capture system 402. In some embodiments, the image processing system 406 generates a mini map representing the area of ​​the physical environment. After capturing images and depth data of the living room of a home using the environment capture system 402, the environment capture system 402 may generate an overhead view of the physical environment. The user may use this information to determine areas of the physical environment for which the user has not yet captured or generated a 3D panoramic image.

[0105] In one embodiment, the environment capture system 700 may interleave image capture using an image capture device with lens assembly 704 with depth information capture using LiDAR 708. For example, an image capture device may capture a segment 1605 of a physical environment (e.g., Fig.16 1605, and then the LiDAR 708 obtains depth information from segment 1605. Once the LiDAR 708 obtains depth information from segment 1605, the image capture device may continue to move to capture an image of another segment 1610, and then the LiDAR 708 obtains depth information from segment 1610, thereby interleaving image capture and depth information capture.

[0106] In some embodiments, the LiDAR 708 may have a field of view of at least 145°, and depth information of all objects in the 360° view of the environment capture system 700 may be obtained in three or four scans by the environment capture system 700. In another example, the LiDAR 708 may have a field of view of at least 150°, 180°, or between 145° and 180°.

[0107] The increase in the field of view of the lens reduces the amount of time required to obtain visual and depth information of the physical environment surrounding the environment capture system 700. In various embodiments, the LiDAR 708 has a minimum depth range of 0.5 m. In one embodiment, the LiDAR 708 has a maximum depth range of greater than 8 meters.

[0108] The LiDAR 708 can utilize a mirror assembly 712 to direct the laser at different scanning angles. In one embodiment, an optional vertical motor 718 has the ability to vertically move the mirror assembly 712. In some embodiments, the mirror assembly 712 can be a dielectric mirror with a hydrophobic coating or layer. The mirror assembly 712 can be coupled to a vertical motor 718 that rotates the mirror assembly 712 when in use.

[0109] The reflectors of the reflector assembly 712 may, for example, include the following specifications:

[0110]

[0111] The reflectors of reflector assembly 712 may, for example, include the following specifications for materials and coatings:

[0112] S1L1 Material Dielectric S1L2 Material Hydrophobicity S2L1 Material Black paint emulsion Powder suspended in paint Base Material Schott B270I

[0113] The hydrophobic coating of the reflector of reflector assembly 712 may, for example, include a contact angle number >105.

[0114] The reflectors of reflector assembly 712 may include the following quality specifications:

[0115]

[0116]

[0117] The vertical motor may include, for example, the following specifications:

[0118]

[0119] Due to the RGB capture device and LiDAR 708, the environmental capture system 700 can capture images outdoors in bright sunlight, or capture images indoors with bright light or sunlight glare from windows. In systems that utilize different devices (e.g., structured light devices), they may not be able to operate in bright environments, whether indoors or outdoors. These devices are typically limited to use only indoors and only during dawn or sunset to control light. Otherwise, bright spots in the room create artifacts or "holes" in the image that must be filled or corrected. However, the environmental capture system 700 can be used both indoors and outdoors in bright sunlight. The capture device and LiDAR 708 may be able to capture images and depth data in bright environments without artifacts or holes caused by glare or bright light.

[0120] In one embodiment, the GPS antenna 714 receives global positioning system (GPS) data. The GPS data can be used to determine the location of the environment capture system 700 at any given time.

[0121] In various embodiments, the display 720 allows the environment capture system 700 to provide the current status of the system, such as updating, warming up, scanning, scan complete, error, etc.

[0122] The battery pack 722 provides power to the environment capture system 700. The battery pack 722 may be removable and rechargeable, allowing the user to put in a new battery pack 722 while charging the depleted battery pack. In some embodiments, the battery pack 722 may allow for continuous use of at least 1000 SWLs or at least 250 SWLs before recharging. The environment capture system 700 may utilize a USB-C plug for recharging.

[0123] In some embodiments, the mount 724 is a connector that provides for the environment capture system 700 to be connected to a platform such as a tripod or a mount. The horizontal motor 726 can rotate the environment capture system 700 around the xy plane. In some embodiments, the horizontal motor 726 can provide information to a grid coordinate system to determine the (x, y, z) coordinates associated with each laser pulse. In various embodiments, the horizontal motor 726 can enable the environment capture system 700 to scan quickly due to the wide field of view of the lens, the positioning of the lens around the axis of rotation, and the LiDAR device.

[0124] In one example, the horizontal motor 726 may have the following specifications:

[0125]

[0126] In various embodiments, the mount 724 may include a quick release adapter.The holding torque may be, for example, >2.0 Nm, and the durability of the capture operation may be up to or exceeding 70,000 cycles.

[0127] For example, the environment capture system 700 can enable the construction of a 3D mesh of a standard home where the distance between sweeps is greater than 8m. The time to capture, process, and align an indoor sweep can be under 45 seconds. In one example, the time frame from the start of the sweep capture to when the user can move the environment capture system 700 can be less than 15 seconds.

[0128] In various embodiments, these components provide the environment capture system 700 with the ability to align outdoor and indoor scanning locations and thus create a seamless roaming experience between indoor and outdoor (which can be a high priority for hotels, vacation rentals, real estate, construction documentation, CRE, and as-built modeling and verification). The environment capture system 700 can also generate an "outdoor dollhouse" or outdoor mini map. As shown herein, the environment capture system 700 can also improve the accuracy of 3D reconstruction, primarily from a measurement perspective. For scanning density, the ability of the user to tune it can also be a favorable factor. These components can also enable the environment capture system 700 to capture wide empty spaces (e.g., longer ranges). In order to generate a 3D model of a wide empty space, the environment capture system may be required to scan and capture 3D data and depth data from a larger distance range than to generate a 3D model of a smaller space.

[0129] In various embodiments, these components enable the environment capture system 700 to align SWL and reconstruct 3D models in a similar manner for indoor and outdoor use. These components can also enable the environment capture system 700 to perform geo-location of the 3D model (which can be easily integrated into Google Street View and help align outdoor panoramas if desired).

[0130] The image capture device of the environment capture system 700 may be capable of providing DSLR-like images with a quality printable at 8.5"x11" for a 70° VFOV and RGB image type.

[0131] In some embodiments, the environment capture system 700 can take an RGB image using an image capture device (e.g., using a wide angle lens), and then move the lens (a total of four moves using a motor) before taking the next RGB image. While the horizontal motor 726 rotates the environment capture system 90 degrees, the LiDAR 708 can capture depth data. In some embodiments, the LiDAR 708 includes an APD array.

[0132] In some embodiments, the image and depth data can then be sent to a capture application (e.g., a device that communicates with the environment capture system 700, such as an image capture system on a smart device or network). In some embodiments, the environment capture system 700 can send the image and depth data to the image processing system 406 for processing and generating a 2D panoramic image or a 3D panoramic image. In various embodiments, the environment capture system 700 can generate a sweep list of captured RGB images and depth data from a 360-degree rotation of the environment capture system 700. The sweep list can be sent to the image processing system 406 for stitching and alignment. The output of the sweep can be a SWL, which includes image data from the lens assembly 704 and depth data from the LiDAR 708 and the properties of the sweep, including GPS location and a timestamp when the sweep occurred.

[0133] In various embodiments, the LiDAR, vertical reflector, RGB lens, tripod mount, and horizontal drive are rigidly mounted within the housing to allow the housing to be opened without recalibrating the system.

[0134] Figure 9a A block diagram 900 of an example of an environment capture system according to some embodiments is depicted. The block diagram 900 includes a power supply 902, a power converter 904, an input / output (I / O) printed circuit board assembly (PCBA), a system on module (SOM) PCBA, a user interface 910, a LiDAR 912, a reflector brushless DC (BLCD) motor 914, a drive train 916, a wide FOV (WFOV) lens 918, and an image sensor 920.

[0135] The power supply 902 may be Figure 7 Battery pack 722. The power source can be a removable, rechargeable battery, such as a lithium-ion battery (e.g., 4 x 18650 lithium-ion batteries), capable of providing power to the environmental capture system.

[0136] The power converter 904 can change the voltage level from the power supply 902 to a lower or higher voltage level so that it can be utilized by the electronic components of the ambient capture system. The ambient capture system can utilize 4x 18650 lithium ion batteries in a 4S1P configuration or four series connections and one parallel connection configuration.

[0137] In some embodiments, the I / O PCBA 906 may include elements that provide an IMU, Wi-Fi, GPS, Bluetooth, an inertial measurement unit (IMU), a motor driver, and a microcontroller. In some embodiments, the I / O PCBA 906 includes a microcontroller for controlling a horizontal motor and encoding a horizontal motor control and controlling a vertical motor and encoding a vertical motor control.

[0138] The SOM PCBA 908 may include a central processing unit (CPU) and / or a graphics processing unit (GPU), a memory, and a mobile interface. The SOM PCBA 908 may control the LiDAR 912, the image sensor 920, and the I / O PCBA 906. The SOM PCBA 908 may determine the (x, y, z) coordinates associated with each laser pulse of the LiDAR 912 and store the coordinates in a memory component of the SOM PCBA 908. In some embodiments, the SOM PCBA 908 may store the coordinates in an image processing system of the environment capture system 400. In addition to the coordinates associated with each laser pulse, the SOM PCBA 908 may also determine additional attributes associated with each laser pulse, including the intensity of the laser pulse, the number of returns, the current number of returns, the classification point, the RGC value, the GPS time, the scanning angle, and the scanning direction.

[0139] In some embodiments, the SOM PCBA 908 comprises a Nvidia SOM PCBA w / CPU / GPU, DDR, eMMC, Ethernet.

[0140] The user interface 910 may include physical buttons or switches with which a user may interact. The buttons or switches may provide functions such as turning the environment capture system on and off, scanning the physical environment, etc. In some embodiments, the user interface 910 may include a display such as a Figure 7 Display 720.

[0141] In some embodiments, LiDAR 912 captures depth information of the physical environment. LiDAR 912 includes an optical sensing module that can measure the distance to a target or object in a scene by illuminating the target or scene with light using pulses from a laser. The optical sensing module of LiDAR 912 measures the time it takes for photons to travel to the target or object and return to the receiver in LiDAR 912 after reflection, thereby giving the distance of the LiDAR from the target or object. Together with the distance, SOM PCBA 908 can determine the (x, y, z) coordinates associated with each laser pulse. LiDAR 912 can be assembled within a width of 58 mm, a height of 55 mm, and a depth of 60 mm.

[0142] The LiDAR 912 may include a range of 90 m (10% reflectivity), a range of 130 m (20% reflectivity), a range of 260 m (100% reflectivity), a range accuracy of 2 cm (1σ@900 m), a wavelength of 1705 nm, and a beam divergence of 0.28×0.03 degrees.

[0143] The SOM PCBA 908 may determine coordinates based on the position of the drive train 916. In various embodiments, the LiDAR 912 may include one or more LiDAR devices. Multiple LiDAR devices may be utilized to increase LiDAR resolution.

[0144] The mirror brushless DC (BLCD) motor 914 can control Figure 7 The reflector assembly 712 is provided.

[0145] In some embodiments, the transmission system 916 may include Figure 7 The drive train 916 may include a stepper motor Nema 14, a worm and plastic wheel drive train, a clutch, a bushing bearing, and a backlash prevention mechanism. In some embodiments, the environment capture system may be able to complete a scan in less than 17 seconds. In various embodiments, the drive train 916 has a maximum speed of 60 degrees / second, 300 degrees / second, and a rotation speed of 100 degrees / second. 2 The encoder features a maximum acceleration of 100 nm, a maximum torque of 0.5 nm, an angular position accuracy of less than 0.1 degrees, and an encoder resolution of approximately 4096 counts per revolution.

[0146] In some embodiments, the drive train 916 includes a vertical monogon reflector and a motor. In this example, the drive train 916 may include a BLDC motor, an external Hall effect sensor, a magnet (paired with the Hall effect sensor), a reflector bracket, and a reflector. In this example, the drive train 916 may have a maximum speed of 4,000 rpm and a maximum acceleration of 300 degrees / second^2. In some embodiments, the monogon reflector is a dielectric reflector. In one embodiment, the monogon reflector includes a hydrophobic coating or layer.

[0147] The placement of the components of the environment capture system is such that the lens assembly and LiDAR are placed substantially at the center of the axis of rotation. This can reduce image parallax that occurs when the image capture system is not placed at the center of the axis of rotation.

[0148] In some embodiments, the WFOV lens 918 may be Figure 7The lens of the lens assembly 704. The WFOV lens 918 focuses light onto the image capture device. In some embodiments, the WFOV lens can have a FOV of at least 145 degrees. With such a wide FOV, three separate image captures of the image capture device can be used to obtain image capture of a 360-degree view around the environment capture system. In some embodiments, the WFOV lens 918 can be approximately ~60mm in diameter and ~80mm total length of lens (TTL). In one example, the WFOV lens 918 can include a horizontal field of view greater than or equal to 148.3 degrees and a vertical field of view greater than or equal to 94 degrees.

[0149] The image capture device may include a WFOV lens 918 and an image sensor 920. The image sensor 920 may be a CMOS image sensor. In one embodiment, the image sensor 920 is a charge coupled device (CCD). In some embodiments, the image sensor 920 is a red-green-blue (RGB) sensor. In one embodiment, the image sensor 920 is an IR sensor. In various embodiments, the image capture device may have a resolution of at least 35 pixels per degree (PPD).

[0150] In some embodiments, the image capture device may include an F-number of f / 2.4, an image circle diameter of 15.86 mm, a pixel pitch of 2.4 um, a HFOV of >148.3°, a VFOV of >94.0°, pixels per degree of >38.0 PPD, a chief ray angle at full height of 3.0°, a minimum object distance of 1300 mm, a maximum object distance of infinity, a relative illumination of >130%, a maximum distortion of <90%, and a spectral transmittance variation of <=5%.

[0151] In some embodiments, the lens may include an F-number of 2.8, an image circle diameter of 15.86mm, pixels per degree >37, a chief ray angle at full height of 3.0, an L1 diameter of <60mm, a TTL of <80mm, and a relative illumination of >50%.

[0152] The lens may include >85% 52lp / mm (on axis), >66% 104lp / mm (on axis), >45% 1308lp / mm (on axis), >75% 52lp / mm (83% field), >41% 104lp / mm (83% field) and >25% 1308lp / mm (83% field).

[0153] The ambient capture system can have a resolution of >20MP, a green sensitivity of >1.7V / lux*s, a SNR of >65dB (100lux, 1x gain), and a dynamic range of >70dB.

[0154] Figure 9bA block diagram of an example SOM PCBA 908 of an environment capture system according to some embodiments is depicted. The SOM PCBA 908 may include a communication component 922, a LiDAR control component 924, a LiDAR position component 926, a user interface component 928, a classification component 930, a LiDAR data storage area 932, and a captured image data storage area 934.

[0155] In some embodiments, the communication component 922 can be connected to any component of the SOM PCBA 1008 and Figure 9a The environment captures the sending and receiving of requests or data between components of the system.

[0156] In various embodiments, the LiDAR control component 924 can control various aspects of the LiDAR. For example, the LiDAR control component 924 can send a control signal to the LiDAR 912 to start emitting laser pulses. The control signal sent by the LiDAR control component 924 can include instructions regarding the frequency of the laser pulses.

[0157] In some embodiments, the LiDAR position component 926 can utilize GPS data to determine the location of the environment capture system. In various embodiments, the LiDAR position component 926 utilizes the position of the mirror assembly to determine the scan angle and (x, y, z) coordinates associated with each laser pulse. The LiDAR position component 926 can also utilize an IMU to determine the orientation of the environment capture system.

[0158] The user interface component 928 can promote the interaction of the user and the environment capture system. In some embodiments, the user interface component 928 can provide one or more user interface elements that the user can interact with. The user interface provided by the user interface component 928 can be sent to the user system 1110. For example, the user interface component 928 can provide a visual representation of an area of ​​the floor plan of a building to the user system (e.g., a digital device). When the user places the environment capture system in different parts of the floor of the building to capture and generate a 3D panoramic image, the environment capture system can generate a visual representation of the floor plan. The user can place the environment capture system in an area of ​​the physical environment to capture and generate the 3D panoramic image in this area of ​​the house. Once the 3D panoramic image of the area has been generated by the image processing system, the user interface component can update the floor plan view using the overhead view of the living room area depicted in Figure 1b. In some embodiments, the floor plan view 200 can be generated by the user system 1110 after the second sweep of the same family or floor of the building has been captured.

[0159] In various embodiments, the classification component 930 can classify the type of physical environment. The classification component 930 can analyze the objects in the image or the objects in the image to classify the type of physical environment captured by the environment capture system. In some embodiments, the image processing system can be responsible for classifying the type of physical environment captured by the environment capture system 400.

[0160] The LiDAR data store 932 may be any one structure and / or multiple structures suitable for the captured LiDAR data (e.g., an active database, a relational database, a self-referential database, a table, a matrix, an array, a flat file, a document-oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, and / or similar structures). The image data store 408 may store the captured LiDAR data. However, in the event that the communication network 404 is not functional, the LiDAR data store 932 may be used to cache the captured LiDAR data. For example, in the event that the environment capture system 402 and the user system 1110 are in a remote location without a cellular network or in an area without Wi-Fi, the LiDAR data store 932 may store the captured LiDAR data until they can be transferred to the image data store 934.

[0161] Similar to the LiDAR data store, the captured image data store 934 can be any one structure and / or multiple structures suitable for capturing images (e.g., active databases, relational databases, self-referential databases, tables, matrices, arrays, flat files, document-oriented storage systems, non-relational No-SQL systems, FTS management systems such as Lucene / Solar, and / or similar structures). The image data store 934 can store captured images.

[0162] Figure 10a-10c Depicted is a process for an environment capture system 400 to capture images in some embodiments. Figure 10a-10c As shown, the environment capture system 400 can take a burst of images at different exposures. The burst of images can be a group of images, each image having a different exposure. The first image burst occurs at time 0.0. The environment capture system 400 can receive the first frame and then evaluate the first frame while waiting for the second frame. Fig.10a Instructs to blend the first frame before the second frame arrives. In some embodiments, the environment capture system 400 can process each frame to identify pixels, colors, etc. Once the next frame arrives, the environment capture system 400 can process the most recently received frame and then blend the two frames together.

[0163] In various embodiments, the environment capture system 400 performs image processing to blend the sixth frame and further evaluates pixels in the blended frame (e.g., a frame that may include elements from any number of frames in the image burst). During a final step before or during movement (e.g., rotation) of the environment capture system 400, the environment capture system 400 may optionally transfer the blended image from the graphics processing unit to the CPU memory.

[0164] The process Fig.10b Continue in Fig.10b At the beginning, the environment capture system 400 takes another burst. The environment capture system 400 can use JxR to compress all or part of the blended frame and / or the captured frame. Fig.10a , a burst of images can be a group of images, each with a different exposure (the exposure length of each frame of the group can be different from Fig.10a and 10c ). The second image burst occurs at time 2 seconds. The environment capture system 400 can receive the first frame and then evaluate the first frame while waiting for the second frame. Fig.10b Instructs to blend the first frame before the second frame arrives. In some embodiments, the environment capture system 400 can process each frame to identify pixels, colors, etc. Once the next frame arrives, the environment capture system 400 can process the most recently received frame and then blend the two frames together.

[0165] In various embodiments, the environment capture system 400 performs image processing to blend the sixth frame and further evaluates pixels in the blended frame (e.g., a frame that may include elements from any number of frames in the image burst). During a final step before or during movement (e.g., rotation) of the environment capture system 400, the environment capture system 400 may optionally transfer the blended image from the graphics processing unit to the CPU memory.

[0166] After the turn, the environment capture system 400 can continue the process by taking another color burst at approximately time 3.5 seconds (e.g., after turning 180 degrees). The environment capture system 400 can use JxR to compress the blended frame and / or capture all or part of the frame. The burst of images can be a group of images, each image having a different exposure (the exposure length of each frame of the group can be different from the exposure length of the image). Fig.10a and 10c The environment capture system 400 may receive the first frame and then evaluate the first frame while waiting for the second frame. Fig.10bInstructs to blend the first frame before the second frame arrives. In some embodiments, the environment capture system 400 can process each frame to identify pixels, colors, etc. Once the next frame arrives, the environment capture system 400 can process the most recently received frame and then blend the two frames together.

[0167] In various embodiments, the environment capture system 400 performs image processing to blend the sixth frame and further evaluates pixels in the blended frame (e.g., a frame that may include elements from any number of frames in the image burst). During a final step before or during movement (e.g., rotation) of the environment capture system 400, the environment capture system 400 may optionally transfer the blended image from the graphics processing unit to the CPU memory.

[0168] exist Fig.10c , the last burst occurs at time 5 seconds. The environment capture system 400 can use JxR to compress the blended frames and / or capture all or part of the frames. A burst of images can be a group of images, each with a different exposure (the exposure length of each frame of the group can be different from Fig.10a and 10b The environment capture system 400 may receive the first frame and then evaluate the first frame while waiting for the second frame. Fig.10c Instructs to blend the first frame before the second frame arrives. In some embodiments, the environment capture system 400 can process each frame to identify pixels, colors, etc. Once the next frame arrives, the environment capture system 400 can process the most recently received frame and then blend the two frames together.

[0169] In various embodiments, the environment capture system 400 performs image processing to blend the sixth frame and further evaluates pixels in the blended frame (e.g., a frame that may include elements from any number of frames in the image burst). During a final step before or during movement (e.g., rotation) of the environment capture system 400, the environment capture system 400 may optionally transfer the blended image from the graphics processing unit to the CPU memory.

[0170] The dynamic range of an image capture device is a measure of how much light an image sensor can capture. The dynamic range is the difference between the darkest and brightest areas of an image. There are many ways to increase the dynamic range of an image capture device, one of which is to capture multiple images of the same physical environment using different exposures. Images captured with short exposures will capture brighter areas of the physical environment, while long exposures will capture darker areas of the physical environment. In some embodiments, the environment capture system can capture multiple images with six different exposure times. Some or all of the images captured by the environment capture system are used to generate a 2D image with a high dynamic range (HDR). One or more of the captured images can be used for other functions, such as ambient light detection, flicker detection, etc.

[0171] A 3D panoramic image of a physical environment can be generated based on four separate image captures of an image capture device and four separate depth data captures of a LiDAR device of an environmental capture system. Each of the four separate image captures may include a series of image captures with different exposure times. A hybrid algorithm may be used to mix a series of image captures with different exposure times to generate one of four RGB image captures, which may be used to generate a 2D panoramic image. For example, an environmental capture system may be used to capture a 3D panoramic image of a kitchen. An image of one wall of a kitchen may include a window, and an image with an image captured with a shorter exposure may provide a view outside the window, but may underexpose the rest of the kitchen. Conversely, another image captured with a longer exposure may provide a view inside the kitchen. The hybrid algorithm may generate a hybrid RGB image by mixing the view outside the kitchen window from one image with the rest of the kitchen view from another image.

[0172] In various embodiments, a 3D panoramic image may be generated based on three separate image captures by an image capture device and four separate depth data captures by a LiDAR device of an environment capture system. In some embodiments, the number of image captures and the number of depth data captures may be the same. In one embodiment, the number of image captures and the number of depth data captures may be different.

[0173] After capturing a first in a series of images with an exposure time, the blending algorithm receives the first in the series of images, calculates initial intensity weights for the image, and sets the image as a baseline image for combining subsequently received images. In some embodiments, the blending algorithm may utilize a graphics processing unit (GPU) image processing routine, such as a "blend_kernel" routine. The blending algorithm may receive subsequent images that may be blended with previously received images. In some embodiments, the blending algorithm may utilize a variation of the Blend_Kernel GPU image processing routine.

[0174] In one embodiment, the blending algorithm utilizes other methods of blending multiple images, such as determining the difference between the darkest and lightest portions of a baseline image or the contrast of a baseline image, to determine whether the baseline image may be overexposed or underexposed. For example, a contrast value less than a predetermined contrast threshold means that the baseline image is overexposed or underexposed. In one embodiment, the contrast of the baseline image can be calculated by taking the average of the light intensity of an image or a subset of the images. In some embodiments, the blending algorithm calculates the average light intensity for each row or column of the image. In some embodiments, the blending algorithm can determine a histogram for each image received from the image capture device and analyze the histogram to determine the light intensity of the pixels that make up each image.

[0175] In various embodiments, blending may involve sampling colors within two or more images of the same scene, including along objects and seams. If there is a significant difference in color between the two images (e.g., within a predetermined threshold of color, hue, brightness, saturation, and / or the like), a blending module (e.g., on the environment capture system 400 or the user device 1110) may blend the two images of a predetermined size along the location where the difference exists. In some embodiments, the greater the difference in color or image at a location in the image, the greater the amount of space around or near that location that can be blended.

[0176] In some embodiments, after blending, a blending module (e.g., on the environment capture system 400 or the user device 1110) may rescan and sample the colors along the image(s) to determine if there are other differences in the images or colors that exceed a predetermined threshold of color, hue, brightness, saturation, and / or the like. If so, the blending module may identify the portion within the image(s) and continue blending that portion of the images. The blending module may continue resampling the images along the seam until there are no other portions of the images to blend (e.g., any differences in color are below the predetermined threshold(s)).

[0177] Fig.11 A block diagram of an example environment 1100 capable of capturing and stitching images to form a 3D visualization in accordance with some embodiments is depicted. The example environment 1100 includes a 3D and panoramic capture and stitching system 1102, a communication network 1104, an image stitching and processor system 1106, an image data storage area 1108, a user system 1110, and a first scene 1112 of a physical environment. The 3D and panoramic capture and stitching system 1102 and / or the user system 1110 may include an image capture device (e.g., environment capture system 400) that may be used to capture images of an environment (e.g., physical environment 1112).

[0178] The 3D and panoramic capture and stitching system 1102 and the image stitching and processor system 1106 may be part of the same system (e.g., part of one or more digital devices) that is communicatively coupled to the environment capture system 400. In some embodiments, one or more functions of the components of the 3D and panoramic capture and stitching system 1102 and the image stitching and processor system 1106 may be performed by the environment capture system 400. Similarly or alternatively, the 3D and panoramic capture and stitching system 1102 and the image stitching and processor system 1106 may be performed by the user system 1110 and / or the image stitching and processor system 1106.

[0179] The user may utilize the 3D panoramic capture and stitching system 1102 to capture multiple 2D images of an environment, such as the interior of a building and / or the exterior of a building. For example, the user may utilize the 3D and panoramic capture and stitching system 1102 to capture multiple 2D images of a first scene of a physical environment 1112 provided by the environment capture system 400. The 3D and panoramic capture and stitching system 1102 may include an alignment and stitching system 1114. Alternatively, the user system 1110 may include the alignment and stitching system 1114.

[0180] The alignment and stitching system 1114 may be software, hardware, or a combination of both, configured to provide guidance to a user of an image capture system (e.g., on the 3D and panoramic capture and stitching system 1102 or the user system 1110) and / or process images to enable the production of an improved panoramic picture (e.g., by stitching, aligning, cropping, etc.). The alignment and stitching system 1114 may be on a computer-readable medium (described herein). In some embodiments, the alignment and stitching system 1114 may include a processor for performing the functions.

[0181] An example of a first scene of a physical environment 1112 may be any room, real estate, etc. (e.g., a representation of a living room). In some embodiments, the 3D and panoramic capture and stitching system 1102 is used to generate a 3D panoramic image of an indoor environment. In some embodiments, the 3D panoramic capture and stitching system 1102 may be a system for generating a 3D panoramic image of an indoor environment. Figure 4 The environmental capture system 400 is discussed.

[0182] In some embodiments, the 3D panoramic capture and stitching system 1102 can communicate with devices and software for capturing images and depth data (e.g., the environment capture system 400). All or part of the software can be installed on the 3D panoramic capture and stitching system 1102, the user system 1110, the environment capture system 400, or both. In some embodiments, the user can interact with the 3D and panoramic capture and stitching system 1102 via the user system 1110.

[0183] The 3D and panoramic capture and stitching system 1102 or the user system 1110 may obtain multiple 2D images. The 3D and panoramic capture and stitching system 1102 or the user system 1110 may obtain depth data (eg, from a LiDAR device, etc.).

[0184] In various embodiments, an application on the user system 1110 (e.g., the user's smart device, such as a smartphone or tablet computer) or an application on the environment capture system 400 may provide visual or auditory guidance to the user for capturing images using the environment capture system 400. Graphical guidance may include, for example, floating arrows on a display of the environment capture system 400 (e.g., on a viewfinder or LED screen on the back of the environment capture system 400) to guide the user where to position and / or point the image capture device. In another example, the application may provide audio guidance on where to position and / or point the image capture device.

[0185] In some embodiments, guidance may allow a user to capture multiple images of a physical environment without the aid of a stable platform (e.g., a tripod). In one example, the image capture device may be a personal device such as a smartphone, tablet, media tablet, laptop, etc. The application may provide directions about the location for each sweep to approximate a parallax-free point based on the location of the image capture device, location information from the image capture device, and / or previous images of the image capture device.

[0186] In some embodiments, visual and / or auditory guidance enables capturing images that can be stitched together to form a panorama without requiring a tripod and without requiring camera positioning information (e.g., indicating the position, location and / or orientation of the camera from a sensor, GPS device, etc.).

[0187] The alignment and stitching system 1114 may align or stitch 2D images (eg, captured by the user system 1110 or the 3D panoramic capture and stitching system 1102 ) to obtain a 2D panoramic image.

[0188] In some embodiments, the alignment and stitching system 1114 utilizes a machine learning algorithm to align or stitch multiple 2D images into a 2D panoramic image. Parameters of the machine learning algorithm may be managed by the alignment and stitching system 1114. For example, the 3D and panoramic capture and stitching system 1102 and / or the alignment and stitching system 1114 may identify objects within the 2D images to help align the images into a 2D panoramic image.

[0189] In some embodiments, the alignment and stitching system 1114 may utilize the depth data and the 2D panoramic image to obtain a 3D panoramic image. The 3D panoramic image may be provided to the 3D and panoramic stitching system 1102 or the user system 1110. In some embodiments, the alignment and stitching system 1114 determines a 3D / depth measurement associated with an identified object within the 3D panoramic image and / or sends one or more 2D images, depth data, (one or more) 2D panoramic image(s), (one or more) 3D panoramic image(s) to the image stitching and processor system 106 to obtain a 2D panoramic image or a 3D panoramic image having a greater pixel resolution than the 2D panoramic image or the 3D panoramic image provided by the 3D and panoramic capture and stitching system 1102.

[0190] The communication network 1104 may represent one or more computer networks (e.g., LAN, WAN, etc.) or other transmission media. The communication network 1104 may provide communication between the systems 1102, 1106-1110, and / or other systems described herein. In some embodiments, the communication network 104 includes one or more digital devices, routers, cables, buses, and / or other network topologies (e.g., mesh, etc.). In some embodiments, the communication network 1104 may be wired and / or wireless. In various embodiments, the communication network 1104 may include the Internet, one or more wide area networks (WANs) or local area networks (LANs), one or more networks that may be public, private, IP-based, non-IP-based, etc.

[0191] The image stitching and processor system 1106 can process 2D images captured by an image capture device (e.g., the environment capture system 400 or a user device such as a smartphone, a personal computer, a media tablet, etc.) and stitch them into a 2D panoramic image. The 2D panoramic image processed by the image stitching and processor system 106 can have a higher pixel resolution than the panoramic image obtained by the 3D and panoramic capture and stitching system 1102.

[0192] In some embodiments, the image stitching and processor system 1106 receives and processes the 3D panoramic image to generate a 3D panoramic image having a higher pixel resolution than the pixel resolution of the received 3D panoramic image. The higher pixel resolution panoramic image can be provided to an output device, such as a computer screen, a projector screen, etc., having a higher screen resolution than the user system 1110. In some embodiments, the higher pixel resolution panoramic image can provide the panoramic image to the output device in greater detail and can be magnified.

[0193] The image data store 1108 may be any one structure and / or multiple structures suitable for captured image and / or depth data (e.g., an active database, a relational database, a self-referential database, a table, a matrix, an array, a flat file, a document-oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, and / or similar structures). The image data store 1108 may store images captured by an image capture device of the user system 1110. In various embodiments, the image data store 1108 stores depth data captured by one or more depth sensors of the user system 1110. In various embodiments, the image data store 1108 stores properties associated with the image capture device or properties associated with each of multiple image captures or depth captures used to determine a 2D or 3D panoramic image. In some embodiments, the image data store 1108 stores a panoramic 2D or 3D panoramic image. The 2D or 3D panoramic image may be determined by the 3D and panoramic capture and stitching system 1102 or the image stitching and processor system 106.

[0194] The user system 1110 can communicate between the user and other associated systems. In some embodiments, the user system 1110 can be or include one or more mobile devices (e.g., smart phones, cellular phones, smart watches, etc.).

[0195] The user system 1110 may include one or more image capture devices. The one or more image capture devices may include, for example, an RGB camera, an HDR camera, a video camera, an IR camera, and the like.

[0196] The 3D and panoramic capture and stitching system 1102 and / or the user system 1110 may include two or more capture devices that may be arranged on or within the same mobile housing in positions relative to each other such that their common field of view spans up to 360°. In some embodiments, paired image capture devices capable of generating stereoscopic image pairs (e.g., with slightly offset but partially overlapping fields of view) may be used. The user system 1110 may include two image capture devices with a vertical stereo offset field of view capable of capturing a vertical stereo pair of images. In another example, the user system 1110 may include two image capture devices with a vertical stereo offset field of view capable of capturing a vertical stereo pair of images.

[0197] In some embodiments, the user system 1110, the environment capture system 400, or the 3D and panoramic capture and stitching system 1102 may generate and / or provide image capture location and orientation information. For example, the user system 1110 or the 3D and panoramic capture and stitching system 1102 may include an inertial measurement unit (IMU) to help determine location data associated with one or more image capture devices that capture multiple 2D images. The user system 1110 may include a global positioning sensor (GPS) to provide GPS coordinate information associated with multiple 2D images captured by one or more image capture devices.

[0198] In some embodiments, the user may interact with the alignment and stitching system 1114 using a mobile application installed in the user system 1110. The 3D and panoramic capture and stitching system 1102 may provide images to the user system 1110. The user may view images and previews using the alignment and stitching system 1114 on the user system 1110.

[0199] In various embodiments, the alignment and stitching system 1114 may be configured to provide or receive one or more 3D panoramic images from the 3D and panoramic capture and stitching system 1102 and / or the image stitching and processor system 1106. In some embodiments, the 3D and panoramic capture and stitching system 1102 may provide the user system 1110 with a visual representation of a portion of a floor plan of a building that has been captured by the 3D and panoramic capture and stitching system 1102.

[0200] The user of the system 1110 can navigate the space around the area and view different rooms of the house. In some embodiments, as the image stitching and processor system 1106 completes the generation of the 3D panoramic image, the user of the user system 1110 can display the 3D panoramic image, such as the example 3D panoramic image. In various embodiments, the user system 1110 generates a preview or thumbnail of the 3D panoramic image. The preview 3D panoramic image can have a lower image resolution than the 3D panoramic image generated by the 3D and panoramic capture and stitching system 1102.

[0201] Fig.12 1 is a block diagram of an example of an alignment and stitching system 1114 according to some embodiments. The alignment and stitching system 1114 includes a communication module 1202, an image capture location module 1204, a stitching module 1206, a cropping module 1208, a graphic cutting module 1210, a blending module 1211, a 3D image generator 1214, a captured 2D image data storage area 1216, a 3D panoramic image data storage area 1218, and a guidance module 220. It can be appreciated that there can be any number of modules of the alignment and stitching system 1114 that perform one or more different functions as described herein.

[0202] In some embodiments, the alignment and stitching system 1114 includes an image capture module configured to receive images from one or more image capture devices (e.g., cameras). The alignment and stitching system 1114 may also include a depth module configured to receive depth data from a depth device such as LiDAR (if available).

[0203] The communication module 1202 can communicate with any module or data storage area of ​​the alignment and stitching system 1114. Fig.11 The example environment 1100 can send and receive requests, images, or data between components of the example environment 1100. Similarly, the alignment and stitching system 1114 can send and receive requests, images, or data to any device or system on the communication network 1104.

[0204] In some embodiments, the image capture location module 1204 can determine image capture device location data of an image capture device (e.g., a camera that can be a stand-alone camera, a smartphone, a media tablet, a laptop, etc.). The image capture device location data can indicate the position and orientation of the image capture device and / or lens. In one example, the image capture location module 1204 can utilize the IMU of the user system 1110, a camera, a digital device with a camera, or the 3D and panoramic capture and stitching system 1102 to generate the location data of the image capture device. The image capture location module 1204 can determine the current direction, angle, or tilt of one or more image capture devices (or lenses). The image capture location module 1204 can also utilize the GPS of the user system 1110 or the 3D and panoramic capture and stitching system 1102.

[0205] For example, when a user wants to use the user system 1110 to capture a 360° view of a physical environment (such as a living room), the user can hold the user system 1110 in front of them at eye level to begin capturing one of the multiple images that will eventually become a 3D panoramic image. In order to reduce the amount of parallax of the image and capture images that are more suitable for stitching and generating a 3D panoramic image, it would be preferred if one or more image capture devices rotate at the center of the rotation axis. The alignment and stitching system 1114 can receive position information (e.g., from an IMU) to determine the position of the image capture device or lens. The alignment and stitching system 1114 can receive and store the field of view of the lens. The guidance module 1220 can provide visual and / or audio information about the recommended initial position of the image capture device. The guidance module 1220 can make recommendations for positioning the image capture device for subsequent images. In one example, the guidance module 1220 can provide guidance to the user to rotate and position the image capture device so that the image capture device rotates close to the center of rotation. Additionally, guidance module 1220 may provide guidance to the user to rotate and position the image capture device so that subsequent images are substantially aligned based on the field of view and / or characteristics of the image capture device.

[0206] The guidance module 1220 can provide visual guidance to the user. For example, the guidance module 1220 can place a mark or arrow in a viewer or display on the user system 1110 or the 3D and panoramic capture and stitching system 1102. In some embodiments, the user system 1110 can be a smart phone or tablet computer with a display. When taking one or more pictures, the guidance module 1220 can position one or more marks (e.g., different color marks or the same mark) on the output device and / or in the viewfinder. Then, the user can use the marks on the output device and / or the viewfinder to align the next image.

[0207] There are many techniques for guiding a user of the user system 1110 or the 3D and panoramic capture and stitching system 1102 to take multiple images to facilitate stitching the images into a panorama. When a panorama is derived from multiple images, the images can be stitched together. In order to improve the time, efficiency, and effectiveness of stitching images together while reducing the need to correct artifacts or misalignments, the image capture position module 1204 and the guidance module 1220 can help the user take multiple images in a position that improves the quality, time efficiency, and effectiveness of stitching images of the desired panorama.

[0208] For example, after taking a first picture, the display of the user system 1110 may include two or more objects, such as circles. The two circles may appear stationary relative to the environment, and the two circles may move with the user system 1110. When the two stationary circles are aligned with the two circles moving with the user system 1110, the image capture device and / or the user system 1110 may be aligned for the next image.

[0209] In some embodiments, after the image capture device captures the image, the image capture position module 1204 can obtain sensor measurements of the position of the image capture device (e.g., including orientation, tilt, etc.). The image capture position module 1204 can determine one or more edges of the captured image by calculating the positions of the edges of the field of view based on the sensor measurements. Additionally or alternatively, the image capture position module 1204 can determine one or more edges of the image by scanning the image captured by the image capture device, identifying objects within the image (e.g., using the machine learning models discussed herein), determining one or more edges of the image, and positioning an object (e.g., a circle or other shape) at the edge of a display on the user system 1110.

[0210] The image capture position module 1204 can display two objects within the display of the user system 1110, which indicate the positioning of the field of view of the next picture. These two objects can indicate the position of the edge of the environment indicating the presence of the last image. The image capture position module 1204 can continue to receive sensor measurements of the position of the image capture device and calculate two additional objects in the field of view. The two additional objects can be spaced apart by the same width as the first two objects. Although the first two objects can represent the edge of the captured image (e.g., the rightmost edge of the image), the next two additional objects representing the edge of the field of view can be on the opposite edge (e.g., the leftmost edge of the field of view). By allowing the user to physically align the first two objects on the edge of the image with the other two objects on the opposite edge of the field of view, the image capture device can be positioned to capture another image that can be stitched together more effectively without a tripod. This process can continue for each image until the user determines that the desired panorama has been captured.

[0211] Although multiple objects are discussed herein, it should be understood that the image capture location module 1204 can calculate the location of one or more objects used to locate the image capture device. The object can be any shape (e.g., a circle, an oval, a square, an emoji, an arrow, etc.). In some embodiments, the object can have different shapes.

[0212] In some embodiments, there may be a distance between objects representing the edge of a captured image, and there may be a distance between objects in the field of view. The user may be directed to move forward to move out of the way so that sufficient distance can exist between objects. Alternatively, as the image capture device approaches the correct position (e.g., by moving closer or further away from a position that will enable the next image to be taken in a position that will improve image stitching), the size of the objects in the field of view may change to match the size of the objects representing the edge of the captured image.

[0213] In some embodiments, the image capture location module 1204 can estimate the location of the image capture device using objects in the image captured by the image capture device. For example, the image capture location module 1204 can use GPS coordinates to determine the geographic location associated with the image. The image capture location module 1204 can use the location to identify a landmark that can be captured by the image capture device.

[0214] The image capture location module 1204 may include a 2D machine learning model that converts a 2D image into a 2D panoramic image. The image capture location module 1204 may include a 3D machine learning model that converts a 2D image into a 3D representation. In one example, a 3D representation may be utilized to display a three-dimensional tour or visualization of an interior and / or exterior environment.

[0215] The 2D machine learning model can be trained to stitch or help stitch two or more 2D images together to form a 2D panoramic image. The 2D machine learning model can be, for example, a neural network trained with 2D images that include physical objects in the images and object recognition information that trains the 2D machine learning model to recognize objects in subsequent 2D images. Objects in the 2D images can help determine (one or more) locations within the 2D image to help determine the edges of the 2D image, distortions in the 2D image, and to help align the images. In addition, objects in the 2D images can help determine artifacts in the 2D image, blending of artifacts or boundaries between two images, cutting images, and / or cropping images.

[0216] In some embodiments, the 2D machine learning model can be, for example, a neural network trained with 2D images, wherein the 2D images include depth information of the environment (e.g., from a LiDAR device or structured light device of the user system 1110 or a 3D and panoramic capture and stitching system 1102) and physical objects in the image to identify the physical object, the location of the physical object, and / or the location of the image capture device / field of view. The 2D machine learning model can identify the depth of the physical object and other aspects of the 2D image relative to help align and position the two 2D images for stitching (or stitching the two 2D images).

[0217] The 2D machine learning model may include any number of machine learning models (e.g., any number of models generated by a neural network, etc.).

[0218] The 2D machine learning model may be stored on the 3D and panoramic capture and stitching system 1102, the image stitching and processor system 1106, and / or the user system 1110. In some embodiments, the 2D machine learning model may be trained by the image stitching and processor system 1106.

[0219] The image capture position module 1204 can estimate the position of the image capture device (the position of the field of view of the image capture device) based on the seams between two or more 2D images from the stitching module 1206, the image distortion from the cropping module 1208 and / or the graph cut from the graph cut module 1210.

[0220] The stitching module 1206 can combine two or more 2D images to generate a 2D panorama based on the seams between the two or more 2D images from the stitching module 1206, the image warping and / or the graphic cutting from the cropping module 1208, which has a larger field of view than each of the two or more images.

[0221] The stitching module 1206 can be configured to align or "stitch together" two different 2D images that provide different perspectives of the same environment to generate a panoramic 2D image of the environment. For example, the stitching module 1206 can employ known or derived (e.g., using the techniques described herein) information about the capture locations and orientations of the respective 2D images to help stitch the two images together.

[0222] The stitching module 1206 may receive two 2D images. The first 2D image may have been captured just before the second image or within a predetermined time period. In various embodiments, the stitching module 1206 may receive positioning information of an image capture device associated with the first image, and then receive positioning information associated with the second image. The positioning information may be associated with the image based on positioning data from an IMU, GPS, and / or information provided by a user when the image was captured.

[0223] In some embodiments, the stitching module 1206 can utilize a 2D machine learning module to scan the two images to identify objects within the two images, including objects (or portions of objects) that may be shared by the two images. For example, the stitching module 1206 can identify corners, patterns on a wall, furniture, etc. that are shared at opposite edges of the two images.

[0224] The stitching module 1206 can align the edges of the two 2D images based on the positioning of the shared object (or portion of the object), positioning data from the IMU, positioning data from the GPS, and / or information provided by the user, and then combine the two edges of the images (i.e., "stitch" them together). In some embodiments, the stitching module 1206 can identify a portion of the two 2D images that overlap each other, and stitch the images at the overlapping location (e.g., using the positioning data and / or the results of the 2D machine learning model).

[0225] In various embodiments, the 2D machine learning model may be trained to combine or stitch two edges of an image using positioning data from an IMU, positioning data from a GPS, and / or information provided by a user. In some embodiments, the 2D machine learning model may be trained to identify common objects in two 2D images to align and position the 2D images, and then combine or stitch two edges of the images. In further embodiments, the 2D machine learning model may be trained to align and position the 2D images using positioning data and object recognition, and then stitch two edges of the images together to form all or part of a panoramic 2D image.

[0226] The stitching module 1206 can utilize depth information of the respective images (eg, pixels in the respective images, objects in the respective images, etc.) to facilitate aligning the respective 2D images with each other in association with generating a single 2D panoramic image of the environment.

[0227] The cropping module 1208 can solve the problem of two or more 2D images in the case where the image capture device does not remain in the same position when capturing the 2D image. For example, while capturing an image, the user may position the user system 1110 in a vertical position. However, while capturing another image, the user may position the user system at an angle. The resulting images may be misaligned and there may be parallax effects. Parallax effects occur when foreground and background objects are not aligned in the same way in the first image and the second image.

[0228] The cropping module 1208 may utilize a 2D machine learning model (by applying positioning information, depth information, and / or object recognition) to detect changes in the position of the image capture device in two or more images, and then measure the amount of change in the position of the image capture device. The cropping module 1208 may warp one or more 2D images so that when the images are stitched, the images may be able to align together to form a panoramic image while preserving certain characteristics of the images, such as keeping straight lines straight.

[0229] The output of the crop module 1208 may include the number of pixel columns and rows to offset each pixel of the image to straighten the image. The offset for each image may be output in the form of a matrix indicating the number of pixel columns and rows to offset each pixel of the image.

[0230] In some embodiments, the cropping module 1208 can determine the amount of image distortion to be performed on one or more of the multiple 2D images captured by the image capture device of the user system 1110 based on one or more image capture positions from the image capture position module 1204 or gaps between two or more 2D images from the stitching module 1206, graphic cuts from the graphic cut module 1210, or mixing of colors from the mixing module 1211.

[0231] The graphic cutting module 1210 can determine where to cut or slice one or more 2D images captured by the image capture device. For example, the graphic cutting module 1210 can use a 2D machine learning model to identify objects in two images and determine that they are the same object. The image capture position module 1204, the cropping module 1208 and / or the graphic cutting module 1210 can determine that the two images cannot be aligned even if distorted. The graphic cutting module 1210 can use information from the 2D machine learning model to identify segments of two images that can be stitched together (e.g., by cutting off a portion of one or both images to help with alignment and positioning). In some embodiments, the two 2D images may overlap at least a portion of the physical world represented in the image. The graphic cutting module 1210 can identify objects in the two images, such as the same chair. However, even after the image capture positioning and image distortion of the cropping module 1208, the images of the chair cannot be aligned to generate an undistorted panorama and will not correctly represent a portion of the physical world. The graphic cutting module 1210 can select one of the two images of the chair as the correct representation (e.g., based on misalignment, positioning, and / or artifacts of one image when compared to the other image) and cut the chair from the image with the misalignment, positioning error, and / or artifacts. The stitching module 1206 can then stitch the two images together.

[0232] The graphic cutting module 1210 may try two combinations, for example, cutting an image of a chair from a first image and stitching the first image minus the chair to a second image, to determine which graphic cutting generates a more accurate panoramic image. The output of the graphic cutting module 1210 may be a location where one or more of the multiple 2D images corresponding to the graphic cutting are cut, which generates a more accurate panoramic image.

[0233] The graphic cutting module 1210 can determine how to cut or slice one or more 2D images captured by the image capture device based on one or more image capture positions from the image capture position module 1204, the stitching or seams between two or more 2D images from the stitching module 1206, the image distortion from the cropping module 1208, and the graphic cutting from the graphic cutting module 1210.

[0234] The blending module 1211 can colorize at the seam (e.g., stitching) between two images so that the seam is not visible. Changes in lighting and shadows can cause the same object or surface to be output with slightly different colors or shades. The blending module can determine the amount of color blending required based on one or more image capture positions from the image capture position module 1204, stitching, image color along the seam from two images, image distortion from the cropping module 1208, and / or graphic cuts from the graphic cut module 1210.

[0235] In various embodiments, the blending module 1211 may receive a panoramic view from a combination of two 2D images and then sample colors along the seam of the two 2D images. The blending module 1211 may receive seam location information from the image capture location module 1204 to enable the blending module 1211 to sample colors along the seam and determine differences. If there is a significant difference in color along the seam between the two images (e.g., within a predetermined threshold of color, hue, brightness, saturation, and / or the like), the blending module 1211 may blend the two images of a predetermined size along the seam at the location where the difference exists. In some embodiments, the greater the difference in color or image along the seam, the greater the amount of space that can be blended along the seam of the two images.

[0236] In some embodiments, after blending, the blending module 1211 can rescan and sample the colors along the seam to determine if there are other differences in the images or colors that exceed a predetermined threshold of color, hue, brightness, saturation, and / or the like. If so, the blending module 1211 can identify the portion along the seam and continue blending that portion of the image. The blending module 1211 can continue to resample the image along the seam until there are no other portions of the image to blend (e.g., any differences in color are below a predetermined threshold).

[0237] The 3D image generator 1214 can receive a 2D panoramic image and generate a 3D representation. In various embodiments, the 3D image generator 1214 transforms the 2D panoramic image into a 3D representation using a 3D machine learning model. The 3D machine learning model can be trained using a 2D panoramic image and depth data (e.g., from a LiDAR sensor or a structured optical device) to generate a 3D representation. The 3D representation can be tested and reviewed for curation / screening and feedback. In some embodiments, the 3D machine learning model can be used together with the 2D panoramic image and depth data to generate a 3D representation.

[0238] In various embodiments, the accuracy, rendering speed, and quality of the 3D representations generated by the 3D image generator 1214 are greatly improved by utilizing the systems and methods described herein. For example, by rendering a 3D representation from a 2D panoramic image that has been aligned, positioned, and stitched using the methods described herein (e.g., through alignment and positioning information provided by hardware, through improved positioning caused by guidance provided to the user during image capture, by cropping and changing the distortion of the image, by cutting the image to avoid artifacts and overcome distortion, by blending images, and / or any combination), the accuracy, rendering speed, and quality of the 3D representation are improved. In addition, it should be understood that by utilizing 2D panoramic images that have been aligned, positioned, and stitched using the methods described herein, the training of 3D machine learning models can be greatly improved (e.g., in terms of speed and accuracy). In addition, in some embodiments, the 3D machine learning model can be smaller and less complex due to the reduction in processing and learning that will be used to overcome misalignment, positioning errors, distortions, poor graphic cutting, poor blending, artifacts, etc. to generate a reasonably accurate 3D representation.

[0239] The trained 3D machine learning model may be stored in the 3D and panoramic capture and stitching system 1102 , the image stitching and processor system 106 , and / or the user system 1110 .

[0240] In some embodiments, the 3D machine learning model may be trained using multiple 2D images and depth data from the image capture device of the user system 1110 and / or the 3D and panoramic capture and stitching system 1102. Additionally, the 3D image generator 1214 may be trained using image capture position information associated with each of the multiple 2D images from the image capture position module 1204, the seam locations for aligning or stitching each of the multiple 2D images from the stitching module 1206, the pixel offset(s) for each of the multiple 2D images from the cropping module 1208, and / or the graph cuts from the graph cut module 1210. In some embodiments, the 3D machine learning model can be used with the 2D panoramic image, depth data, image capture position information associated with each of the multiple 2D images from the image capture position module 1204, alignment or stitching of seam locations of each of the multiple 2D images from the stitching module 1206, (one or more) pixel offsets of each of the multiple 2D images from the cropping module 1208, and / or graphic cuts from the graphic cut module 1210 to generate a 3D representation.

[0241] The stitching module 1206 may be part of a 3D model that converts multiple 2D images into a 2D panorama or 3D panoramic image. In some embodiments, the 3D model is a machine learning algorithm, such as a predictive neural network model from 2D to 3D. The cropping module 1208 may be part of a 3D model that converts multiple 2D images into a 2D panorama or 3D panoramic image. In some embodiments, the 3D model is a machine learning algorithm, such as a predictive neural network model from 2D to 3D. The graphic cutting module 1210 may be part of a 3D model that converts multiple 2D images into a 2D panorama or 3D panoramic image. In some embodiments, the 3D model is a machine learning algorithm, such as a predictive neural network model from 2D to 3D. The mixing module 1211 may be part of a 3D machine learning model that converts multiple 2D images into a 2D panorama or 3D panoramic image. In some embodiments, the 3D model is a machine learning algorithm, such as a predictive neural network model from 2D to 3D.

[0242] 3D image generator 1214 may generate weights for each of image capture location module 1204, crop module 1208, graph cut module 1210, and blend module 1211, which may represent the reliability or "strength" or "weakness" of the module. In some embodiments, the sum of the weights of the modules is equal to 1.

[0243] In the event that depth data is not available for the plurality of 2D images, the 3D image generator 1214 may determine depth data for one or more objects in the plurality of 2D images captured by the image capture device of the user system 1110. In some embodiments, the 3D image generator 1214 may derive depth data based on images captured by a stereo image pair. Rather than determining depth data based on a passive stereo algorithm, the 3D image generator may evaluate the stereo image pair to determine data about the quality of photometric matching between the images at various depths (a more intermediate result).

[0244] The 3D image generator 1214 may be part of a 3D model that converts multiple 2D images into a 2D panorama or 3D panoramic image. In some embodiments, the 3D model is a machine learning algorithm, such as a predictive neural network model from 2D to 3D.

[0245] The captured 2D image data store 1216 may be any one structure and / or multiple structures suitable for captured image and / or depth data (e.g., an active database, a relational database, a self-referential database, a table, a matrix, an array, a flat file, a document-oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, and / or similar structures). The captured 2D image data store 1216 may store images captured by an image capture device of the user system 1110. In various embodiments, the captured 2D image data store 1216 stores depth data captured by one or more depth sensors of the user system 1110. In various embodiments, the captured 2D image data store 1216 stores image capture device parameters associated with the image capture device, or capture properties associated with each of a plurality of image captures, or depth captures used to determine a 2D panoramic image. In some embodiments, the image data store 1108 stores a panoramic 2D panoramic image. The 2D panoramic image may be determined by the 3D and panoramic capture and stitching system 1102 or the image stitching and processor system 106. Image capture device parameters may include illumination, color, image capture lens focal length, maximum aperture, tilt angle, etc. Capture properties may include pixel resolution, lens distortion, illumination, and other image metadata.

[0246] The 3D panoramic image data storage area 1218 may be any one structure and / or multiple structures suitable for 3D panoramic images (e.g., an active database, a relational database, a self-referential database, a table, a matrix, an array, a flat file, a document-oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, and / or similar structures). The 3D panoramic image data storage area 1218 may store 3D panoramic images generated by the 3D and panoramic capture and stitching system 1102. In various embodiments, the 3D panoramic image data storage area 1218 stores properties associated with an image capture device or properties associated with each of a plurality of image captures or depth captures used to determine a 3D panoramic image. In some embodiments, the 3D panoramic image data storage area 1218 stores 3D panoramic images. A 2D or 3D panoramic image may be determined by the 3D and panoramic capture and stitching system 1102 or the image stitching and processor system 106.

[0247] Fig.13 A flowchart 1300 of a 3D panoramic image capture and generation process according to some embodiments is depicted. In step 1302, the image capture device can use the image sensor 920 and WFOV lens 918 of Figure 9 to capture multiple 2D images. A wider FOV means that the environment capture system 402 will need fewer scans to obtain a 360° view. The WFOV lens 918 can also be wider horizontally and vertically. In some embodiments, the image sensor 920 captures RGB images. In one embodiment, the image sensor 920 captures black and white images.

[0248] In step 1304, the environment capture system may send the captured 2D images to the image stitching and processor system 1106. The image stitching and processor system 1106 may apply a 3D modeling algorithm to the captured 2D images to generate a panoramic 2D image. In some embodiments, the 3D modeling algorithm is a machine learning algorithm for stitching the captured 2D images into a panoramic 2D image. In some embodiments, step 1304 may be optional.

[0249] In step 1306, the LiDAR 912 and WFOV lens 918 of Figure 9 may capture LiDAR data. A wider FOV means that the environment capture system 400 will require fewer scans to obtain a 360° view.

[0250] In step 1308, the LiDAR data may be sent to the image stitching and processor system 1106. The image stitching and processor system 1106 may input the LiDAR data and the captured 2D images into a 3D modeling algorithm to generate a 3D panoramic image. The 3D modeling algorithm is a machine learning algorithm.

[0251] In step 1310, the image stitching and processor system 1106 generates a 3D panoramic image. The 3D panoramic image may be stored in the image data storage area 408. In one embodiment, the 3D panoramic image generated by the 3D modeling algorithm is stored in the image stitching and processor system 1106. In some embodiments, as the environment capture system is used to capture various portions of the physical environment, the 3D modeling algorithm may generate a visual representation of a floor plan of the physical environment.

[0252] In step 1312, the image stitching and processor system 1106 may provide at least a portion of the generated 3D panoramic image to the user system 1110. The image stitching and processor system 1106 may provide a visual representation of a floor plan of the physical environment.

[0253] The order of one or more steps of flowchart 1300 may be changed without affecting the final result of the 3D panoramic image. For example, the environment capture system may interleave image capture using an image capture device with LiDAR data or depth information capture using LiDAR 912. For example, an image capture device may capture an image of a segment of the physical environment using the image capture device, and then LiDAR 912 obtains depth information from the segment 1605. Once LiDAR 912 obtains depth information from a segment, the image capture device may continue to move to capture an image of another segment, and then LiDAR 912 obtains depth information from the segment, thereby interleaving image capture and depth information capture.

[0254] In some embodiments, the devices and / or systems discussed herein employ an image capture device to capture a 2D input image. In some embodiments, the one or more image capture devices 1116 may represent a single image capture device (or image capture lens). According to some of these embodiments, a user of a mobile device housing an image capture device may be configured to rotate about an axis to generate images at different capture orientations relative to the environment, wherein the common field of view of the images spans up to 360° horizontally.

[0255] In various embodiments, the devices and / or systems discussed herein may employ two or more image capture devices to capture 2D input images. In some embodiments, two or more image capture devices may be arranged on or within the same mobile housing in positions relative to each other so that their common field of view spans up to 360°. In some embodiments, paired image capture devices capable of generating stereoscopic image pairs (e.g., with slightly offset but partially overlapping fields of view) may be used. For example, a user system 1110 (e.g., a device comprising one or more image capture devices for capturing 2D input images) may include two image capture devices with horizontal stereo offset fields of view capable of capturing stereoscopic image pairs. In another example, the user system 1110 may include two image capture devices with vertical stereo offset fields of view capable of capturing vertical stereo image pairs. According to any of these examples, each camera may have a field of view spanning up to 360°. In this regard, in one embodiment, the user system 1110 may employ two panoramic cameras with vertical stereo offsets capable of capturing paired panoramic images of stereo pairs (with vertical stereo offsets).

[0256] The positioning component 1118 may include any hardware and / or software configured to capture user system position data and / or user system orientation data. For example, the positioning component 1118 includes an IMU to generate user system 1110 position data associated with the one or more image capture devices of the user system 1110 used to capture multiple 2D images. The positioning component 1118 may include a GPS unit to provide GPS coordinate information associated with multiple 2D images captured by the one or more image capture devices. In some embodiments, the positioning component 1118 can associate the user system's position data and orientation data with corresponding images captured using the one or more image capture devices of the user system 1110.

[0257] Various embodiments of the device provide users with 3D panoramic images of indoor and outdoor environments. In some embodiments, the device can effectively and quickly provide users with 3D panoramic images of indoor and outdoor environments using a single wide field of view (FOV) lens and a single light and detection and ranging sensor (LiDAR sensor).

[0258] The following is an example use case of the example device described herein. The following use case is one of the embodiments. Different embodiments of the device as discussed herein may include one or more features and capabilities similar to those of this use case.

[0259] Fig.14 Depicted is a flow diagram of a 3D and panoramic capture and stitching process 1400 in accordance with some embodiments. Fig.14The flowchart of refers to the 3D and panoramic capture and stitching system 1102 as including an image capture device, but in some embodiments, the data capture device may be a user system 1110 .

[0260] In step 1402, the 3D and panoramic capture and stitching system 1102 may receive a plurality of 2D images from at least one image capture device. The image capture device of the 3D and panoramic capture and stitching system 1102 may be or include a complementary metal oxide semiconductor (CMOS) image sensor. In various embodiments, the image capture device is a charge coupled device (CCD). In one example, the image capture device is a red-green-blue (RGB) sensor. In one embodiment, the image capture device is an IR sensor. Each of the plurality of 2D images may have a field of view that partially overlaps with at least one other image in the plurality of 2D images. In some embodiments, at least some of the plurality of 2D images are combined to produce a 360° view of a physical environment (e.g., indoors, outdoors, or both).

[0261] In some embodiments, all of the multiple 2D images are received from the same image capture device. In various embodiments, at least a portion of the multiple 2D images are received from two or more image capture devices of the 3D and panoramic capture and stitching system 1102. In one example, the multiple 2D images include a set of RGB images and a set of IR images, wherein the IR images provide depth data to the 3D and panoramic capture and stitching system 1102. In some embodiments, each 2D image can be associated with depth data provided by a LiDAR device. In some embodiments, each 2D image can be associated with positioning data.

[0262] In step 1404, the 3D and panoramic capture and stitching system 1102 may receive capture parameters and image capture device parameters associated with each of the received plurality of 2D images. The image capture device parameters may include illumination, color, image capture lens focal length, maximum aperture, field of view, etc. The capture properties may include pixel resolution, lens distortion, illumination, and other image metadata. The 3D and panoramic capture and stitching system 1102 may also receive positioning data and depth data.

[0263] In step 1406, the 3D and panoramic capture and stitching system 1102 may obtain the information received from steps 1402 and 1404 for stitching the 2D images to form a 2D panoramic image. Fig.15 The process of stitching 2D images is further discussed in the flowchart.

[0264] In step 1408, the 3D and panoramic capture and stitching system 1102 can apply the 3D machine learning model to generate a 3D representation. The 3D representation can be stored in the 3D panoramic image data storage area. In various embodiments, the 3D representation is generated by the image stitching and processor system 1106. In some embodiments, as the environment capture system is used to capture various parts of the physical environment, the 3D machine learning model can generate a visual representation of the floor plan of the physical environment.

[0265] In step 1410, the 3D and panoramic capture and stitching system 1102 may provide at least a portion of the generated 3D representation or model to the user system 1110. The user system 1110 may provide a visual representation of a floor plan of the physical environment.

[0266] In some embodiments, the user system 1110 may send the plurality of 2D images, capture parameters, and image capture parameters to the image stitching and processor system 1106. In various embodiments, the 3D and panoramic capture and stitching system 1102 may send the plurality of 2D images, capture parameters, and image capture parameters to the image stitching and processor system 1106.

[0267] The image stitching and processor system 1106 can process multiple 2D images captured by the image capture device of the user system 1110 and stitch them into a 2D panoramic image. The 2D panoramic image processed by the image stitching and processor system 1106 can have a higher pixel resolution than the 2D panoramic image obtained by the 3D and panoramic capture and stitching system 1102.

[0268] In some embodiments, the image stitching and processor system 106 may receive the 3D representation and output a 3D panoramic image having a higher pixel resolution than the pixel resolution of the received 3D panoramic image. The higher pixel resolution panoramic image may be provided to an output device, such as a computer screen, a projector screen, etc., having a higher screen resolution than the user system 1110. In some embodiments, the higher pixel resolution panoramic image may provide the panoramic image to the output device in greater detail and may be magnified.

[0269] Fig.15 Depicted showing Fig.141502, the image capture position module 1204 may determine image capture device position data associated with each image captured by the image capture device. The image capture position module 1204 may utilize the IMU of the user system 1110 to determine the position data of the image capture device (or the field of view of the lens of the image capture device). The position data may include the direction, angle, or tilt of one or more image capture devices when taking one or more 2D images. One or more of the cropping module 1208, the graphic cutting module 1210, or the blending module 1212 may utilize the direction, angle, or tilt associated with each of the multiple 2D images to determine how to warp, cut, and / or blend the images.

[0270] In step 1504, the cropping module 1208 may warp one or more of the plurality of 2D images so that two images can be aligned together to form a panoramic image while preserving certain characteristics of the image, such as keeping straight lines straight. The output of the cropping module 1208 may include the number of pixel columns and rows to offset each pixel of the image to straighten the image. The offset for each image may be output in the form of a matrix representing the number of pixel columns and pixel rows to offset each pixel of the image. In this embodiment, the cropping module 1208 may determine the amount of warping required for each of the plurality of 2D images based on an image capture pose estimate for each of the plurality of 2D images.

[0271] In step 1506, the graphic cutting module 1210 determines where to cut or slice one or more of the plurality of 2D images. In this embodiment, the graphic cutting module 1210 may determine where to cut or slice each of the plurality of 2D images based on image capture pose estimation and image distortion of each of the plurality of 2D images.

[0272] In step 1508, the stitching module 1206 can stitch the two or more images together using the edges of the images and / or cutouts of the images. The stitching module 1206 can align and / or position the images based on objects detected within the images, distortions, cuts of the images, etc.

[0273] In step 1510, the blending module 1212 can adjust the color at the seam (e.g., the stitching of two images) or the location where one image touches or connects to another image. The blending module 1212 can determine the desired amount of color blending based on one or more image capture positions from the image capture position module 1204, the image distortion from the cropping module 1208, and the graphic cut from the graphic cut module 1210.

[0274] The order of one or more steps of the 3D and panoramic capture and stitching process 1400 may be changed without affecting the final result of the 3D panoramic image. For example, the environment capture system may interleave image capture using an image capture device with LiDAR data or depth information capture. For example, an image capture device may capture a portion of the physical environment using an image capture device. Fig.16 1605, and then the LiDAR 612 obtains depth information from segment 1605. Once the LiDAR obtains depth information from segment 1605, the image capture device may continue to move to capture an image of another segment 1610, and then the LiDAR 612 obtains depth information from segment 1610, thereby interleaving image capture and depth information capture.

[0275] Fig.16 A block diagram of an example digital device 1602 is depicted in accordance with some embodiments. Any of the user system 1110, the 3D panoramic capture and stitching system 1102, and the image stitching and processor system may include an instance of the digital device 1602. The digital device 1602 includes a processor 1604, a memory 1606, a storage device 1608, an input device 1610, a communication network interface 1612, an output device 1614, an image capture device 1616, and a positioning component 1618. The processor 1604 is configured to execute executable instructions (e.g., a program). In some embodiments, the processor 1604 includes a circuit or any processor capable of processing executable instructions.

[0276] The memory 1606 stores data. Some examples of the memory 1606 include storage devices such as RAM, ROM, RAM cache, virtual memory, etc. In various embodiments, working data is stored in the memory 1606. The data in the memory 1606 can be cleared or eventually transferred to the storage device 1608.

[0277] Storage device 1608 includes any storage device configured to retrieve and store data. Some examples of storage device 1608 include a flash drive, a hard drive, an optical drive, and / or a magnetic tape. Each of memory 1606 and storage device 1608 includes a computer-readable medium that stores instructions or programs that can be executed by processor 1604.

[0278] Input device 1610 is any device for inputting data (e.g., a touch keyboard, a stylus). Output device 1614 outputs data (e.g., a speaker, a display, a virtual reality headset). It should be understood that storage device 1608, input device 1610, and output device 1614. In some embodiments, output device 1614 is optional. For example, a router / switch may include processor 1604 and memory 1606 as well as devices for receiving and outputting data (e.g., communication network interface 1612 and / or output device 1614).

[0279] The communication network interface 1612 may be coupled to a network (e.g., the communication network 104) via the communication network interface 1612. The communication network interface 1612 may support communication via an Ethernet connection, a serial connection, a parallel connection, and / or an ATA connection. The communication network interface 1612 may also support wireless communications (e.g., 802.16a / b / g / n, WiMAX, LTE, Wi-Fi). It will be apparent that the communication network interface 1612 may support many wired and wireless standards.

[0280] Components may be hardware or software. In some embodiments, components may configure one or more processors to perform functions associated with the components. Although different components are discussed herein, it should be understood that the server system may include any number of components that perform any or all functions discussed herein.

[0281] The digital device 1602 may include one or more image capture devices 1616. The one or more image capture devices 1616 may include, for example, an RGB camera, an HDR camera, a video camera, etc. According to some embodiments, the one or more image capture devices 1616 may also include a video camera capable of capturing video. In some embodiments, the one or more image capture devices 1616 may include an image capture device that provides a relatively standard field of view (e.g., approximately 75°). In other embodiments, the one or more image capture devices 1616 may include a camera that provides a relatively wide field of view (e.g., from approximately 120° to 360°), such as a fisheye camera, etc. (e.g., the digital device 1602 may include the environment capture system 400 or be included in the environment capture system 400).

[0282] Components may be hardware or software. In some embodiments, components may configure one or more processors to perform functions associated with the components. Although different components are discussed herein, it should be understood that the server system may include any number of components that perform any or all functions discussed herein.

Claims

1. A method for an apparatus for acquiring images and depth information in a 360-degree scene, wherein the apparatus comprises a first motor, an image capturing device, and a depth information capturing device, the method comprising: capturing, with the image capture device pointed in a first direction, a plurality of images at different exposures in a first field of view of the 360-degree scene, wherein the first field of view is a first FOV; Rotating the depth information capture device and the image capture device around a first substantially vertical axis by the first motor until the image capture device points to a second direction, and capturing depth information of a first portion of the 360-degree scene by the rotating depth information capture device; capturing, with the image capture device pointed in the second direction, a plurality of images at different exposures in a second FOV overlapping the first FOV of the 360 ​​degree scene; Rotating the depth information capturing device and the image capturing device around the first axis by the first motor until the image capturing device points to a third direction, and capturing depth information of a second portion of the 360-degree scene by the rotating depth information capturing device; as well as A plurality of images at different exposures in a third FOV overlapping the second FOV of the 360 ​​degree scene are captured by the image capture device pointed in the third direction.

2. The method according to claim 1, further comprising: Rotating the depth information capturing device and the image capturing device around the first axis by the first motor until the image capturing device points to a fourth direction; and A plurality of images at different exposures in a fourth FOV overlapping the first FOV and the third FOV of the 360 ​​degree scene are captured by the image capture device pointed in the fourth direction.

3. The method according to claim 2, further comprising: While the depth information capturing device and the image capturing device are rotated around the first axis by the first motor until the image capturing device points to the fourth direction, the depth information capturing device captures depth information of a third portion of the 360-degree scene.

4. The method according to claim 3, wherein capturing the depth information of the first part, the second part and the third part of the 360-degree scene by the depth information capture device includes capturing the depth information of a first plurality of segments, a second plurality of segments and a third plurality of segments of the 360-degree scene by the depth information capture device.

5. The method according to claim 1, further comprising: Stitching one or more of the plurality of images in the first FOV, the second FOV, and the third FOV together into a panoramic image of the 360-degree scene through an image stitching module; as well as The depth information of the first part, the second part and the third part of the 360-degree scene is combined with the panoramic image of the 360-degree scene by a 3D image generation module to generate the 3D panoramic image of the 360-degree scene.

6. The method according to claim 5, further comprising: Two or more of the multiple images in the first FOV, the second FOV and the third FOV are mixed into a first mixed image, a second mixed image and a third mixed image of the first FOV, the second FOV and the third FOV respectively through a mixing module, wherein one or more of the multiple images in the first FOV, the second FOV and the third FOV are stitched together into the panoramic image of the 360-degree scene through the image stitching module, including stitching the first mixed image, the second mixed image and the third mixed image together into the panoramic image of the 360-degree scene.

7. The method according to claim 6, wherein mixing two or more of the plurality of images in the first FOV, the second FOV, and the third FOV into the first mixed image, the second mixed image, and the third mixed image of the first FOV, the second FOV, and the third FOV by the mixing module comprises mixing two or more of the plurality of images in the first FOV, the second FOV, and the third FOV into a first high dynamic range image, a second high dynamic range image, and a third high dynamic range image of the first FOV, the second FOV, and the third FOV, wherein the first high dynamic range image, the second high dynamic range image, and the third high dynamic range image are a first HDR image, a second HDR image, and a third HDR image; and Wherein, stitching one or more of the multiple images in the first FOV, the second FOV, and the third FOV together into the panoramic image of the 360-degree scene through the image stitching module includes stitching the first HDR image, the second HDR image, and the third HDR image in the first FOV, the second FOV, and the third FOV together into the panoramic image of the 360-degree scene.

8. The method of claim 6, wherein the apparatus further comprises a second motor, wherein the depth information capturing device comprises a light detection and ranging device (i.e., a LiDAR device) and a reflector, the method further comprising rotating the reflector about a second substantially horizontal axis by the second motor and emitting a plurality of laser pulses to the reflector by the LiDAR device, the reflector in turn emitting the plurality of laser pulses in one or more revolutions about the second axis and receiving a corresponding plurality of reflected laser pulses, and the LiDAR device in turn receiving the plurality of reflected laser pulses and thereby generating the depth information, wherein the first mixture of the first FOV, the second FOV, and the third FOV The image, the second mixed image and the third mixed image each include a plurality of pixels, wherein each of the plurality of pixels is associated with a digital coordinate identifying a position of the pixel in the scene, and wherein each of the reflected laser pulses is similarly associated with a corresponding digital coordinate identifying a position of the depth information generated thereby, and combining the depth information of the 360-degree scene with the panoramic image of the 360-degree scene by the 3D image generation module to generate the 3D panoramic image of the 360-degree scene includes combining the depth information at a position in the 360-degree scene with the panoramic image at the same position in the 360 ​​scene to generate the 3D panoramic image of the 360-degree scene.

9. A method according to claim 1, wherein the image capture device includes a lens located at a parallax-free point substantially at the center of the first axis, so that as the first motor causes the image capture device to rotate about the first axis to capture the multiple images at different exposures in the corresponding first overlapping FOV, second overlapping FOV and third overlapping FOV while the image capture device is pointed in the first direction, the second direction and the third direction, the parallax effect between the multiple images in the first overlapping FOV, the second overlapping FOV and the third overlapping FOV is reduced or eliminated.

10. The method of claim 1 , wherein the apparatus further comprises a second motor, the depth information capturing device comprises a light detection and ranging device (i.e., a LiDAR device) and a reflector, and the method further comprises rotating the reflector about a second substantially horizontal axis by the second motor and emitting a plurality of laser pulses to the reflector by the LiDAR device, the reflector in turn emitting the plurality of laser pulses in one or more revolutions about the second axis and receiving a corresponding plurality of reflected laser pulses, and the LiDAR device in turn receiving the plurality of reflected laser pulses and thereby generating the depth information, such that as the image capturing device and the LiDAR device rotate about the first axis by the first motor and the reflector rotates about the second axis by the second motor, the LiDAR device captures the depth information of the three dimensions of the 360 ​​scene.

11. A device for collecting image and depth information for use in a 360-degree scene, comprising: First Motor; an image capture device that captures a plurality of images at different exposures in a first field of view (FOV) of the 360 ​​degree scene while pointing in a first direction; and A depth information capture device, wherein the depth information capture device and the image capture device are rotated around a first basically vertical axis by the first motor until the image capture device points to a second direction, the rotating depth information capture device captures depth information of a first portion of the 360-degree scene, and the image capture device pointing in the second direction captures a plurality of images at different exposures in a second FOV overlapping with the first FOV of the 360-degree scene, the depth information capture device and the image capture device are rotated around the first axis by the first motor until the image capture device points to a third direction, the rotating depth information capture device captures depth information of a portion of the 360-degree scene, and the image capture device pointing in the third direction captures a plurality of images at different exposures in a third FOV overlapping with the second FOV of the 360-degree scene.

12. An apparatus according to claim 11, wherein the depth information capture device and the image capture device are rotated around the first axis by the first motor until the image capture device points to a fourth direction; and while pointing in the fourth direction, the image capture device captures a plurality of images at different exposures in a fourth FOV overlapping with the first FOV and the third FOV of the 360-degree scene.

13. The apparatus of claim 12, wherein the depth information capturing device captures depth information of a third portion of the 360 ​​degree scene while being rotated by the first motor about the first axis until the image capturing device points to the fourth direction.

14. The device according to claim 13, wherein the depth information capturing device being rotated captures the depth information of the first part, the second part and the third part of the 360-degree scene, including the depth information capturing device being rotated captures the depth information of the first multiple segments, the second multiple segments and the third multiple segments of the 360-degree scene.

15. The apparatus according to claim 11, further comprising: an image stitching module, which stitches together one or more of the plurality of images in the first FOV, the second FOV, and the third FOV into a panoramic image of the 360-degree scene; as well as A 3D image generation module combines the depth information of the first part, the second part and the third part of the 360-degree scene with the panoramic image of the 360-degree scene to generate the 3D panoramic image of the 360-degree scene.

16. The apparatus according to claim 15, further comprising: a blending module that blends two or more of the plurality of images in the first FOV, the second FOV, and the third FOV into a first blended image, a second blended image, and a third blended image of the first FOV, the second FOV, and the third FOV; The image stitching module stitches one or more of the multiple images in the first FOV, the second FOV and the third FOV together into the panoramic image of the 360-degree scene, including the image stitching module stitching the first mixed image, the second mixed image and the third mixed image together into the panoramic image of the 360-degree scene.

17. The device according to claim 16, wherein the mixing module mixes two or more of the multiple images in the first FOV, the second FOV and the third FOV into the first mixed image, the second mixed image and the third mixed image of the first FOV, the second FOV and the third FOV, including the mixing module mixing two or more of the multiple images in the first FOV, the second FOV and the third FOV into the first high dynamic range image, the second high dynamic range image and the third high dynamic range image of the first FOV, the second FOV and the third FOV, wherein the first high dynamic range image, the second high dynamic range image and the third high dynamic range image are namely the first HDR image, the second HDR image and the third HDR image; and the image stitching module stitches one or more of the multiple images in the first FOV, the second FOV and the third FOV together into the panoramic image of the 360-degree scene, including the image stitching module stitches the first HDR image, the second HDR image and the third HDR image in the first FOV, the second FOV and the third FOV together into the panoramic image of the 360-degree scene.

18. The apparatus of claim 16, further comprising a second motor, wherein the depth information capturing device comprises a light detection and ranging device (i.e., a LiDAR device) and a reflector, the second motor causing the reflector to rotate about a second substantially horizontal axis while the LiDAR device transmits a plurality of laser pulses to the reflector, the reflector in turn transmitting the plurality of laser pulses in one or more revolutions about the second axis and receiving a corresponding plurality of reflected laser pulses, and the LiDAR device in turn receiving the plurality of reflected laser pulses and generating the depth information therefrom, wherein the first mixed image, the second mixed image of the first FOV, the second FOV, and the third FOV The image and the third mixed image each include a plurality of pixels, wherein each of the plurality of pixels is associated with a digital coordinate identifying a position of the pixel in the scene, and wherein each of the reflected laser pulses is similarly associated with a corresponding digital coordinate identifying a position of the depth information generated thereby, and combining the depth information of the 360-degree scene with the panoramic image of the 360-degree scene by the 3D image generation module to generate the 3D panoramic image of the 360-degree scene includes combining the depth information at a position in the 360-degree scene with the panoramic image at the same position in the 360 ​​scene to generate the 3D panoramic image of the 360-degree scene.

19. An apparatus according to claim 11, wherein the image capture device includes a lens located at a parallax-free point substantially at the center of the first axis, so that as the first motor causes the image capture device to rotate about the first axis to capture the multiple images at different exposures in the corresponding first overlapping FOV, second overlapping FOV and third overlapping FOV while the image capture device is pointed in the first direction, the second direction and the third direction, the parallax effect between the multiple images in the first overlapping FOV, the second overlapping FOV and the third overlapping FOV is reduced or eliminated.

20. The apparatus according to claim 11 further comprises a second motor, wherein the depth information capturing device comprises a light detection and ranging device, i.e., a LiDAR device, and a reflector, the reflector being rotated about a second substantially horizontal axis by the second motor while the LiDAR device emits a plurality of laser pulses to the reflector, the reflector in turn emitting the plurality of laser pulses in one or more revolutions about the second axis and receiving a corresponding plurality of reflected laser pulses, the LiDAR device in turn receiving the plurality of reflected laser pulses and thereby generating the depth information, so that as the image capturing device and the LiDAR device are rotated about the first axis by the first motor and the reflector is rotated about the second axis by the second motor, the LiDAR device captures the three-dimensional depth information of the 360 ​​scene.

Images