System and method for capturing and generating panoramic 3D images - Patents.com

Abstract

1. An apparatus comprising: a housing; a mount configured to be coupled to a motor to move the apparatus horizontally; a wide-angle lens positioned above the mount and therefore aligned with an axis of rotation about which the apparatus rotates; an image capture device within the housing, the image capture device configured to receive a two-dimensional image of an environment through the wide-angle lens; and a LiDAR device within the housing, the LiDAR device configured to generate depth data based on the environment.

Description

[Technical field]

[0001] SUMMARY OF THE DISCLOSURE Embodiments of the present invention generally relate to capturing and stitching panoramic images of scenes of a physical environment. [Background technology]

[0002] The popularity of providing three-dimensional (3D) panoramic images of the real world has spawned many solutions capable of capturing two-dimensional (2D) images and creating a 3D image based on the captured 2D images. There are hardware solutions and software applications (i.e., "apps") that can capture multiple 2D images and stitch them together into a panoramic image. Summary of the Invention [Problem to be solved by the invention]

[0003] Technologies exist for capturing and generating 3D data from buildings. However, existing technologies are generally unable to capture and generate 3D renderings of areas with bright light. Windows with sunlight streaming in, or areas of floors or walls with bright light, typically appear as holes in the 3D rendering and may require additional post-production work to fill them. This increases the turnaround time and reliability of the 3D rendering. Furthermore, outdoor environments also pose a challenge to many existing 3D capture devices, as structured lighting is not available for capturing 3D images.

[0004] Other limitations of existing technologies for capturing and generating 3D data include the amount of time required to capture and process the digital images necessary to generate a 3D panoramic image. [Means for solving the problem]

[0005] An example apparatus includes: a housing and a mount configured to be coupled to a motor to move the apparatus horizontally; a wide-angle lens coupled to the housing, the wide-angle lens positioned above the mount and thus aligned with an axis of rotation about which the apparatus rotates when coupled to the motor; an image capture device within the housing, the image capture device configured to receive a two-dimensional image of an environment through the wide-angle lens; and a LiDAR device within the housing, the LiDAR device configured to generate depth data based on the environment.

[0006] The image capture device may include a housing, a first motor, a wide-angle lens, an image sensor, a mount, a LiDAR, a second motor, and a mirror. The housing may have a front and a back. The first motor may be coupled to the housing at a first position between the front and back of the housing, the first motor configured to horizontally turn the image capture device approximately 270° about a vertical axis. The wide-angle lens may be coupled to the housing at a second position between the front and back of the housing along the vertical axis, the second position being a cross point, the wide-angle lens having a field of view away from the front of the housing. The image sensor may be coupled to the housing and may be configured to generate an image signal from light received by the wide-angle lens. The mount may be coupled to the first motor. The LiDAR may be coupled to the housing at a third position, the LiDAR configured to generate laser pulses and generate a depth signal. The second motor may be coupled to the housing. The mirror can be coupled to the second motor, the second motor can be configured to rotate the mirror about a horizontal axis, the mirror including an angled surface configured to receive the laser pulses from the LiDAR and direct the laser pulses about the horizontal axis.

[0007] In some embodiments, the image sensor is configured to generate a first plurality of images at different exposures when the image capture device is stationary and facing a first direction. The first motor may be configured to turn the image capture device about the vertical axis after generating the first plurality of images. In various embodiments, the image sensor does not generate images while the first motor is turning the image capture device, and the LiDAR generates a depth signal based on the laser pulses while the first motor is turning the image capture device. The image sensor may be configured to generate a second plurality of images at different exposures when the image capture device is stationary and facing a second direction, and the first motor is configured to turn the image capture device 90° about the vertical axis after generating the second plurality of images. The image sensor may be configured to generate a third plurality of images at the different exposures when the image capture device is stationary and facing a third orientation, and the first motor is configured to turn the image capture device 90 degrees about the vertical axis after generating the third plurality of images. The image sensor may be configured to generate a fourth plurality of images at the different exposures when the image capture device is stationary and facing a fourth orientation, and the first motor is configured to turn the image capture device 90 degrees about the vertical axis after generating the fourth plurality of images.

[0008] In some embodiments, the system may further comprise a processor configured to blend frames of the first plurality of images before the image sensor generates the second plurality of images. A remote digital device may be in communication with the image capture device and may be configured to generate a 3D visualization based on the first, second, third, and fourth plurality of images and the depth signal, the remote digital device configured to generate the 3D visualization without using images other than the first, second, third, and fourth plurality of images. In some embodiments, the first, second, third, and fourth plurality of images are generated during a combination of turns that turn the image capture device 270° about the vertical axis. The speed or rotation of the mirror about the horizontal axis increases as the first motor turns the image capture device. The angled surface of the mirror may be 90°. In some embodiments, the LiDAR emits the laser pulses in a direction opposite to the front surface of the housing.

[0009] One example method includes: receiving light from a wide angle lens of an image capture device, the wide angle lens being coupled to a housing of the image capture device, the light being received at a field of view of the wide angle lens, the field of view extending away from a front surface of the housing; generating a first plurality of images with the light from the wide angle lens by an image sensor of an image capture device, the image sensor being coupled to the housing, the first plurality of images being at different exposures; turning the image capture device horizontally approximately 270° about a vertical axis by a first motor, the first motor rotating the image capture device horizontally approximately 270° about a vertical axis of the housing. the wide angle lens is at a second position along the vertical axis, the second position being a cross point; rotating a mirror having an angled surface about a horizontal axis by a second motor, the second motor being coupled to the housing; generating laser pulses by the LiDAR, the LiDAR being coupled to the housing at a third position, the laser pulses pointing at the rotating mirror while the image capture device is turning horizontally; and generating a depth signal by the LiDAR based on the laser pulses.

[0010] The step of generating the first plurality of images with the image sensor may occur before the image capture device turns horizontally. In some embodiments, the image sensor does not generate images while the first motor is turning the image capture device, and the LiDAR generates the depth signal based on the laser pulses while the first motor is turning the image capture device.

[0011] The method may further include: generating a second plurality of images at the different exposures by the image sensor while the image capture device is stationary and facing a second direction; and turning the image capture device 90 degrees about the vertical axis by the first motor after generating the second plurality of images.

[0012] In some embodiments, the method may further include: generating a third plurality of images at the different exposures by the image sensor when the image capture device is stationary and facing a third direction; and turning the image capture device 90° about the vertical axis by the first motor after generating the third plurality of images. The method may further include generating a fourth plurality of images at the different exposures by the image sensor when the image capture device is stationary and facing a fourth direction. The method may include generating a 3D visualization using the first, second, third and fourth plurality of images and based on the depth signal, wherein the generating the 3D visualization does not use any other images.

[0013] In some embodiments, the method may further include blending frames of the first plurality of images before the image sensor generates the second plurality of images. The first, second, third and fourth plurality of images may be generated during a combination of turns that rotate the image capture device 270° about the vertical axis. In some embodiments, a speed or rotation of the mirror about the horizontal axis increases as the first motor turns the image capture device. [Brief description of the drawings]

[0014] [Figure 1a] FIG. 1a illustrates a dollhouse view of an exemplary environment, such as a house, according to some embodiments. [Figure 1b]FIG. 1b illustrates a floor plan of the first floor of a house, according to some embodiments. [Diagram 2] FIG. 2 shows an exemplary view from eye level of a living room that may be part of a virtual walkthrough. [Diagram 3] FIG. 3 illustrates an example of an environmental capture system according to some embodiments. [Figure 4] FIG. 4 illustrates a schematic diagram of an environmental capture system in some embodiments. [Diagram 5] FIG. 5 is an illustration of laser pulses from a LiDAR around an environmental capture system in some embodiments. [Figure 6a] FIG. 6a shows a side view of the environmental capture system. [Figure 6b] FIG. 6b shows a top view of the environmental capture system in some embodiments. [Figure 7] FIG. 7 illustrates a component diagram of an example environmental capture system according to some embodiments. [Figure 8a] FIG. 8a shows exemplary lens dimensions in some embodiments. [Figure 8b] FIG. 8b illustrates an exemplary lens design specification according to some embodiments. [Figure 9a] FIG. 9a illustrates a block diagram of an example of an environment capture system according to some embodiments. [Figure 9b] FIG. 9b illustrates a block diagram of an exemplary SOM PCBA of an environmental capture system according to some embodiments. [Figures 10a-10c] 10a-10c illustrate a process of an environment capture system for taking images in some embodiments. [Figure 11] FIG. 11 shows a block diagram of an exemplary environment in which images can be captured and stitched to form a 3D visualization, according to some embodiments. [Figure 12]FIG. 12 is a block diagram of an example alignment and stitching system, according to some embodiments. [Figure 13] FIG. 13 illustrates a flowchart of a 3D panoramic image capture and generation process according to some embodiments. [Figure 14] FIG. 14 illustrates a flowchart of a 3D and panoramic capture and stitching process according to some embodiments. [Figure 15] FIG. 15 shows a flow chart illustrating further details of one step of the 3D and panoramic capture and stitching process of FIG. [Figure 16] FIG. 16 illustrates a block diagram of an exemplary digital device according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Many of the innovations described herein are made with reference to the drawings. Like reference numerals are used to refer to like elements. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding. However, it may be apparent that different innovations can be practiced without these specific details. In other instances, well-known structures and components are shown in block diagram form to facilitate description of the innovations.

[0016] Various embodiments of the device provide a user with 3D panoramic images of indoor and outdoor environments. In some embodiments, the device can efficiently and quickly provide a user with 3D panoramic images of indoor and outdoor environments using a single wide field-of-view (FOV) lens and a single light and detection and ranging sensor (LiDAR sensor).

[0017] The following is an exemplary use example of the exemplary device described herein. The following use example is one of multiple embodiments. As described herein, different embodiments of the device may include one or more features and functions similar to this use example.

[0018] FIG. 1a is a dollhouse view 100 of an exemplary environment, such as a house, according to some embodiments. The dollhouse view 100 provides an overview of the exemplary environment captured by an environment capture system (described herein). A user can interact with the dollhouse view 100 on a user system by switching between different views of the exemplary environment. For example, a user can interact with area 110 to trigger a floor plan of the first floor of the house as shown in FIG. 1b. In some embodiments, a user can interact with icons in the dollhouse view 100, such as icons 120, 130, 140, to provide a walk-through view (e.g., for a 3D walk-through), a floor plan, or a measurement view, respectively.

[0019] Figure 1b shows a floor plan of the first floor of a house, according to some embodiments. The floor plan is an overhead view of the first floor of the house. A user can interact with an area of ​​the floor plan, such as area 150, to trigger an eye-level view of a particular portion of the floor plan, such as the living room. An example of an eye-level view of the living room can be seen in Figure 2, which can be part of the virtual walkthrough.

[0020] A user may interact with a portion of floor plan 200 that corresponds to area 150 in FIG. 1b. The user may move the view around the room as if the user were actually in the living room. In addition to a horizontal 360° view of the living room, the user may also view or manipulate the floor or ceiling of the living room. Furthermore, the user may pass through the living room to other parts of the house by interacting with specific areas of the portion of floor plan 200, e.g. areas 210, 220. When the user interacts with area 220, the environment capture system may provide a walking transition between an area of ​​the house that generally corresponds to the area of ​​the house indicated by area 150 and an area of ​​the house that generally corresponds to the area of ​​the house indicated by area 220.

[0021] 3 illustrates an example of an environmental capture system 300 according to some embodiments. The environmental capture system 300 includes a lens 310, a housing 320, a mount attachment 330, and a movable cover 340.

[0022] In use, the environmental capture system 300 may be positioned in an environment, such as a room. The environmental capture system 300 may be positioned on a support (e.g., a tripod). The movable cover 340 may be moved to expose the LiDAR and the rapidly rotatable mirror. When activated, the environmental capture system 300 may take a burst of images and then turn using the motor. The environmental capture system 300 may turn on the mount attachment 330. When turning, the LiDAR may take measurements (while turning, the environmental capture system cannot take images). Once facing a new direction, the environmental capture system may take a burst of images and then turn to the next direction.

[0023] For example, after positioning, the user may command the environment capture system 300 to begin a sweep. A sweep may be as follows: (1) Exposure estimation and subsequent capture of HDR RGB images Rotate 90 degrees and capture depth data (2) Exposure estimation and subsequent capture of HDR RGB images Rotate 90 degrees and capture depth data (3) Exposure estimation and subsequent capture of HDR RGB images Rotate 90 degrees and capture depth data (4) Exposure estimation and subsequent capture of HDR RGB image Rotate 90° (total 360°) and capture depth data

[0024] For each burst there can be any number of images at different exposures, and the environmental capture system can blend any number of images of a burst together while waiting for another frame and / or while waiting for the next burst.

[0025] The housing 320 may protect the electronic components of the environmental capture system 300 and may provide an interface for user interaction, such as a power button, a scan button, etc. For example, the housing 320 may include a movable cover 340, which may be movable to uncover the LiDAR. The housing 320 may further include electronic interfaces, such as a power adapter and indicator lights. In some embodiments, the housing 320 is a molded plastic housing. In various embodiments, the housing 320 is a combination of one or more of a plastic, a metal, and a polymer.

[0026] The lens 310 may be part of a lens assembly. Further details of the lens assembly can be explained in the description of FIG. 7. The lens 310 is strategically placed at the center of the axis of rotation 305 of the environment capture system 300. In this example, the axis of rotation 305 is on the x-y plane. By placing the lens 310 at the center of the axis of rotation 305, the parallax effect can be eliminated or reduced. Parallax is an error caused by rotation of an image capture device around a non-parallax point (NPP). In this example, the NPP can be found at the center of the entrance pupil of the lens.

[0027] For example, assume a panoramic image of a physical environment is generated using four images captured by the environment capture system 300, where there is 25% overlap between the images of the panoramic image. In the absence of parallax, 25% of an image can be accurately overlapped with another image of the same area of ​​the physical environment. Eliminating or reducing the parallax effect of multiple images captured by the image sensor through the lens 310 can aid in stitching multiple images into a 2D panoramic image.

[0028] The lens 310 may include a wide field of view (e.g., the lens 310 may be a fisheye lens). In some embodiments, the lens may have a horizontal FOV (HFOV) of at least 148° and a vertical FOV (VFOV) of at least 94°.

[0029] Mount attachment 330 may enable environmental capture system 300 to be attached to a mount. The mount may enable environmental capture system 300 to be coupled to a tripod, a flat surface, or a motorized mount (e.g., for moving environmental capture system 300). In some embodiments, the mount may enable environmental capture system 300 to be rotated along a horizontal axis.

[0030] In some embodiments, environmental capture system 300 may include a motor for turning environmental capture system 300 horizontally about mount attachment 330 .

[0031] In some embodiments, a motorized mount may move the environmental capture system 300 along a horizontal axis, a vertical axis, or both. In some embodiments, the motorized mount may rotate or move in the x-y plane. The mount attachment 330 may be used to couple the environmental capture system 300 to a motorized mount, tripod, or the like to stabilize the environmental capture system 300 so that sway can be reduced or minimized. In another example, the mount attachment 330 may be coupled to a motorized mount that can rotate the 3D environmental capture system 300 at a stable, known rate, which aids the LiDAR in determining the (x, y, z) coordinates of each laser pulse of the LiDAR.

[0032] 4 illustrates, in some embodiments, a schematic diagram of an environmental capture system 400. The schematic diagram illustrates the environmental capture system 400 (which may be one example of the environmental capture system 300 of FIG. 3) from various views, such as a front view 410, a top view 420, a side view 430, and a back view 440. In these schematic diagrams, the environmental capture system 400 may include any hollow portions shown in the side view 430.

[0033] In some embodiments, environmental capture system 400 has a width of 75 mm, a height of 180 mm, and a depth of 189 mm. It will be understood that environmental capture system 400 may have any width, height, or depth. In various embodiments, the ratio of width to depth in the first example is maintained regardless of the specific measurements.

[0034] The housing of the 3D environmental capture system 400 may protect the electronic components of the environmental capture system 400 and may provide an interface for user interaction (e.g., the screen of the rear view 440). The housing may further include electronic interfaces such as a power adapter and indicator lights. In some embodiments, the housing is a molded plastic housing. In various embodiments, the housing is a combination of one or more of plastic, metal, and polymer. The environmental capture system 400 may include a movable cover, which may be movable to uncover the LiDAR and to protect the LiDAR from the elements when not in use.

[0035] The lens shown in front view 410 may be part of a lens assembly. Similar to environmental capture system 300, the lens of environmental capture system 400 is strategically located at the center of rotation axis 305. The lens may include a wide field of view. In various embodiments, the lens shown in front view 410 is concave and the housing is flared so that the wide angle lens is just at the cross point (e.g., directly above the midpoint of the mount and / or motor) but still able to capture images without interference from the housing.

[0036] A mount attachment at the base of environmental capture system 400 may allow the environmental capture system to be attached to a mount. The mount may allow environmental capture system 400 to be coupled to a tripod, a flat surface, or a motorized mount (e.g., for moving environmental capture system 400). In some embodiments, the mount may be coupled with an internal motor for turning environmental capture system 400 around the mount.

[0037] In some embodiments, the mount can allow the environmental capture system 400 to rotate along a horizontal axis. In various embodiments, a motorized mount can move the environmental capture system 400 along a horizontal axis, a vertical axis, or both. The mount attachment can allow the environmental capture system 400 to be coupled to a motorized mount, tripod, or the like to stabilize the environmental capture system 400, thereby reducing or minimizing sway. In another example, the mount attachment can be coupled to a motorized mount that can rotate the environmental capture system 400 at a stable, known rate, which aids the LiDAR in determining the (x, y, z) coordinates of each laser pulse of the LiDAR.

[0038] In view 430, a mirror 450 is exposed. The LiDAR may emit a laser pulse toward the mirror (in a direction opposite to the view of the lens). The laser pulse may strike the mirror 450, which may be angled (e.g., at a 90° angle). The mirror 450 may be coupled with an internal motor that turns the mirror so that the LiDAR's laser pulses can be emitted and / or received at many different angles around the environmental capture system 400.

[0039] 5 is a diagram of laser pulses from a LiDAR around an environmental capture system 400 in some embodiments. In this example, the laser pulses are emitted at a rapidly rotating mirror 450. The laser pulses may be emitted and received perpendicular to the horizontal axis 602 (see FIG. 6) of the environmental capture system 400. The mirror 450 may be angled so that the laser pulses from the LiDAR are pointed away from the environmental capture system 400. In some examples, the angle of the angled surface of the mirror may be 90°, or 60°, 120°, or between 60° and 120°.

[0040] In some embodiments, when the environmental capture system 400 is stationary and in motion, the environmental capture system 400 can take bursts of images through a lens. The environmental capture system 400 can turn on a horizontal motor between bursts of images. While turning along the mount, a LiDAR of the environmental capture system 400 can emit and / or receive laser pulses that strike the rapidly rotating mirror 450. The LiDAR can generate a depth signal and / or generate depth data from reflections of the received laser pulses.

[0041] In some embodiments, the depth data may be associated with coordinates relative to the environment capture system 400. Similarly, by associating pixels or portions of an image with coordinates relative to the environment capture system 400, the image and depth data can be used to create 3D visualizations (e.g., images from different directions, 3D walk-throughs, etc.).

[0042] 5, the LiDAR pulses may be blocked by the bottom of environmental capture system 400. It will be appreciated that mirror 450 may continuously rotate at high speed while environmental capture system 400 is moving around the mount, or mirror 450 may rotate at high speed more slowly (e.g., may maintain a constant speed between starting and stopping the mount motors) as environmental capture system 400 begins to move and as environmental capture system 400 decelerates again to a stop.

[0043] The LiDAR can receive depth data from the pulses. As the environmental capture system 400 moves and / or the mirror 450 speeds up or slows down, the depth data for the environmental capture system 400 may not be of consistent density (e.g., more dense in some areas and less dense in other areas).

[0044] 6a shows a side view of the environmental capture system 400. Mirror 450 is shown in this view, which can be rapidly rotated about a horizontal axis. Pulses 604 may be emitted by the LiDAR at the rapidly rotating mirror 450 and may be emitted perpendicular to the horizontal axis 602. Similarly, pulses 604 may be received by the LiDAR in a similar manner.

[0045] Although the LiDAR pulses are described as being perpendicular to the horizontal axis 602, it will be understood that the LiDAR pulses may be at any angle relative to the horizontal axis 602 (e.g., the mirror angle may be any angle including 60-120°). In various embodiments, the LiDAR emits pulses on the opposite side (e.g., front side 604) of the environmental capture system 400 (e.g., in a direction opposite the center of the lens' field of view, or toward the back surface 606).

[0046] As described herein, the environment capture system 400 may turn about the vertical axis 608. In various embodiments, the environment capture system 400 may turn 90° after capturing an image, such that a fourth set of images is captured as the environment capture system 400 completes a 270° turn from the original starting position where the first set of images was captured. Thus, the environment capture system 400 may generate four sets of images during multiple turns totaling 270° (e.g., assuming the first set of images was captured before the first turn of the environment capture system 400). In various embodiments, images (e.g., four sets of images) from a single sweep of the environment capture system 400 (e.g., captured during one rotation or 270° rotation about the vertical axis), along with depth data acquired during the same sweep, are sufficient to generate a 3D visualization without further sweeps or turns of the environment capture system 400.

[0047] It will be appreciated that in this example, the LiDAR pulse is emitted and redirected by a rapidly rotating mirror at a location away from the rotation point of the environmental capture system 400. In this example, the mount is at a distance 608 from the rotation point (e.g., the lens may be at a cross point, but the lens may be at a location behind the lens relative to the front of the environmental capture system 400). Because the LiDAR pulse is redirected by the mirror 450 at a location away from the rotation point, the LiDAR cannot receive depth data from a cylinder that extends from above the environmental capture system 400 to below the environmental capture system 400. In this example, the radius of the cylinder (e.g., this cylinder lacks depth information) can be measured from the center of the rotation point of the motor mount to the point where the mirror 450 redirects the LiDAR pulse.

[0048] Further shown in FIG. 6a is a cavity 610. In this example, the environmental capture system 400 includes a rapidly rotating mirror within the body of the housing of the environmental capture system 400. There is a cut-out section from the housing. After the laser pulse is reflected out of the housing by the mirror, the reflection can be received by the mirror and redirected back to the LiDAR, allowing the LiDAR to create a depth signal and / or depth data. The base of the body of the environmental capture system 400 below the cavity 610 can block a portion of the laser pulse. The cavity 610 can be defined by the base of the environmental capture system 400 and the rotating mirror. As shown in FIG. 6b, there can still be a space between the angled mirror and the housing of the environmental capture system 400 including the LiDAR.

[0049] In various embodiments, the LiDAR is configured to stop emitting laser pulses if the rotational speed of the mirror drops below a rotational safety threshold (e.g., if the motor that spins the mirror at high speed fails or if the mirror is held in place), allowing the LiDAR to be configured for safety and reduce the chance of laser pulses continuing to be emitted in the same direction (e.g., at the user's eye).

[0050] Figure 6b shows a top view of the environmental capture system 400, according to some embodiments. In this example, the front of the environmental capture system 400 is shown concave with the lens, directly above the center of rotation (e.g., directly above the center of the mount). The front of the camera is concave for the lens, and the front of the housing is flared so that the housing does not block the field of view of the image sensor. The mirror 450 is shown facing up.

[0051] 7 shows a schematic diagram of the components of the environmental capture system 300 according to some embodiments. The environmental capture system 700 includes a front cover 702, a lens assembly 704, a structural frame 706, a LiDAR 708, a front housing 710, a mirror assembly 712, a GPS antenna 714, a rear housing 716, a vertical motor 718, a display 720, a battery pack 722, a mount 724, and a horizontal motor 726.

[0052] In various embodiments, the environmental capture system 700 can be configured to scan, register, and create 3D meshes outdoors and indoors under clear skies. This removes the barrier to adopting other systems that are indoor-only tools. The environmental capture system 700 can scan large spaces more quickly than other devices. In some embodiments, the environmental capture system 700 can provide improved depth accuracy by improving single-scan depth accuracy at 90 m.

[0053] In some embodiments, the environmental capture system 700 may weigh at or about 1 kg. In some examples, the environmental capture system 700 may weigh between 1 and 3 kg.

[0054] The front cover 702, the front housing 710, and the rear housing 716 form part of a housing. In one example, the width w of the front cover may be 75 mm.

[0055] The lens assembly 704 may include a camera lens that focuses light onto the image capture device. The image capture device may capture an image of a physical environment. A user may position the environment capture system 700 to capture a portion of a floor of a structure, such as the second structure 422 of FIG. 1, to obtain a panoramic image of the portion of the floor. By moving the environment capture system 700 to another portion of the floor of the structure, a panoramic image of another portion of the floor can be obtained. In one example, the depth of field of the image capture device is from 0.5 meters to infinity. FIG. 8a shows the dimensions of an example lens in some embodiments.

[0056] In some embodiments, the image capture device is a complementary metal-oxide-semiconductor (CMOS) image sensor (e.g., a Sony IMX283 ∼20 Megapixel CMOS MIPI sensor with an NVidia Jetson Nano SOM). In various embodiments, the image capture device is a charged coupled device (CCD). In one example, the image capture device is a red-green-blue (RGB) sensor. In one embodiment, the image capture device is an infrared (IR) sensor. The lens assembly 704 can provide the image capture device with a wide field of view.

[0057] The image sensor may have many different specifications. In one example, the image sensor includes:

[0058] [Table 1]

[0059] An example specification may be as follows:

[0060] [Table 2]

[0061] In various embodiments, looking at the MTF at the F0 relative field of view (i.e., center), the focus shift can vary from +28 micrometers at 0.5 m to -25 micrometers at infinity, for an overall focus shift of 53 micrometers.

[0062] FIG. 8b illustrates an exemplary lens design specification according to some embodiments.

[0063] In some examples, the lens assembly 704 has an HFOV of at least 148° and a VFOV of at least 94°. In some examples, the lens assembly 704 has a field of view of 150°, 180°, or 145°-180°. Image capture of a 360° view around the environment capture system 700 can be obtained, in some examples, by three or four separate image captures from the image capture device of the environment capture system 700. In various embodiments, the image capture device can have a resolution of at least 37 pixels per degree. In some embodiments, the environment capture system 700 includes a lens cap (not shown) to protect the lens assembly 704 when not in use. The output of the lens assembly 704 can be a digital image of an area of ​​the physical environment. A 2D panoramic image of the physical environment can be formed by stitching together multiple images captured by the lens assembly 704. A 3D panorama can be generated by combining depth data captured by the LiDAR 708 with a 2D panoramic image generated by stitching together multiple images from the lens assembly 704. In some embodiments, multiple images captured by the environment capture system 402 are stitched together by the image processing system 406. In various embodiments, the environment capture system 402 generates a “preview” or “thumbnail” version of the 2D panoramic image. The preview or thumbnail version of the 2D panoramic image can be presented on a user system 1110, such as an iPad, personal computer, smartphone, etc. In some embodiments, the environment capture system 402 can generate a minimap of the physical environment that represents an area of ​​the physical environment. In various embodiments, the image processing system 406 generates a minimap that represents an area of ​​the physical environment.

[0064] The image captured by the lens assembly 704 may include capture device location data that identifies or indicates the capture location of the 2D image. For example, in some implementations, the capture device location data may include global positioning system (GPS) coordinates associated with the 2D image. In other implementations, the capture device location data may include location information that indicates the relative location of the capture device (e.g., a camera and / or a 3D sensor) with respect to its environment, such as the relative or calibrated location of the capture device with respect to an object in the environment, another device in the environment, etc. In some implementations, this type of location data may be determined in conjunction with the capture of the image by the capture device (e.g., a camera and / or a device operably coupled to the camera with positioning hardware and / or software) and may be received along with the image. The placement of the lens assembly 704 is not only by design. By placing the lens assembly 704 at or near the center of the axis of rotation, parallax effects can be reduced.

[0065] In some embodiments, the structural frame 706 can hold the lens assembly 704 and the LiDAR 708 in a particular position and help protect the components of the environmental capture system of this example. The structural frame 706 can help mount the LiDAR 708 securely and serve to place the LiDAR 708 in a fixed position. Additionally, the fixed position of the lens assembly 704 and the LiDAR 708 allows for a fixed relationship to align depth data with image information to aid in creating a 3D image. By aligning the 2D image data and depth data captured in the physical environment to a common 3D coordinate space, a 3D model of the physical environment can be generated.

[0066] In various embodiments, the LiDAR 708 captures depth information of the physical environment. When a user places the environmental capture system 700 on a portion of a floor of a second structure, the LiDAR 708 can obtain depth information of the object. The LiDAR 708 can include an optical sensing module that can measure the distance to a target or object within the scene by illuminating the target or scene using a pulse from a laser and measuring the time it takes for a photon to travel to the target and return to the LiDAR 708. The measurements can then be transformed into a grid coordinate system using information derived from the horizontal drive train of the environmental capture system 700.

[0067] In some embodiments, the LiDAR 708 can return depth data points every 10 microseconds with a timestamp (of an internal clock). The LiDAR 708 can sample a partial sphere (with small holes at the top and bottom) every 0.25°. With a data point every 10 microseconds and 0.25°, in some embodiments, this can be 14.40 milliseconds per "disk" of points, and there can be 1440 disks to make a sphere that is nominally 20.7 seconds. Each disk captures back and forth, so the sphere can be captured in a 180° sweep.

[0068] In one example, the specifications of the LiDAR 708 may be as follows:

[0069] [Table 3]

[0070] One advantage of using LiDAR is that using LiDAR at relatively low wavelengths (eg, 905 nm, 900-940 nm, etc.) allows the environment capture system 700 to determine depth information for outdoor environments or brightly lit indoor environments.

[0071] The arrangement of the lens assembly 704 and the LiDAR 708 allows the environment capture system 700 or a digital device in communication with the environment capture system 700 to generate 3D panoramic images using depth data from the LiDAR 708 and the lens assembly 704. In some embodiments, the 2D and 3D panoramic images are not generated by the environment capture system 402.

[0072] The output of the LiDAR 708 may include attributes associated with each laser pulse transmitted by the LiDAR 708, such as: intensity of the laser pulse; number of returns; current return number; classification point; RGC value; GPS time; scan angle; scan direction; or any combination thereof. The depth of field may be (0.5m;infinity), (1m;infinity), etc. In some embodiments, the depth of field is 0.2m to 1m and infinity.

[0073] In some embodiments, the environment capture system 700 captures four separate RBG images with the lens assembly 704 while the environment capture system 700 is stationary. In various embodiments, the LiDAR 708 captures depth data for four different instances while the environment capture system 700 is moving and moving from one RBG image capture location to another RBG image capture location. In one example, a 3D panoramic image is captured by a 360° rotation of the image capture system 700, which may be referred to as a sweep. In various embodiments, a 3D panoramic image is captured by a less than 360° rotation of the environment capture system 700. The output of the sweep may be a sweep list (SWL), which includes image data from the lens assembly 704, depth data from the LiDAR 708, and characteristics of the sweep, including the GPS location and a timestamp of when the sweep was performed. In various embodiments, a single sweep (e.g., a single 360° turn of environment capture system 700) captures sufficient image and depth information to generate a 3D visualization (e.g., by a digital device in communication with environment capture system 700 receiving image and depth data from environment capture system 700 and creating a 3D visualization using only the image and depth data from environment capture system 700 captured in the single sweep).

[0074] In some embodiments, multiple images captured by the environment capture system 402 can be blended, stitched together, and combined with depth data from the LiDAR 708 by an image stitching and processing system described below.

[0075] In various embodiments, the environment capture system 402 and / or an application on the user system 1110 may generate a preview or thumbnail version of the 3D panoramic image. The preview or thumbnail version of the 3D panoramic image may be presented on the user system 1110 and may have a lower image resolution than the 3D panoramic image generated by the image processing system 406. After the lens assembly 704 and the LiDAR 708 capture image and depth data of the physical environment, the environment capture system 402 may generate a minimap that represents an area of ​​the physical environment that the environment capture system 402 has captured. In some embodiments, the image processing system 406 generates a minimap that represents an area of ​​the physical environment. After using the environment capture system 402 to capture image and depth data of the living room of the house, the environment capture system 402 can generate a top-down view of the physical environment. The user can use this information to determine areas of the physical environment for which the user has not captured or generated a 3D panoramic image.

[0076] In one embodiment, the environment capture system 700 can interleave depth information capture by the LiDAR 708 between image capture by the image capture device of the lens assembly 704. For example, the image capture device may capture an image of a section 1605 of the physical environment as seen in FIG. 16, after which the LiDAR 708 obtains depth information from the section 1605. Once the LiDAR 708 obtains depth information from the section 1605, the image capture device may move to capture an image of another section 1610, after which the LiDAR 708 obtains depth information from the section 1610. In this manner, image capture and depth information capture alternate.

[0077] In some embodiments, the LiDAR 708 may have a field of view of at least 145°, and depth information for all objects in a 360° view of the environmental capture system 700 may be obtained in three or four scans by the environmental capture system 700. In other examples, the LiDAR 708 may have a field of view of at least 150°, 180°, or between 145° and 180°.

[0078] The increased field of view of the lens reduces the amount of time required to obtain visual and depth information of the physical environment around the environmental capture system 700. In various embodiments, the LiDAR 708 has a minimum depth range of 0.5 m. In one embodiment, the LiDAR 708 has a maximum depth range of greater than 8 meters.

[0079] The LiDAR 708 can utilize a mirror assembly 712 to direct the laser to different scan angles. In one embodiment, an optional vertical motor 718 functions to move the mirror assembly 712 vertically. In some embodiments, the mirror assembly 712 can be a dielectric mirror having a hydrophobic coating or layer. The mirror assembly 712 can be coupled to the vertical motor 718, which rotates the mirror assembly 712 when in use.

[0080] The mirrors of the mirror assembly 712 may have, for example, the following specifications:

[0081] [Table 4]

[0082] The mirrors of the mirror assembly 712 may have, for example, the following specifications in terms of materials and coatings:

[0083] [Table 5]

[0084] The hydrophobic coating on the mirrors of the mirror assembly 712 may have a contact angle of, for example, greater than 105°.

[0085] The mirrors of the mirror assembly 712 may have the following quality specifications:

[0086] [Table 6]

[0087] The vertical motor may have, for example, the following specifications:

[0088] [Table 7]

[0089] The RGB capture device and LiDAR 708 allow the environment capture system 700 to capture images outdoors under clear skies or indoors where the light is bright or sunlight from windows is glaring. Systems utilizing different devices (e.g., structured lighting devices) may not be able to operate in bright environments, whether indoors or outdoors. These devices are often limited to be used only indoors and only during dawn or dusk to control the light. Otherwise, bright spots indoors would create artifacts or "holes" in the image that need to be filled or corrected. However, the environment capture system 700 can be used in bright sunlight, both indoors and outdoors. The capture device and LiDAR 708 can capture images and depth data in bright environments without artifacts or holes caused by glaring or bright light.

[0090] In one embodiment, the GPS antenna 714 receives Global Positioning System (GPS) data that can be used to determine the location of the environment capture system 700 at any given time.

[0091] In various embodiments, the display 720 allows the environmental capture system 700 to provide the current status of the system, such as updating, warming up, scanning, scan complete, error, etc.

[0092] The battery pack 722 provides power to the environmental capture system 700. The battery pack 722 may be removable and rechargeable, allowing a user to insert a new battery pack 722 while charging a depleted battery pack. In some embodiments, the battery pack 722 may be capable of at least 1000 SWL or at least 250 SWL of continuous use before recharging. The environmental capture system 700 may utilize a USB-C plug for recharging.

[0093] In some embodiments, mount 724 provides a connector for connecting environmental capture system 700 to a platform such as a tripod or mount. Horizontal motor 726 can rotate environmental capture system 700 about an x-y plane. In some embodiments, horizontal motor 726 can provide information to a grid coordinate system to determine the (x, y, z) coordinates associated with each laser pulse. In various embodiments, the wide field of view of the lens, the positioning of the lens about the axis of rotation, and the LiDAR device can allow horizontal motor 726 to rapidly perform scans.

[0094] The horizontal motor 726 may have the following specifications, by way of example:

[0095] [Table 8]

[0096] In various embodiments, the mount 724 may include a quick release adapter. The retention torque may be, for example, greater than 2.0 Nm, and the durability of the capture operation may be up to or greater than 70,000 cycles.

[0097] For example, the environmental capture system 700 may be capable of constructing a 3D mesh of a standard house with sweep-to-sweep distances of over 8 m. The time for capturing, processing, and aligning a sweep indoors may be less than 45 seconds. In one example, the time frame from the start of sweep capture to the point at which the user can move the environmental capture system 700 may be less than 15 seconds.

[0098] In various embodiments, these components provide the environmental capture system 700 with the ability to create a seamless walk-through experience between indoors and outdoors by aligning multiple outdoor and indoor scan locations (which may have high priority for hotel, vacation rental, real estate, construction research, CRE, and as-built modeling and verification). The environmental capture system 700 can also create an "outdoor dollhouse" or outdoor minimap. As shown here, the environmental capture system 700 can also improve the accuracy of the 3D reconstruction, mainly from a measurement perspective. With regard to the density of the scans, it may also be beneficial for the user to be able to fine-tune this. These components can also enable the environmental capture system 700 to capture large empty spaces (e.g., relatively long ranges). To generate a 3D model of a large empty space, the environmental capture system may need to scan and capture 3D and depth data from a larger distance range than to generate a 3D model of a smaller space.

[0099] In various embodiments, these components enable the environment capture system 700 to align multiple SWLs and reconstruct 3D models in a similar manner for indoor and outdoor use. These components can also enable the environment capture system 700 to perform geo-localization of the 3D models (which can facilitate integration into Google Street View and be useful for aligning multiple outdoor panoramas, if necessary).

[0100] The image capture device of the environment capture system 700 may be capable of providing DSLR-like images with printable quality at 8.5 inches by 11 inches for a VFOV of 70 degrees, and an RGB image style.

[0101] In some embodiments, the environment capture system 700 can capture an RGB image with an image capture device (e.g., using a wide-angle lens), move the lens, and then capture the next RGB image (moving the motor a total of four times). The LiDAR 708 can capture depth data while the horizontal motor 726 rotates the environment capture system 90°. In some embodiments, the LiDAR 708 includes an APD array.

[0102] In some embodiments, the image and depth data may then be sent to a capture application (e.g., a device in communication with the environment capture system 700, such as a smart device or image capture system over a network). In some embodiments, the environment capture system 700 may send the image and depth data to the image processing system 406 for processing and generation of a 2D or 3D panoramic image. In various embodiments, the environment capture system 700 may generate a sweep list of the captured RGB image and depth data from a 360° rotation of the environment capture system 700. This sweep list may be sent to the image processing system 406 for stitching and alignment. The output of the sweep may be an SWL, which includes image data from the lens assembly 704, depth data from the LiDAR 708, and characteristics of the sweep, including the GPS location and a timestamp of when the sweep was performed.

[0103] In various embodiments, the LIDAR, vertical mirror, RGB lens, tripod mount, and horizontal drive are securely mounted within the housing so that the housing can be opened without requiring recalibration of the system.

[0104] 9a illustrates an example block diagram 900 of an environmental capture system in accordance with some embodiments. The block diagram 900 includes a power supply 902, a power converter 904, an input / output (I / O) printed circuit board assembly (PCBA), a system on module (SOM) PCBA, a user interface 910, a LiDAR 912, a mirror brushless direct current (BLCD) motor 914, a drive train 916, a wide FOV (WFOV) lens 918, and an image sensor 920.

[0105] The power source 902 may be the battery pack 722 of Figure 7. The power source may be a removable and rechargeable battery, such as a lithium-ion battery (e.g., 4 x 18650 Li-Ion cells), capable of powering the environmental capture system.

[0106] A power converter 904 can convert the voltage level from the power source 902 to a lower or higher voltage for use by the electronic components of the environmental capture system. The environmental capture system may utilize 4×18650 Li-Ion batteries in a 4S1P configuration, i.e., four in series and one in parallel.

[0107] In some embodiments, the I / O PCBA 906 may include elements providing an IMU, Wi-Fi, GPS, Bluetooth, an inertial measurement unit (IMU), motor drives, and a microcontroller, hi some embodiments, the I / O PCBA 906 includes a microcontroller for controlling and encoding control of the horizontal motors and for controlling and encoding control of the vertical motors.

[0108] The SOM PCBA 908 may include a central processing unit (CPU) and / or a graphics processing unit (GPU), memory, and a mobile interface. The SOM PCBA 908 may control the LiDAR 912, the image sensor 920, and the I / O PCBA 906. The SOM PCBA 908 may determine (x,y,z) coordinates associated with each laser pulse of the LiDAR 912 and store the coordinates in a memory component of the SOM PCBA 908. In some embodiments, the SOM PCBA 908 may store the coordinates in an image processing system of the environment capture system 400. In addition to the coordinates associated with each laser pulse, the SOM PCBA 908 may determine further attributes associated with each laser pulse, including the intensity of the laser pulse, the number of returns, the current return number, the classification point, the RGC value, the GPS time, the scan angle, and the scan direction.

[0109] In some embodiments, the SOM PCBA 908 includes an Nvidia SOM PCBA with CPU / GPU, DDR, eMMC, Ethernet.

[0110] The user interface 910 may include physical buttons or switches with which a user can interact. The buttons or switches may provide functions such as turning the environmental capture system on and off, scanning the physical environment, etc. In some embodiments, the user interface 910 may include a display, such as display 720 of FIG. 7.

[0111] In some embodiments, the LiDAR 912 captures depth information of the physical environment. The LiDAR 912 includes an optical sensing module that can measure the distance to a target or object in a scene by illuminating the target or scene with light. The LiDAR 912 optical sensing module gives the LiDAR's distance from the target or object by measuring the time it takes for a photon to travel to the target or object and return to the LiDAR 912's receiver after reflection. The SOM PCBA 908 can determine the distance as well as the (x,y,z) coordinates associated with each laser pulse. The LiDAR 912 can be within the range of 58mm wide, 55mm high, and 60mm deep.

[0112] The LiDAR 912 may have a range (10% reflectivity) of 90m, a range (20% reflectivity) of 130m, a range (100% reflectivity) of 260m, a range accuracy (1σ@900m) of 2cm, a wavelength of 1705nm, and a beam divergence of 0.28×0.03°.

[0113] The SOM PCBA 908 can determine coordinates based on the location of the drive train 916. In various embodiments, the LiDAR 912 can include one or more LiDAR devices. By utilizing multiple LiDAR devices, the resolution of the LiDAR can be improved.

[0114] A mirror brushless direct current (BLCD) motor 914 can control the mirror assembly 712 of FIG.

[0115] In some embodiments, the drive train 916 may include the horizontal motor 726 of FIG. 7. The drive train 916 can provide rotation of the environmental capture system when the environmental capture system is mounted on a platform such as a tripod. The drive train 916 can include a stepper motor Nema 14, a worm and plastic gear drive train, a clutch, a bushing bearing, and an anti-backlash mechanism. In some embodiments, the environmental capture system can complete one scan in less than 17 seconds. In various embodiments, the drive train 916 can provide a maximum speed of 60° / sec, a maximum rotation speed of 300° / sec, and a maximum rotation speed of 100° / sec. 2 It has a maximum acceleration of 1000 Hz, a maximum torque of 0.5 nm, an angular position accuracy of less than 0.1°, and an encoder resolution of approximately 4096 counts per revolution.

[0116] In some embodiments, the drive train 916 includes a vertical monogon mirror and a motor. In this example, the drive train 916 may include a BLDC motor, an external Hall effect sensor, a magnet (paired with the Hall effect sensor), a mirror bracket, and a mirror. The drive train 916 in this example may have a maximum speed of 4,000 RPM and a maximum acceleration of 300° / sec^2. In some embodiments, the monogon mirror is a dielectric mirror. In one embodiment, the monogon mirror includes a hydrophobic coating or layer.

[0117] The arrangement of the components of the environmental capture system is such that the lens assembly and LiDAR are located approximately in the center of the axis of rotation, which reduces image parallax that occurs when the image capture system is not located in the center of the axis of rotation.

[0118] In some embodiments, the WFOV lens 918 may be a lens of the lens assembly 704 of FIG. 7. The WFOV lens 918 focuses light onto the image capture device. In some embodiments, the WFOV lens may have an FOV of at least 145°. With such a wide FOV, 360° image capture around the environmental capture system can be obtained by three separate image captures of the image capture device. In some embodiments, the WFOV lens 918 may have a diameter of about 60 mm and a total track length (TTL) of about 80 mm. In one example, the WFOV lens 918 may have a horizontal field of view of 148.3° or more and a vertical field of view of 94° or more.

[0119] The image capture device may include a WFOV lens 918 and an image sensor 920. The image sensor 920 may be a CMOS image sensor. In one embodiment, the image sensor 920 is a charge-coupled device (CCD). In some embodiments, the image sensor 920 is a red-green-blue (RGB) sensor. In one embodiment, the image sensor 920 is an IR sensor. In various embodiments, the image capture device may have a resolution of at least 35 pixels per degree (PPD).

[0120] In some embodiments, the image capture device may have: an F-number of f / 2.4; an image circle diameter of 15.86 mm; a pixel pitch of 2.4 μm; an HFOV of greater than 148.3°; a VFOV of greater than 94.0°; pixels per degree of greater than 38.0 PPD; a chief ray incidence angle at full height of 3.0°; a minimum focusing distance of 1300 mm; a maximum focusing distance of infinity; a relative light output of greater than 130%; a maximum distortion of less than 90%; and a spectral transmittance variation of 5% or less.

[0121] In some embodiments, the lens may have: an F-number of 2.8; an image circle diameter of 15.86 mm; pixels per degree greater than 37; a chief ray incidence angle at the full height sensor of 3.0; an L1 diameter less than 60 mm; a TTL less than 80 mm; and a relative light output greater than 50%.

[0122] The lenses may have 85% greater than 52 lp / mm (on-axis), 66% greater than 104 lp / mm (on-axis), 45% greater than 208 lp / mm (on-axis), 75% greater than 52 lp / mm (83% of field of view), 41% greater than 104 lp / mm (83% of field of view), and 25% greater than 208 lp / mm (83% of field of view).

[0123] The environmental capture system may have a resolution of >20MP, a green sensitivity of >1.7V / lux*sec, an SNR of >65dB (100 lux, 1x gain), and a dynamic range of >70dB.

[0124] 9b illustrates a block diagram of an example SOM PCBA 908 of an environmental capture system according to some embodiments. The SOM PCBA 908 can include communication components 922, LiDAR control components 924, LiDAR positioning components 926, user interface components 928, classification components 930, a LiDAR data store 932, and a captured image data store 934.

[0125] In some embodiments, the communication component 922 can send and receive requests or data between any of the components of the SOM PCBA 1008 and the components of the environmental capture system of FIG. 9a.

[0126] In various embodiments, the LiDAR control component 924 can control various aspects of the LiDAR. For example, the LiDAR control component 924 can send a control signal to the LiDAR 912 to initiate the emission of a laser pulse. The control signal sent by the LiDAR control component 924 can include instructions for the frequency of the laser pulse.

[0127] In some embodiments, the LiDAR positioning component 926 can utilize GPS data to determine the location of the environmental capture system. In various embodiments, the LiDAR positioning component 926 utilizes the position of the mirror assembly to determine the scan angle and (x,y,z) coordinates associated with each laser pulse. The LiDAR positioning component 926 can also utilize an IMU to determine the orientation of the environmental capture system.

[0128] The user interface component 928 can facilitate user interaction with the environment capture system. In some embodiments, the user interface component 928 can provide one or more user interface elements with which the user can interact. The user interface provided by the user interface component 928 can be sent to the user system 1110. For example, the user interface component 928 can provide a user system (e.g., a digital device) with a visual representation of an area of ​​a building floor plan. The environment capture system can generate a visual representation of the floor plan as the user places the environment capture system in different parts of a floor of the building to capture and generate 3D panoramic images. The user can place the environment capture system in an area of ​​the physical environment to capture and generate a 3D panoramic image of the area of ​​the house. After the 3D panoramic image of the area is generated by the image processing system, the user interface component can update the floor plan with an overhead view of the living room area as shown in FIG. 1b. In some embodiments, the floor plan 200 can be generated by the user system 1110 after capturing a second sweep of a house or a floor of a building.

[0129] In various embodiments, the classification component 930 can classify the type of physical environment. The classification component 930 can analyze objects in an image or objects in an image to classify the type of physical environment captured by the environment capture system. In some embodiments, the image processing system can be responsible for classifying the type of physical environment captured by the environment capture system 400.

[0130] The LiDAR data store 932 may be any structure and / or structures suitable for captured LiDAR data (e.g., an active database, a relational database, a self-referential database, a table, a matrix, an array, a flat file, a document-oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, etc.). The image data store 408 may store the captured LiDAR data. However, the LiDAR data store 932 may be utilized to cache the captured LiDAR data when the communication network 404 is not functioning. For example, when the environment capture system 402 and the user system 1110 are in a remote location without a cellular network or in an area without Wi-Fi, the LiDAR data store 932 may store the captured LiDAR data until it can be transferred to the image data store 934.

[0131] Similar to the LiDAR data store, the captured image data store 934 can be any structure and / or structures suitable for captured imagery (e.g., an active database, a relational database, a self-referential database, a table, a matrix, an array, a flat file, a document-oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, etc.). The image data store 934 can store captured imagery.

[0132] 10a-10c illustrate a process of the environment capture system 400 for taking images in some embodiments. As shown in FIG. 10a-10c, the environment capture system 400 can take a burst of images at different exposures. A burst of images can be a set of images, each at a different exposure. The first image burst is at time 0.0. The environment capture system 400 can receive a first frame and evaluate this frame while waiting for the second frame. FIG. 10a illustrates that the first frame is blended after the arrival of the second frame. In some embodiments, the environment capture system 400 can process each frame to identify pixels, colors, etc. When the next frame arrives, the environment capture system 400 can process the most recently received frame and blend the two frames into one.

[0133] In various embodiments, environment capture system 400 performs image processing to blend the sixth frame and further evaluates pixels in the blended frame (e.g., a frame that may include elements from any number of frames of the image burst). During this final step, before or during a movement (e.g., turn) of environment capture system 400, environment capture system 400 may optionally transfer the blended image from the image processor to CPU memory.

[0134] The process continues in FIG. 10b. Beginning in FIG. 10b, the environment capture system 400 performs another burst. The environment capture system 400 may compress the blended frame and / or all or part of the captured frames using JXR. Similar to FIG. 10a, the burst of images may be a set of images with different exposures (the exposure length of each frame in the set may be the same and may be in the same order as the other bursts encompassed in FIG. 10a, 10c). The second burst of images is at 2 seconds. The environment capture system 400 may receive the first frame and evaluate it while waiting for the second frame. FIG. 10b shows that the first frame is blended after the arrival of the second frame. In some embodiments, the environment capture system 400 may process each frame to identify pixels, colors, etc. When the next frame arrives, the environment capture system 400 may process the most recently received frame and blend the two frames into one.

[0135] In various embodiments, environment capture system 400 performs image processing to blend the sixth frame and further evaluates pixels in the blended frame (e.g., a frame that may include elements from any number of frames of the image burst). During this final step, before or during a movement (e.g., turn) of environment capture system 400, environment capture system 400 may optionally transfer the blended image from the image processor to CPU memory.

[0136] After turning, the environment capture system 400 may continue the process by performing another color burst at approximately 3.5 seconds (e.g., after a 180° turn). The environment capture system 400 may compress the blended frame and / or all or part of the captured frame using JXR. A burst of images may be a set of images with different exposures (each frame of the set may have the same exposure length and may be in the same order as the other bursts included in Figures 10a, 10c). The environment capture system 400 may receive a first frame and evaluate it while waiting for the second frame. Figure 10b shows the first frame being blended after the arrival of the second frame. In some embodiments, the environment capture system 400 may process each frame to identify pixels, colors, etc. When the next frame arrives, the environment capture system 400 may process the most recently received frame and blend the two frames into one.

[0137] In various embodiments, environment capture system 400 performs image processing to blend the sixth frame and further evaluates pixels in the blended frame (e.g., a frame that may include elements from any number of frames of the image burst). During this final step, before or during a movement (e.g., turn) of environment capture system 400, environment capture system 400 may optionally transfer the blended image from the image processor to CPU memory.

[0138] The final burst occurs at 5 seconds in FIG. 10c. The environment capture system 400 may compress the blended frame and / or all or part of the captured frames using JXR. A burst of images may be a set of images with different exposures (each frame of the set may have the same exposure length and may be in the same order as the other bursts contained in FIG. 10a, 10b). The environment capture system 400 may receive a first frame and evaluate it while waiting for the second frame. FIG. 10c shows the first frame being blended after the arrival of the second frame. In some embodiments, the environment capture system 400 may process each frame to identify pixels, colors, etc. When the next frame arrives, the environment capture system 400 may process the most recently received frame and blend the two frames into one.

[0139] In various embodiments, environment capture system 400 performs image processing to blend the sixth frame and further evaluates pixels in the blended frame (e.g., a frame that may include elements from any number of frames of the image burst). During this final step, before or during a movement (e.g., turn) of environment capture system 400, environment capture system 400 may optionally transfer the blended image from the image processor to CPU memory.

[0140] The dynamic range of an image capture device is a measure of the amount of light that an image sensor can capture. The dynamic range is the difference between the darkest and brightest areas of an image. There are many ways to improve the dynamic range of an image capture device, one of which is to capture multiple images of the same physical environment with different exposures. Images captured with short exposures will capture the brightest areas of the physical environment, and long exposures will capture the darker areas of the physical environment. In some embodiments, the environment capture system may capture multiple images with six different exposure times. Some or all of the images captured by the environment capture system are used to generate a high dynamic range (HDR) 2D image. One or more of the captured images may be used for other functions, such as light detection, flicker detection, etc.

[0141] A 3D panoramic image of a physical environment can be generated based on four separate image captures of an image capture device and four separate depth data of a LiDAR device of an environmental capture system. Each of the four separate image captures can include a series of image captures with different exposure times. A blending algorithm can be used to blend the series of image captures with different exposure times to generate one of four RGB image captures, which can be used to generate a 2D panoramic image. For example, the environmental capture system can be used to capture a 3D panoramic image of a kitchen. An image of one wall of the kitchen can include a window, and an image captured with a short exposure can provide a view outside the window, while the rest of the kitchen can remain underexposed. In contrast, another image captured with a long exposure can provide a view inside the kitchen. A blending algorithm can blend a view outside the kitchen window from one image with the rest of the view of the kitchen from another image to generate a blended RGB image.

[0142] In various embodiments, the 3D panoramic image can be generated based on three separate image captures of the image capture device and four separate depth data captures of the LiDAR device of the environmental capture system. In some embodiments, the number of image captures and the number of depth data captures can be the same. In one embodiment, the number of image captures and the number of depth data captures can be different.

[0143] After capturing a first series of images at an exposure time, a blending algorithm receives the first series of images, calculates initial intensity weights for the images, and sets the images as a baseline image for combining subsequently received images. In some embodiments, the blending algorithm may utilize a graphics processing unit (GPU) image processing routine, such as a "blend_kernel" routine. The blending algorithm may receive subsequent images, which may be blended with previously received images. In some embodiments, the blending algorithm may utilize a variation of the blend_kernel GPU image processing routine.

[0144] In one embodiment, the blending algorithm utilizes other methods of blending multiple images, such as determining the difference, or contrast, between the darkest and lightest parts of the baseline image to determine whether the baseline image is overexposed or underexposed. For example, a contrast value below a predetermined contrast threshold means that the baseline image is overexposed or underexposed. In one embodiment, the contrast of the baseline image can be calculated by taking the average of the light intensity of the image, or of a subset of the image. In some embodiments, the blending algorithm calculates the average light intensity for each row or column of the image. In some embodiments, the blending algorithm may determine a histogram of each image received from the image capture device and determine the light intensity of the pixels that make up each image by analyzing the histogram.

[0145] In various embodiments, blending may involve sampling colors in two or more images of the same scene, including along objects and seams. If there is a significant difference in color between the two images (e.g., within a predefined threshold of color, hue, brightness, saturation, etc.), a blending module (e.g., on the environment capture system 400 or user device 1110) may blend both images of a predefined size along the location where the difference exists. In some embodiments, the greater the color or image difference at a location in the images, the more space around that location may be blended.

[0146] In some embodiments, after blending, a blending module (e.g., on environment capture system 400 or user device 1110) may rescan and sample colors along one or more images to determine whether there are other differences in the images or colors that exceed the predetermined thresholds, such as color, hue, brightness, saturation, etc. If so, the blending module may identify those portions in the one or more images and continue blending those portions of the images. The blending module may continue resampling images along seams until there are no more portions of the images to blend (e.g., the color difference is below one or more predetermined thresholds).

[0147] 11 illustrates a block diagram of an example environment 1100 in which images can be captured and stitched to form a 3D visualization, according to some embodiments. The example environment 1100 includes a 3D and panoramic capture and stitching system 1102, a communication network 1104, an image stitching and processor system 1106, an image data store 1108, a user system 1110, and a first scene of a physical environment 1112. The 3D and panoramic capture and stitching system 1102 and / or the user system 1110 may include an image capture device (e.g., environment capture system 400) that can be used to capture images of an environment (e.g., physical environment 1112).

[0148] The 3D and panoramic capture and stitching system 1102 and the image stitching processor system 1106 may be part of a system (e.g., part of one or more digital devices) communicatively coupled to the environment capture system 400. In some embodiments, at least one of the functions of the components of the 3D and panoramic capture and stitching system 1102 and the image stitching processor system 1106 may be performed by the environment capture system 400. Similarly, or alternatively, the 3D and panoramic capture and stitching system 1102 and the image stitching processor system 1106 may be performed by the user system 1110 and / or the image stitching processor system 1106.

[0149] A user can use the 3D panoramic capture and stitching system 1102 to capture multiple 2D images of an environment, such as the inside of a structure and / or the outside of a structure. For example, a user may use the 3D and panoramic capture and stitching system 1102 to capture multiple 2D images of a first scene of a physical environment 1112 provided by the environment capture system 400. The 3D and panoramic capture and stitching system 1102 may include an alignment and stitching system 1114. Alternatively, the user system 1110 may include the alignment and stitching system 1114.

[0150] The alignment and stitching system 1114 may be software, hardware, or a combination of both configured to provide guidance to a user of the image capture system (e.g., for the 3D and panoramic capture and stitching system 1102, or the user system 1110) and / or process images to enable creation of an improved panoramic photograph (by stitching, alignment, cropping, etc.). The alignment and stitching system 1114 may reside on a computer-readable medium (as described herein). In some embodiments, the alignment and stitching system 1114 may include a processor for performing the functions.

[0151] The first scene example of the physical environment 1112 may be any room, property, etc. (e.g., a representation of a living room). In some embodiments, the 3D and panoramic capture and stitching system 1102 is utilized to generate a 3D panoramic image of the indoor environment. In some embodiments, the 3D panoramic capture and stitching system 1102 may be the environment capture system 400 described in connection with FIG. 4.

[0152] In some embodiments, the 3D capture and stitching system 1102 can communicate with devices for capturing image and depth data, as well as software (e.g., the environment capture system 400). All or part of the software can be installed on the 3D panoramic capture and stitching system 1102, the user system 1110, the environment capture system 400, or all of these. In some embodiments, a user can interact with the 3D and panoramic capture and stitching system 1102 via the user system 1110.

[0153] The 3D and panoramic capture and stitching system 1102 or the user system 1110 can obtain multiple 2D images. The 3D and panoramic capture and stitching system 1102 or the user system 1110 can obtain depth data (e.g., from a LiDAR device, etc.).

[0154] In various embodiments, an application on the user system 1110 (e.g., a user's smart device, such as a smartphone or tablet computer) or an application on the environment capture system 400 can provide visual or audio guidance to the user for taking an image with the environment capture system 400. For example, the graphical guidance can include free-floating arrows on the display of the environment capture system 400 (e.g., on a viewfinder or LED screen on the back of the environment capture system 400) to guide the user as to where to position and / or point the image capture device. In another example, the application can provide audio guidance as to where to position and / or point the image capture device.

[0155] In some embodiments, the guidance allows a user to capture multiple images of a physical environment without the aid of a stabilizing platform such as a tripod. In some examples, the image capture device may be a personal device such as a smartphone, tablet, media tablet, laptop, etc. The application can provide directions for each sweep position to approximate a point of difference based on the position of the image capture device, location information from the image capture device, and / or past images of the image capture device.

[0156] In some embodiments, visual and / or audio guidance enables capture of multiple images that can be stitched together to form a panorama, without the use of a tripod and without the use of camera position information (e.g., indicating the camera's location, position, and / or orientation from a sensor, GPS device, etc.).

[0157] The alignment and stitching system 1114 can align or stitch 2D images (e.g., captured by the user system 1110 or the 3D panorama capture and stitching system 1102) to obtain a 2D panoramic image.

[0158] In some embodiments, the alignment and stitching system 1114 utilizes machine learning algorithms to align or stitch multiple 2D images into a 2D panoramic image. Parameters of the machine learning algorithms can be managed by the alignment and stitching system 1114. For example, the 3D and panoramic capture and stitching system 1102 and / or the alignment and stitching system 1114 can assist in aligning the 2D images into a 2D panoramic image by recognizing objects in the images.

[0159] In some embodiments, the registration and stitching system 1114 can utilize the depth data and the 2D panoramic image to obtain a 3D panoramic image. The 3D panoramic image can be provided to the 3D and panoramic stitching system 1102 or the user system 1110. In some embodiments, the registration and stitching system 1114 determines 3D depth measurements associated with objects recognized in the 3D panoramic image and / or sends one or more 2D images, the depth data, one or more 2D panoramic images, and one or more 3D panoramic images to the image stitching processor system 1106, thereby obtaining a 2D panoramic image or a 3D panoramic image having a higher pixel resolution than the 2D panoramic image or the 3D panoramic image provided by the 3D and panoramic capture and stitching system 1102.

[0160] The communication network 1104 may represent one or more computer networks (e.g., LAN, WAN, etc.) or other transmission media. The communication network 1104 may provide communication between the systems 1102, 1106-1110, and / or other systems described herein. In some embodiments, the communication network 104 includes one or more digital devices, routes, cables, buses, and / or other network topologies (e.g., mesh, etc.). In some embodiments, the communication network 1104 may be wired and / or wireless. In various embodiments, the communication network 1104 may include: the Internet; one or more wide area networks (WANs) or local area networks (LANs); one or more networks that may be public, private, IP-based, non-IP-based, etc.

[0161] The image stitching processor system 1106 may process 2D images captured by an image capture device (e.g., the environment capture system 400 or a user device such as a smartphone, personal computer, media tablet, etc.) and stitch them into a 2D panoramic image. The 2D panoramic image processed by the image stitching processor system 1106 may have a higher pixel resolution than the panoramic image obtained by the 3D and panoramic capture and stitching system 1102.

[0162] In some embodiments, the image stitching processor system 1106 receives the 3D panoramic image and processes it to create a 3D panoramic image having a higher pixel resolution than the received 3D panoramic image. This higher pixel resolution panoramic image can be provided to an output device, such as a computer screen, a projector screen, etc., having a higher screen resolution than the user system 1110. In some embodiments, this higher pixel resolution panoramic image can provide a more detailed panoramic image to the output device and can be scaled.

[0163] The image data store 1108 may be any structure and / or structures suitable for captured image and / or depth data (e.g., active databases, relational databases, self-referential databases, tables, matrices, arrays, flat files, document-oriented storage systems, non-relational No-SQL systems, FTS management systems such as Lucene / Solar, etc.). The image data store 1108 may store images captured by an image capture device of the user system 1110. In various embodiments, the image data store 1108 may store depth data captured by one or more depth sensors of the user system 1110. In various embodiments, the image data store 1108 may store characteristics associated with an image capture device or characteristics associated with each of a plurality of image captures or depth captures used to determine a 2D or 3D panoramic image. In some embodiments, the image data store 1108 may store a 2D or 3D panoramic image. The 2D or 3D panoramic image may be determined by the 3D and panoramic capture and stitching system 1102 or the image stitching and processor system 106.

[0164] The user system 1110 can facilitate communication between a user and other associated systems. In some embodiments, the user system 1110 can be or include one or more mobile devices (e.g., a smart phone, a mobile phone, a smart watch, etc.).

[0165] The user system 1110 may include one or more image capture devices. The one or more image capture devices may include, for example, an RGB camera, an HDR camera, a video camera, an IR camera, and the like.

[0166] The 3D and panoramic capture and stitching system 1102 and / or the user system 1110 may include two or more capture devices, which may be arranged on or in the same mobile housing at relative positions to each other such that their combined field of view spans 360°. In some embodiments, multiple pairs of image capture devices (e.g., having slightly offset but partially overlapping fields of view) capable of generating a stereo image pair may be used. The user system 1110 may include two image capture devices with vertical stereo offset fields of view capable of capturing a vertical stereo image pair. In another example, the user system 1110 may be equipped with two image capture devices with vertical stereo offset fields of view capable of capturing a vertical stereo image pair.

[0167] In some embodiments, the user system 1110, the environmental capture system 400, or the 3D and panoramic capture and stitching system 1102 can generate and / or provide image capture position and location information. For example, the user system 1110 or the 3D and panoramic capture and stitching system 1102 may include an inertial measurement unit (IMU) to assist in determining position data associated with one or more image capture devices that capture multiple 2D images. The user system 1110 may include a global positioning sensor (GPS) to provide GPS coordinate information associated with multiple 2D images captured by one or more image capture devices.

[0168] In some embodiments, a user may interact with the alignment and stitching system 1114 using a mobile application installed on the user system 1110. The 3D and panoramic capture and stitching system 1102 may provide images to the user system 1110. The user may use the alignment and stitching system 1114 on the user system 1110 to review images and previews.

[0169] In various embodiments, the alignment and stitching system 1114 may be configured to transmit and receive one or more 3D panoramic images to the 3D and panoramic capture and stitching system 1102 and / or the image stitching and processor system 1106. In some embodiments, the 3D and panoramic capture and stitching system 1102 may provide a visual representation of a portion of a building floor plan captured by the 3D and panoramic capture and stitching system 1102 to the user system 1110.

[0170] A user of the system 1110 can navigate the space around the area to view different rooms in the house. In some embodiments, a user of the user system 1110 can view a 3D panoramic image, such as the exemplary 3D panoramic image, once the image stitching and processor system 1106 completes generating the 3D panoramic image. In various embodiments, the user system 1110 generates a preview or thumbnail of the 3D panoramic image. The preview of the 3D panoramic image may have a lower image resolution than the 3D panoramic image generated by the 3D and panoramic capture and stitching system 1102.

[0171] 12 is a block diagram of an example of an alignment and stitching system 1114 according to some embodiments. The alignment and stitching system 1114 includes a communication module 1202, an image capture location module 1204, a stitching module 1206, a crop module 1208, an image cropping module 1210, a blending module 1211, a 3D image generator 1214, a captured 2D image data store 1216, a 3D panoramic image data store 1218, and a guidance module 220. It can be understood that there can be any number of modules in the alignment and stitching system 1114 that perform one or more different functions as described herein.

[0172] In some embodiments, the alignment and stitching system 1114 includes an image capture module configured to receive images from one or more image capture devices (e.g., cameras). The alignment and stitching system 1114 may also include a depth module configured to receive depth data from a depth device, such as a LiDAR, if available.

[0173] The communications module 1202 can send and receive requests, images, or data between any of the modules or data stores of the registration and stitching system 1114 and components of the example environment 1100 of Figure 11. Similarly, the registration and stitching system 1114 can send and receive requests, images, or data to any device or system via the communications network 1104.

[0174] In some embodiments, the image capture position module 1204 can determine image capture device position data of an image capture device (e.g., a camera that may be a standalone camera, a smartphone, a media tablet, a laptop, etc.). The image capture device position data can be indicative of the position and orientation of the image capture device and / or lens. In one example, the image capture position module 1204 can utilize an IMU of the user system 1110, a camera, a digital device with a camera, or the 3D and panoramic capture and stitching system 1102 to generate the image capture device position data. The image capture position module 1204 can determine the current direction, angle, or tilt of one or more image capture devices (or lenses). The image capture position module 1204 may utilize a GPS of the user system 1110 or the 3D and panoramic capture and stitching system 1102.

[0175] For example, when a user wants to use the user system 1110 to capture a 360° view of a physical environment, such as a living room, the user may hold the user system 1110 at eye level in front of the user and begin capturing one of the images that will ultimately become a single 3D panoramic image. To reduce the amount of parallax for the images and capture images that are more suitable for stitching and generating a 3D panoramic image, it may be preferable for one or more image capture devices to rotate about a center of rotation axis. The alignment and stitching system 1114 may receive position information (e.g., from an IMU) to determine the position of the image capture device or lens. The alignment and stitching system 1114 may receive and store the field of view of the lens. The guidance module 1220 may provide visual and / or audio information regarding a recommended initial position of the image capture device. The guidance module 1220 may provide recommendations for positioning the image capture device for subsequent images. In one example, the guidance module 1220 may provide guidance to the user to rotate and position the image capture device so that the image capture device rotates near the center of rotation. Additionally, the guidance module 1220 can provide guidance to the user for rotating and positioning the image capture device so that subsequent images are generally aligned based on the field of view and / or characteristics of the image capture device.

[0176] The guidance module 1220 may provide visual guidance to the user. For example, the guidance module 1220 may place a marker or an arrow on a viewer or display on the user system 1110 or the 3D and panoramic capture and stitching system 1102. In some embodiments, the user system 1110 may be a smartphone or tablet computer with a display. When taking one or more photographs, the guidance module 1220 may position one or more markers (e.g., markers of different colors or the same markers) on the output device and / or in the viewfinder. The user may then use these markers on the output device and / or the viewfinder to align the next image.

[0177] There are numerous techniques to guide a user of the user system 1110 or the 3D and panorama capture and stitching system 1102 to capture multiple images so that they can be easily stitched together into a panorama. When a panorama is derived from multiple images, the images may be stitched together. To improve the time, efficiency, and effectiveness of stitching the images together while reducing the need for artifact or misalignment correction, the image capture position module 1204 and guidance module 1220 can assist the user in capturing multiple images at positions that improve the quality, time efficiency, and effectiveness of stitching the images for the desired panorama.

[0178] For example, after taking a first photograph, the display of the user system 1110 may include two or more objects, such as circles. The two circles may appear stationary relative to the environment, and the two circles may move with the user system 1110. Once the two stationary circles are aligned with a circle that moves with the user system 1110, the image capture device and / or the user system 1110 may be aligned for the next image.

[0179] In some embodiments, after capturing an image with an image capture device, the image capture position module 1204 can obtain sensor measurements of the position (including, e.g., orientation, tilt, etc.) of the image capture device. The image capture position module 1204 can determine one or more edges of the captured image by calculating locations of edges of the field of view based on the sensor measurements. Additionally or alternatively, the image capture position module 1204 can determine one or more edges of the image by scanning an image captured by the image capture device, identifying objects in the image (e.g., using a machine learning model described herein), determining one or more edges of the image, and positioning the object (e.g., a circle or other shape) to an edge of a display on the user system 1110.

[0180] The image capture position module 1204 can display two objects in the display of the user system 1110 indicating the positioning of the field of view for the next photograph. These two objects can indicate positions in the environment that represent where the edge of the last image is. The image capture position module 1204 can continue to receive sensor measurements of the position of the image capture device and calculate two more objects in the field of view. These two more objects can be the same width apart as the previous two objects. The first two objects can represent one edge of the captured image (e.g., the right-most edge of the image), while the next two more objects representing the edge of the field of view can be at the opposite edge (e.g., the left-most edge of the field of view). By having the user physically align the first two objects at the edge of the image with the two more objects at the opposite edge of the field of view, the image capture device can be positioned to take another image that can be stitched together more effectively without the use of a tripod. This process can continue for each image until the user determines that the desired panorama has been captured.

[0181] Although multiple objects are described herein, it will be understood that the image capture position module 1204 may calculate the position of one or more objects for positioning the image capture device. The objects may be of any shape (e.g., circle, oval, square, glyph, arrow, etc.). In some embodiments, the objects may be of different shapes.

[0182] In some embodiments, there may be distance between objects that represent edges of the captured image, and there may be distance between objects in the field of view. The user may be guided to move forward and away to allow sufficient distance between the objects. Alternatively, the size of the objects in the field of view may change to match the size of the objects that represent edges of the captured image as the image capture device approaches the correct position (e.g., by moving closer to or away from a position that allows the next image to be taken in a position that improves image stitching).

[0183] In some embodiments, the image capture location module 1204 can estimate the location of the image capture device using objects in an image captured by the image capture device. For example, the image capture location module 1204 can use GPS coordinates to determine a geographic location associated with the image. The image capture location module 1204 can use this location to identify landmarks that may be captured by the image capture device.

[0184] The image capture location module 1204 may include 2D machine learning models for converting 2D images into 2D panoramic images. The image capture location module 1204 may include 3D machine learning models for converting 2D images into 3D representations. In one example, the 3D representations may be utilized to display three-dimensional walk-throughs or visualizations of indoor and / or outdoor environments.

[0185] The 2D machine learning model may be trained to perform or assist in the formation of a 2D panoramic image by stitching two or more 2D images. For example, the 2D machine learning model may be trained with 2D images that include physical objects in the images and object identification information, whereby the 2D machine learning model is trained to identify the object in a subsequent 2D image. The object in the 2D image may assist in determining one or more locations in the 2D image, which may assist in determining edges of the 2D image, warping deformations in the 2D image, and aligning the images. Additionally, the object in the 2D image may assist in determining artifacts in the 2D image, blending artifacts or boundaries of two image views, determining where to cut an image, and / or determining where to crop an image.

[0186] In some embodiments, the 2D machine learning model may be, for example, a neural network trained on 2D images that include depth information of the environment (e.g., from a LiDAR device or structured lighting device of the user system 1110 or the 3D and panoramic capture and stitching system 1102) and include a physical object within the image, thereby identifying the physical object, the location of the physical object, and / or the location of the image capture device / field of view. The 2D machine learning model may assist in aligning and positioning two 2D images for stitching (or stitching two images) by identifying the physical object and the depth of the physical object relative to other aspects of the 2D image.

[0187] The 2D machine learning models may include any number of machine learning models (e.g., any number of models generated by neural networks, etc.).

[0188] The 2D machine learning model may be stored in the 3D and panoramic capture and stitching system 1102, the image stitching processor system 1106, and / or the user system 1110. In some embodiments, the 2D machine learning model may be trained by the image stitching processor system 1106.

[0189] The image capture position module 1204 can estimate the position of the image capture device (a portion of the field of view of the image capture device) based on the seams between two or more 2D images from the stitching module 1206, the image warp transformation from the crop module 1208, and / or the image crop from the image cropping module 1210.

[0190] The stitching module 1206 can combine two or more 2D images based on seams between the two or more 2D images from the stitching module 1206, image warping from the crop module 1208, and / or image cropping to generate a 2D panorama having a field of view larger than the field of view of each of the two or more images.

[0191] The stitching module 1206 may be configured to generate a panoramic 2D image of the same environment by aligning or "stitching together" two different 2D images that provide different perspectives of the environment. For example, the stitching module 1206 may use known or derived (e.g., using the techniques herein) information about the capture position and orientation of each 2D image to assist in stitching the two images together.

[0192] The stitching module 1206 may receive two 2D images. The first 2D image may have been captured immediately before or within a predetermined time period of the second 2D image. In various embodiments, the stitching module 1206 may receive image capture device positioning information associated with the first image and positioning information associated with the second image. The positioning information may be associated with the images based on positioning data from an IMU, GPS, and / or user-provided information at the time the images were captured.

[0193] In some embodiments, the stitching module 1206 can utilize a 2D machine learning module to scan both images and recognize objects in both images, including objects (or parts of objects) that may be shared by both images. For example, the stitching module 1206 can identify corners, wall patterns, furniture, etc. that are shared on opposing edges of both images.

[0194] The stitching module 1206 can align edges of two 2D images and combine (i.e., "stitch" them together) the two edges of the images based on positioning of the shared object (or part of the object), positioning data from an IMU, positioning data from a GPS, and / or information provided by a user. In some embodiments, the stitching module 1206 can identify portions of the 2D images that overlap each other and stitch the images together at the overlapping locations (e.g., using positioning data and / or results of a 2D machine learning model).

[0195] In various embodiments, the 2D machine learning model may be trained to combine or stitch two edges of an image using positioning data from an IMU, positioning data from a GPS, and / or information provided by a user. In some embodiments, the 2D machine learning model may be trained to align and register the 2D images by identifying common objects in both 2D images, and to combine or stitch two edges of the images. In further embodiments, the 2D machine learning model may be trained to align and register the 2D images using positioning data and object recognition, and to stitch two edges of the images together to form a full or partial panoramic 2D image.

[0196] The stitching module 1206 can utilize depth information about each image (e.g., pixels within each image, objects within each image, etc.) to facilitate alignment of each 2D image relative to one another associated with generating a single 2D panoramic image of the environment.

[0197] The crop module 1208 can solve problems with two or more 2D images when the image capture device is not held in the same position when the 2D images are captured. For example, during capture of one image, a user may position the user system 1110 in a vertical position. However, during capture of another image, the user may position the user system at an angle. The resulting images may not be aligned and may suffer from parallax effects. Parallax effects can occur when foreground and background objects are not aligned in the same way in the first and second images.

[0198] The crop module 1208 can utilize 2D machine learning models (by applying positioning information, depth information, and / or object recognition) to detect changes in the position of an image capture device in two or more images and measure the amount of change in the position of the image capture device. The crop module 1208 can warp transform one or more 2D images so that when stitched together, the images line up to form a panoramic image, while preserving certain characteristics of the images, such as keeping straight lines straight.

[0199] The output of the crop module 1208 may include the number of pixel columns and rows to offset each pixel of the image to straighten the image. The amount of offset for each image may be output in the form of a matrix representing the number of pixel columns and pixel rows to offset each pixel of the image.

[0200] In some embodiments, the crop module 1208 can determine the amount of image warping transformation to perform on one or more of the multiple 2D images captured by the image capture device of the user system 1110 based on one or more image capture positions from the image capture position module 1204, or seams between two or more 2D images from the stitching module 1206, an image crop from the image crop module 1210, or a color blend from the blend module 1211.

[0201] The image cropping module 1210 can determine where to crop or slice one or more of the 2D images captured by the image capture device. For example, the image cropping module 1210 can use a 2D machine learning model to identify an object in both images and determine that they are the same object. The image capture position module 1204, the crop module 1208, and / or the image cropping module 1210 can determine that the two images cannot be aligned even if they are warped. The image cropping module 1210 can use information from the 2D machine learning model to identify sections of the two images that can be stitched together (e.g., by cropping a portion of one or both images to aid in alignment and positioning). In some embodiments, the two 2D images may overlap at least in a portion of the real world depicted in the images. The image cropping module 1210 can identify an object in both images, e.g., the same chair. However, the images of the chair may not be aligned to produce an undistorted panorama, even after image capture positioning and cropping module 1208 warps the images, resulting in an incorrect representation of the portion of the real world. Image cropping module 1210 can select one of the two images of the chair as the correct representation (e.g., based on misalignment, positioning, and / or artifacts of one image compared to the other) and crop the chair from the image with misalignment, positioning errors, and artifacts. Stitching module 1206 can then stitch the two images together.

[0202] The image cropping module 1210 can try both combinations, for example cropping the image of the chair from the first image and stitching the first image minus the chair into the second image, to determine which image crop produces a more precise panoramic image. The output of the image cropping module 1210 can be a location to crop one or more of the multiple 2D images that corresponds to the image crop that produces the more precise panoramic image.

[0203] The image cropping module 1210 can determine how to crop or slice one or more of the 2D images captured by the image capture device based on one or more image capture locations from the image capture location module 1204; the stitching or seams between two or more 2D images from the stitching module 1206; the image warp transformation from the crop module 1208; and the image crop from the image cropping module 1210.

[0204] The Blending module 1211 can color a seam (e.g., stitching) between two images so that the seam is not visible. Changes in lighting and shadows can cause the same object or surface to be output in slightly different colors or shades. The Blending module can determine the amount of color blending required based on: one or more image capture locations from the Image Capture Location module 1204; the stitching; the image colors along the seam from the two images; the image warp transformation from the Crop module 1208; and / or the image crop from the Image Cropping module 1210.

[0205] In various embodiments, the blending module 1211 may receive a panorama from the combination of two 2D images and sample colors along the seam of the two 2D images. The blending module 1211 may receive seam location information from the image capture location module 1204, which allows the blending module 1211 to sample colors along the seam to determine differences. If there is a significant difference in color along the seam between the two images (e.g., within a predetermined threshold of color, hue, brightness, saturation, etc.), the blending module 1211 may blend both images of a predetermined size along the seam where the difference exists. In some embodiments, the greater the color or image difference along the seam, the more space along the seam of the two images may be blended.

[0206] In some embodiments, after blending, the blending module 1211 may rescan and sample the colors along the seams to determine whether there are other differences in the images or colors that exceed the above-mentioned predetermined thresholds, such as color, hue, brightness, saturation, etc. If so, the blending module 1211 may identify those portions along the seams and continue blending those portions of the images. The blending module 1211 may continue resampling the images along the seams until there are no more portions of the images to blend (e.g., the color differences are below one or more predetermined thresholds).

[0207] The 3D image generator 1214 can receive the 2D panoramic image and generate a 3D representation. In various embodiments, the 3D image generator 1214 utilizes a 3D machine learning model to convert the 2D panoramic image into a 3D representation. The 3D machine learning model can be trained to create the 3D representation using the 2D panoramic image and depth data (e.g., from a LiDAR sensor or structured lighting device). The 3D representation may be tested and reviewed for curation and feedback. In some embodiments, the 3D machine learning model can be used with the 2D panoramic image and depth data to generate the 3D representation.

[0208] In various embodiments, the accuracy, rendering speed, and quality of the 3D representation generated by the 3D image generator 1214 are significantly improved by utilizing the systems and methods described herein. For example, rendering the 3D representation from 2D panoramic images that have been aligned, registered, and stitched using the methods described herein (e.g., by: by hardware-provided alignment and positioning information; by improved positioning resulting from guidance provided to the user during image capture; by image cropping and warping modification; by cropping images to avoid artifacts and overcome warping deformations; by image blending; and / or combinations thereof) improves the accuracy, rendering speed, and quality of the 3D representation. Furthermore, it will be appreciated that by utilizing 2D panoramic images that have been aligned, registered, and stitched using the methods described herein, the training of 3D machine learning models can be significantly improved (e.g., in terms of speed and accuracy). Furthermore, in some embodiments, the 3D machine learning models can be smaller and less complex. This is because reduced processing and learning is required to overcome misalignment, registration errors, warping deformations, poor image cropping, poor blending, artifacts, etc. to produce a reasonably accurate 3D representation.

[0209] The trained 3D machine learning model can be stored in the 3D and panorama capture and stitching system 1102, the image stitching and processor system 106, and / or the user system 1110.

[0210] In some embodiments, the 3D machine learning model may be trained using a plurality of 2D image and depth data from an image capture device of the user system 1110 and / or the 3D and panoramic capture and stitching system 1102. Additionally, the 3D image generator 1214 may be trained using: image capture position information associated with each of the plurality of 2D images from the image capture position module 1204; seam locations for aligning or stitching each of the plurality of 2D images from the stitching module 1206; one or more pixel offsets for each of the plurality of 2D images from the crop module 1208; and / or an image crop from the image crop module 1210. In some embodiments, the 3D machine learning model can be used in conjunction with: the 2D panoramic image; the depth data; image capture position information associated with each of the multiple 2D images from image capture position module 1204; seam locations for aligning or stitching each of the multiple 2D images from stitching module 1206; one or more pixel offsets for each of the multiple 2D images from crop module 1208; and / or an image crop from image crop module 1210 to generate the 3D representation.

[0211] The stitching module 1206 may be part of a 3D model that converts a plurality of 2D images into a 2D panoramic or 3D panoramic image. In some embodiments, the 3D model is a machine learning algorithm, such as a 3D-from-2D predictive neural network model. The crop module 1208 may be part of a 3D model that converts a plurality of 2D images into a 2D panoramic or 3D panoramic image. In some embodiments, the 3D model is a machine learning algorithm, such as a 3D-from-2D predictive neural network model. The image cropping module 1210 may be part of a 3D model that converts a plurality of 2D images into a 2D panoramic or 3D panoramic image. In some embodiments, the 3D model is a machine learning algorithm, such as a 3D-from-2D predictive neural network model. The blending module 1211 may be part of a 3D machine learning model that converts a plurality of 2D images into a 2D panoramic or 3D panoramic image. In some embodiments, the 3D model is a machine learning algorithm, such as a 3D-from-2D predictive neural network model.

[0212] The 3D image generator 1214 may generate a weighting for each of the image capture position module 1204, the crop module 1208, the image crop module 1210, and the blend module 1211, which may represent the reliability or "strength" or "weakness" of the module. In some embodiments, the sum of the weightings of these modules is equal to 1.

[0213] If depth data is not available for the multiple 2D images, the 3D image generator 1214 can determine depth data for one or more objects in the multiple 2D images captured by the image capture device of the user system 1110. In some embodiments, the 3D image generator 1214 may derive depth data based on images captured by a stereo image pair. Rather than determining depth data from a passive stereo algorithm, the 3D image generator can evaluate stereo image pairs to determine data regarding the photometric match quality between images at various depths (a more intermediate result).

[0214] The 3D image generator 1214 may be part of a 3D model that converts multiple 2D images into a 2D panoramic or 3D panoramic image. In some embodiments, the 3D model is a machine learning algorithm, such as a 3D-from-2D predictive neural network model.

[0215] The captured 2D image data store 1216 may be any structure and / or structures suitable for captured image and / or depth data (e.g., active database, relational database, self-referential database, table, matrix, array, flat file, document-oriented storage system, non-relational No-SQL system, FTS management system such as Lucene / Solar, etc.). The captured 2D image data store 1216 can store images captured by an image capture device of the user system 1110. In various embodiments, the captured 2D image data store 1216 stores depth data captured by one or more depth sensors of the user system 1110. In various embodiments, the captured 2D image data store 1216 stores image capture device characteristics associated with an image capture device or capture characteristics associated with each of multiple image captures or depth captures used to determine the 2D panoramic image. In some embodiments, the image data store 1108 stores a 2D panoramic image. The 2D panoramic image can be determined by the 3D and panoramic capture and stitching system 1102 or the image stitching and processor system 106. Image capture device parameters include illumination, color, focal length of the image capture lens, maximum aperture, tilt angle, etc. Capture characteristics include pixel resolution, lens distortion, lighting, and other image metadata.

[0216] The 3D panoramic image data store 1218 may be any structure and / or multiple structures suitable for 3D panoramic images (e.g., active databases, relational databases, self-referential databases, tables, matrices, arrays, flat files, document-oriented storage systems, non-relational No-SQL systems, FTS management systems such as Lucene / Solar, etc.). The 3D panoramic image data store 1218 can store 3D panoramic images generated by the 3D and panoramic capture and stitching system 1102. In various embodiments, the 3D panoramic image data store 1218 stores characteristics associated with an image capture device or characteristics associated with each of multiple image captures or depth captures used to determine the 3D panoramic image. In some embodiments, the 3D panoramic image data store 1218 stores 3D panoramic images. 2D or 3D panoramic images can be determined by the 3D and panoramic capture and stitching system 1102 or the image stitching processor system 106.

[0217] FIG. 13 shows a flowchart 1300 of a 3D panoramic image capture and generation process according to some embodiments. In step 1302, an image capture device may capture multiple 2D images using the image sensor 920 and WFOV lens 918 of FIG. 9. A wider FOV means less scanning is required by the environment capture system 402 to get a 360° view. The WFOV lens 918 may also be wider horizontally and vertically. In some embodiments, the image sensor 920 captures RGB images. In one embodiment, the image sensor 920 captures black and white images.

[0218] In step 1304, the environment capture system may send the captured 2D images to the image stitching processor system 1106. The image stitching processor system 1106 may obtain a panoramic 2D image by applying a 3D modeling algorithm to the captured 2D images. In some embodiments, the 3D modeling algorithm is a machine learning algorithm for stitching the captured 2D images into a panoramic 2D image. In some embodiments, step 1304 may be optional.

[0219] 9 may capture LiDAR data. A wider FOV means that the environment capture system 400 requires fewer scans to get a 360° view.

[0220] In step 1308, the LiDAR data may be sent to the image stitching processor system 1106. The image stitching processor system 1106 may input the LiDAR data and the captured 2D images into a 3D modeling algorithm to generate a 3D panoramic image. The 3D modeling algorithm may be a machine learning algorithm.

[0221] In step 1310, the image stitching processor system 1106 generates a 3D panoramic image. The 3D panoramic image may be stored in the image data store 408. In one embodiment, the 3D panoramic image generated by the 3D modeling algorithm is stored in the image stitching processor system 1106. In some embodiments, the 3D modeling algorithm utilizes an environmental capture system to capture various portions of the physical environment, so that a visual representation of the floor plan of the physical environment can be generated.

[0222] In step 1312, the image stitching processor system 1106 may provide at least a portion of the generated 3D panoramic image to the user system 1110. The image stitching processor system 1106 may provide a visual representation of the floor plan of the physical environment.

[0223] The order of one or more steps of the flowchart 1300 can be changed without affecting the final production of the 3D panoramic image. For example, the environment capture system can sandwich LiDAR data or depth information capture by the LiDAR 912 between image capture by the image capture device. For example, the image capture device may capture an image of a section of the physical environment, and then the LiDAR 912 obtains depth information from this section 1605. Once the LiDAR 912 obtains depth information from this section, the image capture device may move to capture an image of another section, and then the LiDAR 912 obtains depth information from this section. In this manner, image capture and depth information capture alternate.

[0224] In some embodiments, the devices and / or systems described herein capture 2D input images using one image capture device. In some embodiments, one or more image capture devices 1116 can represent a single image capture device (or image capture lens). According to some of these embodiments, a user of a mobile device housing the image capture device can be configured to rotate about an axis to generate images at different capture orientations relative to the environment, with a combined field of view of up to 360° horizontally.

[0225] In various embodiments, the devices and / or systems described herein may capture 2D input images using two or more image capture devices. In some embodiments, the two or more image capture devices may be disposed on or in the same mobile housing at relative positions relative to one another such that their combined field of view spans 360°. In some embodiments, multiple pairs of image capture devices (e.g., having slightly offset but partially overlapping fields of view) may be used that can generate a pair of stereo images. For example, the user system 1110 (e.g., a device that includes one or more image capture devices used to capture 2D input images) may include two image capture devices with horizontal stereo offset fields of view that can capture a pair of stereo images. In another example, the user system 1110 may include two image capture devices with vertical stereo offset fields of view that can capture a pair of vertical stereo images. According to any of these examples, each camera may have a field of view that spans 360°. In this regard, in one embodiment, the user system 1110 may use two panoramic cameras with vertical stereo offsets that can capture a pair of panoramic images that form a stereo pair (with vertical stereo offset).

[0226] The positioning component 1118 may include any hardware and / or software configured to capture user system position data and / or user system location data. For example, the positioning component 1118 may include an IMU to generate position data for the user system 1110 associated with one or more image capture devices of the user system 1110 used to capture the multiple 2D images. The positioning component 1118 may include a GPS unit to provide GPS coordinate information associated with the multiple 2D images captured by the one or more image capture devices. In some embodiments, the positioning component 1118 may correlate the user system position and location data with each image captured using one or more image capture devices of the user system 1110.

[0227] Various embodiments of the device provide a user with 3D panoramic images of indoor and outdoor environments. In some embodiments, the device can efficiently and quickly provide a user with 3D panoramic images of indoor and outdoor environments using a single wide field of view (FOV) lens and a single light detection and ranging (LiDAR) sensor.

[0228] The following is an exemplary use example of the exemplary device described herein. The following use example is one of multiple embodiments. As described herein, different embodiments of the device may include one or more features and functions similar to this use example.

[0229] 14 shows a flowchart of a 3D and panoramic capture and stitching process 1400 according to some embodiments. The flowchart in FIG. 14 illustrates the 3D and panoramic capture and stitching system 1102 as including an image capture device, but in some embodiments, the data capture device may be a user system 1110.

[0230] In step 1402, the 3D and panoramic capture and stitching system 1102 may receive a plurality of 2D images from at least one image capture device. The image capture device of the 3D and panoramic capture and stitching system 1102 may be or may include a complementary metal oxide semiconductor (CMOS) image sensor. In various embodiments, the image capture device is a charge-coupled device (CCD). In one example, the image capture device is a red-green-blue (RGB) sensor. In one embodiment, the image capture device is an IR sensor. Each of the plurality of 2D images may have a field of view that overlaps with at least one other image of the plurality of 2D images. In some embodiments, at least some of the plurality of 2D images are combined to create a 360° view of the physical environment (e.g., indoors, outdoors, or both).

[0231] In some embodiments, the plurality of 2D images are all received from the same image capture device. In various embodiments, at least a portion of the plurality of 2D images are received from two or more image capture devices of the 3D and panoramic capture and stitching system 1102. In one example, the plurality of 2D images includes a set of RGB images and a set of IR images, where the IR images provide depth data to the 3D and panoramic capture and stitching system 1102. In some embodiments, each 2D image can be associated with depth data provided from a LiDAR device. In some embodiments, each 2D image can be associated with positioning data.

[0232] At step 1404, the 3D and panoramic capture and stitching system 1102 may receive capture parameters and image capture device parameters associated with each of the received multiple 2D images. Image capture device parameters include illumination, color, focal length of the image capture lens, maximum aperture, field of view, etc. Capture characteristics include pixel resolution, lens distortion, illumination, and other image metadata. The 3D and panoramic capture and stitching system 1102 may also receive positioning data and depth data.

[0233] In step 1406, the 3D and panoramic capture and stitching system 1102 may use the information received from steps 1402, 1404 to stitch the 2D images together to form a 2D panoramic image. The process of stitching 2D images is further described in conjunction with the flowchart of FIG.

[0234] In step 1408, the 3D and panoramic capture and stitching system 1102 may apply the 3D machine learning model to generate a 3D representation. The 3D representation may be stored in a 3D panoramic image data store. In various embodiments, the 3D representation is generated by the image stitching and processor system 1106. In some embodiments, the 3D machine learning model may generate a visual representation of the floor plan of the physical environment, since the environmental capture system is utilized to capture various portions of the physical environment.

[0235] In step 1410, the 3D and panoramic capture and stitching system 1102 may provide at least a portion of the generated 3D representation or model to a user system 1110. The user system 1110 may provide a visual representation of the floor plan of the physical environment.

[0236] In some embodiments, the user system 1110 may send multiple 2D images, capture parameters, and image capture parameters to the image stitching processor system 1106. In various embodiments, the 3D and panoramic capture and stitching system 1102 may send multiple 2D images, capture parameters, and image capture parameters to the image stitching processor system 1106.

[0237] The image stitching processor system 1106 may process multiple 2D images captured by the image capture devices of the user system 1110 and stitch them into a 2D panoramic image. The 2D panoramic image processed by the image stitching processor system 1106 may have a higher pixel resolution than the 2D panoramic image obtained by the 3D and panoramic capture and stitching system 1102.

[0238] In some embodiments, the image stitching processor system 106 may receive the 3D representation and output a 3D panoramic image having a higher pixel resolution than the received 3D panoramic image. This higher pixel resolution panoramic image can be provided to an output device, such as a computer screen, a projector screen, etc., having a higher screen resolution than the user system 1110. In some embodiments, this higher pixel resolution panoramic image can provide the output device with a more detailed panoramic image and can be scaled.

[0239] FIG. 15 illustrates a flow chart showing further details of one step of the 3D and panoramic capture and stitching process of FIG. 14. In step 1502, the image capture position module 1204 may determine image capture device position data associated with each image captured by the image capture device. The image capture position module 1204 may use an IMU of the user system 1110 to determine the position data of the image capture device (or the field of view of the lens of the image capture device). The position data may include the orientation, angle, or tilt of the one or more image capture devices at the time of capture of the one or more 2D images. One or more of the crop module 1208, the image crop module 1210, and the blend module 1212 may use the orientation, angle, or tilt associated with each of the multiple 2D images to determine how to warp, crop, and / or blend the images.

[0240] In step 1504, the crop module 1208 may warp one or more of the multiple 2D images to allow the two images to line up to form a single panoramic image while preserving certain characteristics of the images, such as keeping straight lines straight. The output of the crop module 1208 may include the number of pixel columns and rows to offset each pixel of the image to straighten the image. The amount of offset for each image may be output in the form of a matrix representing the number of pixel columns and pixel rows to offset each pixel of the image. In this embodiment, the crop module 1208 may determine the amount of warping required for each of the multiple 2D images based on the image capture pose estimate for each of the multiple 2D images.

[0241] At step 1506, the image cropping module 1210 determines where to crop or slice one or more of the plurality of 2D images. In this embodiment, the image cropping module 1210 may determine where to crop or slice each of the plurality of 2D images based on an image capture pose estimate and an image warp transformation for each of the plurality of 2D images.

[0242] In step 1508, the stitching module 1206 may stitch two or more images together using image edges and / or image cropping. The stitching module 1206 may align and / or position the images based on objects detected in the images, warp transformations, image cropping, etc.

[0243] In step 1510, the blending module 1212 may adjust a seam (e.g., stitching two images) or the location of one image that touches or connects to another image. The blending module 1212 can determine the amount of color blending required based on: one or more image capture positions from the image capture position module 1204; the image warp transformation from the crop module 1208; and / or the image crop from the image crop module 1210.

[0244] The order of one or more steps of the 3D and panoramic capture and stitching process 1400 can be changed without affecting the final production of the 3D panoramic image. For example, the environment capture system can sandwich LiDAR data or depth information capture between image capture by the image capture device. For example, the image capture device may capture an image of section 1605 of FIG. 16 of the physical environment, and then the LiDAR 612 obtains depth information from section 1605. Once the LiDAR obtains depth information from section 1605, the image capture device may move to capture an image of another section 1610, and then the LiDAR 612 obtains depth information from section 1610. In this manner, image capture and depth information capture are alternated.

[0245] 16 illustrates a block diagram of an exemplary digital device 1602 according to some embodiments. Any of the user system 1110, the 3D panorama capture and stitching system 1102, and the image stitching and processor system may include an instance of the digital device 1602. The digital device 1602 includes a processor 1604, a memory 1606, a storage 1608, an input device 1610, a communication network interface 1612, an output device 1614, an image capture device 1616, and a positioning component 1618. The processor 1604 is configured to execute executable instructions (e.g., a program). In some embodiments, the processor 1604 includes any processor or circuitry capable of processing executable instructions.

[0246] The memory 1606 stores data. Some examples of the memory 1606 include storage devices such as RAM, ROM, RAM cache, virtual memory, etc. In various embodiments, working data is stored in the memory 1606. The data in the memory 1606 may be cleared or eventually transferred to the storage 1608.

[0247] Storage 1608 includes any storage configured to acquire and store data. Some examples of storage 1608 include a flash drive, a hard drive, an optical drive, and / or a magnetic tape. Memory 1606 and storage 1608 each include a computer-readable medium that stores instructions or programs executable by processor 1604.

[0248] Input device 1610 is any device for inputting data (e.g., touch keyboard, stylus). Output device 1614 outputs data (e.g., speakers, display, virtual reality headset). Storage 1608, input device 1610, and output device 1614 will be understood. In some embodiments, output device 1614 is optional. For example, a router / switcher may include a processor 1604 and memory 1606, as well as devices for receiving and outputting data (e.g., a communications network interface 1612 and / or an output device 1614).

[0249] The communication network interface 1612 may be coupled to a network (e.g., the communication network 104) via the communication network interface 1612. The communication network interface 1612 may support communication via an Ethernet connection, a serial connection, a parallel connection, and / or an ATA connection. The communication network interface 1612 may also support wireless communication (e.g., 802.16 a / b / g / n, WiMAX, LTE, Wi-Fi). It will be apparent that the communication network interface 1612 may support wired and wireless standards.

[0250] A component may be hardware or software. In some embodiments, a component may configure one or more processors to perform the functionality associated with the component. Although various components are described herein, it will be understood that a server system may include any number of components that perform any functionality described herein.

[0251] The digital device 1602 may include one or more image capture devices 1616. The one or more image capture devices 1616 may include, for example, an RGB camera, an HDR camera, a video camera, etc. The one or more image capture devices 1616 may also include a video camera capable of capturing video according to some embodiments. In some embodiments, the one or more image capture devices 1616 may include an image capture device that provides a relatively standard field of view (e.g., about 75°). In other embodiments, the one or more image capture devices 1616 may include a camera that provides a relatively wide field of view (e.g., about 120° to 360°), such as a fisheye camera (e.g., the digital device 1602 may include or be included in the environment capture system 400).

[0252] A component may be hardware or software. In some embodiments, a component may configure one or more processors to perform the functionality associated with the component. Although various components are described herein, it will be understood that a server system may include any number of components that perform any functionality described herein.

Claims

1. 1. An image capture device comprising: a housing, the housing having a front surface and a back surface; a first motor coupled to the housing at a first location between the front and back sides of the housing, the first motor configured to horizontally turn the image capture device approximately 270 degrees about a vertical axis; a wide angle lens coupled to the housing at a second location between the front and back surfaces of the housing along the vertical axis, the second location being a point of difference, the wide angle lens having a field of view away from the front surface of the housing; an image sensor coupled to the housing and configured to generate an image signal from light received by the wide angle lens; a mount coupled to the first motor; a LiDAR coupled to the housing at a third location, the LiDAR configured to generate laser pulses and generate a depth signal; a second motor coupled to the housing; and a mirror coupled to the second motor, the second motor may be configured to rotate the mirror about a horizontal axis, the mirror including an angled surface configured to receive the laser pulses from the LiDAR and direct the laser pulses about the horizontal axis; An image capture device comprising:

2. The image capture device of claim 1 , wherein the image sensor is configured to generate a first plurality of images at different exposures when the image capture device is stationary and facing a first direction.

3. The image capture device of claim 2 , wherein the first motor is configured to turn the image capture device about the vertical axis after generating the first plurality of images.

4. 4. The image capture device of claim 3, wherein the image sensor does not generate images while the first motor is turning the image capture device, and the LiDAR generates a depth signal based on the laser pulses while the first motor is turning the image capture device.

5. 4. The image capture device of claim 3, wherein the image sensor is configured to generate a second plurality of images at the different exposures when the image capture device is stationary and facing a second direction, and the first motor is configured to turn the image capture device 90 degrees about the vertical axis after generating the second plurality of images.

6. 6. The image capture device of claim 5, wherein the image sensor is configured to generate a third plurality of images at the different exposures when the image capture device is stationary and facing a third direction, and the first motor is configured to turn the image capture device 90 degrees about the vertical axis after generating the third plurality of images.

7. 7. The image capture device of claim 6, wherein the image sensor is configured to generate a fourth plurality of images at the different exposures when the image capture device is stationary and facing a fourth direction, and the first motor is configured to turn the image capture device 90 degrees about the vertical axis after generating the fourth plurality of images.

8. 8. The image capture device of claim 7, further comprising a processor configured to blend frames of the first plurality of images before the image sensor generates the second plurality of images.

9. 8. The image capture device of claim 7, further comprising a remote digital device in communication with the image capture device and configured to generate a 3D visualization based on the first, second, third and fourth plurality of images and the depth signal, the remote digital device configured to generate the 3D visualization without using any images other than the first, second, third and fourth plurality of images.

10. 10. The image capture device of claim 9, wherein the first, second, third and fourth plurality of images are generated during a combination of multiple turns that rotate the image capture device 270 degrees about the vertical axis.

11. The image capture device of claim 4 , wherein the speed or rotation of the mirror about the horizontal axis increases as the first motor turns the image capture device.

12. The image capture device of claim 1 , wherein the angled surface of the mirror is 90°.

13. The image capture device of claim 1 , wherein the LiDAR emits the laser pulses in a direction away from the front surface of the housing.

14. 13. A method comprising: receiving light from a wide angle lens of an image capture device, the wide angle lens being coupled to a housing of the image capture device, the light being received at a field of view of the wide angle lens, the field of view extending away from a front surface of the housing; generating a first plurality of images with an image sensor of an image capture device using the light from the wide angle lens, the image sensor being coupled to the housing, the first plurality of images being at different exposures; rotating the image capture device horizontally approximately 270 degrees about a vertical axis with a first motor, the first motor being coupled to the housing at a first position between a rear face and a front face of the housing and the wide angle lens being at a second position along the vertical axis, the second position being a cross point; rotating a mirror having an angled surface about a horizontal axis with a second motor, the second motor coupled to the housing; generating a laser pulse by a LiDAR, the LiDAR being coupled to the housing in a third position and the laser pulse being directed toward the rotating mirror while the image capture device is turning horizontally; and generating a depth signal by the LiDAR based on the laser pulses. A method comprising:

15. The method of claim 14 , wherein the step of generating the first plurality of images with the image sensor occurs before the image capture device turns horizontally.

16. 16. The method of claim 15, wherein the image sensor does not generate images while the first motor is turning the image capture device, and the LiDAR generates the depth signal based on the laser pulses while the first motor is turning the image capture device.

17. generating a second plurality of images at the different exposures with the image sensor while the image capture device is stationary and facing a second direction; and turning the image capture device 90 degrees about the vertical axis by the first motor after generating the second plurality of images. The method of claim 16 further comprising:

18. generating a third plurality of images at the different exposures by the image sensor while the image capture device is stationary and facing a third direction; and after generating the third plurality of images, rotating the image capture device 90 degrees about the vertical axis by the first motor.

20. The method of claim 17, further comprising:

19. 20. The method of claim 18, further comprising generating a fourth plurality of images at the different exposures by the image sensor when the image capture device is stationary and facing a fourth direction.

20. 20. The method of claim 19, further comprising the step of generating a 3D visualization using the first, second, third and fourth plurality of images and based on the depth signal, wherein the step of generating the 3D visualization does not use any other images.

21. 20. The method of claim 17, further comprising blending frames of the first plurality of images before the image sensor generates the second plurality of images.

22. 20. The method of claim 19, wherein the first, second, third and fourth plurality of images are generated during a combination of multiple turns that rotate the image capture device 270 degrees about the vertical axis.

23. The method of claim 14 , wherein the speed or rotation of the mirror about the horizontal axis increases as the first motor turns the image capture device.

Images