System and method for capturing and generating panoramic three-dimensional images

Abstract

A system and method of capturing and generating panoramic three-dimensional images is disclosed. An apparatus comprising a housing, a mount configured to be coupled to a motor to move the apparatus horizontally, a wide-angle lens coupled to the housing, the wide-angle lens positioned above the mount along an axis of rotation about which the apparatus rotates, an image capture device within the housing configured to receive a two-dimensional image of an environment through the wide-angle lens, and a LiDAR device within the housing configured to generate depth data based on the environment.

Description

[0001] This application is a divisional application of Chinese patent application 202080084506.9 (PCT / US2020 / 067474), filed on December 30, 2020, entitled "System and method for capturing and generating panoramic 3D images". Technical Field

[0002] Embodiments of the present invention generally relate to capturing and stitching panoramic images of scenes in a physical environment. Background Technology

[0003] The proliferation of providing three-dimensional (3D) panoramic images of the physical world has spawned numerous solutions capable of capturing multiple two-dimensional (2D) images and generating 3D images based on the captured 2D images. Hardware solutions and software applications (or "apps") exist that can capture multiple 2D images and stitch them together to create a panoramic image.

[0004] Techniques exist for capturing and generating 3D data from buildings. However, existing techniques often cannot capture and generate 3D renderings of areas under bright lighting conditions. Areas where sunlight shines through windows or where floors or walls are exposed under bright lighting conditions often appear as holes in 3D renderings, which may require additional post-production work to fill in. This increases the turnaround time and realism of 3D renderings. Furthermore, outdoor environments present challenges for many existing 3D capture devices because structured light may not be used to capture 3D images.

[0005] Other limitations of existing technologies for capturing and generating 3D data include the amount of time required to capture and process the digital images needed to produce 3D panoramic images. Summary of the Invention

[0006] An example device includes a housing, a mounting, a wide-angle lens, an image capture device, and a LiDAR device. The mounting is configured to be coupled to a motor to move the device horizontally. The wide-angle lens is coupled to the housing and positioned above the mounting along a rotation axis, which is the axis along which the device rotates when coupled to the motor. The image capture device is located within the housing and is configured to receive a two-dimensional image of the environment through the wide-angle lens. The LiDAR device is located within the housing and is configured to generate depth data based on the environment.

[0007] An image capturing device may include a housing, a first motor, a wide-angle lens, an image sensor, a mounting bracket, a LiDAR, a second motor, and a mirror. The housing may have a front side and a rear side. The first motor may be coupled to the housing at a first position between the front and rear sides of the housing, the first motor being configured to rotate the image capturing device horizontally about a vertical axis by approximately 270 degrees. The wide-angle lens may be coupled to the housing at a second position along the vertical axis between the front and rear sides of the housing, the second position being a parallax-free point, the wide-angle lens having a field of view remote from the front side of the housing. The image sensor may be coupled to the housing and configured to generate an image signal from light received by the wide-angle lens. The mounting bracket may be coupled to the first motor. The LiDAR may be coupled to the housing at a third position, the LiDAR being configured to generate laser pulses and generate a depth signal. The second motor may be coupled to the housing. The reflector may be coupled to a second motor, which may be configured to rotate the reflector about a horizontal axis. The reflector includes an angled surface configured to receive the laser pulses from the LiDAR and guide the laser pulses about the horizontal axis.

[0008] In some embodiments, the image sensor is configured to generate a first plurality of images with different exposures when the image capturing device is stationary and pointing in a first direction. The first motor may be configured to rotate the image capturing device about the vertical axis after the first plurality of images have been generated. In various embodiments, the image sensor does not generate images when the first motor rotates the image capturing device, and wherein the LiDAR generates a depth signal based on the laser pulses when the first motor rotates the image capturing device. The image sensor may be configured to generate a second plurality of images with the different exposures when the image capturing device is stationary and pointing in a second direction, and the first motor is configured to rotate the image capturing device 90 degrees about the vertical axis after the second plurality of images have been generated. The image sensor may be configured to generate a third plurality of images with the different exposures when the image capturing device is stationary and pointing in a third direction, and the first motor is configured to rotate the image capturing device 90 degrees about the vertical axis after the third plurality of images have been generated. The image sensor can be configured to generate a fourth plurality of images with different exposures when the image capturing device is stationary and pointing in a fourth direction, and the first motor is configured to rotate the image capturing device 90 degrees around the vertical axis after the fourth plurality of images are generated.

[0009] In some embodiments, the system may further include a processor configured to blend frames of the first plurality of images prior to the generation of the second plurality of images by the image sensor. A remote digital device (RDD) may communicate with the image capture device and is configured to generate a 3D visualization based on the first, second, third, and fourth plurality of images and the depth signal, the DD being configured to generate the 3D visualization using no more than the first, second, third, and fourth plurality of images. In some embodiments, the first, second, third, and fourth plurality of images are generated between rotations, the rotations combining a rotation that causes the image capture device to rotate 270 degrees about the vertical axis. The speed or rotation of the reflector about the horizontal axis increases as the first motor rotates the image capture device. The angled surface of the reflector may be 90 degrees. In some embodiments, the LiDAR emits the laser pulses in a direction opposite to the front side of the housing.

[0010] An example method includes receiving light from a wide-angle lens of an image capturing device, the wide-angle lens being coupled to a housing of the image capturing device, the light being received at the field of view of the wide-angle lens extending away from the front side of the housing; generating a first plurality of images by an image sensor of the image capturing device using the light from the wide-angle lens, the image sensor being coupled to the housing, the first plurality of images being at different exposures; rotating the image capturing device horizontally about a vertical axis by a first motor at a first position between the front and rear sides of the housing, the wide-angle lens at a second position along the vertical axis, the second position being parallax-free; rotating a mirror having an angled surface about a horizontal axis by a second motor, the second motor being coupled to the housing; generating a laser pulse by a LiDAR at a third position coupled to the housing, the laser pulse being directed to the rotating mirror as the image capturing device rotates horizontally; and generating a depth signal by the LiDAR based on the laser pulse.

[0011] The generation of the first plurality of images by the image sensor can occur before the image capturing device rotates horizontally. In some embodiments, the image sensor does not generate an image when the first motor rotates the image capturing device, and wherein the LiDAR generates the depth signal based on the laser pulse when the first motor rotates the image capturing device.

[0012] The method may further include generating a second plurality of images by the image sensor with the different exposures when the image capturing device is stationary and pointing in the second direction, and after generating the second plurality of images, rotating the image capturing device 90 degrees around the vertical axis by the first motor.

[0013] In some embodiments, the method may further include generating a third plurality of images by the image sensor at different exposures when the image capturing device is stationary and pointing in a third direction, and after generating the third plurality of images, rotating the image capturing device 90 degrees about the vertical axis by the first motor. The method may further include generating a fourth plurality of images by the image sensor at different exposures when the image capturing device is stationary and pointing in a fourth direction. The method may include generating a 3D visualization using the first, second, third, and fourth plurality of images and based on the depth signal, wherein generating the 3D visualization does not use any other images.

[0014] In some embodiments, the method may further include blending frames of the first plurality of images before the image sensor generates the second plurality of images. The first, second, third, and fourth plurality of images may be generated between rotations, the rotations combining a rotation that causes the image capturing device to rotate 270 degrees about the vertical axis. In some embodiments, the speed or rotation of the reflector about the horizontal axis increases as the first motor rotates the image capturing device. Attached Figure Description

[0015] Figure 1A A dollhouse view depicting an example environment (such as a house) according to some embodiments is shown.

[0016] Figure 1B A floor plan view of the first floor of a house according to some embodiments is depicted.

[0017] Figure 2 An example eye-level view of a living room is depicted, which could be part of a virtual walkthrough.

[0018] Figure 3 An example of an environment capture system according to some embodiments is described.

[0019] Figure 4 A perspective view of an environment capture system in some embodiments is depicted.

[0020] Figure 5 This is a depiction of laser pulses from LiDAR around an environmental capture system in some embodiments.

[0021] Figure 6AA side view of the environment capture system is depicted.

[0022] Figure 6B A view from above the environment capture system is depicted in some embodiments.

[0023] Figure 7 A perspective view depicting components of an example of an environment capture system according to some embodiments is provided.

[0024] Figure 8A Example lens sizes are depicted in some embodiments.

[0025] Figure 8B Example lens design specifications are depicted in some embodiments.

[0026] Figure 9a A block diagram depicts an example of an environment capture system according to some embodiments.

[0027] Figure 9b A block diagram of an example SOM PCBA of an environment capture system according to some embodiments is depicted.

[0028] Figures 10a-10c The process of an environmental capture system for taking images is described in some embodiments.

[0029] Figure 11 A block diagram depicts an example environment, according to some embodiments, capable of capturing and stitching images to form a 3D visualization.

[0030] Figure 12 This is a block diagram of an example alignment and stitching system according to some embodiments.

[0031] Figure 13 A flowchart is depicted for the 3D panoramic image capture and generation process according to some embodiments.

[0032] Figure 14 A flowchart depicts a 3D and panoramic capture and stitching process according to some embodiments.

[0033] Figure 15 Depicting Figure 14 A flowchart providing further details of one step in the 3D and panoramic capture and stitching process.

[0034] Figure 16 A block diagram of an example digital device according to some embodiments is depicted. Detailed Implementation

[0035] Many of the innovations described herein are made with reference to the accompanying drawings. The same reference numerals are used to denote the same elements. In the following description, numerous specific details are set forth for illustrative purposes to provide a thorough understanding. However, it will be apparent that different innovations can be practiced without these specific details. In other instances, well-known structures and components are shown in block diagram form to facilitate the description of innovations.

[0036] Various embodiments of the device provide users with 3D panoramic images of indoor and outdoor environments. In some embodiments, the device can efficiently and quickly provide users with 3D panoramic images of indoor and outdoor environments using a single wide field of view (FOV) lens and a single light detection and ranging sensor (LiDAR sensor).

[0037] The following are example use cases of the example device described herein. The following use case is one of the embodiments. Different embodiments of the device discussed herein may include one or more features and capabilities similar to those of this use case.

[0038] Figure 1A A dollhouse view 100 depicts an example environment (e.g., a house) according to some embodiments. The dollhouse view 100 provides an overall view of the example environment captured by an environment capture system (discussed herein). A user can interact with the dollhouse view 100 on a user system by switching between different views of the example environment. For example, a user can interact with area 110 to trigger a floor plan view of the first floor of the house, such as... Figure 1B As shown. In some embodiments, a user can interact with icons (such as icons 120, 130, and 140) in the dollhouse view 100 to provide a walkthrough view (e.g., for 3D walkthrough), a floor plan view, or a measurement view, respectively.

[0039] Figure 1B A floor plan view of the first floor of a house according to some embodiments is depicted. The floor plan view is a top view of the first floor of the house. Users can interact with areas of the floor plan view (such as area 150) to trigger an eye-level view of a specific portion of the floor plan (such as the living room). An example of an eye-level view of the living room can be found in... Figure 2 It can be found within a virtual tour.

[0040] Users can interact with the corresponding Figure 1BThe user can interact with a portion of the floor plan 200, specifically area 150. The view can be moved throughout the room as if the user were actually in the living room. In addition to a 360° horizontal view of the living room, the user can also view or navigate the floor or ceiling. Furthermore, the user can traverse the living room to reach other parts of the house by interacting with specific areas (such as areas 210 and 220) of this portion of floor plan 200. When the user interacts with area 220, the environment capture system can provide a walkable transition between areas of the house that substantially correspond to blocks of the house depicted by area 150 and areas of the house that substantially correspond to blocks of the house depicted by area 220.

[0041] Figure 3 An example of an environmental capture system 300 according to some embodiments is depicted. The environmental capture system 300 includes a lens 310, a housing 320, a mounting attachment 330, and a movable cover 340.

[0042] When in use, the environment capture system 300 can be positioned in an environment such as a room. The environment capture system 300 can be positioned on a support (e.g., a tripod). A movable cover 340 can be moved to expose the LiDAR and the rotatable reflector. Once activated, the environment capture system 300 can take a burst of images and then rotate using a motor. The environment capture system 300 can open the mounting attachment 330. While rotating, the LiDAR can perform measurements (the environment capture system may not take images during rotation). Once pointed in a new direction, the environment capture system can take another burst of images before rotating to the next direction.

[0043] For example, once located, the user can command the Environment Capture System 300 to begin scanning. The scan can proceed as follows:

[0044] (1) Exposure estimation, and then capturing HDR RGB images.

[0045] Rotate 90 degrees to capture depth data

[0046] (2) Exposure estimation, and then capturing HDR RGB images.

[0047] Rotate 90 degrees to capture depth data

[0048] (3) Exposure estimation, and then capturing HDR RGB images.

[0049] Rotate 90 degrees to capture depth data

[0050] (4) Exposure estimation, and then capturing HDR RGB images.

[0051] Rotate 90 degrees (360 degrees in total) to capture depth data

[0052] For each burst shot, there can be any number of images at different exposures. The ambient capture system can blend any number of images from the burst shot while waiting for another frame and / or the next burst shot.

[0053] The housing 320 protects the electronic components of the environmental capture system 300 and provides an interface for user interaction with power buttons, scan buttons, etc. For example, the housing 320 may include a removable cover 340, which may be movable to expose the LiDAR. Furthermore, the housing 320 may include electronic interfaces such as a power adapter and indicator lights. In some embodiments, the housing 320 is a molded plastic housing. In various embodiments, the housing 320 is one or more combinations of plastics, metals, and polymers.

[0054] Lens 310 may be part of a lens assembly. Further details of the lens assembly can be found in... Figure 7 The description is as follows. Lens 310 is strategically placed at the center of the rotation axis 305 of the ambient capture system 300. In this example, the rotation axis 305 is in the xy plane. By placing lens 310 at the center of the rotation axis 305, parallax effects can be eliminated or reduced. Parallax is an error caused by the rotation of the image capture device around a point that is not a non-parallax point (NPP). In this example, the NPP can be found at the center of the lens's entrance pupil.

[0055] For example, suppose a panoramic image of the physical environment is generated based on four images captured by the environment capture system 300, where the images in the panoramic image have a 25% overlap. If parallax is not present, then 25% of one image can completely overlap with another image of the same area of ​​the physical environment. Eliminating or reducing the parallax effect of multiple images captured by the image sensor through the lens 310 can help stitch multiple images into a 2D panoramic image.

[0056] Lens 310 may include a large field of view (e.g., lens 310 may be a fisheye lens). In some embodiments, the lens may have a horizontal field of view (HFOV) of at least 148 degrees and a vertical field of view (VFOV) of at least 94 degrees.

[0057] Mounting attachment 330 allows the environmental capture system 300 to be attached to a mounting. The mounting allows the environmental capture system 300 to be coupled to a tripod, a flat surface, or a motorized mounting (e.g., to move the environmental capture system 300). In some embodiments, the mounting allows the environmental capture system 300 to rotate along a horizontal axis.

[0058] In some embodiments, the environmental capture system 300 may include a motor for horizontally rotating the environmental capture system 300 about the mounting attachment 330.

[0059] In some embodiments, the motorized mount can move the environment capture system 300 along a horizontal axis, a vertical axis, or both. In some embodiments, the motorized mount can rotate or move in the xy plane. The use of the mounting attachment 330 can allow the environment capture system 300 to be coupled to the motorized mount, tripod, etc., to stabilize the environment capture system 300 to reduce or minimize swaying. In another example, the mounting attachment 330 can be coupled to the motorized mount, which allows the 3D and environment capture system 300 to rotate at a stable, known speed, which helps the LiDAR determine the (x, y, z) coordinates of each laser pulse of the LiDAR.

[0060] Figure 4 Renderings of an environment capture system 400 in some embodiments are depicted. The renderings show the environment capture system 400 (which may be...) from various views (such as front view 410, top view 420, side view 430, and rear view 440). Figure 3 (Example of environment capture system 300). In these renderings, environment capture system 400 may include an optional hollow portion depicted in side view 430.

[0061] In some embodiments, the environmental capture system 400 has a width of 75 mm, a height of 180 mm, and a depth of 189 mm. It should be understood that the environmental capture system 400 can have any width, height, or depth. In various embodiments, regardless of the specific measurements, the width-to-height-to-depth ratios of the first example are maintained.

[0062] The housing of the 3D and environmental capture system 400 protects the electronic components of the environmental capture system 400 and provides an interface / interface for user interaction (e.g., a screen on the rear view 440). Additionally, the housing may include electronic interfaces such as a power adapter and indicator lights. In some embodiments, the housing is a molded plastic housing. In various embodiments, the housing is a combination of one or more of plastics, metals, and polymers. The environmental capture system 400 may include a removable cover that is movable to expose the LiDAR and protect the LiDAR from components when not in use.

[0063] The lens depicted in front view 410 may be part of a lens assembly. Similar to the ambient capture system 300, the lens of the ambient capture system 400 is strategically positioned at the center of the rotation axis. The lens may include a large field of view. In various embodiments, the lens depicted in front view 410 is recessed, and the housing is flared, such that the wide-angle lens is directly above a parallax-free point (e.g., directly above the midpoint of the mount and / or motor), but can still capture images without interference from the housing.

[0064] A mounting attachment at the base of the environmental capture system 400 allows the environmental capture system to be attached to a mounting. The mounting allows the environmental capture system 400 to be coupled to a tripod, a flat surface, or a motorized mounting (e.g., to move the environmental capture system 400). In some embodiments, the mounting may be coupled to an internal motor for rotating the environmental capture system 400 about the mounting.

[0065] In some embodiments, the mount may allow the environment capture system 400 to rotate along a horizontal axis. In various embodiments, the motorized mount may move the environment capture system 400 along a horizontal axis, a vertical axis, or both. The use of mount attachments may allow the environment capture system 400 to be coupled to a motorized mount, tripod, etc., to stabilize the environment capture system 400 to reduce or minimize swaying. In another example, the mount attachment may be coupled to a motorized mount that allows the environment capture system 400 to rotate at a stable, known speed, which helps the LiDAR determine the (x, y, z) coordinates of each laser pulse of the LiDAR.

[0066] In view 430, a reflector 450 is exposed. The LiDAR can emit laser pulses (in the direction opposite to the lens view) towards the reflector. The laser pulses can strike the reflector 450, which can be angled (e.g., at a 90-degree angle). The reflector 450 can be coupled to an internal motor that rotates the reflector, allowing the LiDAR's laser pulses to be emitted and / or received around the environment capture system 400 at many different angles.

[0067] Figure 5 This is a depiction of laser pulses from a LiDAR around an environment capture system 400 in some embodiments. In this example, the laser pulses are emitted at a rotating reflector 450. The laser pulses can be emitted and received perpendicular to the horizontal axis 602 of the environment capture system 400 (see Figure 6). The reflector 450 can be angled such that the laser pulses from the LiDAR are directed away from the environment capture system 400. In some examples, the angle of the angled surface of the reflector can be 90 degrees, or between 60 and 120 degrees.

[0068] In some embodiments, when the ambient capture system 400 is stationary and in operation, it can capture images in bursts through a lens. The ambient capture system 400 can activate a horizontal motor between images in a burst. As it rotates along the mounting, the LiDAR of the ambient capture system 400 can emit and / or receive laser pulses that impact the rotating mirror 450. The LiDAR can reflect the received laser pulses to generate depth signals and / or depth data.

[0069] In some embodiments, depth data may be associated with coordinates relative to the environment capture system 400. Similarly, pixels or portions of an image may be associated with coordinates relative to the environment capture system 400, so that the generation of 3D visualizations (e.g., images from different directions, 3D walkthroughs, etc.) can be generated using image and depth data.

[0070] like Figure 5 As shown, the LiDAR pulse can be blocked by the bottom of the ambient capture system 400. It should be understood that the reflector 450 can rotate uniformly as the ambient capture system 400 moves around the mount, or the reflector 450 can rotate more slowly when the ambient capture system 400 starts moving and, in addition, when the ambient capture system 400 slows down and stops (e.g., maintaining a constant speed between the start and stop of the mount motor).

[0071] LiDAR can receive depth data from pulses. Due to the movement of the environment capture system 400 and / or the increase or decrease in the speed of the mirror 450, the density of the depth data of the environment capture system 400 may be inconsistent (e.g., denser in some areas and less dense in others).

[0072] Figure 6A A side view of the environment capture system 400 is depicted. In this view, a reflector 450 is depicted, and the reflector 450 is rotatable about a horizontal axis. A pulse 604 can be emitted by a LiDAR at the rotating reflector 450 and can be emitted perpendicular to the horizontal axis 602. Similarly, the pulse 604 can be received by a LiDAR in a similar manner.

[0073] Although the LiDAR pulse is discussed as being perpendicular to the horizontal axis 602, it should be understood that the LiDAR pulse can be at any angle relative to the horizontal axis 602 (e.g., the mirror angle can be any angle including 60 degrees to 120 degrees). In various embodiments, the LiDAR emits a pulse opposite to the front side (e.g., front side 604) of the environment capture system 400 (e.g., in a direction opposite to the center of the lens's field of view or toward the rear side 606).

[0074] As discussed herein, the environment capture system 400 can rotate about a vertical axis 608. In various embodiments, the environment capture system 400 captures images and then rotates 90 degrees, so that a fourth set of images is captured when the environment capture system 400 has completed a 270-degree rotation from the original starting position from which the first set of images was captured. Thus, the environment capture system 400 can generate four sets of images over a total of 270 degrees of rotation (e.g., assuming the first set of images was captured before the initial rotation of the environment capture system 400). In various embodiments, images from a single sweep of the environment capture system 400 (e.g., four sets of images) (e.g., captured in a single full rotation or 270-degree rotation about the vertical axis) together with depth data acquired during the same sweep are sufficient to generate a 3D visualization without any additional sweeps or rotations of the environment capture system 400.

[0075] It should be understood that in this example, the LiDAR pulse is emitted and guided by a rotating mirror at a position away from the rotation point of the ambient capture system 400. In this example, the distance from the rotation point of the mount is 608 (e.g., the lens may be at a parallax-free point, and the lens may be positioned behind the lens relative to the front of the ambient capture system 400). Because the LiDAR pulse is guided by a mirror 450 at a position away from the rotation point, the LiDAR cannot receive depth data from a cylinder extending from above to below the ambient capture system 400. In this example, the radius of the cylinder (e.g., a cylinder without depth information) can be measured from the center of the rotation point of the motor mount to the point where the mirror 450 guides the LiDAR pulse.

[0076] In addition, Figure 6B Cavity 610 is depicted. In this example, the environment capture system 400 includes a rotating mirror within the body of a housing of the environment capture system 400. A cutout section exists from the housing. Laser pulses can be reflected out of the housing by the mirror, and the reflection can then be received by the mirror and guided back to the LiDAR, enabling the LiDAR to generate depth signals and / or depth data. The base of the body of the environment capture system 400 below cavity 610 can block some laser pulses. Cavity 610 can be defined by the base of the environment capture system 400 and the rotating mirror. Figure 6B As shown, space can still exist between the edge of the angled reflector and the housing of the environment capture system 400 containing LiDAR.

[0077] In various embodiments, the LiDAR is configured to stop emitting laser pulses if the rotational speed of the mirror drops below a rotational safety threshold (e.g., if there is a malfunction of the motor that rotates the mirror or if the mirror is held in place). In this way, the LiDAR can be configured for safety and to reduce the likelihood that laser pulses will continue to be emitted in the same direction (e.g., at the user's eye).

[0078] Figure 6B A view from above the ambient capture system 400 is depicted in some embodiments. In this example, the front of the ambient capture system 400 is depicted, with the lens recessed and directly above the center of rotation (e.g., above the center of the mount). The front of the camera is recessed for the lens, and the front of the housing is flared to allow the field of view of the image sensor to be unobstructed by the housing. The reflector 450 is depicted pointing upwards.

[0079] Figure 7 A perspective view of components of an example of an environmental capture system 700 according to some embodiments is depicted. The environmental capture system 700 includes a front cover 702, a lens assembly 704, a structural frame 706, a LiDAR 708, a front housing 710, a reflector assembly 712, a GPS antenna 714, a rear housing 716, a vertical motor 718, a display 720, a battery pack 722, a mounting bracket 724, and a horizontal motor 726.

[0080] In various embodiments, the environmental capture system 700 can be configured to scan, align, and generate 3D meshes both outdoors and indoors under full sunlight. This eliminates the barriers imposed by other systems that are only suitable for indoor use. The environmental capture system 700 can be able to scan large spaces faster than other devices. In some embodiments, the environmental capture system 700 can provide improved depth accuracy by improving single-scan depth accuracy at 90m.

[0081] In some embodiments, the environmental capture system 700 may weigh 1 kg or about 1 kg. In one example, the environmental capture system 700 may weigh 1-3 kg.

[0082] The front cover 702, the front housing 710, and the rear housing 716 constitute part of the housing. In one example, the front cover may have a width w of 75 mm.

[0083] Lens assembly 704 may include a camera lens that focuses light onto an image capturing device. The image capturing device can capture images of the physical environment. A user can position the environment capturing system 700 to capture a portion of a floor of a building (such as the second building 422 in Figure 1) to obtain a panoramic image of that portion of the floor. The environment capturing system 700 can be moved to another portion of a floor of the building to obtain a panoramic image of that other portion of the floor. In one example, the depth of field of the image capturing device ranges from 0.5 meters to infinity. Figure 8A Example lens sizes are depicted in some embodiments.

[0084] In some embodiments, the image capturing device is a complementary metal-oxide-semiconductor (CMOS) image sensor (e.g., a Sony IMX283-20 ​​Megapixel CMOS MIPI sensor with an Nvidia Jetson Nano SOM). In various embodiments, the image capturing device is a charge-coupled device (CCD). In one example, the image capturing device is a red-green-blue (RGB) sensor. In one embodiment, the image capturing device is an infrared (IR) sensor. The lens assembly 704 can provide the image capturing device with a wide field of view.

[0085] Image sensors can have many different specifications. In one example, an image sensor includes the following:

[0086] Pixels per column Pixels 5496 Pixels per row Pixels 3694 resolution MP ＞20 Image circle diameter mm 15.86mm Pixel pitch um 2.4um Pixels per degree (PPD) PPD ＞37 Chief ray angle at full altitude Spend 3.0° Output interface - MIPI Green Sensitivity V / lux*s ＞1.7 SNR(100 lux, 1x gain) dB ＞65 Dynamic range dB ＞70

[0087] Example specifications can be as follows:

[0088] F-number 2.8 Image circle diameter mm 15.86 Minimum object distance mm 500 Maximum object distance mm Infinity The angle of the principal ray at the full height of the sensor Spend 3.0 L1 diameter <![CDATA[ mm ]]> ＜60 Total Lens Length (TTL) mm ＜＝80 Back focal length (BFL) mm Effective focal length (EFL) mm Relative Illuminance ％ ＞50 Maximum distortion ％ ＜5 52 lp / mm (on the shaft) ％ ＞85 104 lp / mm (on the shaft) ％ ＞66 208 lp / mm (on the shaft) ％ ＞45 52 lp / mm (83% field) ％ ＞75 1041p / mm (83% field) ％ ＞41 208 lp / mm (83% field) ％ ＞25

[0089] In various embodiments, when observing the MTF at the F0 relative field (i.e., the center), for a total through focus shift of 53 micrometers, the focus shift can vary from +28 micrometers at 0.5m to -25 micrometers at infinity.

[0090] Figure 8B Example lens design specifications are depicted in some embodiments.

[0091] In some examples, lens assembly 704 has an HFOV of at least 148 degrees and a VFOV of at least 94 degrees. In one example, lens assembly 704 has a field of view of 150°, 180°, or within the range of 145° to 180°. In one example, an image capture of a 360° view around environment capture system 700 can be obtained using three or four separate image captures from image capture devices of environment capture system 700. In various embodiments, the image capture device may have a resolution of at least 37 pixels per degree. In some embodiments, environment capture system 700 includes a lens cap (not shown) to protect lens assembly 704 when not in use. The output of lens assembly 704 may be a digital image of a region of the physical environment. Images captured by lens assembly 704 can be stitched together to form a 2D panoramic image of the physical environment. A 3D panorama can be generated by combining depth data captured by LiDAR 708 with a 2D panoramic image generated by stitching together multiple images from lens assembly 704. In some embodiments, images captured by the environment capture system 402 are stitched together by the image processing system 406. In various embodiments, the environment capture system 402 generates a “preview” or “thumbnail” version of the 2D panoramic image. The preview or thumbnail version of the 2D panoramic image can be displayed on a user system 1110 such as an iPad, personal computer, smartphone, etc. In some embodiments, the environment capture system 402 can generate a mini-map of the physical environment representing an area of ​​the physical environment. In various embodiments, the image processing system 406 generates the mini-map representing that area of ​​the physical environment.

[0092] The image captured by lens assembly 704 may include capture device location data that identifies or indicates the capture location of the 2D image. For example, in some embodiments, the capture device location data may include Global Positioning System (GPS) coordinates associated with the 2D image. In other embodiments, the capture device location data may include location information indicating the relative position of the capture device (e.g., a camera and / or a 3D sensor) relative to its environment, such as the relative or calibrated position of the capture device relative to objects in the environment, another camera in the environment, another device in the environment, etc. In some embodiments, this type of location data may be determined by the capture device associated with the image capture (e.g., a camera and / or a device operatively coupled to the camera, including positioning hardware and / or software) and received along with the image. The placement of lens assembly 704 is not merely a matter of design. By placing lens assembly 704 at or substantially at the center of the rotation axis, parallax effects can be reduced.

[0093] In some embodiments, the structural frame 706 holds the lens assembly 704 and the LiDAR 708 in a specific position and helps protect components of the example environmental capture system. The structural frame 706 can be used to help rigidly mount the LiDAR 708 and place it in a fixed position. Furthermore, the fixed position of the lens assembly 704 and the LiDAR 708 allows the fixation relationship to align depth data with image information to help generate 3D images. 2D image data and depth data captured in the physical environment can be aligned relative to a common 3D coordinate space to generate a 3D model of the physical environment.

[0094] In various embodiments, the LiDAR 708 captures depth information of the physical environment. When a user places the environment capture system 700 within a floor of a second building, the LiDAR 708 can acquire depth information of objects. The LiDAR 708 may include an optical sensing module capable of measuring the distance to a target or object in the scene by illuminating the target or scene with pulses from a laser, and measuring the time it takes for photons to travel to the target and return to the LiDAR 708. The measurements can then be transformed into a grid coordinate system using information derived from the horizontal drive system of the environment capture system 700.

[0095] In some embodiments, the LiDAR 708 can return depth data points with timestamps (according to its internal clock) every 10 microseconds. The LiDAR 708 can sample portions of the sphere (the small holes at the top and bottom) every 0.25 degrees. In some embodiments, with one data point every 10 microseconds and 0.25 degrees, each point "disk" can have 14.40 milliseconds and 1440 disks to form a nominal 20.7-second sphere. Because each disk is captured before and after, the sphere can be captured by scanning 180°.

[0096] In one example, the LiDAR 708 specifications could be as follows:

[0097]

[0098]

[0099] One advantage of using LiDAR is that, in the case of LiDAR at lower wavelengths (e.g., 905nm, 900-940nm, etc.), it allows the environmental capture system 700 to determine depth information of outdoor or indoor environments with bright light.

[0100] The placement of the lens assembly 704 and the LiDAR 708 allows the environment capture system 700 or a digital device communicating with the environment capture system 700 to generate 3D panoramic images using depth data from the LiDAR 708 and the lens assembly 704. In some embodiments, 2D and 3D panoramic images are not generated on the environment capture system 402.

[0101] The output of the LiDAR 708 can include attributes associated with each laser pulse transmitted by the LiDAR 708. These attributes include the laser pulse intensity, number of returns, current return count, classification point, RGC value, GPS time, scan angle, scan direction, or any combination thereof. The depth of field can be (0.5m; infinity), (1m; infinity), etc. In some embodiments, the depth of field is 0.2m to 1m and infinity.

[0102] In some embodiments, while the environment capture system 700 is stationary, it captures four separate RBG images using the lens assembly 704. In various embodiments, while the environment capture system 700 is in motion, moving from one RBG image capture location to another, the LiDAR 708 captures depth data under four different conditions. In one example, a 360° rotation of the environment capture system 700 (which may be referred to as a sweep) is used to capture a 3D panoramic image. In various embodiments, a rotation of less than 360° is used to capture a 3D panoramic image. The output of the sweep may be a sweep list (SWL), which includes image data from the lens assembly 704 and depth data from the LiDAR 708, as well as the nature of the sweep, including GPS location and a timestamp when the sweep occurred. In various embodiments, a single sweep (e.g., a single 360-degree rotation of the environment capture system 700) captures enough image and depth information to generate a 3D visualization (e.g., via a digital device communicating with the environment capture system 700, which receives image and depth data from the environment capture system 700 and uses only the image and depth data from the environment capture system 700 captured in a single sweep to generate the 3D visualization).

[0103] In some embodiments, images captured by the environment capture system 402 can be blended, stitched together, and combined with depth data from LiDAR 708 using the image stitching and processing system discussed herein.

[0104] In various embodiments, applications on the environment capture system 402 and / or user system 1110 can generate preview or thumbnail versions of 3D panoramic images. These preview or thumbnail versions of the 3D panoramic images can be displayed on user system 1110 and may have a lower image resolution than the 3D panoramic image generated by image processing system 406. After the lens assembly 704 and LiDAR 708 capture image and depth data of the physical environment, the environment capture system 402 can generate a minimap representing an area of ​​the physical environment that has been captured by the environment capture system 402. In some embodiments, image processing system 406 generates the minimap representing that area of ​​the physical environment. After capturing image and depth data of a living room in a home using the environment capture system 402, the environment capture system 402 can generate a top-down view of the physical environment. Users can use this information to determine areas of the physical environment that have not yet been captured or for which 3D panoramic images have been generated.

[0105] In one embodiment, the environment capture system 700 can interleave image capture using an image capture device with lens assembly 704 with depth information capture using LiDAR 708. For example, the image capture device can capture a segment 1605 of the physical environment (such as...) Figure 16 The image capture device can move to capture an image of another segment 1610, and then the LiDAR 708 obtains depth information from segment 1605. Once the LiDAR 708 has obtained depth information from segment 1605, the image capture device can continue to move to capture an image of another segment 1610, and then the LiDAR 708 obtains depth information from segment 1610, thus interleaving image capture and depth information capture.

[0106] In some embodiments, the LiDAR 708 may have a field of view of at least 145°, and depth information of all objects in the 360° view of the environment capture system 700 may be obtained by the environment capture system 700 in three or four scans. In another example, the LiDAR 708 may have a field of view of at least 150°, 180°, or between 145° and 180°.

[0107] The increased field of view of the lens reduces the amount of time required to obtain visual and depth information about the physical environment surrounding the environment capture system 700. In various embodiments, the LiDAR 708 has a minimum depth range of 0.5 m. In one embodiment, the LiDAR 708 has a maximum depth range greater than 8 meters.

[0108] The LiDAR 708 can guide the laser at different scanning angles using the mirror assembly 712. In one embodiment, an optional vertical motor 718 has the capability to move the mirror assembly 712 vertically. In some embodiments, the mirror assembly 712 may be a dielectric mirror with a hydrophobic coating or layer. The mirror assembly 712 may be coupled to the vertical motor 718, which rotates the mirror assembly 712 during use.

[0109] The reflector of the reflector assembly 712 may include, for example, the following specifications:

[0110]

[0111] The reflector of the reflector assembly 712 may include, for example, the following specifications regarding materials and coatings:

[0112]

[0113] The hydrophobic coating of the mirror of the mirror assembly 712 may include, for example, a contact angle number >105.

[0114] The reflector of the reflector assembly 712 may include the following mass specifications:

[0115]

[0116]

[0117] Vertical motors may include, for example, the following specifications:

[0118]

[0119] Thanks to the RGB capture device and LiDAR 708, the Ambient Capture System 700 can capture images outdoors in bright sunlight or indoors in bright light or sunlight glare from windows. Systems utilizing different devices (e.g., structured light devices) may not be able to operate in bright environments, whether indoors or outdoors. These devices are typically limited to indoor use only and only during dawn or sunset to control light. Otherwise, bright spots in the room create artifacts or “holes” in the image that must be filled or corrected. However, the Ambient Capture System 700 can be used indoors and outdoors in bright sunlight. The capture device and LiDAR 708 are able to capture images and depth data in bright environments without artifacts or holes caused by glare or bright light.

[0120] In one embodiment, GPS antenna 714 receives Global Positioning System (GPS) data. The GPS data can be used to determine the location of the environmental capture system 700 at any given time.

[0121] In various embodiments, the display 720 allows the environment capture system 700 to provide the current state of the system, such as update, warm-up, scan, scan complete, error, etc.

[0122] Battery pack 722 provides power to environmental capture system 700. Battery pack 722 may be removable and rechargeable, allowing a user to insert a new battery pack 722 while charging a depleted battery pack. In some embodiments, battery pack 722 may allow continuous use of at least 1000 SWLs or at least 250 SWLs before recharging. Environmental capture system 700 may utilize a USB-C connector for recharging.

[0123] In some embodiments, mounting 724 is a connector providing the environment capture system 700 for connection to a platform such as a tripod or mounting bracket. A horizontal motor 726 allows the environment capture system 700 to rotate about the xy-plane. In some embodiments, the horizontal motor 726 can provide information to a grid coordinate system to determine the (x, y, z) coordinates associated with each laser pulse. In various embodiments, the horizontal motor 726 enables the environment capture system 700 to scan rapidly due to the wide field of view of the lens, the positioning of the lens about the axis of rotation, and the LiDAR device.

[0124] In one example, the horizontal motor 726 may have the following specifications:

[0125]

[0126] In various embodiments, mounting 724 may include a quick-release adapter. The holding torque may be, for example, >2.0 Nm, and the durability of the capture operation may be up to or exceed 70,000 cycles.

[0127] For example, the Environment Capture System 700 can construct a 3D mesh for a standard home, where the distance between sweeps is greater than 8 meters. The time to capture, process, and align indoor sweeps can be less than 45 seconds. In one example, the time frame from the start of sweep capture to when the user can move the Environment Capture System 700 can be less than 15 seconds.

[0128] In various embodiments, these components provide the environment capture system 700 with the ability to align scan locations both outdoors and indoors, and thus create a seamless roaming experience between indoors and outdoors (which can be a high priority for hotels, vacation rentals, real estate, architectural documentation, CRE, and as-built modeling and verification). The environment capture system 700 can also generate an “outdoor dollhouse” or outdoor mini-map. As shown herein, the environment capture system 700 can also improve the accuracy of 3D reconstruction primarily from a measurement perspective. The user’s ability to tune the scan density can also be a favorable factor. These components also enable the environment capture system 700 to capture wide blank spaces (e.g., longer ranges). To generate a 3D model with a wide blank space, the environment capture system may need to scan and capture 3D data and depth data from a larger distance range than generating a 3D model with a smaller space.

[0129] In various embodiments, these components enable the Environment Capture System 700 to align the SWL and reconstruct the 3D model in a similar manner for both indoor and outdoor use. These components also enable the Environment Capture System 700 to perform geolocation of the 3D model (which can be easily integrated into Google Street View and assist in aligning outdoor panoramas if needed).

[0130] The image capture device of the Ambient Capture System 700 is capable of providing DSLR-class images with printable quality at 8.5" x 11" for 70° VFOV and RGB image types.

[0131] In some embodiments, the environment capture system 700 may capture RGB images using an image capture device (e.g., using a wide-angle lens), and then move the lens (using a motor for a total of four movements) before capturing the next RGB image. While the horizontal motor 726 rotates the environment capture system 90 degrees, the LiDAR 708 may capture depth data. In some embodiments, the LiDAR 708 includes an APD array.

[0132] In some embodiments, the image and depth data can then be sent to a capture application (e.g., a device communicating with the environment capture system 700, such as a smart device or an image capture system on a network). In some embodiments, the environment capture system 700 can send the image and depth data to an image processing system 406 for processing and generating a 2D or 3D panoramic image. In various embodiments, the environment capture system 700 can generate a sweep list of captured RGB images and depth data from a 360-degree rotation of the environment capture system 700. The sweep list can be sent to the image processing system 406 for stitching and alignment. The output of the sweep can be a sweep list (SWL), which includes image data from the lens assembly 704 and depth data from the LiDAR 708, as well as the nature of the sweep, including GPS location and a timestamp when the sweep occurred.

[0133] In various embodiments, the LiDAR, vertical mirror, RGB lens, tripod mount, and horizontal driver are rigidly mounted within the housing to allow the housing to be opened without requiring system recalibration.

[0134] Figure 9a Block diagram 900 depicts an example of an environmental capture system according to some embodiments. Block diagram 900 includes a power supply 902, a power converter 904, an input / output (I / O) printed circuit board assembly (PCBA), a system-on-module (SOM) PCBA, a user interface 910, a LiDAR 912, a mirror brushless DC (BLCD) motor 914, a drivetrain 916, a wide field of view (WFOV) lens 918, and an image sensor 920.

[0135] Power supply 902 can be Figure 7 The battery pack 722. The power source can be a removable, rechargeable battery, such as a lithium-ion battery (e.g., 4 x 18650 lithium-ion batteries), capable of providing power to the environmental capture system.

[0136] The power converter 904 can change the voltage level from the power supply 902 to a lower or higher voltage level, making it usable by the electronic components of the environmental capture system. The environmental capture system can utilize four 18650 lithium-ion batteries in a 4S1P configuration or a configuration of four in series and one in parallel.

[0137] In some embodiments, the I / O PCBA 906 may include elements providing an IMU, Wi-Fi, GPS, Bluetooth, an inertial measurement unit (IMU), a motor driver, and a microcontroller. In some embodiments, the I / O PCBA 906 includes a microcontroller for controlling a horizontal motor and encoding horizontal motor controls, and for controlling a vertical motor and encoding vertical motor controls.

[0138] The SOM PCBA908 may include a central processing unit (CPU) and / or a graphics processing unit (GPU), memory, and a mobile interface. The SOM PCBA908 can control the LiDAR 912, the image sensor 920, and the I / O PCBA906. The SOM PCBA908 can determine the (x, y, z) coordinates associated with each laser pulse from the LiDAR 912 and store the coordinates in its memory unit. In some embodiments, the SOM PCBA908 may store the coordinates in the image processing system of the environment capture system 400. In addition to the coordinates associated with each laser pulse, the SOM PCBA908 can also determine other attributes associated with each laser pulse, including the laser pulse intensity, number of returns, current number of returns, classification point, RGC value, GPS time, scan angle, and scan direction.

[0139] In some embodiments, the SOM PCBA908 includes Nvidia SOM PCBAw / CPU / GPU, DDR, eMMC, and Ethernet.

[0140] User interface 910 may include physical buttons or switches that a user can interact with. The buttons or switches may provide functions such as turning the environmental capture system on and off, scanning the physical environment, etc. In some embodiments, user interface 910 may include a display, such as... Figure 7 The monitor is 720.

[0141] In some embodiments, the LiDAR 912 captures depth information of the physical environment. The LiDAR 912 includes an optical sensing module capable of measuring the distance to a target or object in a scene by illuminating it with light pulses from a laser. The LiDAR 912's optical sensing module measures the time it takes for photons to travel to the target or object and return to the receiver in the LiDAR 912 after reflection, thus providing the distance of the LiDAR to the target or object. Along with the distance, the SOM PCBA908 can determine the (x, y, z) coordinates associated with each laser pulse. The LiDAR 912 can be mounted within a width of 58 mm, a height of 55 mm, and a depth of 60 mm.

[0142] The LiDAR 912 can include a range of 90m (10% reflectivity), a range of 130m (20% reflectivity), a range of 260m (100% reflectivity), a range accuracy of 2cm (1σ@900m), a wavelength of 1705nm, and a beam divergence of 0.28 × 0.03 degrees.

[0143] The SOM PCBA908 can determine coordinates based on the position of the drivetrain 916. In various embodiments, the LiDAR 912 may include one or more LiDAR devices. Multiple LiDAR devices can be used to increase LiDAR resolution.

[0144] The reflector brushless DC (BLCD) motor 914 can control Figure 7 The reflector assembly 712.

[0145] In some embodiments, the drivetrain 916 may include Figure 7 A horizontal motor 726 is used. When the environmental capture system is mounted on a platform such as a tripod, a drivetrain 916 can provide rotation for the environmental capture system. The drivetrain 916 may include a stepper motor Nema 14, a worm gear and plastic wheel drivetrain, a clutch, bushing bearings, and a backlash prevention mechanism. In some embodiments, the environmental capture system may be able to complete a scan in less than 17 seconds. In various embodiments, the drivetrain 916 has a maximum speed of 60 degrees / second and a maximum speed of 300 degrees / second. 2 Maximum acceleration, maximum torque of 0.5nm, angular position accuracy of less than 0.1 degrees, and encoder resolution of approximately 4096 counts per revolution.

[0146] In some embodiments, the drivetrain 916 includes a vertical monogon mirror and a motor. In this example, the drivetrain 916 may include a BLDC motor, an external Hall effect sensor, a magnet (paired with the Hall effect sensor), a mirror holder, and a mirror. In this example, the drivetrain 916 may have a maximum speed of 4,000 rpm and a maximum acceleration of 300 degrees / second². In some embodiments, the monogon mirror is a dielectric mirror. In one embodiment, the monogon mirror includes a hydrophobic coating or layer.

[0147] The components of the environment capture system are positioned such that the lens assembly and LiDAR are essentially centered on the axis of rotation. This reduces image parallax that occurs when the image capture system is not centered on the axis of rotation.

[0148] In some embodiments, the WFOV lens 918 can be Figure 7The lens assembly 704 contains a lens. The WFOV lens 918 focuses light onto the image capturing device. In some embodiments, the WFOV lens may have a FOV of at least 145 degrees. With such a wide FOV, three separate image captures of the image capturing device can be used to obtain an image capture of a 360-degree view around the environment capturing system. In some embodiments, the WFOV lens 918 may have a diameter of approximately ~60 mm and a total lens length (TTL) of approximately 80 mm. In one example, the WFOV lens 918 may include a horizontal field of view greater than or equal to 148.3 degrees and a vertical field of view greater than or equal to 94 degrees.

[0149] The image capture device may include a WFOV lens 918 and an image sensor 920. The image sensor 920 may be a CMOS image sensor. In one embodiment, the image sensor 920 is a charge-coupled device (CCD). In some embodiments, the image sensor 920 is a red-green-blue (RGB) sensor. In one embodiment, the image sensor 920 is an IR sensor. In various embodiments, the image capture device may have a resolution of at least 35 pixels per degree (PPD).

[0150] In some embodiments, the image capturing device may include an F-number of f / 2.4, an image circle diameter of 15.86 mm, a pixel pitch of 2.4 μm, an HFOV of >148.3°, a VFOV of >94.0°, a pixel density of >38.0 PPD, a principal ray angle of 3.0° at full height, a minimum object distance of 1300 mm, a maximum object distance of infinity, a relative illuminance of >130%, a maximum distortion of <90%, and a spectral transmittance variation of <=5%.

[0151] In some embodiments, the lens may include an F-number of 2.8, an image circle diameter of 15.86 mm, a pixel density of >37, a principal ray angle of 3.0 at full height, an L1 diameter of <60 mm, a TTL of <80 mm, and a relative illumination of >50%.

[0152] Lenses may include >85% of 52 lp / mm (on-axis), >66% of 104 lp / mm (on-axis), >45% of 1308 lp / mm (on-axis), >75% of 52 lp / mm (83% field), >41% of 104 lp / mm (83% field), and >25% of 1308 lp / mm (83% field).

[0153] The environmental capture system can have a resolution of >20MP, a green sensitivity of >1.7V / lux*s, an SNR of >65dB (100lux, 1x gain), and a dynamic range of >70dB.

[0154] Figure 9bA block diagram of an example SOM PCBA 908 of an environmental capture system according to some embodiments is depicted. The SOM PCBA 908 may include a communication component 922, a LiDAR control component 924, a LiDAR positioning component 926, a user interface component 928, a classification component 930, a LiDAR data storage area 932, and a captured image data storage area 934.

[0155] In some embodiments, the communication component 922 can be located in any component of the SOM PCBA 1008 and Figure 9a The components of the environment capture system send and receive requests or data.

[0156] In various embodiments, the LiDAR control unit 924 can control various aspects of the LiDAR. For example, the LiDAR control unit 924 can send control signals to the LiDAR 912 to begin emitting laser pulses. The control signals sent by the LiDAR control unit 924 may include instructions regarding the frequency of the laser pulses.

[0157] In some embodiments, the LiDAR positioning component 926 may utilize GPS data to determine the location of the environmental capture system. In various embodiments, the LiDAR positioning component 926 utilizes the position of the mirror assembly to determine the scan angle and (x, y, z) coordinates associated with each laser pulse. The LiDAR positioning component 926 may also utilize an IMU to determine the orientation of the environmental capture system.

[0158] User interface component 928 can facilitate user interaction with the environment capture system. In some embodiments, user interface component 928 can provide one or more user interface elements with which a user can interact. The user interface provided by user interface component 928 can be sent to user system 1110. For example, user interface component 928 can provide a visual representation of an area of ​​a building's floor plan to the user system (e.g., a digital device). The environment capture system can generate a visual representation of the floor plan when the user places the environment capture system in different parts of the building's floors to capture and generate 3D panoramic images. The user can place the environment capture system in an area of ​​the physical environment to capture and generate a 3D panoramic image of that area of ​​the building. Once the 3D panoramic image of that area has been generated by the image processing system, the user interface component can utilize... Figure 1B The floor plan view is updated using a top view of the living room area depicted in the diagram. In some embodiments, the floor plan view 200 may be generated by the user system 1110 after a second scan of the same family or floor of the building has been captured.

[0159] In various embodiments, the classification component 930 can classify the type of physical environment. The classification component 930 can analyze objects in the image or objects within an image to classify the type of physical environment captured by the environment capture system. In some embodiments, the image processing system can be responsible for classifying the type of physical environment captured by the environment capture system 400.

[0160] LiDAR data storage area 932 can be any and / or multiple structures suitable for captured LiDAR data (e.g., active database, relational database, self-referenced database, table, matrix, array, flat file, document-oriented storage system, non-relational No-SQL system, FTS management system such as Lucene / Solar, and / or similar structures). Image data storage area 408 can store captured LiDAR data. However, in the absence of communication network 404, LiDAR data storage area 932 can be used to cache captured LiDAR data. For example, if environmental capture system 402 and user system 1110 are in remote locations without cellular networks or in areas without Wi-Fi, LiDAR data storage area 932 can store captured LiDAR data until it can be transmitted to image data storage area 934.

[0161] Similar to the LiDAR data storage area, the captured image data storage area 934 can be any and / or multiple structures suitable for captured images (e.g., active database, relational database, self-referential database, table, matrix, array, flat file, document-oriented storage system, non-relational No-SQL system, FTS management system such as Lucene / Solar, and / or similar structures). The image data storage area 934 can store captured images.

[0162] Figures 10a-10c The process of an environmental capture system 400 for capturing images in some embodiments is described. For example... Figures 10a-10c As shown, the environment capture system 400 can capture images in burst mode at different exposures. The burst mode can be a set of images, each with a different exposure. The first image burst occurs at time 0.0. The environment capture system 400 can receive the first frame and then evaluate the first frame while waiting for the second frame. Figure 10a The instruction is to blend the first frame before the second frame arrives. In some embodiments, the ambient capture system 400 may process each frame to identify pixels, colors, etc. Once the next frame arrives, the ambient capture system 400 may process the most recently received frame and then blend the two frames together.

[0163] In various embodiments, the ambient capture system 400 performs image processing to blend a sixth frame and further evaluate the pixels in the blended frame (e.g., a frame that may include elements from any number of frames taken in a burst of images). Optionally, the ambient capture system 400 may transfer the blended image from the graphics processing unit to CPU memory before or during the final step of movement (e.g., rotation) of the ambient capture system 400.

[0164] This process is in Figure 10b Continued in... Figure 10b At the start, the environment capture system 400 takes another burst of shots. The environment capture system 400 can use JxR to compress all or part of the mixed frames and / or captured frames. Similar to... Figure 10a A burst of images can be a series of images, each with a different exposure (the exposure length of each frame in the series can be the same as...). Figure 10a and 10c The other burst shots covered in the same sequence are identical. The second burst shot occurs at 2 seconds. The ambient capture system 400 can receive the first frame and then evaluate it while waiting for the second frame. Figure 10b The instruction is to blend the first frame before the second frame arrives. In some embodiments, the ambient capture system 400 may process each frame to identify pixels, colors, etc. Once the next frame arrives, the ambient capture system 400 may process the most recently received frame and then blend the two frames together.

[0165] In various embodiments, the ambient capture system 400 performs image processing to blend a sixth frame and further evaluate the pixels in the blended frame (e.g., a frame that may include elements from any number of frames taken in a burst of images). Optionally, the ambient capture system 400 may transfer the blended image from the graphics processing unit to CPU memory before or during the final step of movement (e.g., rotation) of the ambient capture system 400.

[0166] After the rotation, the ambient capture system 400 can continue the process by taking another color burst shot at approximately 3.5 seconds (e.g., after rotating 180 degrees). The ambient capture system 400 can use JxR to compress all or part of the mixed frames and / or captured frames. The burst of images can be a set of images, each with a different exposure (the exposure length of each frame in the set can be...). Figure 10a and 10c (The other burst shots covered are the same and in the same order). The environment capture system 400 can receive the first frame and then evaluate that first frame while waiting for the second frame. Figure 10bThe instruction is to blend the first frame before the second frame arrives. In some embodiments, the ambient capture system 400 may process each frame to identify pixels, colors, etc. Once the next frame arrives, the ambient capture system 400 may process the most recently received frame and then blend the two frames together.

[0167] In various embodiments, the ambient capture system 400 performs image processing to blend a sixth frame and further evaluate the pixels in the blended frame (e.g., a frame that may include elements from any number of frames taken in a burst of images). Optionally, the ambient capture system 400 may transfer the blended image from the graphics processing unit to CPU memory before or during the final step of movement (e.g., rotation) of the ambient capture system 400.

[0168] exist Figure 10c In the sequence, the last shot occurs at 5 seconds. The Ambient Capture System 400 can use JxR to compress and blend frames and / or capture all or part of the frames. A burst of images can be a set of images, each with a different exposure (the exposure length of each frame in the set can be...). Figure 10a and 10b (The other burst shots covered are the same and in the same order). The environment capture system 400 can receive the first frame and then evaluate that first frame while waiting for the second frame. Figure 10c The instruction is to blend the first frame before the second frame arrives. In some embodiments, the ambient capture system 400 may process each frame to identify pixels, colors, etc. Once the next frame arrives, the ambient capture system 400 may process the most recently received frame and then blend the two frames together.

[0169] In various embodiments, the ambient capture system 400 performs image processing to blend a sixth frame and further evaluate the pixels in the blended frame (e.g., a frame that may include elements from any number of frames taken in a burst of images). Optionally, the ambient capture system 400 may transfer the blended image from the graphics processing unit to CPU memory before or during the final step of movement (e.g., rotation) of the ambient capture system 400.

[0170] The dynamic range of an image capture device is a measure of how much light an image sensor can capture. Dynamic range is the difference between the darkest and brightest areas of an image. There are many ways to increase the dynamic range of an image capture device, one of which is to use different exposures to capture multiple images of the same physical environment. Images captured with short exposures will capture brighter areas of the physical environment, while long exposures will capture darker areas. In some embodiments, an environment capture system may capture multiple images with six different exposure times. Some or all of the images captured by the environment capture system are used to generate a 2D image with high dynamic range (HDR). One or more of the captured images can be used for other functions, such as ambient light detection, flicker detection, etc.

[0171] A 3D panoramic image of a physical environment can be generated based on four separate image captures from an image capture device and four separate depth data captures from a LiDAR device in an environment capture system. Each of the four separate image captures can include a series of image captures with different exposure times. A blending algorithm can be used to blend a series of image captures with different exposure times to generate one of the four RGB image captures, which can then be used to generate a 2D panoramic image. For example, an environment capture system can be used to capture a 3D panoramic image of a kitchen. An image of one wall of the kitchen can include a window; an image captured with a shorter exposure can provide a view outside the window, but may underexpose the rest of the kitchen. Conversely, another image captured with a longer exposure can provide a view inside the kitchen. The blending algorithm can generate a blended RGB image by blending the view outside the kitchen window from one image with the rest of the kitchen view from another image.

[0172] In various embodiments, a 3D panoramic image can be generated based on three separate image captures by an image capture device and four separate depth data captures by a LiDAR device in an environment capture system. In some embodiments, the number of image captures and the number of depth data captures can be the same. In one embodiment, the number of image captures and the number of depth data captures can be different.

[0173] After capturing the first image in a series with one exposure time, the blending algorithm receives that first image in the series, calculates an initial intensity weight for that image, and sets that image as the baseline image for combining subsequently received images. In some embodiments, the blending algorithm may utilize graphics processing unit (GPU) image processing routines, such as the "blend_kernel" routine. The blending algorithm may receive subsequent images that can be blended with previously received images. In some embodiments, the blending algorithm may utilize a variant of the Blend_Kernel GPU image processing routine.

[0174] In one embodiment, the blending algorithm utilizes other methods of blending multiple images, such as determining the difference between the darkest and brightest parts of a baseline image or the contrast of the baseline image, to determine whether the baseline image may be overexposed or underexposed. For example, a contrast value less than a predetermined contrast threshold means that the baseline image is overexposed or underexposed. In one embodiment, the contrast of the baseline image can be calculated by taking the light intensity of the image or the average of a subset of the image. In some embodiments, the blending algorithm calculates the average light intensity of each row or column of the image. In some embodiments, the blending algorithm may determine a histogram of each image received from the image capturing device and analyze the histogram to determine the light intensity of the pixels that make up each image.

[0175] In various embodiments, blending may involve sampling colors within two or more images of the same scene, including along objects and seams. If there is a significant difference in color between two images (e.g., within a predetermined threshold of color, hue, brightness, saturation, and / or similar amounts), then (e.g., on the environment capture system 400 or user device 1110) the blending module may blend the two images of a predetermined size along the locations where the differences exist. In some embodiments, the greater the color or image difference at a location in the images, the greater the amount of space that can be blended around or near that location.

[0176] In some embodiments, after blending, (e.g., on the ambient capture system 400 or user device 1110) the blending module may rescan and sample colors along one or more images to determine if there are any other differences in the images or colors exceeding predetermined thresholds for color, hue, brightness, saturation, and / or similar amounts. If so, the blending module may identify portions within one or more images and continue blending those portions. The blending module may continue resampling images along the seams until no other portions of the images need to be blended (e.g., any color differences are below one or more predetermined thresholds).

[0177] Figure 11 A block diagram depicts an example environment 1100 capable of capturing and stitching images to form a 3D visualization according to some embodiments. The example environment 1100 includes a 3D and panoramic capture and stitching system 1102, a communication network 1104, an image stitching and processing system 1106, an image data storage area 1108, a user system 1110, and a first scene 1112 of the physical environment. The 3D and panoramic capture and stitching system 1102 and / or the user system 1110 may include an image capture device (e.g., an environment capture system 400) that can be used to capture images of the environment (e.g., the physical environment 1112).

[0178] The 3D and panoramic capture and stitching system 1102 and the image stitching and processing system 1106 may be part of the same system (e.g., part of one or more digital devices) communicatively coupled to the environment capture system 400. In some embodiments, one or more functions of the components of the 3D and panoramic capture and stitching system 1102 and the image stitching and processing system 1106 may be performed by the environment capture system 400. Similarly or alternatively, the 3D and panoramic capture and stitching system 1102 and the image stitching and processing system 1106 may be performed by the user system 1110 and / or the image stitching and processing system 1106.

[0179] Users can utilize the 3D panoramic capture and stitching system 1102 to capture multiple 2D images of environments such as the interior and / or exterior of buildings. For example, users can utilize the 3D and panoramic capture and stitching system 1102 to capture multiple 2D images of a first scene of the physical environment 1112 provided by the environment capture system 400. The 3D and panoramic capture and stitching system 1102 may include an alignment and stitching system 1114. Alternatively, the user system 1110 may include the alignment and stitching system 1114.

[0180] The alignment and stitching system 1114 may be software, hardware, or a combination of both, configured to provide guidance and / or process images to a user of an image capture system (e.g., on the 3D and panoramic capture and stitching system 1102 or user system 1110) to enable the creation of improved panoramic images (e.g., through stitching, alignment, cropping, etc.). The alignment and stitching system 1114 may be on a computer-readable medium (described herein). In some embodiments, the alignment and stitching system 1114 may include a processor for performing functions.

[0181] Examples of the first scene of the physical environment 1112 can be any room, property, etc. (e.g., a representation of a living room). In some embodiments, the 3D and panoramic capture and stitching system 1102 is used to generate a 3D panoramic image of the interior environment. In some embodiments, the 3D panoramic capture and stitching system 1102 can be about Figure 4 The environment capture system 400 is discussed.

[0182] In some embodiments, the 3D panoramic capture and stitching system 1102 can communicate with devices and software (e.g., environment capture system 400) used to capture image and depth data. All or part of the software may be installed on the 3D panoramic capture and stitching system 1102, the user system 1110, the environment capture system 400, or both. In some embodiments, a user can interact with the 3D panoramic capture and stitching system 1102 via the user system 1110.

[0183] The 3D and panoramic capture and stitching system 1102 or the user system 1110 can acquire multiple 2D images. The 3D and panoramic capture and stitching system 1102 or the user system 1110 can acquire depth data (e.g., from LiDAR devices, etc.).

[0184] In various embodiments, an application on the user system 1110 (e.g., the user's smart device, such as a smartphone or tablet) or on the ambient capture system 400 may provide the user with visual or auditory guidance for capturing images using the ambient capture system 400. Graphical guidance may include, for example, floating arrows on the display of the ambient capture system 400 (e.g., on a viewfinder or LED screen on the back of the ambient capture system 400) to guide the user in locating and / or pointing the image capture device. In another example, the application may provide audio guidance on locating and / or pointing the image capture device.

[0185] In some embodiments, guidance can allow a user to capture multiple images of a physical environment without the aid of a stable platform (e.g., a tripod). In one example, the image capture device can be a personal device, such as a smartphone, tablet, media tablet, laptop, etc. The application can provide orientation about the location for each sweep to approximate a blind spot based on the location of the image capture device, location information from the image capture device, and / or previous images from the image capture device.

[0186] In some embodiments, visual and / or auditory guidance enables the capture of images that can be stitched together to form a panorama without a tripod and without camera positioning information (e.g., indicating the orientation, position, and / or orientation of the camera from sensors, GPS devices, etc.).

[0187] Alignment and stitching system 1114 can align or stitch 2D images (e.g., captured by user system 1110 or 3D panoramic capture and stitching system 1102) to obtain 2D panoramic images.

[0188] In some embodiments, the alignment and stitching system 1114 utilizes machine learning algorithms to align or stitch multiple 2D images into a 2D panoramic image. The parameters of the machine learning algorithm can be managed by the alignment and stitching system 1114. For example, the 3D and panoramic capture and stitching system 1102 and / or the alignment and stitching system 1114 can identify objects within the 2D images to aid in aligning the images into a 2D panoramic image.

[0189] In some embodiments, the alignment and stitching system 1114 may utilize depth data and 2D panoramic images to obtain a 3D panoramic image. The 3D panoramic image may be provided to the 3D and panoramic stitching system 1102 or the user system 1110. In some embodiments, the alignment and stitching system 1114 determines 3D / depth measurements associated with an identified object within the 3D panoramic image and / or sends one or more 2D images, depth data, (one or more) 2D panoramic images, and (one or more) 3D panoramic images to the image stitching and processor system 106 to obtain a 2D panoramic image or 3D panoramic image with a larger pixel resolution than the 2D panoramic image or 3D panoramic image provided by the 3D and panoramic capture and stitching system 1102.

[0190] Communication network 1104 may represent one or more computer networks (e.g., LAN, WAN, etc.) or other transmission media. Communication network 1104 can provide communication between systems 1102, 1106-1110, and / or other systems described herein. In some embodiments, communication network 1104 includes one or more digital devices, routers, cables, buses, and / or other network topologies (e.g., mesh, etc.). In some embodiments, communication network 1104 may be wired and / or wireless. In various embodiments, communication network 1104 may include the Internet, one or more wide area networks (WANs) or local area networks (LANs), and may be public, private, IP-based, non-IP-based, or other networks.

[0191] The image stitching and processing system 1106 can process 2D images captured by an image capture device (e.g., an ambient capture system 400 or a user device such as a smartphone, personal computer, media tablet, etc.) and stitch them together into a 2D panoramic image. The 2D panoramic image processed by the image stitching and processing system 106 can have a higher pixel resolution than the panoramic image obtained by the 3D and panoramic capture and stitching system 1102.

[0192] In some embodiments, the image stitching and processing system 1106 receives and processes a 3D panoramic image to generate a 3D panoramic image with a higher pixel resolution than the received 3D panoramic image. The higher pixel resolution panoramic image can be provided to an output device with a higher screen resolution than the user system 1110, such as a computer screen, projector screen, etc. In some embodiments, the higher pixel resolution panoramic image can be provided to the output device in more detail and can be magnified.

[0193] Image data storage area 1108 can be any and / or multiple structures suitable for captured image and / or depth data (e.g., active database, relational database, self-referenced database, table, matrix, array, flat file, document-oriented storage system, non-relational No-SQL system, FTS management system such as Lucene / Solar, and / or similar structures). Image data storage area 1108 can store images captured by the image capture device of user system 1110. In various embodiments, image data storage area 1108 stores depth data captured by one or more depth sensors of user system 1110. In various embodiments, image data storage area 1108 stores properties associated with the image capture device or properties associated with each of multiple image captures or depth captures used to determine a 2D or 3D panoramic image. In some embodiments, image data storage area 1108 stores panoramic 2D or 3D panoramic images. 2D or 3D panoramic images can be determined by 3D and panoramic capture and stitching system 1102 or image stitching and processor system 106.

[0194] User system 1110 can communicate between a user and other associated systems. In some embodiments, user system 1110 may be or include one or more mobile devices (e.g., smartphones, cellular phones, smartwatches, etc.).

[0195] User system 1110 may include one or more image capture devices. The one or more image capture devices may include, for example, an RGB camera, an HDR camera, a video camera, an IR camera, etc.

[0196] The 3D and panoramic capture and stitching system 1102 and / or user system 1110 may include two or more capture devices arranged in opposite positions on or within the same movable housing, such that their common field of view spans up to 360°. In some embodiments, paired image capture devices capable of generating stereoscopic image pairs (e.g., with slightly offset but partially overlapping fields of view) may be used. User system 1110 may include two image capture devices having a vertically offset field of view capable of capturing vertical stereoscopic image pairs. In another example, user system 1110 may include two image capture devices having a vertically offset field of view capable of capturing vertically offset stereoscopic image pairs.

[0197] In some embodiments, user system 1110, environment capture system 400, or 3D and panoramic capture and stitching system 1102 may generate and / or provide image capture location and orientation information. For example, user system 1110 or 3D and panoramic capture and stitching system 1102 may include an inertial measurement unit (IMU) to help determine location data associated with one or more image capture devices that capture multiple 2D images. User system 1110 may include a global positioning sensor (GPS) to provide GPS coordinate information associated with the multiple 2D images captured by one or more image capture devices.

[0198] In some embodiments, a user can interact with the alignment and stitching system 1114 using a mobile application installed on the user system 1110. The 3D and panoramic capture and stitching system 1102 can provide images to the user system 1110. The user can utilize the alignment and stitching system 1114 on the user system 1110 to view and preview the images.

[0199] In various embodiments, the alignment and stitching system 1114 may be configured to provide or receive one or more 3D panoramic images from the 3D and panoramic capture and stitching system 1102 and / or the image stitching and processor system 1106. In some embodiments, the 3D and panoramic capture and stitching system 1102 may provide the user system 1110 with a visual representation of a portion of a floor plan of a building that has been captured by the 3D and panoramic capture and stitching system 1102.

[0200] Users of system 1110 can navigate the space surrounding the area and view different rooms of the house. In some embodiments, as the image stitching and processor system 1106 completes the generation of the 3D panoramic image, users of user system 1110 can display the 3D panoramic image, such as the example 3D panoramic image. In various embodiments, user system 1110 generates a preview or thumbnail of the 3D panoramic image. The preview 3D panoramic image may have a lower image resolution than the 3D panoramic image generated by the 3D and panoramic capture and stitching system 1102.

[0201] Figure 12 This is a block diagram of an example alignment and stitching system 1114 according to some embodiments. The alignment and stitching system 1114 includes a communication module 1202, an image capture position module 1204, a stitching module 1206, a cropping module 1208, a graphic cutting module 1210, a mixing module 1211, a 3D image generator 1214, a captured 2D image data storage area 1216, a 3D panoramic image data storage area 1218, and a guidance module 220. It will be understood that any number of modules may exist in the alignment and stitching system 1114 performing one or more of the different functions described herein.

[0202] In some embodiments, the alignment and stitching system 1114 includes an image capture module configured to receive images from one or more image capture devices (e.g., cameras). The alignment and stitching system 1114 may also include a depth module configured to receive depth data from a depth device such as LiDAR (if available).

[0203] Communication module 1202 can communicate with any module or data storage area of ​​alignment and splicing system 1114. Figure 11 In example environment 1100, components send and receive requests, images, or data. Similarly, alignment and stitching system 1114 can send and receive requests, images, or data to any device or system on communication network 1104.

[0204] In some embodiments, the image capture location module 1204 can determine image capture device location data (e.g., a camera with a standalone camera, a smartphone, a media tablet, a laptop, etc.). The image capture device location data can indicate the position and orientation of the image capture device and / or lens. In one example, the image capture location module 1204 can utilize the user system 1110, a camera, a digital device with a camera, or the IMU of the 3D and panoramic capture and stitching system 1102 to generate image capture device location data. The image capture location module 1204 can determine the current orientation, angle, or tilt of one or more image capture devices (or lenses). The image capture location module 1204 can also utilize the GPS of the user system 1110 or the 3D and panoramic capture and stitching system 1102.

[0205] For example, when a user wants to use user system 1110 to capture a 360° view of a physical environment (such as a living room), the user can hold user system 1110 in front of them at eye level to begin capturing one of several images that will eventually become a 3D panoramic image. To reduce the amount of parallax in the images and capture images more suitable for stitching and generating a 3D panoramic image, it is preferable if one or more image capture devices are rotated at the center of the rotation axis. Alignment and stitching system 1114 can receive position information (e.g., from an IMU) to determine the position of the image capture device or lens. Alignment and stitching system 1114 can receive and store the field of view of the lens. Guidance module 1220 can provide visual and / or audio information about a recommended initial position for the image capture device. Guidance module 1220 can make recommendations for positioning the image capture device for subsequent images. In one example, guidance module 1220 can provide guidance to the user to rotate and position the image capture device such that the image capture device rotates closer to the center of rotation. In addition, the guidance module 1220 can provide guidance to the user to rotate and position the image capturing device so that subsequent images are substantially aligned based on the field of view and / or the characteristics of the image capturing device.

[0206] The guidance module 1220 can provide visual guidance to the user. For example, the guidance module 1220 can place markers or arrows in a viewer or display on the user system 1110 or the 3D and panoramic capture and stitching system 1102. In some embodiments, the user system 1110 can be a smartphone or tablet computer with a display. When one or more images are captured, the guidance module 1220 can position one or more markers (e.g., markers of different colors or the same marker) on the output device and / or in the viewfinder. The user can then use the markers on the output device and / or viewfinder to align the next image.

[0207] Numerous techniques are available to guide the user of system 1110 or 3D and panoramic capture and stitching system 1102 to capture multiple images for stitching into a panorama. When a panorama is obtained from multiple images, it can be stitched together. To improve the time, efficiency, and effectiveness of image stitching while reducing the need to correct artifacts or misalignments, image capture location module 1204 and guidance module 1220 assist the user in capturing multiple images at locations that improve the quality, time efficiency, and effectiveness of image stitching for the desired panorama.

[0208] For example, after capturing the first image, the display of user system 1110 may include two or more objects, such as circles. The two circles may appear stationary relative to the environment, and the two circles may move with user system 1110. When the two stationary circles are aligned with the two circles moving with user system 1110, the image capturing device and / or user system 1110 can be aligned for the next image.

[0209] In some embodiments, after the image capturing device captures an image, the image capturing position module 1204 may acquire sensor measurements of the position of the image capturing device (e.g., including orientation, tilt, etc.). The image capturing position module 1204 may determine one or more edges of the captured image by calculating the position of the edges of the field of view based on the sensor measurements. Alternatively or additionally, the image capturing position module 1204 may determine one or more edges of the image by scanning the image captured by the image capturing device, identifying objects within the image (e.g., using a machine learning model discussed herein), determining one or more edges of the image, and positioning the objects (e.g., circles or other shapes) at the edges of the display on the user system 1110.

[0210] The image capture position module 1204 can display two objects within the display of the user system 1110, indicating the location of the field of view for the next image. These two objects can indicate the location of an edge in the environment representing the presence of the last image. The image capture position module 1204 can continue to receive sensor measurements of the position of the image capture device and calculate two additional objects in the field of view. The two additional objects can be spaced apart by the same width as the first two objects. While the first two objects can represent the edges of the captured image (e.g., the rightmost edge of the image), the next two additional objects representing the edges of the field of view can be on opposite edges (e.g., the leftmost edge of the field of view). By physically aligning the first two objects on the image edges with the other two objects on the opposite edges of the field of view, the image capture device can be positioned to capture another image that can be more effectively stitched together without a tripod. This process can continue for each image until the user determines that the desired panorama has been captured.

[0211] Although multiple objects are discussed herein, it should be understood that the image capture location module 1204 can calculate the position of one or more objects used to position the image capture device. The objects can be of any shape (e.g., a circle, ellipse, square, emoji, arrow, etc.). In some embodiments, the objects can have different shapes.

[0212] In some embodiments, distances may exist between objects representing the edges of the captured image, and also between objects within the field of view. The user can be guided forward to move away to allow sufficient distance between objects. Alternatively, as the image capturing device approaches the correct position (e.g., by moving closer or further away to enable the capture of the next image in a position that will improve image stitching), the size of objects within the field of view can be altered to match the size of objects representing the edges of the captured image.

[0213] In some embodiments, the image capture location module 1204 can estimate the location of the image capture device using objects in an image captured by the image capture device. For example, the image capture location module 1204 can use GPS coordinates to determine the geographic location associated with the image. The image capture location module 1204 can use this location to identify landmarks that can be captured by the image capture device.

[0214] The image capture location module 1204 may include a 2D machine learning model for converting 2D images into 2D panoramic images. The image capture location module 1204 may also include a 3D machine learning model for converting 2D images into 3D representations. In one example, the 3D representation can be used to display a 3D walkthrough or visualization of the interior and / or exterior environment.

[0215] A 2D machine learning model can be trained to stitch together two or more 2D images or assist in stitching them together to form a 2D panoramic image. The 2D machine learning model can be, for example, a neural network trained with 2D images, including physical objects within the images and object recognition information that trains the 2D machine learning model to recognize objects in subsequent 2D images. Objects in the 2D images can help determine one or more locations within the 2D images, aiding in the identification of edges, distortions, and alignment. Furthermore, objects in the 2D images can help determine artifacts, artifacts or boundary blending between two images, and the location of image cropping and / or truncating.

[0216] In some embodiments, the 2D machine learning model may be, for example, a neural network trained with 2D images, wherein the 2D images include depth information of the environment (e.g., from a LiDAR device or structured light device or 3D and panoramic capture and stitching system 1102 of user system 1110) and include physical objects in the image to identify the physical objects, the location of the physical objects, and / or the location of the image capture device / field of view. The 2D machine learning model can identify the physical objects and their depth relative to other aspects of the 2D images to aid in the alignment and positioning of the two 2D images for stitching (or stitching two 2D images).

[0217] 2D machine learning models can include any number of machine learning models (e.g., any number of models generated by neural networks, etc.).

[0218] The 2D machine learning model can be stored on the 3D and panoramic capture and stitching system 1102, the image stitching and processing system 1106, and / or the user system 1110. In some embodiments, the 2D machine learning model can be trained by the image stitching and processing system 1106.

[0219] The image capture position module 1204 can estimate the position of the image capture device (the position of the field of view of the image capture device) based on the seam between two or more 2D images from the stitching module 1206, the image distortion from the cropping module 1208, and / or the graphic cutting from the graphic cutting module 1210.

[0220] The stitching module 1206 can combine two or more 2D images to generate a 2D panorama. Based on the seams between the two or more 2D images from the stitching module 1206, image distortion from the cropping module 1208, and / or graphic cutting, it has a larger field of view than each of the two or more images alone.

[0221] The stitching module 1206 can be configured to align or “stitch together” two different 2D images that provide different perspectives of the same environment to generate a panoramic 2D image of the environment. For example, the stitching module 1206 can use known or (e.g., derived) information about the capture location and orientation of the respective 2D images to help stitch the two images together.

[0222] The stitching module 1206 can receive two 2D images. The first 2D image may have been captured just before the second image or within a predetermined time period. In various embodiments, the stitching module 1206 may receive positioning information of the image capture device associated with the first image, and then receive positioning information associated with the second image. The positioning information may be associated with the image based on positioning data from an IMU, GPS, and / or information provided by the user at the time the image was captured.

[0223] In some embodiments, the stitching module 1206 may utilize a 2D machine learning module to scan two images to identify objects within the two images, including objects (or portions of objects) that may be shared by the two images. For example, the stitching module 1206 may identify corners shared at the opposite edges of the two images, patterns on walls, furniture, etc.

[0224] The stitching module 1206 can align the edges of two 2D images based on the location of a shared object (or a portion of an object), location data from an IMU, location data from GPS, and / or information provided by the user, and then combine the two edges of the images (i.e., "stitch" them together). In some embodiments, the stitching module 1206 can identify portions of the two 2D images that overlap each other and stitch the images at the overlapping locations (e.g., using location data and / or the results of a 2D machine learning model).

[0225] In various embodiments, a 2D machine learning model can be trained to combine or stitch together two edges of an image using positioning data from an IMU, positioning data from GPS, and / or information provided by the user. In some embodiments, the 2D machine learning model can be trained to identify common objects in two 2D images to align and locate the 2D images, and then combine or stitch together the two edges of the images. In other embodiments, the 2D machine learning model can be trained to use positioning data and object recognition to align and locate 2D images, and then stitch together the two edges of the images to form all or part of a panoramic 2D image.

[0226] The stitching module 1206 can utilize the depth information of the corresponding images (e.g., pixels in the corresponding images, objects in the corresponding images, etc.) to align the corresponding 2D images with each other in association with a single 2D panoramic image of the generated environment.

[0227] The cropping module 1208 can handle two or more 2D images when the image capturing devices are not kept in the same position during 2D image capture. For example, while capturing an image, the user can position the user system 1110 vertically. However, while capturing another image, the user can position the user system at an angle. The resulting images may be misaligned and may exhibit parallax effects. Parallax effects occur when foreground and background objects are not aligned in the same way in the first and second images.

[0228] The cropping module 1208 can utilize a 2D machine learning model (by applying positioning information, depth information, and / or object recognition) to detect changes in the position of the image capturing device in two or more images, and then measure the amount of change in the position of the image capturing device. The cropping module 1208 can distort one or more 2D images so that when the images are stitched together, they can be aligned to form a panoramic image, while preserving certain characteristics of the images, such as keeping straight lines straight.

[0229] The output of the cropping module 1208 may include the number of pixel columns and rows to straighten the image by offsetting each pixel. The offset of each image may be output as a matrix representing the number of pixel columns and rows offsetting each pixel of the image.

[0230] In some embodiments, the cropping module 1208 may determine the amount of image distortion to be performed on one or more of the plurality of 2D images captured by the image capture device of the user system 1110 based on one or more image capture positions from the image capture position module 1204 or the gap between two or more 2D images from the stitching module 1206, the graphic cutting from the graphic cutting module 1210, or the color mixing from the mixing module 1211.

[0231] The image cutting module 1210 can determine where to cut or slice one or more 2D images captured by the image capture device. For example, the image cutting module 1210 can utilize a 2D machine learning model to identify objects in two images and determine that they are the same object. The image capture positioning module 1204, the cropping module 1208, and / or the image cutting module 1210 can determine that the two images cannot be aligned even when distorted. The image cutting module 1210 can utilize information from the 2D machine learning model to identify segments of the two images that can be stitched together (e.g., by cutting off a portion of one or both images to aid alignment and positioning). In some implementations, the two 2D images can overlap with at least a portion of the physical world represented in the images. The image cutting module 1210 can identify objects in the two images, such as the same chair. However, even after the image distortion by the image capture positioning and cropping module 1208, the images of the chair cannot be aligned to generate a distortion-free panorama and will not correctly represent a portion of the physical world. The image cutting module 1210 can select one of two images of the chair as the correct representation (e.g., based on misalignment, positioning errors, and / or artifacts in one image compared to the other), and cut the chair from the image with misalignment, positioning errors, and / or artifacts. The stitching module 1206 can then stitch the two images together.

[0232] The graphic cutting module 1210 can try two combinations, for example, cutting an image of a chair from a first image and stitching the first image minus the chair to a second image, to determine which graphic cutting produces a more accurate panoramic image. The output of the graphic cutting module 1210 can be the position of cutting one or more of a plurality of 2D images corresponding to the graphic cutting, which produces a more accurate panoramic image.

[0233] The graphic cutting module 1210 can determine how to cut or slice one or more 2D images captured by the image capture device based on one or more image capture positions from the image capture position module 1204, the stitching or seam between two or more 2D images from the stitching module 1206, the image distortion from the cropping module 1208, and the graphic cutting from the graphic cutting module 1210.

[0234] The blending module 1211 can color the seam (e.g., stitching) between two images, making the seam invisible. Variations in lighting and shadows can cause the same object or surface to be output with slightly different colors or shadows. The blending module can determine the desired amount of color blending based on one or more image capture positions from the image capture position module 1204, the stitching, the image color along the seam from the two images, the image distortion from the cropping module 1208, and / or the graphic cutting from the graphic cutting module 1210.

[0235] In various embodiments, the blending module 1211 may receive a panorama from a combination of two 2D images and then sample colors along the seam between the two 2D images. The blending module 1211 may receive seam location information from the image capture location module 1204, enabling it to sample colors along the seam and determine differences. If a significant color difference exists along the seam between the two images (e.g., within a predetermined threshold for color, hue, brightness, saturation, and / or similar amounts), the blending module 1211 may blend the two images of a predetermined size along the seam at the location of the difference. In some embodiments, the greater the difference in color or image along the seam, the greater the amount of space that can be blended along the seam between the two images.

[0236] In some embodiments, after blending, the blending module 1211 may rescan and sample the colors along the seam to determine if there are any other differences in the image or colors exceeding a predetermined threshold for color, hue, brightness, saturation, and / or similar amounts. If so, the blending module 1211 may identify the portion along the seam and continue blending that portion of the image. The blending module 1211 may continue resampling the image along the seam until no other portions of the image need to be blended (e.g., any color differences are below the predetermined threshold).

[0237] 3D image generator 1214 can receive 2D panoramic images and generate 3D representations. In various embodiments, 3D image generator 1214 utilizes a 3D machine learning model to transform the 2D panoramic image into a 3D representation. The 3D machine learning model can be trained using the 2D panoramic image and depth data (e.g., from a LiDAR sensor or structured light device) to produce the 3D representation. The 3D representation can be tested and reviewed against curation and feedback. In some embodiments, the 3D machine learning model can be used together with the 2D panoramic image and depth data to generate the 3D representation.

[0238] In various embodiments, the accuracy, rendering speed, and quality of the 3D representation generated by the 3D image generator 1214 are significantly improved by utilizing the systems and methods described herein. For example, by rendering a 3D representation from a 2D panoramic image that has already been aligned, positioned, and stitched using the methods described herein (e.g., by alignment and positioning information provided by hardware, by improved positioning resulting from guidance provided to the user during image capture, by cropping and altering image distortion, by cutting the image to avoid artifacts and overcome distortion, by blending images, and / or any combination thereof), the accuracy, rendering speed, and quality of the 3D representation are improved. Furthermore, it should be understood that by utilizing a 2D panoramic image that has already been aligned, positioned, and stitched using the methods described herein, the training of 3D machine learning models can be significantly improved (e.g., in terms of speed and accuracy). Moreover, in some embodiments, the 3D machine learning model can be smaller and less complex due to the reduction in processing and learning required to overcome misalignment, positioning errors, distortion, poor graphic cutting, poor blending, artifacts, etc., to generate a reasonably accurate 3D representation.

[0239] The trained 3D machine learning model can be stored in the 3D and panoramic capture and stitching system 1102, the image stitching and processing system 106, and / or the user system 1110.

[0240] In some embodiments, a 3D machine learning model can be trained using multiple 2D images and depth data from the image capture device of the user system 1110 and / or the 3D and panoramic capture and stitching system 1102. Additionally, the 3D image generator 1214 can be trained using image capture location information associated with each of the multiple 2D images from the image capture location module 1204, seam positions for aligning or stitching each of the multiple 2D images from the stitching module 1206, pixel offsets (one or more) of each of the multiple 2D images from the cropping module 1208, and / or graphic cuts from the graphic cutting module 1210. In some embodiments, the 3D machine learning model can be used with 2D panoramic images, depth data, image capture location information associated with each of a plurality of 2D images from image capture location module 1204, seam positions of alignment or stitching of each of a plurality of 2D images from stitching module 1206, pixel offsets of each of a plurality of 2D images from cropping module 1208, and / or graphic cuts from graphic cutting module 1210 to generate a 3D representation.

[0241] The stitching module 1206 may be part of a 3D model that converts multiple 2D images into 2D or 3D panoramic images. In some embodiments, the 3D model is a machine learning algorithm, such as a predictive neural network model from 2D to 3D. The cropping module 1208 may be part of a 3D model that converts multiple 2D images into 2D or 3D panoramic images. In some embodiments, the 3D model is a machine learning algorithm, such as a predictive neural network model from 2D to 3D. The image cutting module 1210 may be part of a 3D model that converts multiple 2D images into 2D or 3D panoramic images. In some embodiments, the 3D model is a machine learning algorithm, such as a predictive neural network model from 2D to 3D. The blending module 1211 may be part of a 3D machine learning model that converts multiple 2D images into 2D or 3D panoramic images. In some embodiments, the 3D model is a machine learning algorithm, such as a predictive neural network model from 2D to 3D.

[0242] The 3D image generator 1214 can generate weights for each of the image capture location module 1204, cropping module 1208, graphic cutting module 1210, and blending module 1211, which can represent the reliability or "strength" or "weakness" of the module. In some embodiments, the sum of the weights of the modules is equal to 1.

[0243] When depth data is not available for multiple 2D images, the 3D image generator 1214 can determine depth data for one or more objects in multiple 2D images captured by the image capture device of the user system 1110. In some embodiments, the 3D image generator 1214 can derive depth data based on images captured from a stereo image pair. Instead of determining depth data based on a passive stereo algorithm, the 3D image generator can evaluate the stereo image pair to determine data (a more intermediate result) regarding the photometric matching quality between images at various depths.

[0244] The 3D image generator 1214 may be part of a 3D model that converts multiple 2D images into 2D or 3D panoramic images. In some embodiments, the 3D model is a machine learning algorithm, such as a predictive neural network model for 2D to 3D conversion.

[0245] The captured 2D image data storage area 1216 can be any and / or multiple structures suitable for the captured image and / or depth data (e.g., active database, relational database, self-referenced database, table, matrix, array, flat file, document-oriented storage system, non-relational No-SQL system, FTS management system such as Lucene / Solar, and / or similar structures). The captured 2D image data storage area 1216 can store images captured by the image capture device of the user system 1110. In various embodiments, the captured 2D image data storage area 1216 stores depth data captured by one or more depth sensors of the user system 1110. In various embodiments, the captured 2D image data storage area 1216 stores image capture device parameters associated with the image capture device, or capture properties associated with each of a plurality of image captures, or depth captures used to determine the 2D panoramic image. In some embodiments, the image data storage area 1108 stores a panoramic 2D panoramic image. The 2D panoramic image can be determined by the 3D and panoramic capture and stitching system 1102 or the image stitching and processor system 106. Image capture device parameters may include illumination, color, image capture lens focal length, maximum aperture, tilt angle, etc. Capture properties may include pixel resolution, lens distortion, illumination, and other image metadata.

[0246] The 3D panoramic image data storage area 1218 can be any and / or multiple structures suitable for 3D panoramic images (e.g., active database, relational database, self-referenced database, table, matrix, array, flat file, document-oriented storage system, non-relational No-SQL system, FTS management system such as Lucene / Solar, and / or similar structures). The 3D panoramic image data storage area 1218 can store 3D panoramic images generated by the 3D and panoramic capture and stitching system 1102. In various embodiments, the 3D panoramic image data storage area 1218 stores properties associated with the image capture device or properties associated with each of a plurality of image captures or depth captures used to determine the 3D panoramic image. In some embodiments, the 3D panoramic image data storage area 1218 stores 3D panoramic images. 2D or 3D panoramic images can be determined by the 3D and panoramic capture and stitching system 1102 or the image stitching and processor system 106.

[0247] Figure 13 A flowchart 1300 depicts a 3D panoramic image capture and generation process according to some embodiments. In step 1302, the image capture device can capture multiple 2D images using the image sensor 920 of FIG9 and the WFOV lens 918. A wider FOV means that the environment capture system 402 will require fewer scans to obtain a 360° view. The WFOV lens 918 can also be wider horizontally and vertically. In some embodiments, the image sensor 920 captures RGB images. In one embodiment, the image sensor 920 captures black and white images.

[0248] In step 1304, the environment capture system may send the captured 2D image to the image stitching and processing system 1106. The image stitching and processing system 1106 may apply a 3D modeling algorithm to the captured 2D image to generate a panoramic 2D image. In some embodiments, the 3D modeling algorithm is a machine learning algorithm for stitching the captured 2D images into a panoramic 2D image. In some embodiments, step 1304 may be optional.

[0249] In step 1306, the LiDAR 912 and WFOV lens 918 of Figure 9 can capture LiDAR data. The wider FOV means that the environment capture system 400 will require fewer scans to obtain a 360° view.

[0250] In step 1308, LiDAR data can be sent to image stitching and processing system 1106. Image stitching and processing system 1106 can input the LiDAR data and captured 2D images into a 3D modeling algorithm to generate a 3D panoramic image. The 3D modeling algorithm is a machine learning algorithm.

[0251] In step 1310, the image stitching and processing system 1106 generates a 3D panoramic image. The 3D panoramic image can be stored in the image data storage area 408. In one embodiment, the 3D panoramic image generated by the 3D modeling algorithm is stored in the image stitching and processing system 1106. In some embodiments, as the environment capture system is used to capture various parts of the physical environment, the 3D modeling algorithm can generate a visual representation of a planar view of the physical environment.

[0252] In step 1312, the image stitching and processing system 1106 can provide at least a portion of the generated 3D panoramic image to the user system 1110. The image stitching and processing system 1106 can provide a visual representation of a planar view of the physical environment.

[0253] The order of one or more steps in flowchart 1300 can be changed without affecting the final result of the 3D panoramic image. For example, an environment capture system can interleave image capture using an image capture device with capture using LiDAR data or depth information from a LiDAR 912. For instance, the image capture device can capture an image of a segment of the physical environment, and then the LiDAR 912 obtains depth information from segment 1605. Once the LiDAR 912 has obtained depth information from a segment, the image capture device can continue to move to capture an image of another segment, and then the LiDAR 912 obtains depth information from that segment, thus interleaving image capture and depth information capture.

[0254] In some embodiments, the devices and / or systems discussed herein employ an image capture device to capture a 2D input image. In some embodiments, the one or more image capture devices 1116 may represent a single image capture device (or image capture lens). According to some of these embodiments, a user of a mobile device housing the image capture device can be configured to rotate about an axis to generate images with different capture orientations relative to the environment, wherein the common field of view of the images horizontally spans up to 360°.

[0255] In various embodiments, the devices and / or systems discussed herein may employ two or more image capture devices to capture 2D input images. In some embodiments, two or more image capture devices may be arranged in opposite positions on or within the same movable housing such that their common field of view spans up to 360°. In some embodiments, paired image capture devices capable of generating stereo image pairs (e.g., with slightly offset but partially overlapping fields of view) may be used. For example, user system 1110 (e.g., a device including one or more image capture devices for capturing 2D input images) may include two image capture devices having a horizontal stereo offset field of view capable of capturing stereo image pairs. In another example, user system 1110 may include two image capture devices having a vertical stereo offset field of view capable of capturing vertical stereo image pairs. According to any of these examples, each camera may have a field of view spanning up to 360°. In this regard, in one embodiment, user system 1110 may employ two panoramic cameras with a vertical stereo offset capable of capturing paired panoramic images forming (with vertical stereo offset) stereo pairs.

[0256] The positioning component 1118 may include any hardware and / or software configured to capture user system location data and / or user system orientation data. For example, the positioning component 1118 includes an IMU to generate user system 1110 location data associated with the one or more image capture devices used to capture multiple 2D images. The positioning component 1118 may include a GPS unit to provide GPS coordinate information associated with the multiple 2D images captured by the one or more image capture devices. In some embodiments, the positioning component 1118 may associate the user system's location data and orientation data with the corresponding images captured using the one or more image capture devices of the user system 1110.

[0257] Various embodiments of this device provide users with 3D panoramic images of indoor and outdoor environments. In some embodiments, the device can efficiently and quickly provide users with 3D panoramic images of indoor and outdoor environments using a single wide field of view (FOV) lens and a single light and detection and ranging sensor (LiDAR sensor).

[0258] The following are example use cases of the example device described herein. The following use case is one of the embodiments. Different embodiments of the device discussed herein may include one or more features and capabilities similar to those of this use case.

[0259] Figure 14 A flowchart of a 3D and panoramic capture and stitching process 1400 according to some embodiments is depicted. Figure 14The flowchart refers to the 3D and panoramic capture and stitching system 1102 as including an image capture device, but in some embodiments, the data capture device may be the user system 1110.

[0260] In step 1402, the 3D and panoramic capture and stitching system 1102 may receive multiple 2D images from at least one image capture device. The image capture device of the 3D and panoramic capture and stitching system 1102 may be or include a complementary metal-oxide-semiconductor (CMOS) image sensor. In various embodiments, the image capture device is a charge-coupled device (CCD). In one example, the image capture device is a red-green-blue (RGB) sensor. In one embodiment, the image capture device is an IR sensor. Each of the multiple 2D images may have a field of view that overlaps with at least one other portion of the multiple 2D images. In some embodiments, at least some of the multiple 2D images are combined to produce a 360° view of a physical environment (e.g., indoor, outdoor, or both).

[0261] In some embodiments, all plurality of 2D images are received from the same image capture device. In various embodiments, at least a portion of the plurality of 2D images are received from two or more image capture devices of the 3D and panoramic capture and stitching system 1102. In one example, the plurality of 2D images includes a set of RGB images and a set of IR images, wherein the IR images provide depth data to the 3D and panoramic capture and stitching system 1102. In some embodiments, each 2D image may be associated with depth data provided by a LiDAR device. In some embodiments, each 2D image may be associated with positioning data.

[0262] In step 1404, the 3D and panoramic capture and stitching system 1102 can receive capture parameters and image capture device parameters associated with each of the multiple received 2D images. Image capture device parameters may include illumination, color, image capture lens focal length, maximum aperture, field of view, etc. Capture properties may include pixel resolution, lens distortion, illumination, and other image metadata. The 3D and panoramic capture and stitching system 1102 can also receive positioning data and depth data.

[0263] In step 1406, the 3D and panoramic capture and stitching system 1102 can acquire the information received from steps 1402 and 1404 for stitching 2D images to form a 2D panoramic image. (About...) Figure 15 The flowchart further discusses the process of stitching 2D images.

[0264] In step 1408, the 3D and panoramic capture and stitching system 1102 may apply a 3D machine learning model to generate a 3D representation. The 3D representation may be stored in a 3D panoramic image data storage area. In various embodiments, the 3D representation is generated by the image stitching and processing system 1106. In some embodiments, as the environment capture system is used to capture various parts of the physical environment, the 3D machine learning model may generate a visual representation of a planar view of the physical environment.

[0265] In step 1410, the 3D and panoramic capture and stitching system 1102 can provide the user system 1110 with at least a portion of the generated 3D representation or model. The user system 1110 can provide a visual representation of a plan view of the physical environment.

[0266] In some embodiments, user system 1110 may send multiple 2D images, capture parameters, and image capture parameters to image stitching and processing system 1106. In various embodiments, 3D and panoramic capture and stitching system 1102 may send multiple 2D images, capture parameters, and image capture parameters to image stitching and processing system 1106.

[0267] The image stitching and processing system 1106 can process multiple 2D images captured by the image capture device of the user system 1110 and stitch them into a 2D panoramic image. The 2D panoramic image processed by the image stitching and processing system 1106 can have a higher pixel resolution than the 2D panoramic image obtained by the 3D and panoramic capture and stitching system 1102.

[0268] In some embodiments, the image stitching and processing system 106 can receive a 3D representation and output a 3D panoramic image with a higher pixel resolution than the received 3D panoramic image. The higher pixel resolution panoramic image can be provided to an output device with a higher screen resolution than the user system 1110, such as a computer screen, projector screen, etc. In some embodiments, the higher pixel resolution panoramic image can be provided to the output device in more detail and can be magnified.

[0269] Figure 15 Depicting Figure 14A flowchart providing further details of one step in the 3D and panoramic capture and stitching process. In step 1502, the image capture position module 1204 may determine image capture device position data associated with each image captured by the image capture device. The image capture position module 1204 may utilize the IMU of the user system 1110 to determine position data of the image capture device (or the field of view of the lens of the image capture device). The position data may include the orientation, angle, or tilt of one or more image capture devices when capturing one or more 2D images. One or more of the cropping module 1208, the graphic cutting module 1210, or the blending module 1212 may utilize the orientation, angle, or tilt associated with each of the multiple 2D images to determine how to warp, cut, and / or blend the images.

[0270] In step 1504, the cropping module 1208 can distort one or more of the plurality of 2D images such that two images can be aligned together to form a panoramic image while preserving specific characteristics of the images, such as keeping straight lines straight. The output of the cropping module 1208 may include the number of pixel columns and rows to offset each pixel of the image to straighten it. The offset of each image may be output as a matrix representing the number of pixel columns and rows offset per pixel of the image. In this embodiment, the cropping module 1208 may determine the desired amount of distortion for each of the plurality of 2D images based on an image capture pose estimation of each of the plurality of 2D images.

[0271] In step 1506, the image cutting module 1210 determines where to cut or slice one or more of the plurality of 2D images. In this embodiment, the image cutting module 1210 may determine where to cut or slice each of the plurality of 2D images based on image capture pose estimation and image distortion of each of the plurality of 2D images.

[0272] In step 1508, the stitching module 1206 can stitch two or more images together using the edges and / or cuts of the images. The stitching module 1206 can align and / or position the images based on objects detected within the images, image distortions, cuts, etc.

[0273] In step 1510, the blending module 1212 can adjust the color at the seam (e.g., the stitching of two images) or at a location on one image that contacts or connects to another image. The blending module 1212 can determine the desired amount of color blending based on one or more image capture positions from the image capture position module 1204, image distortion from the cropping module 1208, and graphic cutting from the graphic cutting module 1210.

[0274] The order of one or more steps in the 3D and panoramic capture and stitching process 1400 can be changed without affecting the final result of the 3D panoramic image. For example, an environment capture system can interleave image capture using an image capture device with LiDAR data or depth information capture. For example, an image capture device can use an image capture device to capture the physical environment. Figure 16 The image capture device captures an image of segment 1605, and then the LiDAR 612 obtains depth information from segment 1605. Once the LiDAR has obtained depth information from segment 1605, the image capture device can continue to move to capture an image of another segment 1610, and then the LiDAR 612 obtains depth information from segment 1610, thus interleaving image capture and depth information capture.

[0275] Figure 16 A block diagram of an example digital device 1602 according to some embodiments is depicted. Any of the user system 1110, the 3D panoramic capture and stitching system 1102, and the image stitching and processor system may include an instance of the digital device 1602. The digital device 1602 includes a processor 1604, a memory 1606, a storage device 1608, an input device 1610, a communication network interface 1612, an output device 1614, an image capture device 1616, and a positioning component 1618. The processor 1604 is configured to execute executable instructions (e.g., a program). In some embodiments, the processor 1604 includes circuitry or any processor capable of processing executable instructions.

[0276] Memory 1606 stores data. Some examples of memory 1606 include storage devices such as RAM, ROM, RAM cache, virtual memory, etc. In various embodiments, working data is stored in memory 1606. Data in memory 1606 can be erased or eventually transferred to storage device 1608.

[0277] Storage device 1608 includes any storage device configured to retrieve and store data. Some examples of storage device 1608 include flash drives, hard disk drives, optical disk drives, and / or magnetic tapes. Each of memory 1606 and storage device 1608 includes a computer-readable medium storing instructions or programs executable by processor 1604.

[0278] Input device 1610 is any device that inputs data (e.g., a touch keyboard, stylus). Output device 1614 outputs data (e.g., a speaker, display, virtual reality headset). It should be understood that storage device 1608, input device 1610, and output device 1614 are included. In some embodiments, output device 1614 is optional. For example, a router / switch may include processor 1604 and memory 1606, as well as devices for receiving and outputting data (e.g., communication network interface 1612 and / or output device 1614).

[0279] The communication network interface 1612 can be coupled to a network (e.g., communication network 104). The communication network interface 1612 can support communication via Ethernet, serial, parallel, and / or ATA connections. The communication network interface 1612 can also support wireless communication (e.g., 802.16a / b / g / n, WiMAX, LTE, Wi-Fi). It will be apparent that the communication network interface 1612 can support many wired and wireless standards.

[0280] Components can be hardware or software. In some embodiments, a component may be configured with one or more processors to perform functions associated with the component. Although different components are discussed herein, it should be understood that a server system may include any number of components that perform any or all of the functions discussed herein.

[0281] Digital device 1602 may include one or more image capture devices 1616. The one or more image capture devices 1616 may include, for example, an RGB camera, an HDR camera, a video camera, etc. According to some embodiments, the one or more image capture devices 1616 may also include a video camera capable of capturing video. In some embodiments, the one or more image capture devices 1616 may include an image capture device that provides a relatively standard field of view (e.g., approximately 75°). In other embodiments, the one or more image capture devices 1616 may include a camera that provides a relatively wide field of view (e.g., from approximately 120° to 360°), such as a fisheye camera, etc. (e.g., digital device 1602 may include or be included in environmental capture system 400).

[0282] Components can be hardware or software. In some embodiments, a component may be configured with one or more processors to perform functions associated with the component. Although different components are discussed herein, it should be understood that a server system may include any number of components that perform any or all of the functions discussed herein.

Claims

1. A device for acquiring image and depth information used in an environment, the device comprising: First motor; An image capturing device is configured to capture a first plurality of images at different exposures within the field of view of the image capturing device while pointing in a first direction in the environment. The image capturing device includes a lens attached to the frame of the device at a first position between the front and rear sides of the frame of the device. Depth information capture device; as well as A first motor is configured to rotate the depth information capturing device and the image capturing device about a first axis until the image capturing device points in a second direction. The depth information capturing device is configured to capture first depth information of a first portion of the environment while being rotated. The image capturing device is also configured to capture, within the environment and pointing in the second direction, a second plurality of images at different exposures that at least partially overlap with one or more of the first plurality of images. The first motor is further configured to rotate the depth information capturing device and the image capturing device about the first axis until the image capturing device points to a third direction. The depth information capturing device is also configured to capture second depth information of a second portion of the environment while being rotated. The image capturing device is also configured to capture a third plurality of images at different exposures that at least partially overlap with one or more of the second plurality of images while pointing in the third direction.

2. The device of claim 1, wherein the depth information capturing device and the image capturing device are configured to be rotated by the first motor about the first axis until the image capturing device points to a fourth direction, and while pointing to the fourth direction, the image capturing device is configured to capture a fourth plurality of images at different exposures in the fourth direction, the fourth plurality of images at different exposures captured in the fourth direction at least partially overlapping one or more of the first plurality of images and the third plurality of images.

3. The device of claim 2, wherein the depth information capturing device is configured to capture third depth information of a third portion of the environment while being rotated by the first motor about the first axis until the image capturing device points to the fourth direction.

4. The device of claim 3, wherein the depth information capturing device that captures depth information of the first portion, the second portion, and the third portion of the environment while being rotated includes the depth information capturing depth information of a first plurality of segments, a second plurality of segments, and a third plurality of segments of the environment while being rotated.

5. The device according to claim 1, further comprising: A communication module configured to provide one or more of the plurality of images of the first, second, and third portions of the environment to a digital device to form a panoramic image of the environment, wherein the depth information capturing device is configured to capture third depth information of the third portion of the environment while being rotated by the first motor about the first axis, and the digital device is configured to stitch together one or more of the plurality of images of the first, second, and third portions to form a panoramic image of the environment and to combine the depth information of the first, second, and third portions of the environment with the panoramic image of the environment to generate a 3D panoramic image of the environment.

6. The device according to claim 5, further comprising: The digital device is also configured to mix two or more of the first plurality of images, the second plurality of images, and the third plurality of images to generate a first mixed image, a second mixed image, and a third mixed image. The digital device is configured to stitch together one or more of the first plurality of images, the second plurality of images, and the third plurality of images to form a panoramic image of the environment, including stitching together the first composite image, the second composite image, and the third composite image to form a panoramic image of the environment.

7. The device according to claim 1, further comprising: Second motor, The depth information acquisition device includes a light detection and ranging device, namely a LiDAR device, and a reflector. The second motor is configured to cause the reflector to rotate about a second axis while the LiDAR device emits multiple laser pulses toward the reflector. The reflector is configured to provide the multiple laser pulses and receive corresponding multiple reflected laser pulses in one or more rotations about the second axis. The LiDAR device is also configured to receive the multiple reflected laser pulses and generate the depth information therefrom.

8. The device according to claim 6, further comprising: Second motor, The depth information acquisition device includes a light detection and ranging device, namely a LiDAR device, and a reflector. The second motor is configured to rotate the reflector about a second axis while the LiDAR device emits multiple laser pulses toward the reflector. The reflector is configured to provide and receive the multiple laser pulses during one or more rotations about the second axis, and the LiDAR device is further configured to receive the multiple reflected laser pulses and generate the depth information therefrom. The first, second, and third blended images each include a plurality of pixels, wherein each pixel is associated with a digital coordinate that identifies the position of that pixel in the environment, and each reflected laser pulse is associated with a corresponding digital coordinate that identifies the position of the depth information generated by that reflected laser pulse. The digital device is further configured to combine the depth information of the environment with the panoramic image of the environment to generate the 3D panoramic image of the environment using at least one set of digital coordinates from the plurality of pixels and at least one set of digital coordinates from the depth information.

9. The device of claim 1, wherein the lens is located substantially at or near the center of the first axis at a parallax-free point, such that as the first motor rotates the image capturing device about the first axis to capture images while the image capturing device is pointing in the first direction, the second direction, and the third direction, parallax effects between the first plurality of images, the second plurality of images, and the third plurality of images are reduced or eliminated.

10. The device according to claim 1, further comprising: Second motor, The depth information capturing device includes a light detection and ranging device, namely a LiDAR device, and a reflector. The second motor is configured to rotate the reflector about a second axis while the LiDAR device emits multiple laser pulses toward the reflector. The reflector is configured to provide the multiple laser pulses and receive corresponding multiple reflected laser pulses in one or more rotations about the second axis. The LiDAR device is also configured to receive the multiple reflected laser pulses and generate the depth information therefrom. The device is further configured such that as the image capturing device and the LiDAR device rotate about the first axis by the first motor and the reflector rotates about the second axis by the second motor, the LiDAR device captures the depth information in three dimensions of the environment.

11. A method for acquiring image and depth information in an environment using an apparatus, wherein the apparatus includes a first motor, an image capturing device, and a depth information capturing device, the method comprising: The image capturing device, which is directed in a first direction, captures a first plurality of images of the environment at different exposures. The image capturing device includes a lens attached to the frame of the device at a first position, which is located between the front and rear sides of the frame of the device. The first motor rotates the depth information capturing device and the image capturing device around the first axis until the image capturing device points to the second direction, and the rotating depth information capturing device captures the first depth information of the first part of the environment; The image capturing device, pointing in the second direction, captures a second plurality of images that overlap with one or more portions of the first plurality of images; The first motor rotates the depth information capturing device and the image capturing device around the first axis until the image capturing device points to a third direction, and the rotating depth information capturing device captures the second depth information of the second part of the environment; and The image capturing device, directed towards the third party, captures a third plurality of images at different exposures that overlap with one or more portions of the second plurality of images.

12. The method of claim 11, further comprising: The first motor rotates the depth information capturing device and the image capturing device around the first axis until the image capturing device points to the fourth direction; and The image capturing device, pointing in the fourth direction, captures a fourth plurality of images that overlap with one or more portions of the first plurality of images and the third plurality of images at different exposures.

13. The method of claim 12, further comprising, while the depth information capturing device and the image capturing device are rotated by the first motor about the first axis until the image capturing device points to the fourth direction, the depth information capturing device captures third depth information of a third portion of the environment.

14. The method of claim 13, wherein capturing depth information of the first portion, the second portion, and the third portion of the environment by the depth information capturing device includes capturing depth information of a first plurality of segments, a second plurality of segments, and a third plurality of segments of the environment by the depth information capturing device.

15. The method of claim 11, further comprising: The first plurality of images, the second plurality of images, and the third plurality of images are provided to the digital device; The digital device stitches together one or more of the first plurality of images, the second plurality of images, and the third plurality of images to form a panoramic image of the environment; and The digital device combines the first depth information, the second depth information, and the third depth information of the first and second portions and the third portion of the environment with the panoramic image of the environment to generate a 3D panoramic image of the environment, wherein the depth information capturing device captures the third depth information of the third portion of the environment while rotating the depth information capturing device about the first axis.

16. The method of claim 15, further comprising: mixing two or more of the plurality of images of the first plurality of images, the second plurality of images, and the third plurality of images by the digital device to generate a first mixed image, a second mixed image, and a third mixed image, wherein stitching one or more of the plurality of images of the first plurality of images, the second plurality of images, and the third plurality of images together by the digital device to form the panoramic image of the environment includes stitching the first mixed image, the second mixed image, and the third mixed image together to form the panoramic image of the environment.

17. The method of claim 11, wherein the image capturing device further comprises a second motor, and the depth information capturing device comprises a light detection and ranging device, i.e., a LiDAR device, and a reflector, and the method further comprises: The second motor rotates the reflector about the second axis. The LiDAR device emits multiple laser pulses toward the reflector; The reflector provides the plurality of laser pulses and receives the corresponding plurality of reflected laser pulses during one or more rotations about the second axis; as well as The LiDAR device receives the plurality of reflected laser pulses and generates the depth information from the plurality of reflected laser pulses.

18. The method of claim 16, wherein the image capturing device further comprises a second motor, and the depth information capturing device comprises a light detection and ranging device, i.e., a LiDAR device, and a reflector, and the method further comprises: The second motor rotates the reflector about the second axis. The LiDAR device emits multiple laser pulses toward the reflector; The reflector provides the plurality of laser pulses and receives the corresponding plurality of reflected laser pulses during one or more rotations about the second axis; as well as The LiDAR device receives the plurality of reflected laser pulses and generates the depth information therefrom; The first, second, and third blended images each include a plurality of pixels, wherein each of the plurality of pixels is associated with a digital coordinate that identifies the position of that pixel in the environment, and Each of the reflected laser pulses is associated with a corresponding digital coordinate that identifies the location of the depth information generated by the reflected laser pulse, and the combination of the depth information of the environment with the panoramic image of the environment by the digital device to generate the 3D panoramic image of the environment includes: combining the depth information of a location in the environment with the panoramic image of that location in the environment to generate the 3D panoramic image of the environment using at least one set of digital coordinates from the plurality of pixels and at least one set of digital coordinates from the depth information.

19. The method of claim 11, wherein the lens is substantially located at or near the center of the first axis at a parallax-free point, such that as the first motor rotates the image capturing device about the first axis to capture images while the image capturing device is pointing toward the first direction, the second direction, and the third direction, parallax effects between the first plurality of images, the second plurality of images, and the third plurality of images are reduced or eliminated.

20. The method of claim 11, wherein the image capturing device further comprises a second motor, and the depth information capturing device comprises a light detection and ranging device, i.e., a LiDAR device, and a reflector, and the method further comprises: The second motor rotates the reflector about the second axis. The LiDAR device emits multiple laser pulses toward the reflector; The reflector provides the plurality of laser pulses and receives the corresponding plurality of reflected laser pulses during one or more rotations about the second axis; and The LiDAR device receives the plurality of reflected laser pulses and generates the depth information from the plurality of reflected laser pulses, such that as the image capturing device and the LiDAR device rotate about the first axis by the first motor and the reflector rotates about the second axis by the second motor, the LiDAR device captures the depth information in three dimensions of the environment.

Images