Systems and methods for simulating flight of light

Abstract

The present invention describes systems and methods related to simulated light flight imaging. A computing device can execute a light flight engine to process an optical image and a corresponding depth map to produce a light flight image or video that simulates the propagation of a wavefront across a scene. The light flight engine can generate an optical simulation output by transforming the depth map into a three-dimensional lifting domain to produce a three-dimensional data structure, and then convolving the three-dimensional data structure with a convolution operator that can define a Gaussian sphere to produce a filtered three-dimensional data structure. The light flight engine then affine transforms the optical image with the identified slices of the filtered three-dimensional data structure to produce a light flight image.

Description

Background Art

[0001] Conventional affordable camera systems are only capable of capturing static optical images of a scene. Such conventional systems are unable to capture the complex paths of light packets as they travel through space and bounce off objects in a given scene, and therefore are unable to capture images or videos depicting such light packets in motion across such a scene. The concept of capturing images of the paths that light packets take as they move across objects in a scene is sometimes referred to as "light flight" imaging. For conventional camera systems to perform light flight imaging, the system requires relatively expensive sensitive time-stamping hardware, and further requires ultra-sensitive cameras that operate at trillions of frames per second. Therefore, due to these costs and equipment complexities, the hardware required for light flight imaging cannot be easily commercialized. Summary of the invention

[0002] In an example, a method includes:

[0003] generating an optical simulation output based on a depth map corresponding to the optical image, the optical simulation output simulating a light scattering effect of a simulated wavefront; and producing a light flight image by adjusting the brightness of a subset of pixels of the optical image based on the optical simulation output and based on the position of the simulated wavefront at a time corresponding to a defined timestamp.

[0004] According to a further example, a method may include one or more (eg, all) of the following features (or any combination thereof).

[0005] Generating optical simulation outputs can include:

[0006] generating a three-dimensional data structure based on the depth map; and

[0007] The three-dimensional data structure is convolved using a convolution operator to generate a filtered three-dimensional data structure.

[0008] Generating a three-dimensional data structure may include:

[0009] For a given pixel of the depth map, the given pixel is mapped to a corresponding position of the three-dimensional data structure based on the two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.

[0010] The convolution operator can define a Gaussian sphere.

[0011] The method may further comprise:

[0012] Identifying includes filtering a slice of a subset of voxels of the three-dimensional data structure, the slice centered at a wavefront depth corresponding to a position of a simulated wavefront at a time corresponding to a defined timestamp.

[0013] Adjusting the brightness of a subset of pixels of the optical image based on the optical simulation output may include:

[0014] An affine transformation is performed on the optical image using the identified slices.

[0015] The method may further comprise:

[0016] A wavefront depth is determined based on the speed of light and based on the time corresponding to the defined timestamp, wherein the wavefront depth indicates a position of a simulated wavefront in the optical image at the time corresponding to the defined timestamp.

[0017] In another example, a mobile device includes:

[0018] an optical camera configured to capture an optical image;

[0019] a depth camera configured to capture a depth map corresponding to the optical image;

[0020] At least one processor configured to:

[0021] generating an optical simulation output based on the depth map, the optical simulation output simulating a light scattering effect of a simulated wavefront; and

[0022] The brightness of a subset of pixels of the optical image is adjusted based on the optical simulation output and based on the position of the simulated wavefront at a time corresponding to the defined timestamp to produce a light flight image.

[0023] According to further examples, the mobile device may include one or more (eg, all) of the following features (or any combination thereof). The mobile device and / or components included within the mobile device may be adapted to perform the following features.

[0024] To generate an optical analog output, the processor may be further configured to:

[0025] generating a three-dimensional data structure based on the depth map; and

[0026] The three-dimensional data structure is convolved with a convolution operator to generate a filtered three-dimensional data structure, wherein the optical simulation output includes the filtered three-dimensional data structure.

[0027] To generate the three-dimensional data structure, the processor may be configured to:

[0028] For a given pixel of the depth map, the given pixel is mapped to a corresponding position of the three-dimensional data structure based on the two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.

[0029] The convolution operator can define a Gaussian sphere.

[0030] The processor can be configured to:

[0031] Identifying includes filtering a slice of a subset of voxels of the three-dimensional data structure, the slice centered at a wavefront depth corresponding to a position of a simulated wavefront at a time corresponding to a defined timestamp.

[0032] To adjust the brightness of a subset of pixels of the optical image based on the optical simulation output, the processor may be configured to:

[0033] The optical image is affine transformed with the identified slices to produce a light-flight image.

[0034] The processor can be further configured to:

[0035] A wavefront depth is determined based on the speed of light and based on the time corresponding to the defined timestamp, wherein the wavefront depth indicates a position of a simulated wavefront in the optical image at the time corresponding to the defined timestamp.

[0036] In another example, a method includes:

[0037] generating a plurality of light flight image frames based on the optical image, a depth map corresponding to the optical image, and one or more timing parameters; and

[0038] Multiple light flight image frames are combined into a light flight video, wherein the light flight video simulates the propagation of a wavefront of a light pulse across a scene depicted in the optical image.

[0039] According to a further example, a method may include one or more (eg, all) of the following features (or any combination thereof).

[0040] Generating a light flight image frame in the plurality of light flight image frames may include:

[0041] Generate a three-dimensional data structure based on the depth map;

[0042] convolving the three-dimensional data structure using a convolution operator to generate a filtered three-dimensional data structure; and

[0043] The brightness of a subset of pixels of the optical image is adjusted based on the optical simulation output to generate a light-flight image frame, wherein the optical simulation output includes filtering the three-dimensional data structure.

[0044] Generating a three-dimensional data structure may include:

[0045] For a given pixel of the depth map, the given pixel is mapped to a corresponding position of the three-dimensional data structure based on the two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.

[0046] The convolution operator can define a Gaussian sphere.

[0047] Generating a light flight image frame in the plurality of light flight image frames may further include:

[0048] Identifying includes filtering a slice of a subset of voxels of the three-dimensional data structure, the slice centered at a wavefront depth corresponding to a position of a wavefront in a scene depicted by the optical image at a time corresponding to a defined timestamp.

[0049] Adjusting the brightness of a subset of pixels of the optical image based on the optical simulation output may include:

[0050] The identified slices are affine transformed using the optical simulation output to produce light flight image frames.

[0051] Generating a light flight image frame in the plurality of light flight image frames may further include:

[0052] A wavefront depth is determined based on the speed of light and based on the time corresponding to the defined timestamp, wherein the wavefront depth indicates a position of the wavefront in a scene depicted by the optical image at the time corresponding to the defined timestamp.

[0053] In another example, a computer program product may store thereon instructions that, when executed by a processor, may cause the processor to perform the steps of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present disclosure may be better understood, and its numerous features and advantages become apparent to those skilled in the art by referring to the accompanying drawings.The use of the same reference symbols in different drawings indicates similar or identical items.

[0055] Figure 1 is a block diagram of a mobile device that executes a light flight engine according to some embodiments.

[0056] Figure 2 is a sequence of depth images according to some embodiments, the sequence depicting positions of wavefront pixels corresponding to different timestamps as the wavefront traverses the scene depicted by the depth images.

[0057] Figure 3 Schematic diagram of a short pulsed beam from a laser, illustrating the beam broadening effect that occurs as the beam leaves the laser.

[0058] Figure 4 It depicts the Figure 2 Schematic diagram of the Gaussian light spot formed when a light beam is incident on an object.

[0059] Figure 5 is a schematic diagram illustrating conversion of a two-dimensional depth map to a three-dimensional lifting domain according to some embodiments.

[0060] Figure 6is a schematic diagram illustrating convolving a two-dimensional depth map in the lifting domain with a convolution operator that accounts for the effects of short pulse scattering, beam expansion, and light dispersion, according to some embodiments.

[0061] Figure 7 is a flow chart of a method of generating a light flight video according to some embodiments.

[0062] Figure 8 is a flow chart of a method of generating a light flight image according to some embodiments. DETAILED DESCRIPTION

[0063] Embodiments of the present disclosure provide systems and methods by which light flight imaging can be simulated using a camera system of a smartphone or other device. For example, the light flight effect can be simulated using a light flight image that uses a red-green-blue (RGB) image (sometimes referred to herein as an "optical image") and a depth image (sometimes referred to herein as a "depth map") pair as input, which can be captured using an RGB camera (sometimes referred to herein as an "optical camera") and a depth camera of a device (such as a smartphone, a tablet computer, a pair of smart glasses, or another applicable device that includes such a camera).

[0064] In some embodiments, a device may simulate light flight imaging of a wavefront that corresponds to a simulated emission of light pulses from the device or from an object (e.g., a light source) in an augmented reality (AR) scene. For example, when generating a light flight image frame based on a captured RGB image, the light flight engine calculates the position of the wavefront in the scene depicted by the RGB image at a particular time based on depth information obtained from a depth map corresponding to the RGB image and based on timing parameters defining the timing of the wavefront moving across the scene (such as the speed at which the wavefront traverses the scene and / or the time period over which the wavefront traverses the scene). In some embodiments, to improve the perceptibility of the wavefront and to better simulate how light packets of the wavefront bounce off objects of the scene, the light flight engine may transform the RGB image by generating an optical simulation of the wavefront (such as by applying a convolution operator to the wavefront pixels of the corresponding depth map and then affine transforming the original RGB image with the output of the optical simulation to produce the light flight image frame). Affine transforming the original RGB image with the output of the optical simulation modifies the brightness of the wavefront pixels and the pixels within the influence area of ​​the wavefront pixels in three-dimensional space, thereby improving the perceptibility of the wavefront pixels.

[0065] In some embodiments, the optical simulation can be generated at multiple sequential timestamps and applied to the original RGB image to generate a light flight video effect. For example, since the wavefront moves across a given scene over time, the light flight engine can generate light flight image frames for multiple timestamps so that the sequence of light flight image frames depicts the movement of the wavefront across the original RGB image scene. The light flight engine can then combine the sequence of light flight image frames to produce a light flight video or animated graphics interchange format (GIF) file, which can be stored on a storage device (e.g., a solid-state drive, a hard drive, a flash drive, a memory, etc.) that is included in or coupled to a device executing the light flight engine.

[0066] Simulating light flight imaging in this manner enables a variety of applications. For example, the light flight engine can be used to apply unique photo filters to captured images, allowing photographers to introduce new simulated lighting to the captured scene. As another example, the light flight engine can be applied in an AR scene to simulate the movement of light packets from a selected light source in such an AR scene, which may be required for its aesthetic visual effects or may be used to teach students about the mechanics of light paths.

[0067] Figure 1 1 is a block diagram illustrating a mobile device 100 configured for simulated light flight imaging. As shown, the mobile device 100 includes one or more RGB cameras 102 (sometimes referred to herein as "optical cameras 102"), one or more depth cameras 104, one or more processors 106, a display 108, and one or more storage devices 110. Each of the components 102, 104, 106, 108, and 110 is coupled (physically, communicatively, and / or operatively) via a communication channel 120. In some examples, the communication channel 120 may include a system bus, a network connection, an inter-process communication data structure, or any other method of transferring data.

[0068] The processor(s) 106 may be configured to implement functionality and / or process instructions for execution. For example, the processor(s) 106 may be capable of processing instructions stored on the storage device(s) 110. The processor(s) 106 may include any or each of the following: a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or equivalent logic circuitry.

[0069] The storage device(s) 110 may be configured to store information within the mobile device 100 during operation. For example, in some embodiments, the storage device(s) 110 include one or more computer-readable storage media, computer-readable storage devices, temporary storage devices, and / or volatile memory, such as random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), or other suitable volatile memory. In some examples, the storage device(s) are used to store program instructions for execution by the processor(s) 106. In some examples, the storage device(s) may further include one or more non-volatile storage elements, such as in the form of a magnetic hard disk, optical disk, floppy disk, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable (EEPROM) memory. In some embodiments, the storage device(s) 110 is used by software or applications (e.g., light flight engine 112) running on the mobile device 100 to temporarily store information during program execution.

[0070] The RGB camera(s) 102 may include one or more cameras including a standard image sensor through which a visual (e.g., color) image of a scene of a person or object is acquired. The optical images captured by the RGB camera(s) 102 may be stored in the storage device(s) 110. In this document, the optical images captured by an RGB camera (such as the RGB camera(s) 102) may sometimes be referred to as "RGB images."

[0071] The depth camera(s) 104 can capture a depth map of a scene that defines the distance between the mobile device 100 and some or all objects in the scene. In some embodiments, for each RGB image captured by the RGB camera(s) 102, the depth camera(s) 104 can capture a corresponding depth map. In some embodiments, the depth camera(s) 104 (sometimes referred to as "range cameras") can include one or more time-of-flight (ToF) depth cameras that use pulsed light or continuous wave amplitude modulation to illuminate a scene with infrared light and measure the ToF of the infrared light. ToF can be characterized as a time period spanning from the time when the infrared light source of the depth camera emits infrared light to the time when the infrared light is reflected back by an object in the scene and then returns to the infrared light sensor of the depth camera. It should be understood that according to various embodiments, other types of depth cameras, such as stereo triangulation cameras and coded aperture cameras, can be additionally or alternatively included in the depth camera(s) 104.

[0072] The mobile device 100 may include Figure 1Additional components are not shown in the figure to avoid clutter. For example, the mobile device 100 may include a battery for providing power to the components of the mobile device 100.

[0073] exist Figure 1 In an example of , the storage device(s) 110 may store RGB images 114 captured by the RGB camera(s) 102 and depth maps 116 captured by the depth camera(s) 104. The storage device(s) 110 may include instructions for executing a light flight engine 112 with one or more processors 106. The light flight engine 112 receives the RGB images 114 and the corresponding depth maps 116, and generates light flight images 118 based on the RGB images 114 and the depth maps 116. For example, for each RGB image 114, the light flight engine 112 generates a sequence of light flight images 118 that depicts a wavefront moving across a scene of the RGB image. In this document, the term "wavefront" or "simulated wavefront" generally corresponds to a simulated emission of short light pulses. In some embodiments, the light flight engine 112 may simulate the emission of such short light pulses from a source located at the depth camera(s) 104 or the RGB camera 102. In some embodiments, the light flight engine 112 may simulate the emission of such short light pulses from a source disposed at a location within the scene depicted by the RGB image or at a location outside the scene. In some embodiments, the light flight engine 112 may simulate the emission of such short light pulses from a source at a location in the environment, which may correspond to a physical object in the environment or may correspond to a virtual object associated with the location as part of an augmented reality scene or a virtual reality scene.

[0074] When simulating light flight imaging of a wavefront of a light pulse moving across a scene of one of the RGB images 114 over a given period of time, the light flight engine 112 can determine which pixels in the RGB image 114 and which pixels in the corresponding depth map 116 correspond to the depth of the wavefront at a given time after the emission of the corresponding light pulse based on the pixel depth information defined in the depth map 116. For example, like the RGB image, the depth map includes an array of pixels, each with a corresponding pixel value. However, in addition to representing a color value, each pixel of the depth map represents a corresponding depth value that corresponds to the distance between the depth camera that captured the depth map and the portion of the scene corresponding to the pixel (i.e., an "object patch").

[0075] Each depth value of the depth map is a function of the time it takes for the light to reach the depth camera multiplied by the speed of light (e.g., d=c*t, where d is the distance, c is the speed of light, and t is time). This means that the light flight engine can determine the amount of time it takes for the wavefront to travel to a position corresponding to a particular depth in the scene based on the speed of light. Therefore, when simulating a wavefront traversing a scene of an RGB image, the light flight engine 112 uses the corresponding depth map of the scene to determine the position of the wavefront in the scene at a given time after the simulated emission of the wavefront. That is, pixels of the RGB image that correspond to the position of the wavefront at a given time during the simulated wavefront's traversal of the scene can be identified by the light flight engine 112 based on the depth values ​​of the corresponding pixels of the depth map 116 associated with the RGB image 114.

[0076] In some embodiments, the light flight engine 112 can determine a time period based on defined timing parameters stored in the storage device 110 during which the wavefront will be simulated to traverse the scene in a given RGB image. In some embodiments, the light flight engine 112 can generate timestamps corresponding to discrete times occurring within the determined time period, and can generate corresponding light flight images for each timestamp or a subset of timestamps. In this document, "timestamp" refers to a record of discrete times occurring within a time period during which the propagation of the wavefront across the scene is simulated or will be simulated. In some embodiments, the light flight engine 112 can generate a light flight video by combining a sequence of generated light flight images (e.g., time-ordered light flight images).

[0077] Alternatively, in some embodiments, the light flight engine 112 can generate a single light flight image that corresponds to a single timestamp within the determined time period, which timestamp can be defined in the timing parameters or defined by user input provided by a user of the mobile device 100. For example, in some photography applications, it may be desirable to generate only one light flight image from an RGB image at a defined timestamp. It should be understood that in some embodiments, in addition to the mobile device that captures the RGB image and the corresponding depth map, the computing device can execute the light flight engine 112 after receiving the RGB image and the depth map. Therefore, the embodiments of the light flight engine described herein are not limited to being executed by a mobile device, or specifically by a mobile device that captures the image processed by the light flight engine.

[0078] In order to visualize the position of the wavefront at a given time after the wavefront is emitted, depending on the desired effect, the light flight engine 112 can increase the brightness of each identified pixel (sometimes referred to herein as a "wavefront pixel") relative to the remaining pixels of the RGB image (sometimes referred to herein as "non-wavefront pixels"), or can reduce the brightness of each non-wavefront pixel in the RGB image relative to the brightness of each wavefront pixel.

[0079] Figure 2 A sequence 200 of three different depth images corresponding to three different timestamps of a simulated wavefront traversing a scene is depicted, wherein each depth image indicates pixels in the scene corresponding to a location (e.g., depth) of the simulated wavefront at the corresponding time. In the present example, the light flight engine 112 simulates the wavefront as if it were a short pulse emitted from a mobile device that has captured images and corresponding depth maps. The sequence 200 includes: a first depth image 202 corresponding to a first time and including a first wavefront pixel set 204; a second depth image 206 corresponding to a second time and including a second wavefront pixel set 208; and a third depth image 210 corresponding to a third time and including a third wavefront pixel set 212, wherein the first time occurs before the second time, the second time occurs before the third time, and the first time, the second time, and the third time occur during a time period of the simulated wavefront traversing the scene depicted by the depth images 202, 206, and 210 and the corresponding RGB images, respectively. In some embodiments, pixels of depth images 202, 206, and 210 with lower pixel values ​​correspond to objects that are closer to the mobile device, while pixels with higher pixel values ​​correspond to objects that are farther away from the mobile device. That is, the value of each pixel of a given depth image 202, 206, 210 corresponds to the depth of the pixel in the scene depicted by the depth map and the depth of the corresponding pixel in the corresponding RGB image. Therefore, the pixel value of a pixel of a depth map may be referred to as a "depth value" in this article. Pixels of a depth map with the same pixel value are approximately the same distance from the mobile device. It should be understood that wavefront pixels 204, 208, and 212 have been darkened to improve their visibility, and therefore, the depth of wavefront pixels 204, 208, and 212 is not determined by the pixel value. Figure 2 The diagrams are indicated by their color or brightness.

[0080] To identify which pixels of the depth map are wavefront pixels at a given time, the light flight engine 112 identifies a depth value corresponding to the time and identifies all pixels in the depth map having the identified depth value as wavefront pixels. In the present example, the wavefront pixel 204 is disposed at a position closer to the mobile device than the wavefront pixel 208, and the wavefront pixel 208 is disposed at a position closer to the mobile device than the wavefront pixel 212, so that the wavefront is shown as moving away from the mobile device over time.

[0081] Return to Figure 1For example, while in some embodiments, the light flight engine 112 may simply brighten the wavefront pixels identified in each light flight image, such brightening may be difficult to perceive, depending on the scene depicted by the original RGB image. Additionally, simply brightening the identified wavefront pixels does not take into account the light scattering that may occur when the wavefront hits various objects within the scene, the beam broadening that occurs when the wavefront propagates across the scene, and the light dispersion that occurs when the wavefront propagates across the scene. Therefore, in some embodiments, the perceptibility of the simulated wavefront and the accuracy of the simulation can be improved by simulating how light packets of the wavefront are reflected or scattered by objects of the scene (e.g., light scattering) and by applying a convolution operator to the wavefront pixels to simulate the effect of beam broadening. For example, the light flight engine 112 can generate a light flight image for the RGB image, simulating the pulse wavefront at a given timestamp, by generating an optical simulation of the wavefront and performing an affine transformation on the original RGB image with the output of the optical simulation. When generating the optical simulation, the light flight engine 112 can apply a convolution operator to the three-dimensional transformation of the pixels of the depth map to produce a simulated output that approximates the non-ideal factors of the wavefront propagating across the scene. Slices of the simulated output centered at the wavefront depth at a given timestamp can then be affine transformed with the original RGB image to produce a light flight image frame. For example, the convolution operator can approximate the effects of scattering of short light pulses, beam broadening, and light dispersion in space.

[0082] For example, Figure 3 How short pulses of light 304 output by laser 302 propagate over time is illustrated. As shown, the cross-sectional area 306 of light 304 can expand outward along the x-axis and y-axis over time. When light 304 travels away from laser 302, it may experience various non-ideal factors, such as scattering, beam broadening and dispersion. The effect of this non-ideal factor on the amplitude of light 304 in the x, y and t directions can be defined as a corresponding Gaussian function. Here, the x-axis and y-axis define a plane transverse to the propagation vector of the emitted light 304. The t-axis is orthogonal to the transverse plane, parallel to the propagation vector of light 304, and represents both the time and distance traveled by light 304.

[0083] For example, a pulse function (represented here as a vector v) that defines the amplitude of light 304 over time T ) can be approximated by a Gaussian function because it is not precisely located in time. Additionally, at a given point along its transverse plane, light 304 can define a Gaussian light spot 308. Figure 4 A graph 400 is depicted that illustrates the beam amplitude of a Gaussian light spot 308 in a transverse plane. As shown, the amplitude of the light 304 can be approximated by corresponding Gaussian functions in the x and y directions. The amplitude of the light 304 in the x direction is represented here as the vector v x , and the amplitude of the light 304 in the y direction is represented here as vy Therefore, the amplitude of light 304 in three-dimensional space can be approximated according to equations 1 to 4:

[0084] v t =G(t) Equation 1

[0085] v x =G(x) Equation 2

[0086] v y =G(y) Equation 3

[0087]

[0088] where G(t), G(x), and G(y) are Gaussian functions on the t-axis, x-axis, and y-axis, respectively, and H is the vector v x 、v y and v t The outer product of makes H represent a three-dimensional matrix. Return to Figure 1 For example, by applying H of Equation 4 as a convolution operator to the pixels of the depth map associated with the original RGB image to produce an optical simulation output, the light flight engine 112 can simulate non-ideal factors such as scattering of quanta of the simulated pulse wavefront, beam broadening and dispersion (e.g., as Figure 3 (shown as light 304).

[0089] When generating optical simulation output for a given RGB image, the light flight engine 112 may transform the corresponding depth map of the RGB image into a three-dimensional data structure, such as a cubic data structure, referred to herein as a transformation to a “lifting domain,” wherein pixels are transformed into voxels having a position in space defined by the position of the original pixel in the depth map and the depth value of the original pixel relative to a sparsely populated cube. Figure 5 An example of the transformation of a depth map from a two-dimensional domain to a three-dimensional lifting domain is shown. As shown, the two-dimensional depth map 512 initially includes an array of pixels, each of which defines a corresponding depth value D i,j, where i and j define the coordinates of a given pixel in the array. In the present example, the light flight engine 112 transforms the depth map 512 to the lifting domain, where the previous two-dimensional depth map 512 is represented as a three-dimensional cubic data structure 520. The cubic data structure 520 is sparsely populated with three-dimensional pixels (i.e., voxels) 524. A subset of voxels 524 corresponding to the wavefront depth at a given timestamp may be referred to herein as "wavefront voxels". After transformation to the lifting domain, the position of a given voxel relative to the x-axis and y-axis is maintained, while the position of the voxel along the new third axis (t-axis or "time axis") is determined based on the depth value defined by the corresponding depth map pixel. For example, the depth value of a given pixel divided by the speed of light defines the position of the corresponding voxel along the t-axis because, as described above, d=c*t. In some embodiments, the third axis may explicitly define depth, rather than time.

[0090] After transforming the depth map to the lifting domain, the light flight engine 112 applies a convolution operator to the voxels of the resulting cubic data structure. Figure 6 The convolution operator 610 is illustrated as being applied to a voxel 624 of a cubic data structure 620. The cubic data structure 620 may correspond to a lifting domain transform of a two-dimensional depth map corresponding to an RGB image / depth map pair. In the cubic data structure 620, the position of each voxel 624 may be defined, for example, by assigning a wavefront voxel 624 to 1 or another non-zero value or assigning all other voxels of the cubic data structure 620 to zero. The convolution operator 610 may define a Gaussian sphere 612, for example, which corresponds to the convolution operator H of Equation 4 and may therefore be defined by corresponding Gaussian functions along the x-axis, the y-axis, and the t-axis. The light flight engine 112 convolves the convolution operator 610 with the cubic data structure 620 to produce a filtered cubic data structure 630 including Gaussian spheres 634, each centered at the position of a corresponding voxel in the voxel 624. For example, the size and amplitude profile of each Gaussian sphere 634 of the filter cube data structure 630 can be the same. In some embodiments, the dimensions of the Gaussian sphere 634 along the x-axis, y-axis, and t-axis can be customized by the user to achieve different visual effects when simulating the wavefront. By convolving the convolution operator 610 with the voxels 624 of the cubic data structure 620, the voxel values ​​around each voxel 624 are increased according to the distribution defined by the Gaussian sphere 634. For example, according to the (multiple) distributions of (multiple) Gaussian spheres defined by H, the amount by which the amplitude of a given non-wavefront voxel is increased depends on its proximity to one or more voxels 624, wherein the closer the affected voxel is to the voxel 624, the greater its amplitude. In this article, the filter cube data structure 630 is considered to be a simulated output.

[0091] When generating a light flight video, the light flight engine 112 may logically divide the filter cube data structure 630 into a number of slices, each slice being centered at a different wavefront depth associated with a corresponding timestamp (e.g., where each wavefront depth corresponds to the position of the wavefront at the associated timestamp), and each having a width corresponding to the diameter H of the t-dimension (i.e., along the t-axis). It should be understood that in some embodiments, the slices may overlap one another along the t-axis. The light flight engine 112 may then perform an affine transform on each slice of the filter cube data structure 630 with the original RGB image to generate a sequence of light flight image frames. The light flight engine 112 may then combine the light flight image frames to produce a light flight video.

[0092] For example, return to Figure 1 After generating the optical simulation output, the light flight engine 112 may perform an affine transform on the original RGB image with the slices of the optical simulation output to generate the light flight image. Equation 5 shows an example of an affine transform that may be performed by the light flight engine 112 to generate a sequence of light flight images.

[0093]

[0094] Where S(I) is the set of light flight images output by the light flight engine 112 for the original RGB image, I, D are depth maps corresponding to the original RGB image, Y is the optical simulation output, H is the convolution operator, X is a cubic data structure corresponding to the depth map, F(D) is a function used to transform the depth map to the lifting domain to produce the cubic data structure X, α is a constant that determines the weight assigned to the simulated output (e.g., the amount by which the brightness of the affected pixels of the original RGB image is increased), k is the index of the sequence of light flight images, and N is the number of light flight images in the sequence. For example, each affine transformation of the original RGB image performed by the light flight engine 112 effectively adjusts the brightness of a pixel of the original RGB image where light from the wavefront would be directly incident (i.e., the wavefront pixel) and where scattered light from the wavefront would be incident at a given timestamp (i.e., approximated by the convolution operator used to generate the optical simulation output). Although Equation 5 indicates the generation of a sequence of light flight images, it should be understood that Equation 5 can also be used to generate a single light flight image.

[0095] The light flight engine 112 may be configured to generate a light flight video by generating a sequence of light flight frames (e.g., as provided in conjunction with Equation 5) that simulate the motion of a wavefront across a scene depicted by the original RGB image. The speed at which the wavefront moves across the scene of such a light flight video (i.e., a "wavefront speed") may be predefined, may be defined by a user of a device (e.g., mobile device 100) performing the light flight image, or may be automatically defined based on a defined time period (e.g., a predefined or user-defined time period of the light flight video). The light flight engine 112 may determine a sequence of timestamps to be used when generating light flight image frames for the light flight video based on one or more predefined or user-defined timing parameters, which may include the wavefront speed, a desired light flight video length, and / or a predefined sampling rate. In some embodiments, the light flight engine 112 may determine a number of light flight frames to be generated based on the depth of the scene depicted by the RGB image and one or more timing parameters. With respect to light flight video, it should be understood that the discrete movement of a wavefront across a scene at the speed of light is generally not visible to the human eye or cannot be reproduced on a conventional display, and therefore, the light flight engine 112 can generate a light flight video to simulate the wavefront moving across the scene at a fraction of the speed of the actual light wavefront movement or over an extended time period so that the discrete movement of the wavefront is perceptible to an observer. The amount by which the light flight engine 112 modifies the speed of the wavefront or extends the time period during which the wavefront traverses the scene for playing the light flight video can be selected based on one or more defined timing parameters (e.g., these parameters can be stored in (multiple) storage devices 110), which can be preset or user-defined. For example, the timing parameters can include one or more of the following: a wavefront speed, which defines the speed at which the wavefront is to move in the light flight video; a wavefront time period, which defines the time period during which the wavefront traverses the scene in the light flight video and can be defined as the length of the light flight video to be generated; or a frame rate, which defines the number of light flight image frames to be generated within a given time period of the light flight video.

[0096] Figure 7 The process flow of the method 700 for generating a light flight video is shown. For example, the method 700 can be performed by a mobile device having an RGB camera and a depth camera. In the current example, the method 700 is performed in Figure 1 The present invention is described in the context of a mobile device 100 .

[0097] At block 702, the RGB camera 102 and the depth camera 104 of the mobile device 100 capture an RGB image and a corresponding depth map. The RGB image and the depth map may be stored in the storage device 110 of the mobile device 100 for subsequent processing.

[0098] At box 704, the light flight engine 112 generates optical simulation output for the RGB image and the depth map by transforming the depth map pixels into a three-dimensional domain (e.g., a cubic data structure) and then applying a convolution operator H to the non-zero voxels of the cubic data structure to produce an optical simulation output (e.g., filtering the cubic data structure).

[0099] At block 706, the light flight engine 112 determines the number N of light flight image frames to generate. In some embodiments, the light flight engine 112 determines the number of light flight image frames to generate based on one or more timing parameters, which may include one or more of: wavefront speed, wavefront time period, frame rate, and / or another applicable timing parameter.

[0100] At block 708, the light flight engine 112 determines a timestamp sequence including a corresponding timestamp for each light flight image frame to be generated, wherein the timestamp of a given light flight image frame determines the position / depth at which the wavefront is simulated in the given light flight image frame. In some embodiments, adjacent timestamps in the sequence may be equally spaced.

[0101] At block 710 , the light flight engine 112 sets a variable k equal to 1, where k represents the index of the next light flight image frame to be generated with respect to the sequence of frames.

[0102] At block 712, the light flight engine 112 generates a kth light flight image frame of the sequence corresponding to the kth timestamp of the timestamp sequence. Figure 8 Blocks 808 and 810 of the method are used to generate each image frame.

[0103] At block 714, the light flight engine 112 determines whether the most recently generated light flight image frame is the last image frame in the sequence of light flight image frames by determining whether k=N-1. If the light flight engine 112 determines that k=N-1, the method 700 proceeds to block 718. Otherwise, if the light flight engine 112 determines that k<N-1, the method 700 proceeds to block 716.

[0104] At block 716 , the light flight engine 112 increases the value of k by 1. The method 700 then returns to block 712 , where the light flight engine 112 generates the next light flight image frame in the sequence.

[0105] At block 718 , the light flight engine 112 combines the sequence of light flight image frames into a light flight video that depicts the scene of the simulated wavefront traversing the RGB image.

[0106] At block 720 , the light flight engine 112 causes the light flight video to be stored on the storage device 110 .

[0107] Figure 8 The process flow of the method 800 for generating a light flight image is shown. For example, when generating an optical simulation output and a light flight image frame for a light flight image, the Figure 7 Some or all of the steps of method 800 may be performed at blocks 704 and 712 of the method. As another example, method 800 may be performed to generate a single independent light flight image. For example, method 800 may be performed by a mobile device having an RGB camera and a depth camera. In the present example, method 800 is performed at Figure 1 The present invention is described in the context of a mobile device 100 .

[0108] At step 802, the light flight engine 112 receives an RGB image, a depth map corresponding to the RGB image, a timestamp, and one or more predefined or user-defined timing parameters. The timestamp defines the time that the wavefront traverses the scene depicted by the RGB image to generate the light flight image, and the position of the wavefront at that time can be obtained from the timestamp. The timing parameters may include one or more of the following: wavefront speed, wavefront time period, frame rate, and / or another applicable timing parameter.

[0109] At step 804, the light flight engine 112 transforms the depth map into the three-dimensional domain. In some embodiments, the light flight engine 112 transforms the depth map into the lifting domain to produce a three-dimensional structure, referred to as a cubic data structure in the present example (e.g., Figure 5 The coordinates of a given voxel in the cubic data structure may be selected based on the coordinates of a corresponding pixel in the original depth map and the depth value of the corresponding pixel.

[0110] At step 806, the light flight engine 112 applies a convolution operator (e.g., the convolution operator H of Equation 4) to the cubic data structure to generate a filter cubic data structure. For example, the filter cubic data structure may include one or more Gaussian spheres, each centered at a non-zero voxel of the cubic data structure.

[0111] At step 808, the light flight engine 112 identifies a slice of the filter cube data structure of the optical simulation output that corresponds to the current timestamp for generating the light flight image frame. For example, the identified slice of the filter cube data structure can be a matrix of a subset of the filter cube data structure. The matrix can be composed of all voxel filter cube data structures within a depth range (i.e., a range of values ​​along the t-axis) centered at a depth corresponding to the current timestamp (i.e., corresponding to the position of the wavefront at the time represented by the timestamp), wherein the size of the depth range corresponds to the diameter of the convolution operator H in the t-direction. At step 812, the light flight engine 112 affine transforms the original RGB image with the identified slice of the optical simulation output to generate the light flight image frame. For example, the affine transformation can increase the brightness of the pixel in the RGB image corresponding to the simulated wavefront incident on the surface at the time of the timestamp, and can increase the brightness of the pixel corresponding to the simulated scattered light from the wavefront incident on the surface at the time of the timestamp.

[0112] In some embodiments, certain aspects of the above-described techniques may be implemented by one or more processors of a processing system that executes software. The software includes one or more sets of executable instructions stored on a non-transitory computer-readable medium or otherwise tangibly embodied. The software may include instructions and certain data that, when executed by one or more processors, manipulate one or more processors to perform one or more aspects of the above-described techniques. For example, a non-transitory computer-readable storage medium may include a disk or optical disk storage device, a solid-state storage device (such as flash memory, cache, random access memory (RAM), or one or more other non-volatile storage devices), etc. The executable instructions stored on a non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executed by one or more processors.

[0113] Computer-readable storage media may include any storage media or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact disks (CDs), digital versatile disks (DVDs), Blu-ray disks), magnetic media (e.g., floppy disks, tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical system (MEMS)-based storage media. Computer-readable storage media may be embodied in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disk or flash memory based on a universal serial bus (USB)), or coupled to a computer system via a wired or wireless network (e.g., a network-accessible storage device (NAS)).

[0114] It is to be noted that not all of the activities or elements described above in the general description are required, that a portion of a particular activity or device may not be required, and that one or more further activities may be performed, or that elements other than those described may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Likewise, the concepts have been described with reference to specific embodiments. However, one of ordinary skill in the art recognizes that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the following claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

[0115] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and any (multiple) features that may cause any good, advantage, or solution to arise or become more obvious should not be construed as key, essential, or essential features of any or all claims. In addition, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter can be modified and practiced in different but effective ways that are obvious to those skilled in the art who have the benefit of the teachings herein. The details of construction or design shown herein are not subject to any restrictions except as described in the following claims. Therefore, it is apparent that the specific embodiments disclosed above may be changed or modified, and all such changes are considered to be within the scope of the disclosed subject matter. Therefore, the protection claimed herein is as described in the following claims.

Claims

1. A method for simulating light flight, the method include: Generating an optical simulation output based on a depth map corresponding to the optical image, the optical simulation output simulating a light scattering effect of a simulated wavefront, wherein generating the optical simulation output comprises: generating a three-dimensional data structure based on the depth map; and convolving the three-dimensional data structure with a convolution operator to generate a filtered three-dimensional data structure; and A light flight image is generated by adjusting the brightness of a subset of pixels of the optical image based on the optical simulation output and based on the position of the simulated wavefront at a time corresponding to a defined timestamp.

2. The method according to claim 1, in, Generating the three-dimensional data structure includes: For a given pixel of the depth map, the given pixel is mapped to a corresponding position of the three-dimensional data structure based on the two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.

3. The method according to claim 1, in, The convolution operator defines a Gaussian sphere.

4. The method according to claim 1, further comprising: include: A slice comprising a subset of voxels of the filtered three-dimensional data structure is identified, the slice centered at a wavefront depth corresponding to the position of the simulated wavefront at a time corresponding to the defined timestamp.

5. The method according to claim 4, in, Adjusting the brightness of the subset of pixels of the optical image based on the optical simulation output includes: An affine transformation is performed on the optical image using the identified slice.

6. The method according to claim 4, further comprising: include: The wavefront depth is determined based on a speed of light and based on the time corresponding to the defined timestamp, wherein the wavefront depth indicates the position of a simulated wavefront in the optical image at the time corresponding to the defined timestamp.

7. A mobile device, include: an optical camera configured to capture an optical image; a depth camera configured to capture a depth map corresponding to the optical image; at least one processor, the at least one processor being configured to: generating an optical simulation output based on the depth map, the optical simulation output simulating a light scattering effect of a simulated wavefront, wherein, to generate the optical simulation output, the processor is further configured to: generating a three-dimensional data structure based on the depth map; and convolving the three-dimensional data structure with a convolution operator to generate a filtered three-dimensional data structure, wherein the optical simulation output includes the filtered three-dimensional data structure; and The brightness of a subset of pixels of the optical image is adjusted based on the optical simulation output and based on the position of the simulated wavefront at a time corresponding to a defined timestamp to produce a light flight image.

8. The mobile device according to claim 7, in, To generate the three-dimensional data structure, the processor is configured to: For a given pixel of the depth map, the given pixel is mapped to a corresponding position of the three-dimensional data structure based on the two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.

9. The mobile device according to claim 7, in, The convolution operator defines a Gaussian sphere.

10. The mobile device according to claim 7, in, The processor is configured to: A slice comprising a subset of voxels of the filtered three-dimensional data structure is identified, the slice being centered at a wavefront depth corresponding to a position of the simulated wavefront at a time corresponding to the defined timestamp.

11. The mobile device according to claim 10, in, To adjust the brightness of the subset of pixels of the optical image based on the optical simulation output, the processor is configured to: The optical image is affine transformed using the identified slice to generate the light flight image.

12. The mobile device according to claim 10, in, The processor is further configured to: The wavefront depth is determined based on the speed of light and based on the time corresponding to the defined timestamp, wherein the wavefront depth indicates the position of the simulated wavefront in the optical image at the time corresponding to the defined timestamp.

13. A method for simulating light flight, the method include: Generating a plurality of light flight image frames based on an optical image, a depth map corresponding to the optical image, and one or more timing parameters, wherein generating a light flight image frame of the plurality of light flight image frames comprises: generating a three-dimensional data structure based on the depth map; convolving the three-dimensional data structure with a convolution operator to generate a filtered three-dimensional data structure; and adjusting brightness of a subset of pixels of the optical image based on an optical simulation output to generate the light-flight image frame, wherein the optical simulation output includes the filtered three-dimensional data structure; and The plurality of light flight image frames are combined into a light flight video, wherein the light flight video simulates the propagation of a wavefront of a light pulse across a scene depicted in the optical image.

14. The method according to claim 13, in, Generating the three-dimensional data structure includes: For a given pixel of the depth map, the given pixel is mapped to a corresponding position of the three-dimensional data structure based on the two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.

15. The method according to claim 13, in, The convolution operator defines a Gaussian sphere.

16. The method according to claim 13, in, Generating the light flight image frame of the plurality of light flight image frames further comprises: A slice comprising a subset of voxels of the filtered three-dimensional data structure is identified, the slice centered at a wavefront depth corresponding to a location of the wavefront in the scene depicted in the optical image at a time corresponding to a defined timestamp.

17. The method according to claim 16, in, Adjusting the brightness of the subset of pixels of the optical image based on the optical simulation output includes: The identified slice is affine transformed using the optical simulation output to generate the light-of-flight image frame.

18. The method according to claim 16, in, Generating the light flight image frame of the plurality of light flight image frames further comprises: The wavefront depth is determined based on the speed of light and based on the time corresponding to the defined timestamp, wherein the wavefront depth indicates the position of the wavefront in the scene depicted in the optical image at the time corresponding to the defined timestamp.

19. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the steps of the method according to any one of claims 1 to 6 and 13 to 18.

Images