System and method for simulating flying lights - Patents.com
The method simulates light scattering effects using depth maps and convolution operators to generate flying light images, addressing the limitations of conventional cameras and enabling realistic light path visualization on devices like smartphones.
Patent Information
- Application Number
- JP2023542998
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-01-15
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2041-01-15
AI Technical Summary
Conventional camera systems are unable to capture the complex paths that light packets take as they travel through space and bounce off objects in a scene, requiring expensive and complex hardware for over-the-air imaging, making commercialization difficult.
A method and system that simulates light scattering effects using depth maps and convolution operators to generate flying light images, allowing devices like smartphones to capture and simulate the movement of light packets across a scene.
Enables the simulation of flying light images on devices like smartphones, providing unique photographic filters and enhancing the perception of light paths, suitable for augmented reality applications and educational purposes.
Smart Images

Figure 0007591662000003 
Figure 0007591662000004 
Figure 0007591662000005
Abstract
Description
[Background technology]
[0001] background Conventional affordable camera systems can only capture still optical images of a scene. Such conventional systems cannot capture the complex paths that light packets take as they travel through space and bounce off objects in a given scene, and therefore cannot capture images or video that represent such light packets moving across such a scene. The concept of capturing images of the paths that light packets take as they move across objects in a scene is sometimes referred to as "light-in-flight" imaging. For a conventional camera system to perform light-in-flight imaging, the system would require sensitive time-tagging hardware that is relatively expensive, and would further require an ultra-sensitive camera that operates at a trillion frames per second. Thus, the hardware required for light-in-flight imaging cannot be easily commercialized due to the cost and equipment complexity. Summary of the Invention [Means for solving the problem]
[0002] overview In one example, the method comprises: generating an optical simulation output simulating light scattering effects for the simulated wavefront based on a depth map corresponding to the optical image; and generating a flight light image by adjusting the brightness of a subset of pixels of the optical image based on the optical simulation output and based on the position of the simulated wavefront at a time corresponding to a defined timestamp.
[0003] According to further examples, the method may include one or more (eg, all) of the following features (or any combination thereof).
[0004] The step of generating an optical simulation output includes: generating a three-dimensional data structure based on the depth map; and convolving the three-dimensional data structure with a convolution operator to generate a filtered three-dimensional data structure.
[0005] The step of generating a three-dimensional data structure includes: For a given pixel of the depth map, the method may include mapping the given pixel to a respective location in the three-dimensional data structure based on the two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.
[0006] The convolution operator may define a Gaussian sphere. The method further comprises: The method may include identifying a slice that includes a subset of voxels of the filtered three-dimensional data structure centered at a wavefront depth corresponding to a position of the simulated wavefront at a time corresponding to a specified timestamp.
[0007] Adjusting the brightness of a subset of pixels of the optical image based on the optical simulation output includes: This may include performing an affine transformation of the optical image using the identified slices.
[0008] The method further comprises: The method may include determining a wavefront depth based on the speed of light and based on a time corresponding to the defined timestamp, the wavefront depth indicating a position of the simulated wavefront in the optical image at a time corresponding to the defined timestamp.
[0009] In another example, the mobile device an optical camera configured to capture an optical image; a depth camera configured to capture a depth map corresponding to the optical image; and at least one processor, the at least one processor comprising: generating an optical simulation output simulating light scattering effects for the simulated wavefront based on the depth map; To generate the flight light image, the system is configured to adjust the brightness of a subset of pixels of the optical image based on the optical simulation output and based on the position of the simulated wavefront at a time corresponding to a defined timestamp.
[0010] According to further examples, a mobile device may include one or more (e.g., all) of the following features (or any combination thereof): The mobile device, and / or components included within the mobile device, may be adapted to perform the following features:
[0011] To generate the optical simulation output, the processor further comprises: Generate a 3D data structure based on the depth map; The optical simulation may be configured to convolve the three-dimensional data structure with a convolution operator to generate a filtered three-dimensional data structure. The optical simulation output includes the filtered three-dimensional data structure.
[0012] To generate the three dimensional data structure, the processor For a given pixel in the depth map, it may be configured to map the given pixel to a respective location in the three-dimensional data structure based on the two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.
[0013] The convolution operator may define a Gaussian sphere. The processor The method may be configured to identify a slice that includes a subset of voxels of the filtered three-dimensional data structure centered at a wavefront depth corresponding to a position of the simulated wavefront at a time corresponding to a specified timestamp.
[0014] To adjust the brightness of a subset of pixels of the optical image based on the optical simulation output, the processor: The identified slices may be used to affinely transform the optical image to generate a flight optical image.
[0015] The processor further comprises: The system may be configured to determine a wavefront depth based on the speed of light and based on a time corresponding to the defined timestamp, the wavefront depth indicating a position of a simulated wavefront in the optical image at a time corresponding to the defined timestamp.
[0016] In another example, the method comprises: generating a plurality of flight light image frames based on the optical images, a depth map corresponding to the optical images, and one or more timing parameters; Assembling the plurality of flying light image frames into a flying light video; Flying light imaging simulates the propagation of a wavefront of a light pulse across a scene represented in an optical image.
[0017] According to further examples, the method may include one or more (eg, all) of the following features (or any combination thereof).
[0018] The step of generating a flying light image frame from among the plurality of flying light image frames includes: generating a three-dimensional data structure based on the depth map; convolving the three-dimensional data structure with a convolution operator to generate a filtered three-dimensional data structure; and adjusting a brightness of a subset of pixels of the optical image based on the optical simulation output to generate a flight light image frame, the optical simulation output including a filtered three-dimensional data structure.
[0019] The step of generating a three-dimensional data structure includes: For a given pixel of the depth map, the method may include mapping the given pixel to a respective location in the three-dimensional data structure based on the two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.
[0020] The convolution operator may define a Gaussian sphere. The step of generating a flying light image frame from among the plurality of flying light image frames includes: The method may further include identifying a slice comprising a subset of voxels of the filtered three-dimensional data structure centered at a wavefront depth corresponding to a position of the wavefront in the scene represented in the optical image at a time corresponding to a specified timestamp.
[0021] The step of adjusting the brightness of a subset of pixels of the optical image based on the optical simulation output comprises: The method may include affinely transforming the identified slices using the optical simulation output to generate a flight light image frame.
[0022] The step of generating a flying light image frame from among the plurality of flying light image frames includes: The method may further include determining a wavefront depth based on the speed of light and based on a time corresponding to the defined timestamp, the wavefront depth indicating a position of the wavefront in the scene represented in the optical image at a time corresponding to the defined timestamp.
[0023] In another example, a computer program product may have instructions stored thereon that, when executed by a processor, may cause the processor to perform method steps as described herein.
[0024] BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference symbols in different drawings indicates similar or identical items. [Brief description of the drawings]
[0025] [Figure 1] FIG. 1 is a block diagram of a mobile device implementing a flight light engine, according to some embodiments. [Diagram 2] 4 is a sequence of depth images representing positions of wavefront pixels corresponding to different timestamps as the wavefront progresses across a scene represented in the depth images, according to some embodiments. [Diagram 3] FIG. 1 is a diagram of a beam output by a laser via a short pulse, showing the beam broadening effect that occurs as the beam travels away from the laser. [Figure 4] 3 is a diagram illustrating the Gaussian spot formed when the beam of FIG. 2 is incident on an object. [Diagram 5] FIG. 2 illustrates the conversion of a 2D depth map to a 3D lift domain in accordance with some embodiments. [Figure 6] FIG. 1 illustrates the convolution of a two-dimensional depth map in the lift domain with a convolution operator that accounts for the effects of short pulse scattering, beam expansion, and optical dispersion, according to some embodiments. [Figure 7] FIG. 2 is a flow diagram of a method for generating flying light images, according to some embodiments. [Figure 8] FIG. 2 is a flow diagram of a method for generating a flying light image, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] Detailed Description The embodiments of the present disclosure provide a system and method in which flying light imaging may be simulated using a camera system of a smartphone or other device. For example, flying light effects may be simulated using flying light imaging that takes as input a pair of a red-green-blue (RGB) image (sometimes referred to herein as an "optical image") and a depth image (sometimes referred to herein as a "depth map"). The images may be captured using an RGB camera (sometimes referred to herein as an "optical camera") and a depth camera, respectively, of a device such as a smartphone, tablet, smart glasses, or another applicable device (including the cameras below).
[0027] In some embodiments, the device may simulate flight light imaging of a wavefront corresponding to a simulated emission of light pulses from the device or from an object (e.g., a light source) in an augmented reality (AR) scene. For example, when generating a flight light image frame based on a captured RGB image, the flight light engine calculates where the wavefront is located in the scene represented in the RGB image at a particular time based on depth information derived from a depth map corresponding to the RGB image and based on timing parameters that define the timing of the movement of the wavefront relative to the scene (e.g., the speed at which the wavefront should move across the scene and / or the period during which the wavefront should move across the scene). In some embodiments, to improve the perceptibility of the wavefront and to better simulate how light packets of the wavefront bounce off objects in the scene, the flight light engine may generate an optical simulation of the wavefront, such as by applying a convolution operator to the wavefront pixels of the corresponding depth map, and then transform the RGB image by affinely transforming the original RGB image with the output of the optical simulation to generate the flight light image frame. Affine transforming the original RGB image with the output of the optical simulation modifies not only the wavefront pixels, but also the luminance of pixels within the wavefront pixels' region of influence in 3D space, thus improving the perceptibility of the wavefront pixels.
[0028] In some embodiments, optical simulations can be generated and applied to the original RGB image over multiple consecutive timestamps to generate a flying light video effect. For example, as a wavefront moves across a given scene over time, the flying light engine can generate a sequence of flying light image frames for multiple timestamps, such that the sequence of flying light image frames represents the movement of the wavefront across the scene in the original RGB image. The flying light engine can then assemble the sequence of flying light image frames to generate, for example, a flying light video or an animated graphics interchange format (GIF) file, which can be stored on a storage device (e.g., solid state drive, hard disk drive, flash drive, memory, etc.) included in or coupled to the device executing the flying light engine.
[0029] Simulating flying light imaging in this manner enables a variety of applications. For example, the flying light engine may be used to apply unique photographic filters to captured images, allowing photographers to introduce simulated new lighting to a captured scene. As another example, the flying light engine may be applied in an AR scene to simulate the movement of light packets from a selected light source in such an AR scene, which may be desirable for its aesthetic visual effect, or which may be used to teach students about the dynamics of light paths.
[0030] 1 is a block diagram illustrating a mobile device 100 configured to simulate flight optical imaging. As illustrated, the mobile device 100 includes one or more RGB cameras 102 (sometimes referred to herein as “optical cameras 102”), one or more depth cameras 104, one or more processors 106, a display 108, and one or more storage devices 110. Each of the components 102, 104, 106, 108, and 110 are coupled (physically, communicatively, and / or operatively) via a communication channel 120. In some examples, the communication channel 120 may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.
[0031] Processor 106 may be configured to implement functionality and / or process instructions for execution. For example, processor 106 may be capable of processing instructions stored on storage device 110. Processor 106 may include any or each of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or equivalent logic circuitry.
[0032] The storage device 110 may be configured to store information within the mobile device 100 during operation. For example, in some embodiments, the storage device 110 includes one or more computer-readable storage media, computer-readable storage devices, temporary memory devices, and / or volatile memories, such as random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), or other applicable volatile memories. In some examples, the storage device is used to store program instructions for execution by the processor 106. In some examples, the storage device may further include one or more non-volatile storage elements, such as a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or a form of electrically programmable memory (EPROM) or electrically erasable and programmable memory (EEPROM). In some embodiments, the storage device 110 is used by software or applications (e.g., the flying light engine 112) executing on the mobile device 100 to temporarily store information during program execution.
[0033] RGB camera 102 may include one or more cameras including a standard image sensor that captures visual images (e.g., color images) of a scene of people and objects. Optical images captured by RGB camera 102 may be stored in storage device 110. As used herein, optical images captured by an RGB camera, such as RGB camera 102, may be referred to as "RGB images."
[0034] The depth camera 104 may capture a depth map of the scene that defines the distance between the mobile device 100 and some or all of the objects in the scene. In some embodiments, for each RGB image captured by the RGB camera 102, the depth camera 104 may capture a corresponding depth map. In some embodiments, the depth camera 104 (sometimes referred to as a "range camera") may include one or more Time-of-Flight (ToF) depth cameras. They use pulsed light or continuous wave amplitude modulation to illuminate the scene with infrared light and measure the ToF of the infrared light. The ToF may be characterized as the period from when infrared light is emitted from the depth camera's infrared light source to when the infrared light returns to the depth camera's infrared light sensor after being reflected off an object in the scene. It should be understood that in addition or instead, other types of depth cameras, such as stereo triangulation cameras and coded aperture cameras, may be included in the depth camera 104 according to various embodiments.
[0035] The mobile device 100 may include additional components not shown in FIG 1 to reduce clutter. For example, the mobile device 100 may include a battery to provide power to the components of the mobile device 100.
[0036] In the example of FIG. 1, the storage device 110 may store an RGB image 114 captured by the RGB camera 102 and a depth map 116 captured by the depth camera 104. The storage device 110 may include instructions for executing the flight light engine 112 using one or more processors 106. The flight light engine 112 receives the RGB image 114 and the corresponding depth map 116, and generates a flight light image 118 based on the RGB image 114 and the depth map 116. For example, for each RGB image 114, the flight light engine 112 generates a sequence of flight light images 118 representing a wave front moving across the scene of that RGB image. In this specification, the term "wave front" or "simulated wave front" generally corresponds to a simulated emission of a short light pulse. In some embodiments, the flight light engine 112 may simulate the emission of such a short light pulse from a source located at the depth camera 104 or the RGB camera 102. In some embodiments, the flying light engine 112 may simulate the emission of such short light pulses from a source located at a position within the scene represented in the RGB image or at a position outside the scene. In some embodiments, the flying light engine 112 may simulate the emission of such short light pulses from a source located at a position in an environment. The source may correspond to a physical object in the environment, or may correspond to a virtual object associated with the position as part of an augmented reality or virtual reality scene.
[0037] When simulating flight light imaging of a wavefront of a light pulse moving across a scene in one of the RGB images 114 over a given time period, the flight light engine 112 can determine which pixels of the RGB image 114 and which pixels of the corresponding depth map 116 correspond to the depth of the wavefront at a given time following the emission of the corresponding light pulse based on the pixel depth information defined in the depth map 116. For example, similar to an RGB image, the depth map includes an array of pixels, each having a respective pixel value. However, rather than representing a color value, each pixel of the depth map represents a respective depth value that corresponds to the distance between the depth camera that captured the depth map and the portion of the scene (i.e., the "object patch") to which the pixel corresponds.
[0038] Each depth value in the depth map is a function of the time it takes light to reach the depth camera multiplied by the speed of light (e.g., d=c * t, where d is distance, c is the speed of light, and t is time. This means that the flight light engine can determine the amount of time it will take for the wave front to travel to a position corresponding to a particular depth in the scene based on the speed of light. Thus, when simulating a wave front traveling across a scene in an RGB image, the flight light engine 112 uses the corresponding depth map of the scene to determine the position of the wave front in the scene at a given time following the simulated emission of the wave front. That is, the pixel of the RGB image corresponding to the position of the wave front at a given time during the simulated traversal of the scene of the wave front can be identified by the flight light engine 112 based on the depth value of the corresponding pixel in the depth map 116 associated with the RGB image 114.
[0039] In some embodiments, the flight light engine 112 may determine a period during which the traversal of a wavefront across a scene in a given RGB image should be simulated based on defined timing parameters stored in the storage device 110. In some embodiments, the flight light engine 112 may generate timestamps corresponding to distinct times occurring within the determined period, and generate a respective flight light image for each timestamp, or for a subset of the timestamps. As used herein, a "timestamp" refers to a record of a distinct time occurring during a period during which the propagation of a wavefront across a scene is or should be simulated. In some embodiments, the flight light engine 112 may generate a flight light video by assembling a sequence of generated flight light images (e.g., flight light images that are consecutive in time).
[0040] Alternatively, in some embodiments, the flight light engine 112 may generate a single flight light image corresponding to a single time stamp within a determined period, which may be defined in the timing parameters or via user input provided by a user of the mobile device 100. For example, it may be desirable to generate only one flight light image from an RGB image at a defined time stamp in some photo applications. It should be understood that in some embodiments, a computing device other than the mobile device that captured the RGB image and the corresponding depth map may execute the flight light engine 112 after receiving the RGB image and the depth map. Thus, the flight light engine embodiments described herein are not limited to execution by a mobile device, or in particular, by a mobile device that captured the image being processed by the flight light engine.
[0041] To visualize the position of the wavefront at a given time following wavefront emission, the flight light engine 112 may increase the brightness of each of the identified pixels (sometimes referred to herein as "wavefront pixels") relative to the remaining pixels of the RGB image (sometimes referred to herein as "non-wavefront pixels"), or may decrease the brightness of each of the non-wavefront pixels of the RGB image relative to the brightness of each of the wavefront pixels, depending on the desired effect.
[0042] 2 shows a sequence 200 of three different depth images corresponding to three different time stamps of a simulated traverse of a wavefront across a scene, with each depth image showing a pixel in the scene corresponding to the location (e.g., depth) of the simulated wavefront at the corresponding time. In this example, the flying light engine 112 simulates the wavefront as if it were a short pulse emitted from a mobile device that captured the image and the corresponding depth map. The sequence 200 includes a first depth image 202 corresponding to a first time and including a first set of wavefront pixels 204, a second depth image 206 corresponding to a second time and including a second set of wavefront pixels 208, and a third depth image 210 corresponding to a third time and including a third set of wavefront pixels 212. The first time occurs before the second time, the second time occurs before the third time, and the first, second and third times each occur during a period during which a wavefront is simulated to traverse the scene represented in the depth images 202, 206 and 210 and the corresponding RGB image. In some embodiments, pixels in the depth images 202, 206 and 210 having lower pixel values correspond to objects closer to the mobile device, while pixels having higher pixel values correspond to objects farther from the mobile device. That is, the value of each pixel in a given depth map 202, 206, 210 corresponds to the depth of that pixel in the scene represented in the depth map and that of the corresponding pixel in the corresponding RGB image. Thus, the pixel values of pixels in a depth map may be referred to herein as "depth values." Pixels in a depth map having the same pixel value are approximately the same distance away from the mobile device. It should be understood that wavefront pixels 204, 208, and 212 have been darkened to improve their visibility, and therefore the depth of wavefront pixels 204, 208, and 212 is not indicated by their color or brightness shown in the diagram of FIG.
[0043] To identify which pixels in the depth map are wavefront pixels at a given time, the flying light engine 112 identifies a depth value that corresponds to that time and identifies all pixels in the depth map having the identified depth value as being wavefront pixels. In this example, wavefront pixel 204 is located closer to the mobile device than wavefront pixel 208, which is located closer to the mobile device than wavefront pixel 212, as shown as the wavefront moving further away from the mobile device over time.
[0044] Returning to the example of FIG. 1, in some embodiments, the flight light engine 112 may brighten only the identified wavefront pixels in each flight light image, but this brightening may be difficult to perceive depending on the scene represented in the original RGB image. In addition, brightening only the identified wavefront pixels does not take into account the light scattering that may occur when the wavefront hits various objects in the scene, the beam widening that occurs as the wavefront propagates across the scene, and the light dispersion that occurs as the wavefront propagates across the scene. Thus, in some embodiments, the perceptibility of the simulated wavefront and the accuracy of the simulation may be improved by simulating how the light packets of the wavefront are reflected or scattered from objects in the scene (i.e., light scattering) and simulating the effects of beam widening and light dispersion by applying a convolution operator to the wavefront pixels. For example, the flight light engine 112 may generate an optical simulation of the wavefront and generate a flight light image for the RGB image that simulates the pulse wavefront at a given timestamp by affinely transforming the original RGB image with the output of the optical simulation. When generating an optical simulation, the flight light engine 112 may apply a convolution operator to a three-dimensional transformation of the pixels of the depth map to generate a simulation output that approximates the non-idealities of the wavefront propagation across the scene. A slice of the simulation output centered on the wavefront depth for a given timestamp may then be affine transformed with the original RGB image to generate a flight light image frame. For example, the convolution operator may approximate the effects of short light pulse scattering, beam widening, and light dispersion in space.
[0045] For example, FIG. 3 shows how a short pulse of light 304 output by a laser 302 propagates over time. As shown, the cross-sectional area 306 of the light 304 may expand outward along the x-axis and y-axis over time. As the light 304 travels away from the laser 302, it may experience various non-idealities, such as scattering, beam broadening, and dispersion. The effect of such non-idealities on the amplitude of the light 304 in the x-, y-, and t-directions may be defined as respective Gaussian functions. Here, the x- and y-axes define a plane that intersects the vector of propagation of the emitted light 304. The t-axis is orthogonal to the transverse plane and parallel to the vector of propagation of the light 304, and represents both the time and distance traveled by the light 304.
[0046] For example, define the amplitude of light 304 over time, where vector v T The pulse function, represented as x, can be approximated by a Gaussian function since it is not precisely localized in time. In addition, at a given point along its cross-section, the light 304 can define a Gaussian spot 308. FIG. 4 depicts a chart 400 illustrating the beam amplitude of the Gaussian spot 308 in the cross-section. As shown, the amplitude of the light 304 can be approximated by respective Gaussian functions along the x and y directions. The amplitude of the light 304 in the x direction is now represented by the vector v x where the amplitude of the light 304 in the y direction is v y Therefore, the amplitude of the light 304 over three-dimensional space may be approximated according to Equations 1-4:
[0047]
number
[0048] where G(t), G(x), and G(y) are Gaussian functions on the t, x, and y axes, respectively, and H is the vector v x , v y , and v twhere H represents a three-dimensional matrix. Returning to the example of FIG. 1, the flight light engine 112 may simulate non-idealities such as scattering, beam broadening, and dispersion of the quanta of the simulated pulse wavefront (e.g., as demonstrated for light 304 in FIG. 3) by applying H from Equation 4 as a convolution operator to pixels of a depth map associated with the original RGB image to generate an optical simulation output.
[0049] When generating an optical simulation output for a given RGB image, the flight light engine 112 may convert the corresponding depth map of the RGB image into a three-dimensional data structure, such as a cube data structure, referred to herein as conversion to the "lift domain," in which pixels are converted into voxels having positions in space defined relative to a cube sparsely populated by the original pixel's position in the depth map and the original pixel's depth value. Figure 5 illustrates an example of a conversion of a depth map from a two-dimensional domain to a three-dimensional lift domain. As shown, a two-dimensional depth map 512 is initially converted into voxels with respective depth values D i、j , where i and j define the coordinates of a given pixel in the array. In this example, the flight light engine 112 converts the depth map 512 to the lift domain where the previously two-dimensional depth map 512 is represented as a three-dimensional cubic data structure 520. The cubic data structure 520 is sparsely populated with three-dimensional pixels (i.e., voxels) 524. The subset of voxels 524 that correspond to the depth of the wavefront at a given timestamp may be referred to herein as "wavefront voxels." Once converted to the lift domain, the location of a given voxel with respect to the x-axis and y-axis is maintained, while the location of the voxel along a new third axis, the t-axis or "time" axis, is determined based on the depth value defined by the corresponding depth map pixel. For example, the depth value of a given pixel divided by the speed of light defines the location of the corresponding voxel along the t-axis. Because, as established above, d=c *In some embodiments, the third axis may explicitly define depth rather than time.
[0050] After transforming the depth map into the lift domain, the flight light engine 112 applies a convolution operator to the voxels of the resulting cubic data structure. FIG. 6 illustrates the application of the convolution operator 610 to voxels 624 of the cubic data structure 620. The cubic data structure 620 may correspond to a lift domain transformation of a two-dimensional depth map corresponding to an RGB image / depth map pair. In the cubic data structure 620, the position of each voxel 624 may be defined, for example, by assigning a value of one or another non-zero value to the wavefront voxel 624 and a value of zero to all other voxels of the cubic data structure 620. The convolution operator 610 defines a Gaussian sphere 612, which corresponds, for example, to the convolution operator H of Equation 4, and thus may be defined by respective Gaussian functions along the x-axis, y-axis, and t-axis. The flight light engine 112 convolves the convolution operator 610 with the cube data structure 620 to generate a filtered cube data structure 630 including Gaussian spheres 634 each centered on the respective location of the voxels 624. For example, the size and amplitude profile of each Gaussian sphere 634 of the cube data structure 630 may be the same. In some embodiments, the dimensions of the Gaussian spheres 634 along the x-axis, y-axis, and t-axis are customizable by the user to achieve different visual effects in simulating a wavefront. By convolving the convolution operator 610 with the voxels 624 of the cube data structure 620, the values of the voxels surrounding each voxel 624 are increased according to the distribution defined by the Gaussian spheres 634. For example, the amount by which the amplitude of a given non-wavefront voxel is increased depends on its proximity to one or more of voxels 624 according to the distribution of a Gaussian sphere defined by H, with affected voxels having larger amplitudes the closer they are to voxel 624. For the purposes of this specification, the filtered cube data structure 630 is considered the simulation output.
[0051] When generating a flight light image, the flight light engine 112 may logically divide the filtered cube data structure 630 into a number of slices. Each slice is centered on a different wavefront depth associated with a respective timestamp (e.g., each wavefront depth corresponds to the position of the wavefront at the associated timestamp), and each has a width corresponding to the diameter of H in the t dimension (i.e., along the t axis). It should be understood that in some embodiments, the slices may overlap each other along the t axis. The flight light engine 112 may then perform an affine transformation of each slice of the filtered cube data structure 630 with the original RGB image to generate a sequence of flight light image frames. The flight light engine 112 may then assemble the flight light image frames to generate a flight light image.
[0052] For example, returning to FIG. 1, upon generating the optical simulation output, the flight light engine 112 may perform an affine transformation of the original RGB image with slices of the optical simulation output to generate flight light images. Equation 5 shows an example of an affine transformation that may be performed by the flight light engine 112 to generate a sequence of flight light images:
[0053]
number
[0054] where S(I) is a set of flight light images output by the flight light engine 112 for the original RGB image I, D is a depth map corresponding to the original RGB image, Y is the optical simulation output, H is a convolution operator, X is a cubic data structure corresponding to the depth map, F(D) is a function used to transform the depth map to the lift domain to generate the cubic data structure X, α is a constant that determines the weight assigned to the simulation output (e.g., the amount by which the brightness of the affected pixels of the original RGB image is increased), k is the index of the sequence of flight light images, and N is the number of flight light images in the sequence. For example, each affine transformation of the original RGB image performed by the flight light engine 112 effectively adjusts the brightness of pixels in the original RGB image where the light from the wavefront would have directly entered (i.e., the wavefront pixel) and the scattered light from the wavefront would have entered (i.e., approximated by the convolution operator used to generate the optical simulation output) for a given time stamp. It should be understood that although Equation 5 illustrates the generation of a sequence of flying light images, Equation 5 may also be used to generate a single flying light image.
[0055] The flying light engine 112 may be configured to generate a flying light image by generating a sequence of flying light frames (e.g., as provided in conjunction with Equation 5) that simulates the movement of a wavefront across a scene represented in the original RGB image. The speed at which the wavefront is represented moving across a scene in such a flying light image (i.e., "wavefront velocity") may be predefined, may be predefined by a user of a device (e.g., the mobile device 100) that executes the flying light image, or may be predefined automatically based on a predefined period (e.g., a predefined or user-defined period for the flying light image). The flying light engine 112 may determine a sequence of timestamps to be used in generating the flying light image frames of the flying light image based on one or more predefined or user-defined timing parameters. The timing parameters may include the wavefront velocity, the length of the desired flying light image, and / or a predefined sampling rate. In some embodiments, the flying light engine 112 may determine the number of flying light frames to be generated based on one or more of the depth of the scene represented in the RGB image and the timing parameters. It should be understood that for the flying light image, since the individual movement of the wave front across the scene at the speed of light is generally not visible to the human eye or reproducible on a conventional display, the flying light engine 112 may generate the flying light image to simulate the movement of the wave front across the scene at a fraction of the speed at which the actual light wave front moves or over an extended period of time to allow the viewer to perceive the individual movement of the wave front. The amount by which the flying light engine 112 modifies the speed of the wave front or extends the period of time the wave front crosses the scene for the playback of the flying light image may be selected based on one or more defined timing parameters (which may be stored in the storage device 110, for example) that may be preset or may be defined by the user.For example, the timing parameters may include one or more of a wavefront velocity, which specifies the speed at which the wavefront should move in the flying light image, a wavefront period, which specifies the period over which the wavefront should traverse a scene in the flying light image and which may be defined as the length of the flying light image to be generated, or a frame rate, which specifies the number of flying light image frames to be generated over a given period of the flying light image.
[0056] 7 shows a process flow for a method 700 of generating a flying light image. For example, the method 700 may be performed by a mobile device having an RGB camera and a depth camera. In this example, the method 700 is described in the context of the mobile device 100 of FIG.
[0057] At block 702, the RGB camera 102 and the depth camera 104 of the mobile device 100 capture an RGB image and a corresponding depth map. The RGB image and the depth map may be stored in the storage device 110 of the mobile device 100 for subsequent processing.
[0058] In block 704, the flight light engine 112 generates optical simulation outputs for the RGB image and the depth map by transforming the depth map pixels into a three-dimensional domain (e.g., a cubic data structure) and then applying a convolution operator H to the non-zero voxels of the cubic data structure to generate optical simulation outputs (e.g., a filtered cubic data structure).
[0059] At block 706, the flight light engine 112 determines the number N of flight light image frames to be generated. In some embodiments, the flight light engine 112 determines the number of flight light image frames to be generated based on one or more timing parameters, which may include one or more of a wavefront velocity, a wavefront period, a frame rate, and / or another applicable timing parameter.
[0060] At block 708, the flying light engine 112 determines a sequence of timestamps including a respective timestamp for each flying light image frame to be generated. Here, the timestamp for a given flying light image frame determines the position / depth at which the wavefront is to be simulated in the given flying light image frame. In some embodiments, adjacent timestamps in the sequence may be equally spaced.
[0061] At block 710, the flying light engine 112 sets variable k to be equal to 1. Here, k represents the index of the next flying light image frame to be generated with respect to the sequence of frames.
[0062] At block 712, the flying light engine 112 generates the k-th flying light image frame of the sequence corresponding to the k-th timestamp of the sequence of timestamps. For example, the flying light engine 112 may generate each image frame according to blocks 808 and 810 of method 800 of FIG. 8.
[0063] At block 714, the flying light engine 112 determines whether the most recently generated flying light image frame is the last image frame in the sequence of flying light image frames by determining whether k = N - 1. If the flying light engine 112 determines that k = N - 1, method 700 proceeds to block 718. Otherwise, if the flying light engine 112 determines that k < N - 1, method 700 proceeds to block 716.
[0064] At block 716, the flying light engine 112 increments the value of k by 1. Method 700 then returns to block 712, where the flying light engine 112 generates the next flying light image frame in the sequence.
[0065] At block 718, the flying light engine 112 assembles the sequence of flying light image frames into a flying light video representing a simulated wavefront traveling across the scene of the RGB image.
[0066] In block 720 , the flying light engine 112 stores the flying light image on the storage device 110 .
[0067] FIG. 8 illustrates a process flow for a method 800 of generating a flight light image. For example, some or all of the steps of the method 800 may be performed in generating flight light image frames for the optical simulation output and the flight light image at blocks 704 and 712 of FIG. 7. As another example, the method 800 may be performed to generate a single standalone flight light image. For example, the method 800 may be performed by a mobile device having an RGB camera and a depth camera. In this example, the method 800 is described in the context of the mobile device 100 of FIG. 1.
[0068] In step 802, the flight light engine 112 receives an RGB image, a depth map corresponding to the RGB image, a timestamp, and one or more predefined or user-defined timing parameters. The timestamp defines a time during the traverse of a wavefront across a scene represented in the RGB image at which a flight light image is to be generated, and a position of the wavefront at that time can be derived from the timestamp. The timing parameters can include one or more of a wavefront velocity, a wavefront period, a frame rate, and / or another applicable timing parameter.
[0069] In step 804, the flying light engine 112 converts the depth map to a three-dimensional domain. In some embodiments, the flying light engine 112 converts the depth map to the lift domain to generate a three-dimensional structure, referred to in this example as a cube data structure (e.g., cube data structure 520 of FIG. 5). The coordinates of a given voxel in the cube data structure may be selected based on the coordinates of the corresponding pixel in the original depth map and the depth value of the corresponding pixel.
[0070] In step 806, the flying light engine 112 applies a convolution operator (e.g., the convolution operator H in Equation 4) to the cubic data structure to generate a filtered cubic data structure. For example, the filtered cubic data structure may include one or more Gaussian spheres each centered on a non-zero voxel of the cubic data structure.
[0071] In step 808, the flight light engine 112 identifies a slice of the filtered cubic data structure of the optical simulation output that corresponds to a current timestamp for which the flight light image frame is being generated. For example, the identified slice of the filtered cubic data structure may be a matrix that is a subset of the filtered cubic data structure. The matrix may be composed of all voxels of the filtered cubic data structure within a depth range (i.e., a range of values along the t-axis) centered at a depth that corresponds to the current timestamp (i.e., corresponds to the position of the wavefront at the time represented by the timestamp). Here, the size of the depth range corresponds to the diameter of the convolution operator H in the t-direction. In step 812, the flight light engine 112 performs an affine transformation of the original RGB image with the identified slice of the optical simulation output to generate the flight light image frame. For example, the affine transformation may increase the brightness of pixels in the RGB image that correspond to a surface that is simulated to have the wavefront incident at the timestamp and increase the brightness of pixels that correspond to a surface that is simulated to have scattered light from the wavefront incident at the timestamp.
[0072] In some embodiments, certain aspects of the above-described techniques may be realized by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by the one or more processors, operate the one or more processors to perform one or more aspects of the above-described techniques. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device, such as a flash memory, a cache, a random access memory (RAM), or other non-volatile memory device(s), etc. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executable by the one or more processors.
[0073] A computer-readable storage medium may include any storage medium, or combination of storage media, that is accessible by a computer system to provide instructions and / or data to the computer system during use. Such storage media may include, but are not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-ray disc), magnetic media (e.g., floppy disk, magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical system (MEMS) based storage media. The computer readable storage medium may be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., a magnetic hard drive), removably attached to the computing system (e.g., an optical disk or Universal Serial Bus (USB)-based flash memory), or coupled to the computer system via a wired or wireless network (e.g., network accessible storage (NAS)).
[0074] It should be noted that not all of the activities or elements described above in the general description are required, that some of the specific activities or devices may not be required, and that one or more additional activities may be performed or elements may be included in addition to those described. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, concepts have been described with reference to specific embodiments. However, those skilled in the art will appreciate that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the claims. Thus, the specification and drawings should be regarded in an illustrative rather than restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.
[0075] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, these benefits, other advantages, solutions to problems, and any features that may cause any benefit, other advantages, or solutions to occur or become more pronounced should not be construed as critical, required, or essential features of any or all of the claims. Moreover, the specific embodiments disclosed above are merely exemplary, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design shown herein, other than as described in the claims. It is therefore apparent that the specific embodiments disclosed above may be altered or modified, and all such variations are contemplated within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims.
Claims
1. generating an optical simulation output simulating light scattering effects for the simulated wavefront based on a depth map corresponding to the optical image; and generating a flight light image by adjusting brightness of a subset of pixels of the optical image based on the optical simulation output and based on a position of the simulated wavefront at a time corresponding to a defined timestamp.
2. The step of generating an optical simulation output includes: generating a three-dimensional data structure based on the depth map; and convolving the three-dimensional data structure with a convolution operator to generate a filtered three-dimensional data structure.
3. The step of generating the three-dimensional data structure comprises:
3. The method of claim 2, comprising, for a given pixel of the depth map, mapping the given pixel to a respective location in the three-dimensional data structure based on two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.
4. The method of claim 2 or 3, wherein the convolution operator defines a Gaussian sphere.
5. 5. The method of claim 2, further comprising identifying a slice comprising a subset of voxels of the filtered three-dimensional data structure centered at a wavefront depth corresponding to the position of the simulated wavefront at the time corresponding to the defined timestamp.
6. Adjusting the luminance of the subset of pixels of the optical image based on the optical simulation output includes: The method of claim 5 , further comprising performing an affine transformation of the optical image using the identified slices.
7. determining the wavefront depth based on a speed of light and based on the time corresponding to the defined timestamp; The method of claim 5 or 6, wherein the wavefront depth indicates the position of a simulated wavefront in the optical image at the time corresponding to the defined timestamp.
8. 1. A mobile device comprising: an optical camera configured to capture an optical image; a depth camera configured to capture a depth map corresponding to the optical image; at least one processor, the at least one processor comprising: generating an optical simulation output based on the depth map, the optical simulation output simulating light scattering effects for the simulated wavefront; and a mobile device configured to adjust a brightness of a subset of pixels of the optical image based on the optical simulation output and based on a position of the simulated wavefront at a time corresponding to a defined timestamp to generate a flight light image.
9. To generate the optical simulation output, the processor further comprises: generating a three-dimensional data structure based on the depth map; configured to convolve the three-dimensional data structure with a convolution operator to generate a filtered three-dimensional data structure; The mobile device of claim 8 , wherein the optical simulation output comprises the filtered three-dimensional data structure.
10. To generate the three-dimensional data structure, the processor 10. The mobile device of claim 9, configured to, for a given pixel of the depth map, map the given pixel to a respective location of the three-dimensional data structure based on two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.
11. The mobile device of claim 9 or 10, wherein the convolution operator defines a Gaussian sphere.
12. The processor, 12. The mobile device of claim 9, configured to identify a slice comprising a subset of voxels of the filtered three-dimensional data structure centered at a wavefront depth corresponding to the position of the simulated wavefront at the time corresponding to the defined timestamp.
13. To adjust the luminance of the subset of pixels of the optical image based on the optical simulation output, the processor: The mobile device of claim 12 , configured to affinely transform the optical image using the identified slices to generate the flight light image.
14. The processor further comprises: configured to determine the wavefront depth based on the speed of light and based on the time corresponding to the defined timestamp; The mobile device of claim 12 or 13, wherein the wavefront depth indicates the position of the simulated wavefront in the optical image at the time corresponding to the defined timestamp.
15. generating a plurality of flight light image frames based on an optical image, a depth map corresponding to the optical image, and one or more timing parameters; Assembling the plurality of flying light image frames into a flying light video; The method, wherein the flight of light images simulates the propagation of a wavefront of a light pulse across a scene represented in the optical image.
16. The step of generating a flying light image frame from among the plurality of flying light image frames includes: generating a three-dimensional data structure based on the depth map; convolving the three-dimensional data structure with a convolution operator to generate a filtered three-dimensional data structure; and adjusting a brightness of a subset of pixels of the optical image based on an optical simulation output to generate the flight light image frame. The method of claim 15 , wherein the optical simulation output comprises the filtered three-dimensional data structure.
17. The step of generating the three-dimensional data structure comprises:
17. The method of claim 16, comprising, for a given pixel of the depth map, mapping the given pixel to a respective location in the three-dimensional data structure based on two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.
18. The method of claim 16 or 17, wherein the convolution operator defines a Gaussian sphere.
19. The step of generating the flying light image frame from the plurality of flying light image frames includes:
19. The method of any one of claims 16 to 18, further comprising identifying a slice comprising a subset of voxels of the filtered three-dimensional data structure centred at a wavefront depth corresponding to a position of the wavefront in the scene represented in the optical image at a time corresponding to a defined timestamp.
20. adjusting the luminance of the subset of pixels of the optical image based on the optical simulation output, 20. The method of claim 19, comprising affinely transforming the identified slices using the optical simulation output to generate the flight light image frame.
21. The step of generating the flying light image frame from the plurality of flying light image frames includes: determining the wavefront depth based on the speed of light and based on a time corresponding to the defined timestamp; 21. The method of claim 19 or 20, wherein the wavefront depth indicates the position of the wavefront in the scene represented in the optical image at the time corresponding to the defined timestamp.
22. A computer program having instructions stored thereon which, when executed by a processor, cause the processor to perform the steps of the method of any one of claims 1 to 7 and 15 to 21.
Citation Information
Patent Citations
Energy propagation modeling device
JP2002526843A
