Interactive motion blur on mobile devices

Abstract

A novel method and system are described for providing interactive motion blur to an image with motion input from the movement of a mobile device displaying the image. The device can process the motion blur with modules that provide motion blur parameter estimation, blur application and image synthesis based on metadata from an encoder and baseline images. A preloaded filter bank can provide the blur kernels for blur application.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 290,322, filed December 16, 2021, and European Patent Application No. 21214983.5, filed December 16, 2021, each of which is incorporated herein by reference in its entirety. Technical Field This disclosure relates to improvements for providing motion blur on still images. More particularly, the present disclosure relates to a method and system for providing interactive motion blur on a mobile device. [Background technology]

[0002] Motion blur is an effect in photography that is typically caused by movement of either the camera or the subject while the photograph is being taken. The effect may be used intentionally to enhance a photograph, for example, by showing movement, by emphasizing certain areas of the photograph over others, by creating an emotional feel for the scene being depicted, or to create some other artistic effect. Blur can also be artificially added to a photograph by post-processing the image, such as with Photoshop™. [Prior art documents] [Non-patent literature]

[0003] Image Blur References, all incorporated herein by reference. [Non-Patent Document 1] Motion Blur: en.wikipedia.org / wiki / Motion_blur [Non-patent document 2] www.nyfa.edu / student-resources / how-to-motion-blur-photography / [Non-patent document 3] Motion blur method (MATLAB): www.mathworks.com / help / images / ref / fspecial.html, section “motion filter” Summary of the Invention [Problem to be solved by the invention]

[0004] A system for adding interactive blur to an image is provided by a mobile device that decodes the image, where motion of the mobile device is converted into instructions for the decoder to selectively apply blur to the image. Metadata provided by the encoder side (the sender of the image) can further control how the blur is applied. The decoder side of the system can be implemented on a mobile device running any operating system, such as iOS™ or Android™. [Means for solving the problem]

[0005] An embodiment is described for achieving a method for providing interactive blur on an image viewed on a mobile device. The method includes measuring at least one sensor output from the mobile device; receiving metadata related to motion blur from an encoder; performing a motion blur estimation based on the at least one sensor output and the metadata; selecting at least one blur kernel from a filter bank based on the motion blur estimation for corresponding pixels of the image; and applying blur to the image by using the at least one blur kernel at each corresponding pixel to generate a blurred image, wherein applying blur includes using an angle map to adjust the blur direction at the corresponding pixel. An embodiment of a decoder configured to perform the method is disclosed, the encoder comprising: an image decoder; a motion blur estimation module; a filter bank; and a blur application module. Disclosed are embodiments of an encoder configured to provide an image and metadata to a decoder, the encoder being configured to encode the image into an encoded image, generate a depth map, a mask map, and an angle map for the image based on preferences, and multiplex the encoded image, the depth mask, the mask map, and the angle map as output to the decoder. [Brief explanation of the drawings]

[0006] [Figure 1] 1 shows an exemplary decoder diagram for implementing immersive motion blur.

[0007] [Figure 2] 1 shows an exemplary flowchart for immersive motion blur.

[0008] [Figure 3] 1 shows an exemplary encoder diagram for supporting immersive motion blur.

[0009] [Figure 4] 1 shows an exemplary flowchart for deriving a blur kernel from a filter bank.

[0010] [Figure 5] 1 shows an exemplary flow diagram of motion blur parameter estimation.

[0011] [Figure 6] Figures A-B show examples of blur types based on device interaction.

[0012] [Figure 7] An example of a radial blur center is shown.

[0013] [Figure 8] 1 shows an example of a depth map with multiple radial blur centers.

[0014] [Figure 9] 10 shows exemplary shifted image centers for multiple radial blur centers.

[0015] [Figure 10] An example of bilinear weight derivation is shown.

[0016] [Figure 11] Shows examples of different maps used for interactive motion blur.

[0017] [Figure 12] An example of a blur vector map.

[0018] [Figure 13] 10 shows an example of a modified intensity map.

[0019] [Figure 14] 1 shows an exemplary flow diagram for applying blur.

[0020] [Figure 15] An example of ghost artifacts is shown.

[0021] [Figure 16] An example of a blur kernel overlapping the foreground.

[0022] [Figure 17] An example of foreground blur is shown below.

[0023] [Figure 18] 10 illustrates an exemplary process of a processed mask for foreground blur.

[0024] [Figure 19]1 illustrates an exemplary process for processing a depth map.

[0025] [Figure 20] 1 shows an example graph of a depth transfer function for a depth map.

[0026] The embodiments in the drawings are intended to be illustrative and not limiting of the scope of the invention. DETAILED DESCRIPTION OF THE INVENTION

[0027] The methods and systems described herein describe a way to emulate blur for still images sent to a mobile device. The blur is controlled by a combination of metadata and the motion of the mobile device. Modern mobile devices typically include sensors, such as accelerometers, that detect how the device is moving while being held, both in terms of the direction and speed of the motion. Feedback from these sensors can provide the device with intuitive instructions on how to apply a blur effect to the photo. For example, a fast sideways motion can be interpreted as applying a strong motion blur in the direction of the motion, while a slow forward (z-axis) motion can apply a slight radial blur. Metadata provided with the image (e.g., from an encoder) can control how different parts of the image (e.g., foreground, background) are blurred. In this way, the sender of the image can control which elements are blurred and in which direction objects in the image should appear to be moving, and the recipient (viewer) of the image can control the type and intensity of the blur. Furthermore, additional metadata can provide further control for both the sender and the viewer. Further processing can be performed to remove artifacts produced by the blurring.

[0028] As used herein, "blur" refers to the manipulation of an image to emulate a smearing or fuzzing effect on the background and / or foreground of the image. "Motion blur" refers to a linear streaking blur to indicate movement in a still image. "Radial blur" refers to a linear streaking blur that radiates from a selected point (radial center) that can be within the image or outside the image frame. Motion blur can be thought of as a special case of radial blur in which the radial center is a large distance from the image (e.g., effectively at infinity) so that all streaks in the image are parallel. Types of blur include panning blur (e.g., unblurred foreground object with motion blur in the background), zoom blur (e.g., radially blurred foreground and background), and others.

[0029] As used herein, a "mobile device" refers to any computing device that can be held, carried, or worn by a person and that can display / project images. Examples include smartphones, tablet computers, smart glasses, smart watches, digital picture frames, wearable digital screens (e.g., integrated into clothing), etc. The device may be a general-purpose device with software (an "app") loaded to implement interactive blur, or a device specifically designed to utilize interactive blur implemented in software, hardware, and / or firmware. As used herein, the terms "user" and "viewer" are used interchangeably to refer to a person using a mobile device to view images. While some embodiments may involve one person performing the mobile device movements (generating interaction data) while another person views the screen, it is believed that the likely typical use will be one in which both functions are performed by a single person.

[0030] As used herein, the term "blur kernel" refers to a 2D low-pass filter that is applied to portions of an image.

[0031] As used herein, a "module" is a hardware and / or software component that performs a function.

[0032] Figure 1 shows an example implementation from the decoder (mobile device) side. Input from the sender is demultiplexed (110) into image data that is decoded by the decoder (115) into a baseline image (125) and metadata (120) that is used by the blur system modules (135, 145, 150). Device interaction data (130) (e.g., data from the mobile device about its movement) is sent to the motion blur parameter estimation module (135).

[0033] The motion blur parameter estimation module (135) receives as input the device interaction data (130) (e.g., velocity) and associated metadata (120) (e.g., depth information, expected blur direction, radial blur center offset, etc.), interprets them, and outputs motion blur parameters. The parameters include the motion blur intensity and direction information. The parameters are sent to the filter bank (140) to fetch the appropriate blur kernel.

[0034] The blur application module (145) receives blur parameters from the motion blur parameter estimation module (135) and uses them to load the appropriate blur kernel from the filter bank (140). It also receives as input the baseline image (125) and associated metadata (120) (e.g., background blur flag, mask map, etc.), processes them, and outputs data for the image synthesis module (150). When simulating foreground object blur, the blur application module (145) can also process the mask map with the blur kernel so that the processed mask map can cover an expanded area of ​​the now blurred foreground object. In this case, the processed mask map is also sent to the image synthesis module (150).

[0035] The image synthesis module (150) receives as input the baseline image (125), data from the blur application module (145) (e.g., blurred image), and associated metadata (120) (e.g., mask map, background / foreground blur flags, etc.) and uses them to synthesize a final output image (190) that is presented on the mobile device (e.g., viewed on a screen or projected).

[0036] Figure 2 shows an example flow diagram of a decoder using interactive blur. When the decoder is initialized, it loads blur kernels into a filter bank (210). This filter bank is continuously loaded into memory so that the appropriate blur kernel can be fetched for any instance of device speed. The decoder then demultiplexes the bitstream received from the encoder and reads the baseline image and metadata (215). The decoder checks the associated metadata to see if the encoder specified not to perform blur on the entire image (e.g., NUM_OBJ=1&BLUR_FLAG=0; see example metadata herein). If true, the decoder proceeds to the image synthesis module (240), where the baseline input image is displayed as the output image (290). If false, the decoder reads the device velocity (225) and synthesizes the final motion-blurred image by going through the three aforementioned modules from Figure 1: motion blur parameter estimation (230), blur application (235), and image synthesis (240). This procedure is repeated, with changes in device velocity resulting in corresponding changes in the output image, thus making the image responsive to device motion in real time.

[0037] Figure 3 shows an example of an encoder designed to support a decoder using interactive blur. The encoder generates a decoder-compatible bitstream. For example, in Figure 3, the encoder receives a set of preference inputs (310), which may be set by the viewer and / or by the entity providing the image to the viewer. This is used to read various maps (322, 323, 324) and form metadata that can be interpreted by the decoder. These maps are presented in an input folder (320) along with an input image (321) to be blurred. The image (321) is encoded (330), and the preference inputs (310), maps (322, 323, 324), and encoded image are multiplexed (335) into an output bitstream (390), which is used as input to the decoder. The maps may be generated using different algorithms depending on the desired level of accuracy, speed, and implementation complexity.

[0038] Filter Bank Generation Filter bank generation requires quantization of the blur strength and direction. The motion blur strength and direction can be represented as S(i) and θ(i), respectively, for the i-th pixel. S(i) and θ(i) are subject to constant variation from the time-varying device velocity. Since generating a blur kernel on the fly every time for a given (S(i),θ(i)) is computationally expensive and tedious, we pre-generate blur kernels corresponding to various combinations of (S,θ) and load them into memory.

[0039] To generate a blur kernel from (S,θ), a motion blur method such as that provided by MATLAB™ (see the "Motion Filters" section at https: / / www.mathworks.com / help / images / ref / fspecial.html, incorporated herein by reference) can be used, where the intensity and direction of the blur are specified and the corresponding blur kernel is output. However, the motion blur method does not need to be constrained to this method; any method (e.g., a 2D Gaussian kernel) that can reflect different levels of blur intensity and direction can be used.

[0040] Depending on (S, θ) provided by the motion blur parameter estimation module, different blur kernels may be invoked. In some embodiments, all blur kernels are loaded into memory and kept there at all times, allowing the appropriate kernel to be fetched on the fly for a given (S, θ). Covering all possible blur kernels that may be generated from all combinations of (S, θ) may be memory intensive. In some embodiments, instead of considering all combinations of (S, θ), it may be desirable to jump by a certain quantization step for each intensity and direction to reduce the number of combinations. Of course, the quantization step may involve perceptual degradation, and therefore a trade-off between memory size and perceptual effect may be considered. In some embodiments, the step sizes for θ and S are set to θ, respectively. step and S step is given as θ step =6, S step= 2 is used. The filter coefficients may be stored, for example, in an unsigned 8-bit format. In some embodiments, the quantized (S, θ) is (S', θ'), where θ' = 0:6:180 and S' = 0:2:150 (global MAX_BLUR), the total number of blur kernels is 2356, the number of kernel coefficients is ∼5,789K, and the memory size is 1 byte × 5,789K or approximately 5.8MB.

[0041] After the decoder is initialized and the filter bank is loaded, the decoder reads the metadata and device velocity and sends them to the motion blur parameter estimation module. The module outputs (S,θ), which is quantized to (S',θ') and sent to the blur application module, where quantization is S'=round(S / S step ) and θ'=round(θ / θ step ) The blurring module uses the information (S'(i),θ'(i)) to fetch the appropriate blur kernel for the ith pixel from the filter bank.

[0042] Figure 4 shows an example flowchart for fetching blur kernels from a filter bank. After the decoder is initialized and the filter bank is loaded, the decoder reads the metadata (410) and device velocity (415) and sends them to the motion blur parameter estimation module (420). The module (620) outputs the magnitude and direction parameters (S, θ) (425), which are quantized (430) to (S', θ') and sent to the blur application module (440). Here, the quantization is S' = round(S / S step ) and θ'=round(θ / θ step The blurring module uses the information (S'(i), θ'(i)) to fetch the appropriate blur kernel for the ith pixel from the filter bank (435).

[0043] Figure 5 shows an example flowchart for a motion blur parameter estimation module. The motion blur parameter estimation module is the part of the decoder that interprets the portable device velocity in relation to the provided metadata. The device velocity is processed from measurements of the device's accelerometer sensor. The primary role of this module is to generate blur parameters, e.g., blur strength and direction, for each pixel location. The output from this module can be a blur strength and direction map containing parameter values ​​for each pixel location.

[0044] The motion blur parameter estimation module first collects device velocity measurements and metadata (505) and then undergoes two main operations: 1) blur vector map generation and 2) intensity and direction map derivation. The blur vector map generation operation has two branches (510) based on metadata (e.g., RAD_BLUR_CENTER_FLAG and NUM_CENTERS, further described later in the "Exemplary Metadata" section): one for the standard single blur center case and the other for the multiple blur center case. The standard operation follows a procedure that first determines (530) the radial blur center through interpretation of device velocity in relation to the associated metadata. Then, based on the determined radial blur center, it generates a blur vector map with vectors starting from the radial blur center to each pixel location (535). In the multiple blur center case, it first generates multiple blur vector maps from the multiple blur centers specified by the encoder (through CENTER_COORDS) (515). The weights to be applied to each blur vector map are then derived (520), which is done through device velocity interpretation. Finally, a weighted average of the blur vector maps is performed (525) to construct the final blur vector map. In either case, the output of this stage should be a single blur vector map. The next operation then utilizes this generated blur vector map to derive the final intensity and direction map. In this process, the module checks whether the decoder has received any relevant metadata, such as depth (555) and / or angle (540) information, and, if necessary, utilizes them (545, 560) to generate the final intensity and direction map (590). Otherwise, a standard map is derived (550).

[0045] Figures 6A and 6B show example cases of velocity in different directions and the corresponding expected blur direction.

[0046] Here, "x" refers to the "side-to-side" (horizontal) lateral movement axis relative to the device screen (on which the image is shown), "y" refers to the "up-down" (vertical) lateral movement axis relative to the device screen, and "z" refers to the "front-to-back" lateral movement axis perpendicular to the device screen. Positive values ​​relate to "tangential to the screen, to the viewer's right" for x, "tangential to the screen, upward in relation to the viewed image scene" for y, and "normal to the screen, towards the viewer's face" for z. Other conventions may be used, and calculations can be adjusted accordingly.

[0047] Figure 6A shows a translational (planar) blur that occurs in the direction opposite to the direction of movement of the device (610) when the x (or y) direction dominates the device velocity. The planar blur (620) is created by placing the radial blur center (630) far from the image.

[0048] FIG. 6B shows zoom blur, presented as a radial blur arising from the image center area (640) when the z-component dominates the device (610) velocity.

[0049] Hybrid blurs that incorporate a combination of these blurs are also possible. These various blur experiences can be simulated through control of the radial blur position. For example, when simulating a unidirectional (planar) blur, as in Figure 6A, the system can send the radial blur center far away in the direction opposite to the blur direction. By doing so, the vector from the radial blur center to the plane of interest (image) is a unidirectional vector, allowing the system to use a specified blur kernel for each pixel location. When simulating a radial blur, as in Figure 6B, the system can position the blur center near the image center so that the vectors from the blur center to each pixel location are directed radially.

[0050] Hybrid blur, which considers a combination of these blurs, can also be simulated by placing an off-center radial blur center depending on velocity direction and magnitude. As we can see, interpreting device velocity to locate the radial blur center is a key process for providing a realistic interactive motion blur experience.

[0051] Using this unified radial blur framework can be extended to more general cases: for example, different motion blur experiences can be provided by controlling the position of the radial blur center, which involves interpreting the relative contribution of each directional component of velocity.

[0052] Radial blur center derivation An example is presented of how to interpret device velocity to locate the radial blur center. The radial blur center determines the intensity and direction of the blur for each pixel.

number

number

[0053] The x and y components of the velocity are then normalized as follows:

number

[0054] Normalized velocity (for convenience, ^v x , ^v y captures the relative contribution of the z velocity through a normalization factor N, which is effective in locating the radial center blur.

[0055] The center of the radial blur is R=(R x ,R y ) Then, R is

number

[0056] where (O x ,O y ) refers to the offset coordinate set by the RAD_BLUR_CENTER_FLAG. (Note: A full description of the exemplary metadata flags used herein and their semantics is provided later in the "Exemplary Metadata" section.)

[0057] If RAD_BLUR_CENTER_FLAG=0, (O x ,O y )=(width / 2,height / 2).

[0058] If RAD_BLUR_CENTER_FLAG=1, (O xc ,O yc )=CENTER_COORDS[c], where c ranges from 0 to NUM_CENTERS-1. CENTER_COORDS is specified by the encoder. Note that if there are multiple center coordinates (c>1), equation (4) does not need to be used due to different velocity interpretation algorithms.

[0059] If RAD_BLUR_CENTER_FLAG=2, (O x ,O y ) is chosen to be the coordinate with the smallest value from DEPTH_WEIGHT to take into account the fact that more distant regions are much more likely to be radial blur centers. x ,O y )=argmin{DEPTH_WEIGHT(x,y)}.

[0060] β x and β y are the normalized velocities ^vx and ^v y This refers to the scaling factor applied to the , which controls the sensitivity of the blur in each direction. More specifically,

number

[0061] Referring again to the examples in Figures 6A and 6B, let RAD_BLUR_CENTER_FLAG=0, VL_X=5, and VL_Y=5. Figure 6A shows the case with device velocity in only the x-direction (v x ,0,0). In this example, according to equation (4), the radial blur center is

number

[0062] Figure 7 shows an example of blur vector map generation from the radial blur center and image plane. Let X(i) and Y(i) represent the coordinates of the ith pixel on the x-axis and y-axis, respectively. Then, the direction and intensity of the blur at each pixel location are determined by calculating the vector from the radial blur center to each pixel location. The blur vector for the ith pixel is defined as → Let B(i) = (bx(i),by(i)) (For convenience, let vector B be → Sometimes written as B.

number

[0063] Note that the y component of the vector is multiplied by -1 due to the fact that most programming languages ​​consider the +y direction on the y axis of the image plane to point downwards, rather than the definition used here (+y points up). → B is passed to the next action.

[0064] Metadata may be specified to have multiple blur center offsets by having RAD_BLUR_CENTER_FLAG=1 and NUM_CENTERS>1. A typical scenario that may require such a case is an image with multiple far-away regions, as shown in Figure 8. These far-away regions are likely to be radial blur centers, and one needs to consider how the blur from those multiple centers interacts for different device motions.

[0065] Now, considering the expected motion blur behavior for the case in Figure 8, if a user pulls the mobile device toward themselves and simultaneously to the right, they expect to experience a zoom blur toward the road on the left. This means that the blur vector from the left radial blur center contributes more to the final motion blur. Conversely, if they pull the device toward themselves and to the left, the right blur center contributes more, thus simulating a zoom blur toward the road on the right. If they pull the device directly toward themselves, the two blur centers will contribute similarly to the result. To link these blur experiences to device velocity, we first generate blur vector maps from those generated from each radial blur center. We then interpret the device velocity to determine which blur vector map contributes more. This information can be used collectively to generate the final blur vector map.

[0066] Multiple blur map generation Consider the case where NUM_CENTERS = 2. However, the algorithm is not restricted to NUM_CENTERS = 2 and can be extended to the case where NUM_CENTERS > 2. Figure 9 shows a scene with two far-away regions.

[0067] The module first reads the offset centers from the metadata CENTER_COORDS. Let us denote the two (in our example) offset centers as O1 and O2. Then, O1 = (O x1 ,O y1 )=CENTER_COORDS[0] and O2=(O x2 ,O y2) = CENTER_COORDS[1]. Note that in Figure 12, O1 and O2 are overlaid on the depth map. This is to convey that the CENTER_COORDS chosen in this example correspond to far-away regions. However, this does not mean that a depth map or DEPTH_WEIGHT are required for a multiple blur center experience. Specifying NUM_CENTER and CENTER_COORDS is sufficient.

[0068] After reading the offset centers from the metadata, we generate a blur vector map for each of the offset centers. This is for O1 and O2 respectively. → B1 and → This is done using equation (7), where (R x ,R y )teeth → B1 and → For B2, respectively (O x1 ,O y1 ) and (O x2 ,O y2 )

[0069] Blur Vector Map Aggregation Once → B1 and → Once B2 is generated, the final vector map → B is obtained by performing a weighted sum of the vector maps as follows:

number

[0070] The weights ω1 and ω2 are the weights of the shifted image center (C' x ,C' y ) is determined based on the distance of each offset center to the shifted image center (C' x ,C' y), which are denoted as l1 and l2 for O1 and O2, respectively. Weights are then determined using bilinear weighting, where ω1 = l2 / (l1 + l2) and ω2 = l1 / (l1 + l2).

[0071] Shifted image center (C' x ,C' y ) plays an important role in determining the weights for the blur vector map. (C' x ,C' y )teeth

number

[0072] Here, β x and β y is defined similarly to equations (5) and (6), which can be expressed as (C' x ,C' y ), and thus which offset center contributes more to the final blur during a given device motion.

[0073] Moving the device only in the z direction (e.g. towards the user) x =0 and ^v y =0, so (C' x ,C' y )=(C x ,C y ) becomes. (C' x ,C' y ) is closer to O2, so the blur vector from O2 receives a higher weight. Thus, the user will have a zoom blur experience to the road on the right. When the device moves in the z direction (towards the user) and to the right, ^v x >0 and ^v y =0, so the system is (C' x ,C' y ) to (C x ,C y ) to the left. Now we have (C'x ,C' y ) is closer to O1, so the blur vector from O1 receives a higher weight. Hence, the user will have a zoom blur experience towards the left road.

[0074] Similar to the single blur center case, the generated aggregate blur vector map from Eq. (8) → B is passed to the next operation: intensity and direction map derivation.

[0075] Intensity and direction map derivation Blur Vector Map → B=(b x ,b y ) from the previous operation (either single or multiple blur centers), the next step is to derive a blur strength (S) direction (θ) map from it.

[0076] Standard Method Unless ANG_FLAG=1, the standard method is used to compute the intensity and direction maps. The blur kernel direction (θ) for each ith pixel location is

number

[0077] To derive the intensity of the blur kernel, first we use a vector plane → The normalized absolute value plane ^M of B

number

number

[0078] Next, the velocity absolute value factor ^V is

number

[0079] The final intensity (S) of the blur kernel for each ith pixel location is

number

[0080] ^M is a plane with the same dimensions as the image, where each position contains a value ranging from 0 to 1. The values ​​correspond to the normalized filter strength at each pixel position based on its distance from the radial blur center. The further a pixel is from the radial blur center, the stronger the blur strength. ^V is a scalar that accounts for the absolute value of the velocity and scales the blur strength accordingly, thus giving different levels of blur for velocities of constant direction and varying absolute value. MAX_BLUR is specified by the encoder and corresponds to the maximum level of blur strength that can be reached for the current image. Its value can be in the range of 0 to 150.

[0081] Using angle information If ANG_FLAG=1, the blur strength (S) and direction (θ) can be derived differently than the standard way. In this case, it indicates that the content provider (encoder) has also specified ANG_X and ANG_Y, which contain the expected blur direction for each pixel. Thus, the direction (θ) of the blur kernel for each i-th pixel is

number

[0082] Blur Vector Map →Although B was not used to derive the orientation (θ) map, it can be used to derive the intensity (S) map. Figure 11 shows an example visualization of the various related maps (panel (a) angle map, panel (b) blur vector map, and panel (c) blur magnitude map), and Figure 12 shows the vector and magnitude map from (ANG_X, ANG_Y). → B shows the figures superimposed together. The thin arrow (1210) in Figure 12 indicates the vector specified by ANG_X and ANG_Y. The thick arrow (1220) in Figure 12 indicates the blur vector map obtained through device velocity interpretation. → In particular, the thick arrow (1220) shown in FIG. 12 corresponds to the vector obtained from the vector (b x ,b y ) corresponds to the vector (b x (i), b y (ANG_X(i),ANG_Y(i)) would have indicated the blur direction and intensity of the pixel for the current device velocity. However, now the expected direction of the blur is specified, so (b x (i), b y (i)) and only consider the magnitude of its directional component. x (i), b y If the angle formed by (ANG_X(i),ANG_Y(i)) and (ANG_X(i),ANG_Y(i)) is denoted as φ(i), then Eq.

number

[0083] Panel (c) of Figure 11 visualizes this M-map, where (ANG_X(i),ANG_Y(i)) and (b x (i), b y Regions where the vectors from (i)) point in the same (or similar) direction have larger absolute values. The rest of the (S) map calculation is done according to equation (14).

[0084] Applying DEPTH_WEIGHT If DEPTH_FLAG=1, the intensity (S) map can be further modified using DEPTH_WEIGHT. This indicates that the content provider (encoder) has requested the use of depth information provided by the metadata parameter DEPTH_WEIGHT. DEPTH_WEIGHT contains a weight applied to the blur intensity at each pixel location. The weight is determined based on the depth information, with smaller values ​​corresponding to more distant areas less likely to be affected by motion blur. The modified intensity for the i-th pixel is calculated as S(i) × DEPTH_WEIGHT(i). Figure 13 shows an illustration of how the intensity map (S) is modified. The original intensity map in panel (a) is scaled by DEPTH_WEIGHT in panel (b), resulting in the modified intensity map shown in panel (c).

[0085] Through these operations, the final intensity (S) and direction (θ) maps for the current device velocity and metadata can be derived. These motion blur parameters are then passed to the next step, which generates the blurred image.

[0086] Blur applied The output (S, θ) from the estimation module is quantized to (S', θ') before being sent to the blur application module. The quantization is done as follows: S'=round(S / S step ) and θ'=round(θ / θ step ) can be done by the following: Exemplary values ​​are S step =2 and θ step= 6. The blur application module receives these quantized parameters (S',θ') and uses them to load the appropriate blur kernel from the filter bank. It also receives the baseline image and associated metadata (e.g., background blur flag) as input and processes them to produce a blurred image and a processed mask map, which are sent to the image synthesis module.

[0087] FIG. 14 shows an example flow diagram of the blur application module operation. The module first receives (1405) a baseline image, metadata, and a quantized orientation map (S', θ') (1410). The module then applies blur on a pixel-by-pixel basis. Using the information (S'(i), θ'(i)), the filter bank is searched for a blur kernel for the i-th pixel of the baseline image. The dimensions of the blur kernel are then extracted, and the region to be convolved is identified. The corresponding image region is filtered to produce a blurred pixel. In some embodiments, the system can skip filtering if S'(i)=0 or 1.

[0088] Depending on what type of motion blur experience is being addressed (1420), and depending on whether there is a foreground region separate from the background (1415), there are three main branches of behavior.

[0089] The first is the case where we apply a blur to the entire image, where we simply apply the blur kernel to the entire image (1422) to generate the output blurred image (1424). If NUM_OBJ = 1 (1415), this indicates the case of global blur. In such a case, we apply the blur kernel to the input image. First, we fetch the appropriate blur kernel for the i-th pixel from the filter bank using the (S'(i),θ'(i)) information (1421). We then convolve the image portion and apply the blur kernel to that region (1422) to generate the blurred pixel response. This procedure is repeated until all pixel locations are covered (1423).

[0090] Second, when background blur is applied, an additional operation is used to modify the blur kernel coefficients to prevent foreground smearing artifacts. If NUM_OBJ > 1 (1415) and BLUR_FLAG[0] = 1 (1420), this indicates a background blur case, where foreground objects remain unblurred while the background is blurred according to device movement. First, BLUR_FLAG[1~NUM_OBJ] = 0 (1425) is set to remove any non-zero blur flags that may have been erroneously assigned to foreground object regions. Then, MASK_MAP is processed (1430) to derive a binarization map that indicates background regions as 1 and otherwise as 0. This binarization map is considered the processed mask map and is used for blur kernel modification and is also sent to the image synthesis module. Next, we use the (S'(i),θ'(i)) information to fetch 1435 the appropriate blur kernel for the i-th pixel from the filter bank.

[0091] The blur kernel can then be modified 1440 to have its coefficients only in background regions, so that they sum to 1, to avoid interference with foreground objects (see Figure 15, see discussion of "Ghosting Artifacts" below). The modified blur kernel is applied 1445 to the image. After all pixels have been covered in this way 1450, the processed mask map is used to output 1490 the blurred image.

[0092] Third, when applying a blur to a foreground object, an additional operation can be used to process the mask map and expand that foreground object mask according to the device movement so that it can fully cover the foreground pixels being blurred out.

[0093] If NUM_OBJ > 1 and BLUR_FLAG[n] = 1, where n is the index of the foreground object the author wants to blur, then MASK_MAP is processed to indicate which areas of the image have BLUR_FLAG set to on (1460). For each pixel, a blur kernel is fetched from the filter bank using the (S'(i),θ'(i)) information (1461). The blur kernel is applied to the image using its mask (1462). After all pixels have been covered in this way (1463), hard and then soft expansions are performed on the mask (1464) to create a natural blend of foreground and background in the blurred output image (1465) (see Figure 18 described below).

[0094] Ghost Artifacts Applying a blur kernel as is can result in ghosting artifacts where the foreground object appears to smear out against the background. An example is shown in Figure 15 where a foreground object (1505) is smeared to the left (1501) and right (1502) of the foreground region.

[0095] Applying a blur kernel to an image results in a fully blurred image, as shown in panel (b) of Figure 15. In this example, BLUR_FLAG=[1,0] indicates that only the background portion is blurred. Therefore, when combining the final image, areas corresponding to white regions from the binarization mask (panel (c) of Figure 15) take pixels from the blurred image (panel (b) of Figure 15), while areas corresponding to black regions in the binarization mask take pixels from the original image. However, the blurred foreground objects in panel (b) smear out into the background areas, shown in white in panel (c), resulting in ghosting artifacts.

[0096] The solution to this is to modify the blur kernel when it touches a foreground object position. Figure 16 shows an example where the blur kernel (1605) touches the foreground region (1610), shown in black. In such a case, ignore any non-zero blur kernel coefficients in the foreground region. Then, rescale the remaining coefficients in the background region so that they sum to one. In this process, there should be a small positive number in the rescale denominator to prevent division by zero (which can occur when the blur kernel is completely in the foreground region). Pseudocode for generating the modified blur kernel is shown below. Again, this modification of the blur kernel is switched on only if NUM_OBJ>1 and BLUR_FLAG[0]=1, which corresponds to a background blur experience. [Table 1]

[0097] Foreground Blur An encoder can provide motion blur only on foreground objects by specifying the metadata as NUM_OBJ>1 and setting BLUR_FLAG[n]=1, where n is the index of the foreground object that the author wants to receive blur. In such a case, the foreground object may be blurred out from the foreground mask boundary.

[0098] Figure 17 shows an example of the problem in the case of foreground blur. Panel (a) shows the image synthesis process in which an input image and a blurred image are applied with their respective masks. Only the areas shown in white are taken from each image. However, when a foreground mask shown in white is superimposed on the blurred image, as shown in panel (b), the blur effect of the foreground object may extend outside the foreground mask boundary. Therefore, as seen in panel (c), the final synthesized image has the blur effect of the foreground object beyond the foreground mask removed. This problem is solved by the operation shown in Figure 14.

[0099] Figure 18 shows a visualization of the mask at each step. First, we generate a binarized mask in which foreground object regions have the BLUR_FLAG set to 1 and 0 elsewhere. Note that panel (a) shows an example image with a single object (a scooter rider). However, the binarized mask can also contain multiple foreground objects with a BLUR_FLAG set to 1 if they receive a BLUR_FLAG of 1. We then apply the same blur kernel to the binarized mask as we did to the image. This allows non-zero values ​​in the binarized map to be expanded by the same amount as the blurred foreground object. An example of such a blurred mask map is provided in panel (b). We then perform a "hard expand" operation on this blurred map, binarizing it by setting non-zero values ​​to 1 (and 0 elsewhere). The goal of this hard expand operation is to expand the foreground mask to completely cover the blurred foreground object. A visual example of a hard expand mask is provided in panel (c). A "soft dilation" operation is then performed, where a generalized Gaussian blur filter is applied to the dilated mask. In this example, a 2D Gaussian filter with a standard deviation of 5 is used. This soft dilated mask is sent to the image compositing module as the final processed mask map. The purpose of this soft dilation is to create a mask that allows for a more natural blending of the blurred foreground with the rest of the region in later image compositing stages. A visual example of the soft dilated mask is shown in panel (d).

[0100] The pseudocode for the hard and soft dilation operations is shown below. [Table 2]

[0101] The coefficients for a 1D Gaussian kernel g with a standard deviation of 5 (in this example) can be generated in the following way: [Table 3]

[0102] The final image composition without the blending operation exhibits unnatural visual artifacts where the blurred foreground and background boundaries appear discontinuous. However, including the blending operation results in a more natural rendering of the blurred foreground and background boundaries.

[0103] Image Synthesis Module The Image Compositing module receives the blurred image and processed mask map generated from the Blur Application module. It also receives the baseline image and metadata and uses them to composite the final output image. The following are example cases of image compositing that can be signaled through different combinations of metadata:

[0104] 1) No blur applied An encoder can choose not to apply any blur to an image. In such a case, the encoder can signal metadata with, for example, BLUR_FLAG[0]=0.

[0105] If NUM_OBJ=1, the decoder treats the whole image as one object, in other words, there is no need to separate foreground objects from the background, and the same operations (e.g., no blurring) are applied to the whole image.

[0106] By setting the BLUR_FLAG corresponding to the entire image to 0, the encoder is indicating that no operation will be performed on the image. Optionally, the image may contain foreground objects, and the BLUR_FLAG for each foreground object may also be set to 0, but this is extra data to achieve the same effect.

[0107] If NUM_OBJ=1, there is no need to send MASK_MAP.

[0108] 2) Apply a blur to the entire image An encoder can choose to apply blur to the entire image by configuring the metadata similar to the "no blur for the entire image" option, except that BLUR_FLAG[0]=1 (or, for foreground images, BLUR_FLAG[n]=1 for all n).

[0109] 3) Background blur If an image has background and foreground objects, the encoder can choose to apply blur only to the background by configuring the metadata in the following way:

[0110] Set NUM_OBJ=N (where N=total number of foreground and background objects), which means that the decoder will treat the image to be composited as one background and (N-1) foreground objects.

[0111] Set BLUR_FLAG=[1,0,0,...,0], where the array is an Nx1 array and the first element of the array corresponds to the background. By setting BLUR_FLAG[0]=1, the encoder specified that the background is blurred.

[0112] It sends a MASK_MAP containing a label n ranging from 0 to (N-1), where locations with label "0" correspond to background regions and locations with label n point to regions corresponding to the nth object. This MASK_MAP is processed by the Blur Application module based on the BLUR_FLAG information as described herein. The resulting processed mask map is then sent to the Image Composition module.

[0113] 4) Foreground Blur If an image has background and foreground object(s), the encoder can choose to apply blur only to the foreground object(s) (all or a subset) by configuring the metadata in the following way:

[0114] Set NUM_OBJ=N as in case 3).

[0115] BLUR_FLAG=[0,f1,f2,…,f N-1 ] (again an Nx1 array). Let n be the label in the range 1 to (N-1) corresponding to the nth foreground object. BLUR_FLAG[0]=0, BLUR_FLAG[f n By setting BLUR_FLAG[f ]=1, the encoder is specifying that foreground objects with a flag value of 1 are blurred. n ] may be a mix of 1's and 0's, indicating that some ('1's) should be blurred, while others ('0's) should not be blurred.

[0116] It sends a MASK_MAP containing a label n ranging from 0 to (N-1), where locations with label "0" correspond to background regions and locations with label n point to regions corresponding to the nth object. This MASK_MAP is processed by the Blur Application module based on the BLUR_FLAG information as described herein. The resulting processed mask map is then sent to the Image Composition module.

[0117] Depth Map Information In 3D computer graphics and computer vision, a depth map is an image or image channel that contains information about the distance of a scene object's surface from the viewpoint. In one embodiment, the syntax parameter DEPTH_WEIGHT (depth map metadata) can define the depth map of an image. This section describes how traditional depth map data, as generated by techniques known in the art, can be converted into DEPTH_WEIGHT metadata values ​​for use with the present method.

[0118] Converting depth maps to DEPTH_WEIGHT metadata Areas further away may have a smaller magnitude of motion blur, while areas closer may have a more significant level of blur. In such cases, the encoder can set DEPTH_FLAG=1 and send DEPTH_WEIGHT, which controls the intensity of the blur generated from the motion blur parameter estimation module. DEPTH_WEIGHT is generated from the encoder side and contains a weight applied to the blur intensity at each pixel location. The weight is determined based on the depth information, and smaller values ​​should correspond to more distant areas that are less likely to be affected by motion blur. The modified intensity for the i-th pixel is calculated by S(i) × DEPTH_WEIGHT(i).

[0119] It may be the case that an image and a corresponding ground truth depth map are ready, but as shown above, it may also be necessary to generate a depth map from a given single image.

[0120] The depth map information is converted to DEPTH_WEIGHT, and the sensitivity to different depth levels can be controlled through configuration on the encoder side. In one example, assume that a depth map is provided along with an image (ground truth or generated). Figure 19 shows the procedure for processing the depth map into DEPTH_WEIGHT. Since depth maps (D) obtained from different methods may contain different value ranges, we apply min-max normalization to constrain the depth map value range from 0 to 1.

[0121] This normalized depth map (D') is then applied to a depth transfer function that converts D' to a DEPTH_WEIGHT. In this process, the encoder can specify which of three transfer functions to use. Linear: The normal conversion of D' to DEPTH_WEIGHT, where the formula is DEPTH_WEIGHT=1-D' is given by Cosine Squared: Makes DEPTH_WEIGHT less sensitive to different depth levels. This means that the blur strength is reduced only for significantly far areas. The formula is DEPTH_WEIGHT=cos 2 ((π / 2.5)×D') is given by Exponential: Makes DEPTH_WEIGHT more sensitive to different depth levels. This means that the blur strength is reduced even for areas that are a little far away. The formula is DEPTH_WEIGHT=exp(-3×D') is given by

[0122] Figure 20 shows a transfer function curve plotting D' versus DEPTH_WEIGHT, along with a visualization of the DEPTH_WEIGHT map. Again, the encoder can configure which transfer function to use to generate the DEPTH_WEIGHT based on its intent. If the encoder wants the blur strength to be less sensitive to different depth levels, it can select a cosine-squared transfer function. It can also select an exponential function for higher sensitivity, or a linear function for standard sensitivity.

[0123] This transfer calculation does not need to be done on the decoder side: the encoder only needs to provide the available depth map along with the transfer function selection and DEPTH_FLAG=1. Then, the min-max normalization of the depth map and the transfer function calculation can all be done on the encoder side, and the resulting DEPTH_WEIGHT can be sent to the decoder side with DEPTH_FLAG=1.

[0124] Example This section provides, but is not limited to, various examples for defining the metadata parameters defined herein.

[0125] In Example 1, consider defining exemplary metadata for applying only background blur to an image. For example, the image may contain a foreground person posing in front of a stationary background (e.g., a bridge). The encoder input configuration may be as follows: [Table 4]

[0126] This specifies that there is one background and one foreground via NUM_OBJ=2. Blurring only the background is specified by BLUR_FLAG=[1,0]. Sending depth information (DEPTH_FLAG=1) ensures that distant areas of the background are blurred less, to add realism to the motion blur.

[0127] Note that there is a parameter "DEPTH_TF" that is not part of the metadata. DEPTH_TF refers to the transfer function applied to convert the depth map to DEPTH_WEIGHT. The value can be one of -1, 0, and 1, where respectively, it refers to cosine squared (less sensitive), linear (normal sensitivity), and exponential (more sensitive). Having lower sensitivity means less sensitivity to different depth levels, thus reducing the blur strength only for areas that are significantly far away. This transfer calculation does not need to be done on the decoder side. Therefore, it can be handled on the encoder side, and only the final output DEPTH_WEIGHT is sent as metadata. Configured Metadata [Table 5]

[0128] This is the configured metadata (for this example) that is sent from the encoder to the decoder side for motion blur calculation. The creator (encoder side) needs to have any necessary metadata maps prepared in the same folder as the input image, so that they will be called in the metadata when flagged from the encoder input configuration. In this example, MASK_MAP and DEPTH_WEIGHT were loaded and sent to the decoder.

[0129] Device Speed ​​Configuration In this example, the absolute value of the device's velocity is increased linearly in each direction until it reaches MAX_VEL. For this example, the velocity direction is set as diagonal (going in both the x and y directions). Therefore, background blur occurs in the diagonal direction, with larger absolute values ​​of velocity giving higher levels of blur.

[0130] Background blur is observed in diagonal directions, with larger absolute motion resulting in higher levels of motion blur, and areas further away from the DEPTH_WEIGHT are less affected by motion blur.

[0131] In Example 2, consider an example metadata map for an object with horizontal motion blur (e.g., a skater or a car moving in the foreground). The encoder input configuration could be: [Table 6]

[0132] In this example image, the encoder expects horizontal motion in the foreground (e.g., the scooter rider), and therefore simulating panning blur on this sequence should give a motion blur on the background that occurs horizontally. Therefore, the encoder sets ANG_FLAG=1 and provides an angle map that gives the expected blur direction pointing to the left. Configured Metadata [Table 7]

[0133] Device Speed ​​Configuration: For this example, the mobile device's motion sets the input velocity direction diagonally (going in both the x and y directions). However, in this case, the angle map metadata specifies (ANG_X,ANG_Y) to point to the horizontal (left) direction, so the decoder projects its blur interpretation in this metadata direction, thus giving a background blur that occurs only in the horizontal direction.

[0134] As expected, background blurring is observed in the horizontal direction, even though the device provides diagonal input speeds.

[0135] In Example 3, we consider an example of a metadata map visualization for another object with motion blur in the foreground on a background with large image depth variations (e.g., a moving train coming towards the camera holder). The encoder input configuration could be as follows: [Table 8]

[0136] The corresponding configured metadata is [Table 9]

[0137] Metadata Map Visualization In this example, (ANG_X, ANG_Y) on the foreground region is expected to be set so that its direction matches the direction in which the foreground object (for example, a train) is heading.

[0138] Device Speed ​​Configuration: For this example, the mobile device's motion sets the input velocity direction to be horizontal and traveling in the +x direction. This would originally shape the blur to be unidirectional, pointing to the left. However, because the encoder specifies the direction of the foreground object (the train) via (ANG_X,ANG_Y), the train will be blurred in the direction it is heading.

[0139] As intended, the foreground object (the train) is blurred in the direction the train is moving, even with purely horizontal device movement. The magnitude of the blur in closer parts of the train is larger compared to the magnitude of the blur in more distant parts of the train, which is the expected experience. This was achieved by setting DEPTH_FLAG=1 and applying DEPTH_WEIGHT to the blur strength. This is a good example of how various metadata maps contribute to adding realism to the motion blur experience.

[0140] Exemplary Metadata The metadata can be used to provide instructions to the decoder on how to apply the blur.

[0141] For example, the metadata may include object information for separating an image into one or more foreground regions (e.g., objects) and background regions. Each object may have a "blur flag" that indicates whether that region should be blurred (e.g., "0" = no blur, "1" = blur applied), and a mask map that identifies which pixels are associated with which object. For example, the metadata may include "NUM_OBJ," "BLUR_FLAG[n]," and "MASK_MAP[i][j]" metadata.

[0142] In this example, NUM_OBJ specifies the number of objects the encoder wants to consider separately for the input image. In this example, the value is an integer greater than 0. A value of 1 indicates that the entire image is considered as one object and blur is applied (or not) to the entire image depending on BLUR_FLAG. A NUM_OBJ value greater than 1 may indicate that there is 1 background object and NUM_OBJ-1 foreground objects that may be considered separately. The default value may be 1.

[0143] In this example, BLUR_FLAG[n] specifies which objects receive blur. The value n ranges from 0 to NUM_OBJ-1, and BLUR_FLAG for each n contains a flag indicating whether to blur the corresponding region. The value of BLUR_FLAG for each n can be 0 or 1, where 1 indicates that the region receives blur and 0 indicates that it does not. The first entry of the parameters, BLUR_FLAG[0], corresponds to the flag for the background, while subsequent entries correspond to the flags for each of the foreground objects (if any). An encoder may be configured to blur the background by setting BLUR_FLAG[0] = 1 while setting the other entries to zero. An encoder may also be configured to blur all (or a subset) of the foreground objects by setting BLUR_FLAG[0] = 0 while setting all (or a subset) of the remaining entries to 1. Bluring the entire image may be achieved by setting NUM_OBJ = 1 and BLUR_FLAG[0] = 1. Preserving the original image can be achieved by having NUM_OBJ=1 and BLUR_FLAG[0]=0. The default value can be BLUR_FLAG=[0].

[0144] In this example, MASK_MAP[i][j] specifies which object pixel (i,j) corresponds to. It is an object ID map that specifies which image regions correspond to which objects. The value for each pixel location may be an integer n ranging from 0 to NUM_OBJ-1, with each value acting as a label for MASK_MAP. Pixels with value 0 correspond to background regions, and pixels with value n correspond to regions showing the nth object. In some embodiments, MASK_MAP is sent only if NUM_OBJ>1. The default option may be to not send a map.

[0145] The metadata can include information about the radial blur center(s).

[0146] RAD_BLUR_CENTER_FLAG can specify which offset coordinate(s) should be used to determine the radial blur center(s). More specifically, the radial blur center is determined by adding some shift amount to the offset coordinate that depends on the movement of the portable device. Details of how the radial blur center is determined were provided earlier in this disclosure. The value of RAD_BLUR_CENTER_FLAG shall be 0, 1, or 2. A value of 0 indicates that the offset coordinate is selected to be the image center given as (width / 2, height / 2). A value of 1 indicates that the offset is selected to be the center specified in the metadata, with related information further provided through the metadata NUM_CENTERS and CENTER_COORDS. A value of 2 indicates that only the offset coordinate is extracted from DEPTH_WEIGHT. This selection assumes DEPTH_FLAG=1 and selects a single coordinate with argMIN(DEPTH_WEIGHT), taking into account the fact that "farther away" regions are very likely to be the radial blur center. The default value for this metadata is 0.

[0147] NUM_CENTERS specifies the number of offset coordinates given by the encoder. The value can be any integer greater than or equal to 1. The default is to not send this parameter.

[0148] CENTER_COORDS[c] specifies the cth center coordinate specified by the encoder. The value c ranges from 0 to NUM_CENTERS-1, and for each c, CENTER_COORDS contains two entries corresponding to the column and row coordinates of the offset. The dimension of CENTER_COORDS is 2 x NUM_CENTERS. The default is to not send this parameter.

[0149] The metadata information can include depth information. This metadata group allows you to control the intensity of the blur based on the depth information, giving users a more realistic motion blur experience.

[0150] DEPTH_FLAG specifies the use of depth map information. Its value can be 0 or 1. A value of 0 indicates that no depth related information is being sent. A value of 1 indicates that a DEPTH_WEIGHT containing depth information is being sent. The default value is 0.

[0151] DEPTH_WEIGHT[i][j] specifies the weight (scaling factor) applied to the blur intensity at each pixel location (i,j). The weight is determined based on the depth map information. The value for each pixel location ranges from 0 to 1, with smaller values ​​corresponding to more distant regions that are less likely to be affected by motion blur. DEPTH_WEIGHT can be derived from the encoder using DEPTH_WEIGHT, which is sent to the decoder through metadata. This DEPTH_WEIGHT can be derived by applying a transfer function to the depth map information. However, this can later be changed to storing a signal for the depth map and transfer function in the metadata. This is because this framework can be extended to include other interactive experiences, in which case depth map information may be required at the decoder. The default option for this metadata is to not send the map.

[0152] The metadata can contain information about the direction (angle) of the motion blur, which is useful when there is an expected direction of motion in the image (e.g., a moving train).

[0153] ANG_FLAG specifies the use of angle map information. Its value can be 0 or 1. A value of 0 indicates that no information about the expected blur direction is sent. A value of 1 indicates that information about the expected blur direction is sent through the parameters ANG_X and ANG_Y. The default value is 0.

[0154] ANG_X[i][j] specifies the x component of the expected direction vector for the (i,j) pixel. ANG_X is sent only if ANG_FLAG=1. The default option is to not send a map.

[0155] ANG_Y[i][j] specifies the y component of the expected direction vector for the (i,j) pixel. ANG_Y is sent only if ANG_FLAG=1. The default option is to not send a map.

[0156] Other data can also be included in the metadata, such as:

[0157] MAX_BLUR specifies the maximum level of blur that can be reached for this image through movement of the portable device. The value can be, for example, in the range 0 to 255, and the unit can be pixels. The default value can be set to, for example, 65.

[0158] MAX_VEL[m] specifies the maximum level of velocity values ​​(e.g., in SI units of m / s) that can be reached through the movement of the portable device for each axis. The value m ranges from 0 to 2, corresponding to the x, y, and z axes, respectively. This MAX_VEL acts as a normalization factor for velocity interpretation and thus controls the sensitivity of the blur to the movement of the portable device. Note that each device may have a different value range. The default value may be, for example, [500,500,250].

[0159] VL_X specifies the sensitivity of the device to motion in the x direction relative to the other (y and z) directions, with higher values ​​indicating greater sensitivity. The value can be any non-negative number. A recommended value for normal experience is VL_X=5. You can also set VL_X=0 to limit the blur response to motion in the y and z directions only. An exemplary default value can be VL_X=5.

[0160] VL_Y specifies the sensitivity of the device to motion in the y direction relative to the other (x and z) directions, with higher values ​​indicating greater sensitivity. The value can be any non-negative number. The recommended value for normal experience is VL_Y=5. You can also set VL_Y=0 to limit the blur response to motion in the x and z directions only. For example, when simulating zoom blur, you can set both VL_X and VL_Y to zero to constrain the blur to motion in the z direction only. An exemplary default value can be VL_Y=5.

[0161] The map may be transmitted using any suitable compression algorithm as known in the art.

[0162] Although several embodiments of the present disclosure have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following enumerated example embodiments (EEE) that describe the structure, features, and functionality of some portions of the present invention.

[0163] EEE1: A method for providing interactive blur on an image viewed on a mobile device, the method comprising: measuring at least one sensor output from the mobile device; receiving metadata related to motion blur from an encoder; performing motion blur estimation based on the at least one sensor output and the metadata; selecting at least one blur kernel from a filter bank for corresponding pixels of the image based on the motion blur estimation; and applying blur to the image by using the at least one blur kernel at each corresponding pixel to generate a blurred image, the applying using an angle map to adjust the blur direction at the corresponding pixels.

[0164] EEE2: The method of EEE1, further comprising performing image synthesis on the blurred image to generate an output image.

[0165] EEE3: The method of any of EEE1 or 2, further wherein applying the blur includes using a depth map to adjust blur intensity at each corresponding pixel.

[0166] EEE4: A method according to any one of EEE1 to EEE3, further comprising separating foreground objects from background objects using a mask map, and wherein applying the blur is selectively performed on foreground and background objects based on the metadata.

[0167] EEE5: The method of any one of EEE1 to EEE4, further comprising the steps of determining a number of radial blur centers from the metadata, and if there is more than one radial blur center, generating multiple blur vector maps, deriving blur vector map weights, and aggregating the multiple blur vector maps to create an aggregate blur vector map used for applying the blur.

[0168] EEE6: The method of any one of EEE1 to EEE5, wherein the sensor output includes velocity data related to movement of the mobile device.

[0169] EEE7. The method of any one of EEE1 to EEE6, further comprising deriving a radial blur center based on said metadata.

[0170] EEE8: The method of EEE7, further comprising generating a blur vector map based on the radial blur centers.

[0171] EEE9: The method of any one of EEE1 to EEE8, further comprising determining from the metadata whether the entire image should be blurred, only background regions of the image should be blurred, or the foreground image should be blurred.

[0172] EEE10: The method according to EEE9, further comprising generating an aggregated blur vector map based on said blur vector map and derived blur vector map weights.

[0173] EEE11. The method of any one of EEE1 to EEE10, further comprising generating a motion blur map intensity and direction based on the metadata.

[0174] EEE12. The method of any one of EEE1 to 11, further comprising modifying the blur kernel to have coefficients only in background regions of the image.

[0175] EEE13: The method of any one of EEE1 to 12, further comprising receiving mask maps of a background region of the image and one or more foreground images, the mask maps being identified in the metadata.

[0176] EEE14: The method of any one of EEE1 to EEE13, wherein the motion blur estimation comprises using a mask map to mask between foreground objects and background in the image.

[0177] EEE15: The method of any one of EEE1 to EEE14, wherein the motion blur estimation comprises using a depth map to provide a blur weighting value.

[0178] EEE16: The method of any one of EEE1 to 15, wherein the angle map is used to apply a scaling factor that controls the sensitivity of the blur in each direction.

[0179] EEE17. The method of any one of EEE7 to 16, further comprising calculating a vector from the radial blur center to each pixel location.

[0180] EEE18: The method according to any one of EEE8 to 17, wherein the strength of each blur kernel of the at least one blur kernel is calculated as a combination of the normalized filter strength, the absolute value of the velocity and the maximum blur value of the corresponding pixel.

[0181] EEE19: The method of EEE18, wherein the normalized filter strength is calculated from the blur vector map.

[0182] EEE20: A decoder configured to perform the method of any one of EEE1 to EEE17, comprising an image decoder, a motion blur estimation module, a filter bank and a blur application module.

[0183] EEE21: The decoder according to EEE20, further comprising an image synthesis module.

[0184] EEE22: An encoder configured to provide an image and metadata to a decoder according to EEE20 or 21, the encoder being configured to: encode the image into an encoded image; generate a depth map, a mask map and an angle map for the image based on preferences; and multiplex the encoded image, the depth mask, the mask map and the angle map as output to the decoder.

[0185] EEE23: The encoder according to EEE22, further configured to provide metadata related to said interactive blurring to said decoder.

[0186] EEE24. The encoder of EEE22 or 23, wherein the encoder is further configured to convert the depth map into depth weight metadata using a transfer function.

[0187] EEE25: The encoder according to EEE24, wherein the transfer function is one of a linear, a cosine squared, or an exponential function.

[0188] EEE26: ​​A mobile device configured to perform a method according to any one of EEE1 to 20, the mobile device having a screen and at least one sensor configured to measure a velocity of the mobile device.

[0189] EEE27: The mobile device according to EEE26, wherein the sensor is an accelerometer.

[0190] EEE28: A mobile device according to EEE26 or 27, wherein the mobile device is one of a smartphone, a tablet computer or a smartwatch.

[0191] The foregoing examples are provided to those skilled in the art as a complete disclosure and description of how to make and use the embodiments of the present disclosure, and are not intended to limit the scope of what the inventors regard as their disclosure.

[0192] Modifications of the above-described modes for carrying out the methods and systems disclosed herein that are obvious to those skilled in the art are intended to be within the scope of the following claims. All patents and publications mentioned in this specification are indicative of the level of skill of those skilled in the art to which this disclosure pertains. All references cited in this disclosure are incorporated by reference as if each reference was individually incorporated by reference in its entirety.

[0193] It is understood that the present disclosure is not limited to particular methods or systems, which can, of course, vary. It is also understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. The term "plurality" includes two or more referents unless the content clearly dictates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

Claims

1. 1. A method of providing motion blur to an image viewed on a mobile device, comprising: measuring at least one sensor output from the mobile device; generating motion blur parameters based on the at least one sensor output and metadata provided with the image; selecting at least one blur kernel from a filter bank for a corresponding pixel of the image based on the motion blur parameters; applying a blur to the image by using the at least one blur kernel at each corresponding pixel to generate a blurred image; performing image synthesis based on the image and the blurred image to generate an output image; presenting the output image on the mobile device; the at least one sensor output includes velocity data related to movement of the mobile device; method.

2. The method of claim 1 , further comprising repeating the step of creating the output image in response to temporal changes in the at least one sensor output.

3. The method of claim 1 , wherein applying the blur includes using an angle map to adjust the blur direction at the corresponding pixel.

4. The method of claim 1 , wherein applying the blur includes using a depth map to adjust blur intensity at each corresponding pixel.

5. 10. The method of claim 1, further comprising separating foreground objects from background objects using a mask map, and wherein applying blur is selectively performed on foreground and background objects based on the metadata.

6. 2. The method of claim 1, further comprising: determining a number of radial blur centers from the metadata; and if there is more than one radial blur center, generating multiple blur vector maps, deriving blur vector map weights, and aggregating the multiple blur vector maps to create an aggregate blur vector map used for applying the blur.

7. The method of claim 1 , further comprising deriving a radial blur center based on the metadata.

8. The method of claim 7 , further comprising generating a blur vector map based on the radial blur centers.

9. The method of claim 1 , further comprising determining from the metadata whether the entire image should be blurred, only a background region of the image should be blurred, or a foreground image should be blurred.

10. The method of claim 6 , further comprising generating an aggregated blur vector map based on the blur vector map and derived blur vector map weights.

11. The method of claim 1 , further comprising generating a motion blur map intensity and direction based on the metadata.

12. The method of claim 1 , further comprising modifying the blur kernel to have coefficients only in background regions of the image.

13. The method of claim 1 , further comprising receiving a mask map of a background region of the image and one or more foreground images, the mask map being identified in the metadata.

14. The method of claim 1 , wherein generating the motion blur parameters includes using a mask map to mask between foreground objects and background in the image.

15. The method of claim 1 , wherein generating the motion blur parameters includes using a depth map to provide a blur weighting value.

16. The method of claim 3 , wherein the angle map is used to apply a scaling factor that controls the sensitivity of the blur in each direction.

17. The method of claim 7 , further comprising calculating a vector from the radial blur center to each pixel location.

18. The method of claim 8 , wherein the strength of each blur kernel of the at least one blur kernel is calculated as a combination of the normalized filter strength, the absolute value of the velocity, and the maximum blur value of the corresponding pixel.

19. The method of claim 18 , wherein the normalized filter strength is calculated from the blur vector map.

20. 20. A decoder configured to perform a method according to any one of claims 1 to 19, comprising: an image decoder; a motion blur estimation module; a filter bank; a blur application module; decoder.

21. 21. The decoder of claim 20, further comprising an image synthesis module.

22. 21. An encoder configured to provide images and metadata to a decoder according to claim 20, the encoder comprising: encoding the image into an encoded image; generating a depth map, a mask map and / or an angle map for the image based on the preferences; multiplexing the encoded image, the depth map, the mask map, and the angle map as outputs to the decoder; The encoder is configured as follows:

23. The encoder of claim 22, further configured to provide metadata related to interactive blurring to the decoder.

24. The encoder of claim 22 , wherein the encoder is further configured to convert the depth map into depth weight metadata using a transfer function.

25. 25. The encoder of claim 24, wherein the transfer function is one of a linear, a cosine squared, or an exponential function.

26. 20. A mobile device configured to perform the method of any one of claims 1 to 19, the mobile device having a screen and at least one sensor configured to measure the speed of the mobile device.

27. 27. The mobile device of claim 26, wherein the sensor is an accelerometer.

28. 27. The mobile device of claim 26, wherein the mobile device is one of a smartphone, a tablet computer, or a smartwatch.

Images