Characteristics of computational photography using depth

Abstract

The method includes receiving an image as a portion of a real-world space, placing an anchor on the image, determining a position of the anchor, determining a depth associated with the position of the anchor, applying a depth-based image editing algorithm to the captured image, and rendering the edited image.

Description

[Technical field]

[0001] Field Embodiments relate to digital photography image editing. [Background technology]

[0002] background Computational photography techniques in mobile devices include using machine learning based segmentation to enhance the visual appearance of an image by applying screen space enhancements, filters, etc., and / or background blurring, modification, etc. For example, a filter can be applied in screen space to modify the coloring (e.g., change the blue color, change to grayscale, brighten the color, etc.). In another example, a learned algorithm can distinguish between a human and a background in screen space and blur the background (or insert a different background). In another example, if a first portion of an image is foreground and a second portion of the image is background, the background can be blurred based on depth. In other words, a depth d0 is identified, and a depth df less than d0 is foreground and a depth db greater than d0 is background. Then, the entire background portion of the image (depth db) is blurred, while the entire foreground portion of the image (depth df) is left untouched. Summary of the Invention [Means for solving the problem]

[0003] overview In a general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method can perform a process using a method that includes receiving an image as a portion of a real-world space, placing an anchor on the image, determining a position of the anchor, determining a depth associated with the position of the anchor, applying a depth-based image editing algorithm to the captured image, and rendering the edited image.

[0004] Implementations may include one or more of the following features. For example, the captured image may be a live image, the rendering of which is constantly modified using a depth-based image editing algorithm. The anchor may place a focal point at a depth in the captured image, and the viewpoint of the captured image may be changed and the focal point may remain on the depth. Applying a depth-based image editing algorithm to the captured image may include applying a first algorithm to a first portion of pixels that includes pixels of the captured image within a depth threshold distance, and applying a second algorithm to a second portion of pixels that includes the remaining pixels of the captured image.

[0005] For example, applying a depth-based image editing algorithm to the captured image may include determining an object associated with the image, determining edges of the object based on the depth, applying a first algorithm to a first portion of pixels of the captured image including pixels of the captured image associated with the edges of the object, and applying a second algorithm to a second portion of pixels including remaining pixels of the captured image. Determining a depth associated with the location of the anchor may include obtaining a depth map associated with the captured image.

[0006] For example, determining a depth associated with a position of the anchor may include obtaining a depth map associated with the captured image, determining a minimum depth value associated with the captured image, determining a maximum depth value associated with the captured image, normalizing the depth map based on the minimum and maximum depth values, and determining a depth associated with the position as the normalized depth value. Determining a depth associated with a position of the anchor may include obtaining a layered depth associated with the captured image, the depth of the anchor having a depth index. Statue editionThe method further includes the step of selecting a collection tool, the image editing tool corresponding to the image editing algorithm.

[0007] Exemplary embodiments will become more fully understood from the detailed description provided herein below and the accompanying drawings, in which like elements are represented by like reference numerals and which are provided by way of example only and not limitation of the exemplary embodiments. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a portion of a user interface in accordance with at least one exemplary embodiment. [Diagram 2] FIG. 1 illustrates a photographic representation of a portion of a user interface in accordance with at least one exemplary embodiment. [Diagram 3] FIG. 1 illustrates a flowchart of a method for editing an image in accordance with at least one example embodiment. [Figure 4] FIG. 1 illustrates a flowchart of a method for editing an image in accordance with at least one example embodiment. [Diagram 5] FIG. 1 is a block diagram of an image processing system in accordance with at least one example embodiment. [Figure 6] FIG. 1 illustrates an example of a computing device and a mobile computing device in accordance with at least one exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] It should be noted that these figures are intended to illustrate general characteristics of methods, structures and / or materials utilized in certain exemplary embodiments and are intended to supplement the description provided below. However, these figures are not to scale and may not precisely reflect the exact structural or performance characteristics of any given embodiment, and should not be construed as defining or limiting the range of values ​​or characteristics encompassed by the exemplary embodiments. For example, the relative thicknesses and positioning of molecules, layers, regions, and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various figures is intended to indicate the presence of similar or identical elements or features.

[0010] Detailed Description of the Embodiments Using the techniques described herein, depth maps and / or depth layers can be used to create new options for computational photography. Computational photography techniques using depth features can enhance the user experience by enabling image manipulation at identified depths. For example, computational photography techniques using depth features can enable pixel enhancement at the target depth and pixel distortion at other depths to enable a user to place enhanced focus on elements of the image that are at the target depth. Depth can be a distance from a reference point. Depth can be a distance and a direction from a reference point. Each pixel in an image can have a depth. Thus, each pixel in an image can have an associated distance (and optionally a direction) from a reference point. The reference point can be based on the location of a device (e.g., a mobile device, a tablet, a camera, etc.) that includes the camera. The reference point can be based on the device's location on Earth (e.g., using a global positioning system (GPS)).

[0011] In one exemplary implementation, an anchor or virtual anchor may be fixed and / or locked in screen space (e.g., in an image displayed on a device) for use with depth-of-field calculations. The anchor may assist in taking a video while fixing the focus at a depth in real-world space (e.g., a user's environment). While the device moves in real-world space (e.g., changing the viewpoint), the depth of view for the device may be tracked in real-world space, e.g., using device sensors and an application programming interface (API) configured to use these sensors to track the depth from the device. In other words, the anchor may be locked at a depth and tracked as the device moves (e.g., without maintaining a fixed focal length). For example, if a marker is placed at a depth of 6 meters (not necessarily placed on a subject at 6 meters) and the device is moved to 5 meters (e.g., a user of the device moves forward), the depth of the marker changes to 5 meters as the device moves. Contrast this with object tracking, where the focus is locked on an object and as the object moves (or the device moves), the focus remains on the object as long as the object is within the field of view. In some implementations, tracking can be a feature included in an augmented reality API.

[0012] For example, a user may take a video of a subject in his / her environment and apply an image effect (e.g., an enhancement) at a certain depth (e.g., not necessarily on the subject). To apply the image effect, the user may touch the depth (on a touch screen display of the device taking the video) to place an anchor at that depth in real world space (as displayed on the device or in screen space). Then, once the anchor is placed, the user may walk around in his / her environment and the focus will remain at the anchored depth. In one exemplary implementation, the user may move around (e.g., change viewpoint) and even look away and the focus may remain at that depth. Also, multiple such anchor points may be used (e.g., placed in screen space).

[0013] In this example, the object may (but is not necessarily) at the depth of the anchor, enabling new object tracking techniques. Depth tracking is an improvement over current techniques for object tracking because if the object is the focus in screen space using current techniques, the user focuses on the object to, for example, take a picture of the object. If the user moves to another viewpoint to take another picture, currently the camera uses an object recognition algorithm (or an acoustic autosensor) to refocus on the object. Using an object recognition algorithm has many drawbacks, including distinguishing between two of the same object, not refocusing if the object is no longer in view and / or has come back into view, requiring a fast processor (or using a large amount of processing resources), needing specialized sensors, etc. In contrast, an exemplary implementation can retain the position and depth of the object (e.g., in three dimensions) and can maintain the same focus. In some implementations, tracking can include using object recognition and depth tracking.

[0014] In this description, the screen space may be the visual area of ​​an image as shown on a display of a device (e.g., a mobile device). The real-world space may be the visual environment in which the device is operating (e.g., taking video). The viewing space may be the visual area of ​​a rendered image (e.g., may be the same or a different visual area as the screen space).

[0015] 1 illustrates a block diagram of a portion of a user interface in accordance with at least one exemplary embodiment. As shown in FIG. 1, interface 105 includes a plurality of objects 110-1, 110-2, 110-3, 110-4, 110-5, a slide bar 115, and an anchor 125. Slide bar 115 includes a slider 120. Anchor 125 may be configured to identify a depth associated with a processing action (e.g., enhancement, filter, fog, blur, etc.). Slide bar 115 may be configured to control a processing action (e.g., amount of blur, amount of fog, etc.).

[0016] In one exemplary implementation, the interface 105 may be part of a user interface of an image (or video, or frame of video) processing application. The anchor 125 may be placed (e.g., using a touch screen action, a mouse action, etc.) on the object (e.g., object 110-3) that causes the depth determination and / or at a depth of the image that includes objects 110-1, 110-2, 110-3, 110-4, 110-5. Each of these objects is at a different depth in the image and / or two or more of these objects are at the same depth. The image may include depth information as captured by a camera (e.g., the camera may be an RGBD sensor) and / or determined based on the image (e.g., using depth from a motion algorithm that creates a depth map). In other words, the image may have an associated depth map that identifies the depth of each pixel in the image. Identifying the depth of each pixel may make it possible to determine (or help determine) the relative depth for each of the objects 110-1, 110-2, 110-3, 110-4, 110-5.

[0017] A user of a device including the interface can select an image processing action to be taken based on the depth associated with the anchor 125. For example, pixels at the depth (or within the depth range) of the anchor can be enhanced (e.g., improved quality), and pixels not at the depth (or within the depth range) of the anchor can be manipulated (e.g., distorted, blurred, clouded, etc.). A slide bar can be used (by the user) to control, for example, the level of distortion and / or the range of depth. The depth can be a depth associated with a pixel (e.g., of a depth map) of a selected object and / or a layer (e.g., z-layer) of a selected object.

[0018] 2 illustrates a photographic representation of a portion of a user interface in accordance with at least one exemplary embodiment. As shown in FIG. 2, the portion of the user interface includes an image in a first state 205, an image in a second state 210, a selection tool 215, an anchor (illustrated as a ball) at a first location 220, and an anchor (illustrated as a ball) at a second location 225. The selection tool 215 is used to select an image from the image. Statue edition The selection tool 215 is configured to select a focus (e.g., a processing action, enhance, focus, distort, blur, fog, filter, etc.) as shown in FIG. 2. The selection tool 215 is already used (e.g., via a touch screen or swipe) to select focus as a processing action.

[0019] The image in the first state 205 shows the anchor at a first position 220. The anchor at the first position 220 causes the image in the first state 205 to have a focal point that is deep or far away in the image (as selected by the selection tool 215). As shown in the image in the first state 205, the distant object (gym equipment) is in focus while the closer object is blurred. As shown in the image in the second state 210, the closer object (gym equipment) is in focus while the farther object is blurred. In one exemplary implementation, changing the position of the anchor from the first position 220 to the second position 225 causes pixel manipulation (e.g., focus and blur) at different depths. This pixel manipulation may be referred to as a depth of field effect.

[0020] In one exemplary implementation, a portion of a real-world space is previewed as an image or a screen space image on a device (e.g., a display, a viewfinder, etc.). A user of the device can place at least one anchor on the image (e.g., using a touch screen of the device). The device can be configured to determine a position of the anchor (e.g., coordinates (x,y) in screen space) and a depth associated with the position (and thus the anchor). For example, the depth can be based on a pixel depth (e.g., using a depth map) at the anchor position. A depth-based image editing algorithm can be applied to the previewed image. The edited image can be rendered on a display of the device. The previewed image can be a live image, the rendering of which is constantly modified using the depth-based image editing algorithm. When the anchor is moved from a first position 220 to a second position 225, a depth determination (e.g., using a position on the depth map) is triggered and a change of focus is triggered based on the determined depth.

[0021] In another implementation, the image in the first state 205 is a captured image displayed on the device using an application. When the anchor is moved from the first position 220 to the second position 225, a depth determination (e.g., using a position on a depth map) is made and triggers an image editing process. The image editing process can cause an image enhancement algorithm to be implemented for pixels at (or within) the determined depth, and an image blurring algorithm to be implemented for pixels not at (or within) the determined depth. The image blurring algorithm can vary the blurring based on the depth. For example, the further away from the determined depth, the more blurring is applied.

[0022] In another implementation, the image is a captured image with high resolution at all depths. The high resolution captured image can be displayed on the device using an application. The anchor can be placed at the first location 220 to cause a depth determination (e.g., by the application) to be made (e.g., using a position on a depth map) and trigger an image editing process. The image editing process can be an image blurring algorithm that is implemented for pixels that are not at the determined depth (or are not within the determined depth range). The image blurring algorithm can vary the blurring based on the depth. For example, the further away from the determined depth, the more blurring is applied. When the anchor is moved from the first location 220 to the second location 225, a new depth determination (e.g., using a position on a depth map) is made and triggers another image editing process. The image editing process can cause a deblurring process to be implemented for pixels at (or within) the determined depth, or can cause reprocessing on the original image and an image blurring algorithm to be implemented for pixels not at (or within) the determined depth. The image blurring algorithm can vary the blur based on depth, e.g., the further away from the determined depth, the more blurring is applied.

[0023] In one exemplary implementation, image editing includes editing pixels as shown in Table 1. In Table 1, d0 is the determined depth and D is the depth of the pixel. The example in Table 1 illustrates a focus / blur editing process.

[0024] [Table 1]

[0025] 3 and 4 are flow charts of methods according to example embodiments. The methods described with respect to Fig. 3 and 4 may be performed by execution of software code stored in a memory (e.g., a non-transitory computer-readable storage medium) associated with the device and executed by at least one processor associated with the device.

[0026] However, alternative embodiments are contemplated, such as the system being embodied as a dedicated processor. The dedicated processor may be a graphics processing unit (GPU). The GPU may be a component of a graphics card. The graphics card may also include a video memory, a random access memory digital-to-analog converter (RAMDAC), and driver software. The video memory may be a frame buffer that stores digital data representing an image, a frame of video, an object of an image, or a scene of a frame. The RAMDAC may be configured to read the contents of the video memory, convert the contents to an analog RGB signal, and send the analog signal to a display or monitor. The driver software may be software code stored in the memory described above. The software code may be configured to implement the methods described herein.

[0027] Although the methods described below are described as being performed by a processor and / or a dedicated processor, the methods are not necessarily performed by the same processor, in other words, at least one processor and / or at least one dedicated processor may perform the methods described below with respect to Figures 3 and 4.

[0028] FIG. 3 illustrates a flowchart of a method for editing an image according to at least one exemplary embodiment. As shown in FIG. 3, in step S305, an image is selected. The image may be selected from a memory associated with a device configured to edit an image. The image may be a frame of a video selected from a memory associated with a device configured to edit a video. The image may be selected as a preview image before the image is captured and stored in the device. The image may be selected as a preview image when a frame of a video is captured and stored in the device. In other words, the image or frame may be edited during the capture process (e.g., as a function of a camera application running on the device). In one exemplary implementation, the selected image includes depth information.

[0029] The depth information may include a depth map having a depth value for each pixel in the image. The depth map may be a color (e.g., RGB, YUV, etc.) image or an image associated with a frame of video. The depth map may be a black and white, grayscale, etc. image or an image associated with a frame of video. The depth map may store a distance value for each pixel in the image or frame. The depth value may have an 8-bit representation, for example, having a value between 0 and 255, where 255 (or 0) represents the closest depth value and 0 (or 255) represents the furthest depth value. In one exemplary implementation, the depth map may be normalized. For example, the depth map may be converted to a range of [0,1]. The normalization may include converting the actual range of the depth values. For example, if the depth map includes values ​​between 43 and 203, 43 will be converted to 0, 203 will be converted to 1, and the remaining depth values ​​will be converted to values ​​between 0 and 1.

[0030] The depth map may be generated by a camera that includes a depth (D) sensor. This type of camera may be referred to as an RGBD camera. The depth map may be generated using an algorithm implemented as a post-processing function of the camera while capturing an image or a frame of video. For example, the image may be generated from multiple (e.g., 3, 6, 9, or 10 or more) images captured by a camera (e.g., a camera of a mobile device). The algorithm may be implemented in an application programming interface (API) associated with capturing images of real-world spaces (e.g., for an augmented reality application).

[0031] In high dynamic range (HDR) image capture, multiple frames (e.g., 3, 6, 9, or 10 or more) can be combined into a single image. As a result, the camera can have the ability to average multiple frames to reduce noise, which improves image quality. In addition, the camera can generate a depth map using multiple images because the images can be from different angles that can be compared to estimate the distance to every pixel. In some cases, the depth map can become more accurate as the device moves around in real-world space.

[0032] The depth information may be represented as a 3D point cloud. In other words, the point cloud may be a depth map in three dimensions. The point cloud may be a collection of 3D points (X,Y,Z) that represent the exterior of the scene and may include color information.

[0033] The depth information may include depth layers, each having a number (e.g., an index or z-index) indicating a layer order. The depth information may be a layered depth image (LDI) having multiple ordered depths for each pixel in the image. The color information may be a color (e.g., RGB, YUV, etc.) for each pixel in the image. The depth image may be an image in which each pixel represents a distance from a camera position.

[0034] A layered depth image (LDI) may be an image-based representation of a three-dimensional (3D) scene. An LDI may include a two-dimensional (2D) array or group of layered depth pixels. Each layered depth pixel may include a set of LDI samples sorted along a line of sight as seen from a single camera position or viewpoint. The camera may be referred to as an LDI camera. Other ways of referencing an LDI sample may include, but are not limited to, a point, a depth pixel, or a layered depth pixel sample. For each LDI sample, a camera, referred to as a source camera, provides data associated with the LDI sample. The representation of an LDI pixel may include color information, alpha channel information, depth information (distance between the pixel and the camera), an identifier for the source camera for the LDI sample (e.g., a number, a pointer, or a reference to a camera), and other attributes that can support the rendering of the LDI in three-dimensional (3D) space.

[0035] In step S310, an image editing tool is selected. For example, a user of the device may select an image processing action to be used to edit the image. The image processing action may be configured to cause execution of an algorithm on the image. The algorithm may include at least image enhancement, image blur, image fogging, image filters, image cropping, image color change, image special effects, image distortion, etc. In some implementations, the image processing action may cause execution of more than one algorithm on the image. For example, image enhancement may be performed on a portion of the image (e.g., selected pixels) and image blurring may be performed on another portion of the image (e.g., selected pixels).

[0036] At step S315, a depth is selected. For example, as described above, an image may have associated depth information. The depth information may include a depth map, a depth per pixel, a layer, etc. Thus, selecting a depth may include selecting a distance from a depth map, a distance from a pixel, a layer (e.g., a z index), etc. As described above, an anchor may be placed at a location on an image. An anchor may be placed on an object, an anchor may be placed at a depth, etc. In one exemplary implementation, a depth is selected based on the placement and / or location of the anchor.

[0037] In step S320, an image editing tool is executed. In one exemplary implementation, the image editing tool is implemented based on the selected depth. For example, an algorithm may be executed on pixels of the image that are at the selected depth. Alternatively, or in addition, an algorithm may be executed on pixels of the image that are within a range of the selected depth. The range may be based on a threshold number of pixels (e.g., based on depth, closer, and / or farther). The range may be a number of pixels based on a threshold distance from the depth. The range may be variable (e.g., may be changed using a slide bar). In some implementations, an image processing action (based on the selected image editing tool) may cause the execution of more than one algorithm on the image. For example, image enhancement may be performed on pixels of the image that are at the selected depth (or within a range of the selected depth), and image blurring may be performed on the remainder of the pixels (or remaining pixels) in the image.

[0038] In step S325, the modified image is displayed. For example, the image modified using image editing tools may be displayed on the device (e.g., in the user interface of an application). In one exemplary implementation, the image or frame may be edited during the capture process (e.g., as a function of a camera application running on the device). Thus, an algorithm may be run continuously (e.g., frame by frame or number of frames) on the pixels of the image at the selected depth, and the modified image may be continuously updated and displayed. In other words, the image editing may include real-time depth-based image editing. In one implementation where an anchor places focus or emphasis on an object, the anchored object may remain in focus and / or emphasis as the device moves (e.g., changes viewpoint). The user interface may give the user options to modify image editing settings and select an option to take a snapshot of the display and save the image to the device's memory.

[0039] FIG. 4 illustrates a flowchart of a method for editing an image according to at least one exemplary embodiment. As shown in FIG. 4, in step S405, an image is selected. The image may be selected from a memory associated with a device configured to edit an image. The image may be a digital image obtained by digital photography. The image may be a frame of a video selected from a memory associated with a device configured to edit a video. The image may be selected as a preview image before the image is captured and stored in the device. The image may be selected as a preview image when a frame of a video is captured and stored in the device. In other words, the image or frame may be edited during the capture process (e.g., as a function of a camera application running on the device). In this implementation, the image may be a live image of a real-world space.

[0040] In one exemplary implementation, the selected image includes depth information. The depth information may include a depth map having a depth value for each pixel in the image. The depth information may be represented as a 3D point cloud. In other words, the point cloud may be a depth map in three dimensions. The depth information may include depth layers, each having a number (e.g., an index or z-index) indicating a layer order.

[0041] In step S410, an anchor (or a virtual anchor) is placed on the image. For example, the anchor may be placed at a location on the image. The anchor may be placed on an object, the anchor may be placed at a depth, etc. The anchor may be placed on the image via a touchscreen action of the device. The anchor may be placed on the image via a gestural action detected by the device. In one implementation, the image may be a live image of a real-world space. Thus, the anchor may be placed at a real-world location within the live image of the real-world space.

[0042] In step S415, the anchor is converted to a coordinate position of the image. For example, the image may have a two-dimensional (2D) representation (e.g., xy coordinates) based on the length and width of the device's display (e.g., screen space). In other words, the anchor may be represented by coordinates in screen space.

[0043] Step S42 0, pixel variables of the image are edited. For example, as described above, the image may be edited using algorithms. The algorithms may include at least image enhancement, image focus, image blur, image haze, image filters, image cropping, image color change, image special effects, image distortion, etc. For example, pixel-by-pixel blurring according to a measured physical distance to an anchor point algorithm may be performed. Distance-based pixel-by-pixel blurring may include blurring pixels such that the pixel becomes more blurred in the rendered result as the pixel moves further away from the anchor. As described above, the image may be a live image of a real-world space. Thus, pixel-by-pixel blurring may simulate a depth-of-field effect on the device in real time. It is used.

[0044] Step S4 25 A modified image is rendered. For example, a depth-based modified image may be rendered on a display of the device (e.g., within a user interface of an application). In one exemplary implementation, an image or frame may be edited during the capture process (e.g., as a function of a camera application running on the device). Thus, an algorithm may be run constantly (e.g., every frame or number of frames) on the pixels of the image at the selected depth, and the modified image may be constantly updated and displayed. In other words, the image editing may include real-time depth-based image editing.

[0045] In an exemplary implementation, image editing may be performed on live image capture of the real-world space. Thus, as the device moves, the screen space is recalculated, the depth map is regenerated, and the anchor's coordinates are recalculated. The image is re-edited and rendered on the device. In other words, the techniques described with respect to Figures 3 and 4 may be continually repeated (e.g., performed in a loop). In an implementation where an anchor places focus or emphasis on an object, the anchored object may remain in focus and / or emphasis as the device moves (e.g., changes viewpoint). The user interface may give the user options to modify image editing settings and select an option to take a snapshot of the display and save the image to the device's memory.

[0046] As an example use case, performing a cloudy effect as an image editing tool is described in the context of FIG. 3 and / or FIG. 4. Using a user interface (e.g., as described with respect to FIG. 1 and / or FIG. 2), a user selects a cloudy effect and sets a cloud thickness, a cloud distance, and / or a cloud color. A live image (e.g., RGB, YUV, etc.) and depth data (e.g., a depth map) of a real-time real-world space are captured. From the depth map, a minimum (MIN) depth value and a maximum (MAX) depth value can be determined. The depth range can be normalized by converting the depth map to the range [0,1]. An anchor can be placed, for example, on an object in the live image of the real-world space.

[0047] In one exemplary implementation, an anchor position in screen space is determined. The anchor position (or anchor positions) may be represented by an index in screen space. However, in an alternative or additional approach, the anchor position (or anchor positions) may be stored, for example, based on a screen touch in screen space. The depth of the anchor may be requested (e.g., a function call to an API) to obtain a depth value associated with the screen coordinates of the anchor. The screen space position of the anchor may be converted to a real-world spatial position of the device. The real-world position (e.g., 3D position) of the anchor may be stored to allow device movement without losing the anchor position when the anchor position is not in screen space. Finally, the anchor position may be converted to a view space position, and the depth may be normalized (e.g., to cause emphasis on the subject that is in focus) based on a minimum (MIN) depth value and a maximum (MAX) depth value. Displaying data on a device with a two-dimensional (2D) display may include mimicking a 3D display by filtering the 2D data (e.g., with a Gaussian filter) to appear as 3D when rendered. This technique is sometimes called rendering as 2.5D RGBD data. Thus, the depth of field data may be rendered as 2.5D RGBD data.

[0048] An anchor depth position is determined and normalized. The normalized depth may be D_a=(d_a-MIN) / MAX-MIN), where D_a is the anchor depth and D_a is the normalized depth. Then, each pixel is edited based on a fog effect algorithm. The fog effect algorithm may be: fog_factor=saturate((-depth_value-fog_distance) / fog_thickness) for each pixel. An alternative fog effect algorithm is: fog_factor=1.0-exp((-depth_value-fog_distance) for each pixel. *The fog factor can be 1.0-exp((-depth_value-fog_distance)*fog_thickness).

[0049] The display of a foggy live image can be a linear interpolation between the original pixel color (as captured by the camera) of the live image of the real-world space and the fog color due to the fog factor (fog_factor). Final color = original color * (1-cloudiness factor)+cloudiness factor * The fog color (Final_color = original color*(1-fog_factor)+fog_factor*fog_color) holds. The resulting rendered image can be displayed on the device. The user interface can give the user the option to make the fog effect changes and select an option to take a snapshot of the display and save the image to the device's memory.

[0050] The following listing (Listing 1) shows another use case as a line code: This use case is an exemplary anchored focus with aperture effect.

[0051]

number

[0052] 5 illustrates a block diagram of an image processing system according to at least one example embodiment. As shown in FIG. 5, the image processing system 500 includes at least one processor 505, at least one memory 510, a controller 520, a buffer 525, an anchor module 530, and an algorithm module 535. The at least one processor 505, the at least one memory 510, the controller 520, the buffer 525, the anchor module 530, and the algorithm module 535 are communicatively coupled via a bus 515.

[0053] The at least one processor 505 may be utilized to execute instructions stored on the at least one memory 510, thereby implementing various features and functions described herein, or additional or alternative features and functions. The at least one processor 505 may be a general-purpose processor. The at least one processor 505 may be a graphics processing unit (GPU). The at least one processor 505 and the at least one memory 510 may be utilized for various other purposes. In particular, the at least one memory 510 may represent one example of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

[0054] The at least one memory 510 may be configured to store data and / or information associated with the image processing system 500. For example, the at least one memory 510 may be configured to store code associated with implementing a user interface to capture and / or edit an image. For example, the at least one memory 510 may be configured to store code associated with editing an image, a frame of video, a preview image, etc. The at least one memory 510 may be a non-transitory computer-readable medium having code that, when executed by the processor 505, causes the processor 505 to implement one or more of the techniques described herein. The at least one memory 510 may be a shared resource. For example, the image processing system 500 may be an element of a larger system (e.g., a server, a personal computer, a mobile device, etc.). Thus, the at least one memory 510 may be configured to store data and / or information associated with other elements (e.g., image / video serving, web browsing, or wired / wireless communication) in the larger system.

[0055] The controller 520 may be configured to generate various control signals and communicate the control signals to various blocks in the image processing system 500. The controller 520 may be configured to generate the control signals to implement the techniques described herein. The controller 520 may be configured to control the algorithm module 535 to modify pixels of an image, a portion of an image, multiple images, etc. according to an exemplary embodiment. For example, the controller 520 may generate control signals corresponding to parameters for implementing image editing.

[0056] The algorithm module 535 may be configured to store algorithms for editing images. The algorithm module 535 may be configured to cause execution (e.g., using the processor 505) of algorithms for editing images. The algorithms may include at least image (and / or object) enhancement, blurring, fogging, filters, edge highlighting, cropping, color changing, special effects, distortion, etc. Although a number of image and / or object editing algorithms (or algorithm functions) are listed, implementations are not limited thereto. The buffer 525 may be a memory (e.g., temporary or volatile memory) configured to store images before and / or after image editing. The buffer 525 may be a spare portion of the at least one memory 510. Alternatively or additionally, the buffer 525 may be a standalone memory.

[0057] In some implementations, the anchor module 530 may be configured (or may assist) in placing an anchor on an image, determining a position of the anchor in the image, determining a depth of the anchor in the image, determining a normalized depth of the anchor, determining an object at the anchor position in the image, etc. (e.g., other anchor functions described herein). In an example implementation, the anchor module 530 and the algorithm module 535 together may be configured to implement the techniques described above with respect to FIGS.

[0058] 6 illustrates an example of a computing device 600 and a mobile computing device 650 that may be used with the techniques described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smart phones, and other similar computing devices. The components illustrated, their connections and relationships, and their functions are intended to be merely exemplary and are not intended to limit the implementation of the invention as described and / or claimed herein.

[0059] Computing device 600 includes a processor 602, a memory 604, a storage device 606, a high-speed interface 608 connecting to memory 604 and a high-speed expansion port 610, and a low-speed interface 612 connecting to a low-speed bus 614 and storage device 606. Each of the components 602, 604, 606, 608, 610, and 612 are interconnected using various buses and may be mounted on a common motherboard or otherwise as appropriate. Processor 602 is capable of processing instructions that are executed within computing device 600, including instructions stored in memory 604 or on storage device 606, for displaying graphical information for a GUI on an external input / output device, such as a display 616 coupled to high-speed interface 608. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memories, as appropriate. Multiple computing devices 600 may also be connected, with each device providing a portion of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0060] The memory 604 stores information within the computing device 600. In one implementation, the memory 604 is one or more volatile memory units. In another implementation, the memory 604 is one or more non-volatile memory units. The memory 604 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0061] The storage device 606 can provide mass storage for the computing device 600. In one implementation, the storage device 606 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 604, the storage device 606, or a memory on the processor 602.

[0062] high speed Interface 608 manages bandwidth intensive operations for the computing device 600, while the low speed controller 612 manages less bandwidth intensive operations. Such an allocation of functions is merely exemplary. In one implementation, the high speed Interface608 is coupled to memory 604, a display 616 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 610, which may accept various expansion cards (not shown). In this implementation, a low-speed controller 612 is coupled to the storage device 606 and a low-speed bus 614. The low-speed expansion ports, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or to a networking device, such as a switch or router, for example, via a network adapter.

[0063] Computing device 600 may be implemented in many different forms as shown in the figure. For example, it may be implemented as a standard server 620 or multiple times in a group of such servers. It may also be implemented as part of a rack server system 624. In addition, it may be implemented in a personal computer, such as a laptop computer 622. Alternatively, components from computing device 600 may be combined with other components in a mobile device (not shown), such as device 650. Each such device may include one or more of computing devices 600, 650, and the entire system may be made up of multiple computing devices 600, 650 in communication with each other.

[0064] Computing device 650 includes, among other components, a processor 652, a memory 664, an input / output device such as a display 654, a communication interface 666, and a transceiver 668. Device 650 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 650, 652, 664, 654, 666, and 668 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in other manners as appropriate.

[0065] The processor 652 can execute instructions in the computing device 650, including instructions stored in the memory 664. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor may provide coordination between other components of the device 650, such as, for example, a user interface, applications executed by the device 650, and control of wireless communication by the device 650.

[0066] The processor 652 may communicate with a user via a control interface 658 and a display interface 656 coupled to a display 654. The display 654 may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display), or an OLED (Organic Light Emitting Diode) display, or other suitable display technology. The display interface 656 may include appropriate circuitry for driving the display 654 to present graphical and other information to a user. The control interface 658 may receive commands from a user and convert them for sending to the processor 652. Additionally, an external interface 662 may be provided in communication with the processor 652 to enable near-area communication between the device 650 and other devices. The external interface 662 may, for example, provide for wired communication in some implementations and wireless communication in other implementations, and multiple interfaces may also be used.

[0067] The memory 664 stores information within the computing device 650. The memory 664 may be realized as one or more of one or more computer readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 674 may also be provided and connected to the device 650 via an expansion interface 672. The expansion interface 672 may include, for example, a Single In Line Memory Module (SIMM) card interface. Such expansion memory 674 may provide extra storage space for the device 650 or may also store applications or other information for the device 650. In particular, the expansion memory 674 may include instructions for performing or supplementing the above-mentioned processes, and may also include secure information. Thus, for example, the expansion memory 674 may be provided as a security module for the device 650 and may be programmed with instructions that allow secure use of the device 650. In addition, secure applications may be provided via a SIMM card along with additional information, such as placing identification information on the SIMM card in an unhackable manner.

[0068] The memory may include, for example, flash memory and / or NVRAM memory as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable or machine-readable medium, such as memory 664, expansion memory 674, or memory on processor 652, and may be received, for example, through transceiver 668 or external interface 662.

[0069] Device 650 may communicate wirelessly via communication interface 666, which may include digital signal processing circuitry as necessary. Communication interface 666 may provide for communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, via radio frequency transceiver 668. In addition, short-range communication may occur using Bluetooth, Wi-Fi, or other such transceivers (not shown), etc. In addition, a Global Positioning System (GPS) receiver module 670 may provide additional navigation-related and location-related wireless data to device 650, which may be used by applications executing on device 650 as appropriate.

[0070] Device 650 may also communicate voice using a voice codec 660, which may receive verbal information from a user and convert it into usable digital information. Voice codec 660 may also generate sounds that are audible to the user, such as through a speaker in a handset of device 650. Such sounds may include sounds from a voice telephone call, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on device 650.

[0071] The computing device 650 may be implemented in many different forms as shown in the figure. For example, it may be implemented as a mobile phone 680. It may also be implemented as part of a smartphone 682, personal digital assistant, or other similar mobile device.

[0072] In a general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method can perform a process using a method that includes receiving an image as a portion of a real-world space, placing an anchor on the image, determining a position of the anchor, determining a depth associated with the position of the anchor, applying a depth-based image editing algorithm to the captured image, and rendering the edited image.

[0073] Implementations may include one or more of the following features. For example, the captured image may be a live image, the rendering of which is constantly modified using a depth-based image editing algorithm. The anchor may place a focal point at a depth in the captured image, and the viewpoint of the captured image may be changed and the focal point may remain on the depth. Applying a depth-based image editing algorithm to the captured image may include applying a first algorithm to a first portion of pixels that includes pixels of the captured image within a depth threshold distance, and applying a second algorithm to a second portion of pixels that includes the remaining pixels of the captured image.

[0074] For example, applying a depth-based image editing algorithm to the captured image may include determining an object associated with the image, determining edges of the object based on the depth, applying a first algorithm to a first portion of pixels of the captured image including pixels of the captured image associated with the edges of the object, and applying a second algorithm to a second portion of pixels including remaining pixels of the captured image. Determining a depth associated with the location of the anchor may include obtaining a depth map associated with the captured image.

[0075] For example, determining a depth associated with a position of the anchor may include obtaining a depth map associated with the captured image, determining a minimum depth value associated with the captured image, determining a maximum depth value associated with the captured image, normalizing the depth map based on the minimum and maximum depth values, and determining a depth associated with the position as the normalized depth value. Determining a depth associated with a position of the anchor may include obtaining a layered depth associated with the captured image, the depth of the anchor having a depth index. Statue edition The method further includes the step of selecting a collection tool, the image editing tool corresponding to the image editing algorithm.

[0076] While exemplary embodiments may include various modifications and alternative forms, such embodiments have been shown by way of example in the drawings and will herein be described in detail. It is to be understood, however, that there is no intention to limit the exemplary embodiments to the particular forms disclosed, but rather, the exemplary embodiments encompass all modifications, equivalents, and alternatives falling within the scope of the claims. Like numbers refer to like elements throughout the description of the figures.

[0077] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general purpose, and may be coupled to transmit and receive data and instructions from a storage system, at least one input device, and at least one output device. Various implementations of the systems and techniques described herein may be realized as and / or generally referred to herein as circuits, modules, blocks, or systems that may combine software and hardware aspects. For example, a module may include functions / acts / computer program instructions executed on a processor (e.g., a processor formed on a silicon substrate, GaAs substrate, etc.) or some other programmable data processing apparatus.

[0078] Some of the above-described exemplary embodiments are described as processes or methods that are depicted as flow charts. Although these flow charts describe operations as sequential processes, many of the operations may be performed in parallel, simultaneously, or simultaneously. In addition, the order of operations may be rearranged. Processes may be terminated when their operations are completed, but may also have additional steps not included in the figures. These processes may correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0079] The methods described above, some of which are illustrated by flow charts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. If implemented in software, firmware, middleware, or microcode, the program code or code segments performing the necessary tasks may be stored in a machine-readable or computer-readable medium such as a storage medium. A processor may perform the necessary tasks.

[0080] Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments, however, example embodiments may be embodied in many alternative forms and should not be construed as being limited to only the embodiments set forth herein.

[0081] It will be understood that terms such as first, second, etc. may be used herein to describe various elements, but these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the example embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0082] When an element is referred to as being connected or coupled to another element, it will be understood that the element may be directly connected or coupled to the other element, or there may be intervening elements. In contrast, when an element is referred to as being directly connected or coupled to another element, there are no intervening elements. Other language used to describe relationships between elements should be construed in an analogous manner (e.g., "between" and "directly between," "adjacent" and "directly adjacent," etc.).

[0083] The terms used herein are merely for the purpose of describing particular embodiments and are not intended to be limitations of example embodiments. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise. It will be further understood that the terms "comprises, comprising" and / or "includes, including", when used herein, specify the presence of the mentioned features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0084] Also, in some alternative implementations, the functions / acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed concurrently or may sometimes be executed in the reverse order, depending on the functionality / acts involved.

[0085] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments belong. Furthermore, it will be understood that terms, for example as defined in commonly used dictionaries, should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and not in an idealized or overly formal sense unless expressly so defined herein.

[0086] Portions of the exemplary embodiments above and corresponding detailed description are presented in terms of software, or algorithms and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the ones by which those skilled in the art effectively convey the substance of their work to others skilled in the art. An algorithm, as the term is used here, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. These steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0087] In the exemplary embodiments described above, references to symbolic representations of acts and operations that may be implemented as program modules or functional processes (e.g., in the form of flowcharts) include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types, and that may be written and / or implemented using existing hardware with existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, or field programmable gate array (FPGA) computers, etc.

[0088] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, or as will be apparent from the description, terms such as display processing, computing, calculation, or determination refer to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical electronic quantities in the computer system's registers and memory, and convert such data into other data similarly represented as physical quantities in the computer system's memory, registers, or other such information storage, transmission, or display devices.

[0089] Also, the software-implemented aspects of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard drive) or optical (e.g., a compact disk read only memory, or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or some other suitable transmission medium known in the art. The exemplary embodiments are not limited by these aspects of any given implementation.

[0090] Finally, although the appended claims recite particular combinations of features described herein, the scope of the disclosure is not limited to that particular combination claimed, but instead extends to encompass any combination of features or embodiments disclosed herein, whether or not that particular combination is currently specifically recited in the appended claims.

Claims

1. receiving an image as part of a real world space; determining a location of an anchor within the image; determining a depth associated with the position of the anchor, the anchor having a size according to the depth; applying the depth-based image editing algorithm to the captured image; and rendering the edited image.

2. The method of claim 1 , wherein the captured images are live images, the rendering of which is constantly modified using the depth-based image editing algorithm.

3. the anchor positions a focal point at a depth in the captured image; The viewpoint of the captured image is changed; The method of claim 1 or 2, wherein the focus remains on the depth.

4. receiving an image as part of a real world space; determining a location of an anchor within the image; determining a depth associated with the position of the anchor; applying the depth-based image editing algorithm to the captured image; and rendering the edited image.

20. The method of claim 19, further comprising: applying a first algorithm to a first portion of pixels including pixels of the captured image within a threshold distance of the depth, the first algorithm including focusing on the anchor; and applying a second algorithm, different from the first algorithm, to a second portion of pixels comprising remaining pixels of the captured image.

5. receiving an image as part of a real world space; determining a location of an anchor within the image; determining a depth associated with the position of the anchor; applying the depth-based image editing algorithm to the captured image; and rendering the edited image.

20. The method of claim 19, further comprising: determining an object associated with the image; determining edges of the object based on the depth; applying a first algorithm to a first portion of pixels including pixels of the captured image associated with the edge of the object, the first algorithm including focusing on the anchor; and applying a second algorithm, different from the first algorithm, to a second portion of pixels comprising remaining pixels of the captured image.

6. The method of any one of claims 1 to 5, wherein determining the depth associated with the position of the anchor comprises obtaining a depth map associated with the captured image.

7. The step of determining the depth associated with the position of the anchor comprises: obtaining a depth map associated with the captured image; determining a minimum depth value associated with the captured image; determining a maximum depth value associated with the captured image; normalizing the depth map based on the minimum depth value and the maximum depth value; and determining the depth associated with the location as a normalized depth value.

8. Determining the depth associated with the position of the anchor includes obtaining a layer depth associated with the captured image; The method of any one of claims 1 to 5, wherein the depth of the anchor has a depth index.

9. receiving an image as part of a real world space; determining a location of an anchor within the image; determining a depth associated with the position of the anchor; applying the depth-based image editing algorithm to the captured image; and rendering the edited image. selecting an image editing tool as the GUI; The method, wherein the image editing tool corresponds to the image editing algorithm and includes focus on the anchor.

10. A computer program comprising instructions which, when executed, cause a processor of a computer system to perform any of the steps of the method according to any one of claims 1 to 9.

11. A computer system for editing images, configured to perform any of the steps of the method according to any one of the preceding claims.

12. 1. A system comprising: a memory for storing a set of instructions; a processor configured to execute the set of instructions to cause the system to perform an operation, the operation comprising: receiving an image as part of a real world space; determining a location of an anchor within the image; determining a depth associated with the position of the anchor, the anchor having a size according to the depth; applying the depth-based image editing algorithm to the captured image; and and rendering the edited image.

13. The system of claim 12 , wherein the captured images are live images, the rendering of which is constantly modified using the depth-based image editing algorithm.

14. the anchor positions a focal point at a depth in the captured image; The viewpoint of the captured image is changed; The system of claim 12 or 13, wherein the focus remains on the depth.

15. 1. A system comprising: a memory for storing a set of instructions; a processor configured to execute the set of instructions to cause the system to perform an operation, the operation comprising: receiving an image as part of a real world space; determining a location of an anchor within the image; determining a depth associated with the position of the anchor; applying the depth-based image editing algorithm to the captured image; and and rendering the edited image. applying the depth-based image editing algorithm to the captured image, applying a first algorithm to a first portion of pixels including pixels of the captured image within a depth threshold distance, the first algorithm including focusing on the anchor; and applying a second algorithm, different from the first algorithm, to a second portion of pixels including the remaining pixels of the captured image.

16. 1. A system comprising: a memory for storing a set of instructions; a processor configured to execute the set of instructions to cause the system to perform an operation, the operation comprising: receiving an image as part of a real world space; determining a location of an anchor within the image; determining a depth associated with the position of the anchor; applying the depth-based image editing algorithm to the captured image; and and rendering the edited image. applying the depth-based image editing algorithm to the captured image, determining an object associated with the image; determining edges of the object based on the depth; applying a first algorithm to a first portion of pixels including pixels of the captured image associated with the edge of the object, the first algorithm including focusing on the anchor; and applying a second algorithm, different from the first algorithm, to a second portion of pixels including the remaining pixels of the captured image.

17. The system of any one of claims 12 to 16, wherein determining the depth associated with the position of the anchor comprises obtaining a depth map associated with the captured image.

18. Determining the depth associated with the position of the anchor comprises: obtaining a depth map associated with the captured image; determining a minimum depth value associated with the captured image; determining a maximum depth value associated with the captured image; normalizing the depth map based on the minimum depth value and the maximum depth value; and determining the depth associated with the location as a normalized depth value.

19. Determining the depth associated with the position of the anchor includes obtaining a layer depth associated with the captured image; The system of any one of claims 12 to 16, wherein the depth of the anchor has a depth index.

20. 1. A system comprising: a memory for storing a set of instructions; a processor configured to execute the set of instructions to cause the system to perform an operation, the operation comprising: receiving an image as part of a real world space; determining a location of an anchor within the image; determining a depth associated with the position of the anchor; applying the depth-based image editing algorithm to the captured image; and and rendering the edited image. selecting an image editing tool as the GUI; The image editing tool corresponds to the image editing algorithm and includes a focus on the anchor.

Images