Stylized Image Painting

Abstract

A photo filter (e.g., art / stylized painting) light field effect system includes an eyewear device with a frame, temples connected to the sides of the frame, and a depth capture camera. The stylized image painting effect system is configured by a processor to execute programming to apply the photo filter selection to: (i) the left original image or the left processed image to create the left photo filter image, and (ii) the right original image or the right processed image to create the right photo filter image. By mixing the left photo filter image and the right photo filter image based on the left image disparity map and the right image disparity map, respectively, the stylized image painting effect system can generate a photo filter stylized painting effect image with a spatial rotation or movement appearance.

Description

[0001] This application claims priority to U.S. Utility Patent Application No. 16 / 587,015, filed on September 29, 2019, entitled “STYLIZED IMAGE PAINTING,” the contents of which are incorporated herein by reference in their entirety. Background Art

[0002] Computing devices available today, such as wearable devices, including portable eyewear (e.g., smart glasses, head-mounted devices, and headbands); mobile devices (e.g., tablets, smartphones, and laptops); and personal computers, integrate image displays and cameras. Currently, computing device users can use photo filters to create special effects on images. Various photo decoration applications provide tools such as stickers, emojis, and text for editing images. Attached photos

[0003] The accompanying drawings are intended to illustrate one or more embodiments, which are intended to provide examples and not limitations. In the accompanying drawings, the same reference numerals represent the same or similar elements.

[0004] Figure 1A is a right view of an example hardware configuration of an eye-mounted device used in a stylized image painting system for photo filters (e.g., artistic / stylized painting), where the style of a selected image is transferred to a specified portion of a target image to create a stylized image.

[0005] Figure 1B yes Figure 1A A top-down cross-sectional view of the right assembly of the eye-mounted device is shown, showing the visible light camera and circuit board to the right of the depth capture camera.

[0006] Figure 1C yes Figure 1A Left view of an example hardware configuration for an eye-mounted device, showing the visible light camera to the left of the depth capture camera.

[0007] Figure 1D yes Figure 1C A top-down cross-sectional view of the left assembly of the eye-mounted device is shown, showing the visible light camera and circuit board to the left of the depth capture camera.

[0008] Figure 2A is a left view of another example hardware configuration of an eye-mounted device utilized in a photo filter (e.g., artistic / stylized painting) stylized image painting system, showing a depth sensor of a visible light camera and a depth capture camera on the right to generate a depth image.

[0009] Figure 2B and 2C is a rear view of an example hardware configuration for a goggles, including two different types of image displays.

[0010] Figure 3 Show Figure 2A A rear perspective cutaway view of the eye-mounted device is shown, showing the infrared camera of the depth sensor, the front of the frame, the back of the frame, and the circuit board.

[0011] Figure 4 It passes through Figure 3 A cross-sectional view of the infrared camera and frame of the eyewear device shown.

[0012] Figure 5 Show Figure 2A A rear perspective view of the eye-mounted device, showing the infrared emitter of the depth sensor, the infrared camera of the depth sensor, the front of the frame, the back of the frame, and the circuit board.

[0013] Figure 6 It passes through Figure 5 A cross-sectional view of the infrared emitter and frame of the eyewear device shown.

[0014] Figure 7 An example of an infrared light pattern emitted by the eye-mounted device's depth sensor infrared emitter and changes in reflection of the infrared light emission pattern captured by the depth sensor's infrared camera are shown to measure pixel depth in the original image and generate a depth image.

[0015] Fig. 8A Shown is an example of infrared light captured by a depth sensor infrared camera as an infrared image, and visible light captured by a visible light camera as a raw image, used to generate a depth image of a three-dimensional scene.

[0016] Figure 8B Shown is an example of visible light captured by a left visible light camera as a left raw image, and visible light captured by a right visible light camera as a right raw image to generate a depth image of a three-dimensional scene.

[0017] Fig. 9 is a high-level functional block diagram of an example filter (e.g., artistic / stylized painting) light field effect system, including an eye-mounted device (equipped with a depth capture camera to generate the filter stylized painting effect image) and a user input device (e.g., touch sensor), a mobile device, and a server system connected via various networks.

[0018] Fig.10 Show Fig. 9 Photo Filter (e.g., Artistic / Stylized Painting) Light Field Effects System An example hardware configuration for a mobile device including a user input device (such as a touch screen device) for receiving indicia and image selection for migrating to an original image or a processed image to generate a filter effect image.

[0019] Fig.11A is a flow chart of a method that can be implemented in a filter (e.g., artistic / stylized painting) light field effect system, applied to an original image or a processed image to generate a photo filter stylized painting effect image.

[0020] Fig. 11B is a flowchart of the method used to create a stylized painting image.

[0021] Fig. 12A An example of a first presented original image is shown, which is a processed (eg, corrected) image.

[0022] Fig. 12B An example of a first presented original image with user markings is shown.

[0023] Fig. 12C Shows from Fig. 12A The first example of an image created with a filter (for example, a stylized painting effect) that renders an original image.

[0024] Fig.12D Shows from Fig. 12B Example of a first filter (e.g., artistic / stylized painting) light field effect image generated by a photo filter image with spatial translation or rotation tilted to the left.

[0025] Fig.12E Shows from Fig. 12B Example of a second filtered (e.g., artistic / stylized painting) light field effect image generated from a photo-filtered image with spatial translation or rotation tilted to the right.

[0026] Fig.13 An example of a photo filtered (eg, artistic / stylized painting) image is shown, blending the stylized painting with the original image within the marked area. Details

[0027] The example described in this article involves transferring the style of a selected image to a tag for a subject image. This allows users to "paint" (selectively apply) the style of a subject image, providing more artistic freedom and improving the user experience.

[0028] The following detailed description sets forth numerous specific details by way of example in order to provide a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that the present teachings may be practiced without such details. In other cases, well-known methods, procedures, components, and circuits have been described to a considerable extent without detailed description in order to avoid unnecessarily obscuring aspects of the present teachings.

[0029] As used herein, the term "photo filter" refers to a graphic effect that edits, alters, or changes a photograph or image to transform specific pixels by applying a style such as pop art (e.g., many paintings such as Andy Warhol's Marilyn Monroe and Evard Munch's The Scream, 1893), or by using deep neural networks (such as the Neural Style Transfer algorithm (NST)) to learn style from the following elements: painting; graphics (e.g., hats, beards, jewelry, frames, stickers, and graphic overlays); texture; light saturation; color exposure; color; sharpness; themes (sepia, dramatic, sepia, grayscale, black and white, retro, disco, color fantasy, and vignette); and image quality enhancement (brightness, contrast, saturation, blur, etc.). The term "artistic effect" refers to editing or changing a photograph or image by applying a style transfer type of pop art or filter. The term "stylized painting effect" refers to editing or changing areas of a photo or picture (i.e., areas of the photo or picture identified by a user, for example, by tagging) by applying a style transfer type of pop art or filters.

[0030] In general, the term "light field" refers to the luminosity of a point in a given direction. The term "light field effect" refers to rendering different views of an image scene to provide the appearance of spatial movement or rotation, as if the viewer is viewing the scene from different angles or perspectives. The term "photo filter light field effect" refers to rendering different filtered scene views of a photo filter image to provide the appearance of spatial movement or rotation, as if the viewer is viewing the photo filter scene from different angles or perspectives. The term "artistic light field effect" refers to rendering different artistic scene views of an artistic image to provide the appearance of spatial movement or rotation, as if the viewer is viewing the artistic scene from different angles or perspectives. The term "stylized painterly light field effect" refers to rendering different artistic / stylized painterly scene views of an artistic / stylized painterly image (including stylized paintings) to provide the appearance of spatial movement or rotation, as if the viewer is viewing the artistic scene with stylized paintings from different angles or perspectives.

[0031] A light field effect camera can capture light from different directions and move it around to create a three-dimensional or four-dimensional scene (e.g., using multiple lenses). However, such processing in three-dimensional space (X, Y, and Z) and four-dimensional space (X, Y, Z and time) is relatively complex and can require a lot of computation. As described herein, two visible light cameras 114A-B can be used to create a simplified light field effect from two images by operating in only two-dimensional space, which is less computationally intensive.

[0032] As used herein, the terms "coupled" or "connected" refer to any logical, optical, physical or electrical connection, link, etc., by which a signal or light generated or provided by one system element is transmitted to another coupled or connected element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements or communication media that can modify, manipulate or carry light or signals.

[0033] The orientations of the eye-mounted devices, related components, and any overall devices incorporating depth capture cameras shown in the accompanying drawings are examples only and are used for illustration and discussion purposes. In actual operation, for photo filters (e.g., artistic / stylized paintings) light field effects, the eye-mounted devices can be oriented in any other direction suitable for the specific application of the eye-mounted devices, such as up, down, sideways, or any other direction. In addition, as used herein, any directional terms, such as front, back, inside, outside, direction, left, right, lateral, longitudinal, top, bottom, upward, downward, top, bottom, side, horizontal, vertical, and diagonal, are used only as examples and do not limit the direction or orientation of any depth capture camera or depth capture camera assembly constructed as otherwise described herein.

[0034] The following description will partially set forth other objects, advantages and novel features of the examples. Such other objects, advantages and novel features will be apparent to those skilled in the art after reviewing the following and accompanying drawings, or may be understood and grasped by making or operating the examples. The objects and advantages of the subject matter may be realized and obtained by the methods, means and combinations particularly pointed out in the appended claims.

[0035] Reference will now be made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0036] Figure 1A1 is a right view of an example hardware configuration of the eye-mounted device 100 used in a photo filter (e.g., artistic / stylized painting) light field effect system, showing a visible light camera 114B to the right of the depth capture camera for generating a depth image. As further described below, in a photo filter (e.g., artistic / stylized painting) light field effect system, a filter selection input (e.g., including stylized artistic aspects within a user-identified image area) is applied to an original image or a processed image to create a photo filter image with a photo filter scene. The photo filter images can be blended together based on a disparity map to create a photo filter light field effect image including artistic / stylized painting aspects. The photo filter light field effect image can provide an appearance of spatial movement or rotation of the photo filter scene around the photo filter image. In one example, the photo filter type is a stylized painting effect. Thus, in this example, the stylized painting effect selection input from the user is applied to the original image or processed image identified by the user to create an artistic / stylized painting effect image, which is then blended to generate a stylized painterly light field effect image with a stylized painterly scene. The stylized painterly light field effect image may provide the appearance of spatial movement or rotation of the stylized painterly scene surrounding the artistic effect image.

[0037] The eye-mounted device 100 includes a right optical component 180B that uses an image display to present an image, such as based on a left original image, a processed left image, a right original image, a processed right image, a photo filter image (e.g., a stylized painting effect image), or a photo filter light field effect image (e.g., a stylized painting light field effect image). Figure 1A -B, the eye-mounted device 100 includes a right visible light camera 114B. The eye-mounted device 100 may include a plurality of visible light cameras 114A-B, which form a passive depth capture camera, such as a stereo camera, wherein the right visible light camera 114B is located on the right block 110B. As shown in FIG. 1C-D, the eye-mounted device 100 may also include a left visible light camera 114A. Alternatively, in Figure 2A In the example of FIG. 1 , the depth capture camera may be an active depth capture camera, which includes a single visible light camera 114B and a depth sensor (see FIG. Figure 2A Element 213 in).

[0038] The left and right visible light cameras 114A-B are sensitive to a range of visible light wavelengths. In the visible light cameras 114A-B, each camera has a different forward field of view that overlaps to allow the generation of a three-dimensional depth image, for example, the right visible light camera 114B with the right field of view 111B shown. In general, a "field of view" is a portion of a scene that is visible to a camera at a specific location and orientation in space. When a visible light camera captures an image, objects or object features outside the field of view 111A-B are not recorded in the original image (e.g., a photograph or picture). The field of view describes the angular range or range of electromagnetic radiation picked up by the visible light camera 114A-B image sensor in a captured image of a given scene. The field of view can be expressed as the angular size of the viewing cone, i.e., the viewing angle. The viewing angle can be measured horizontally, vertically, or diagonally.

[0039] In one example, visible light camera 114A-B has a field of view of between 15° and 30° (e.g., 24°) and a resolution of 480 x 480 pixels. “Coverage angle” describes the coverage angle of visible light camera 114A-B or infrared camera 220 (see Figure 2A ) is the range of angles over which a lens can effectively image. Typically, the image circle produced by a camera lens is large enough to cover the film or sensor, possibly including some vignetting (i.e., a decrease in image brightness or saturation toward the edges compared to the center of the image). If the angular coverage of a camera lens does not fill the sensor, the image circle will become visible, usually with strong vignetting toward the edges, and the effective viewing angle will be limited by the coverage angle.

[0040] Examples of such visible light cameras 114A-B include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640 x 480 pixels, 0.3 megapixels total), 720p, or 1080p. As used herein, the term "overlap" when referring to the field of view means that the pixel matrices overlap by 30% or more in the resulting raw or infrared image of the scene. As used herein, the term "substantially overlap" when referring to the field of view means that the pixel matrices overlap by 50% or more in the resulting raw or infrared image of the scene.

[0041] Image sensor data from visible light cameras 114A-B is captured along with geographic location data, digitized by an image processor, and then stored in memory. The left and right raw images captured by the respective visible light cameras 114A-B are in a two-dimensional spatial domain and include a matrix of pixels on a two-dimensional coordinate system including an X-axis representing horizontal position and a Y-axis representing vertical position. Each pixel includes a color attribute (e.g., a red pixel light value, a green pixel light value, and / or a blue pixel light value); and a position attribute (e.g., an X position coordinate and a Y position coordinate).

[0042] To provide stereoscopic vision, the visible light cameras 114A-B can be coupled with an image processor ( Fig. 9 The image processor 912 is coupled to the visible light camera 114A-B to perform digital processing and time stamping of the scene image captured. The image processor 912 includes a circuit system for receiving signals from the visible light cameras 114A-B and processing the signals from the visible light cameras 114 into a format suitable for storage in the memory. The timestamp can be added by the image processor or other processor, which is responsible for controlling the operation of the visible light cameras 114A-B. The visible light cameras 114A-B allow the depth capture camera to simulate human binocular vision. The depth capture camera has the ability to reproduce a three-dimensional image based on two captured images from the visible light cameras 114A-B with the same timestamp. Such three-dimensional images can provide an immersive and realistic experience, for example, virtual reality or video games.

[0043] For stereoscopic vision, a pair of raw red, green, and blue (RGB) images are captured from the scene at a given moment - one for each of the left and right visible light cameras 114A-B (e.g., a stereo pair). A depth image is generated when a pair of captured raw images from the left and right visible light cameras 114A-B forward to the left and right fields of view 111A-B are processed (e.g., by an image processor). The depth image can be based on a three-dimensional model, which can include a three-dimensional mesh (e.g., a triangle mesh) and textures that are uploaded to a graphics processing unit (GPU) as vertices along with a texture map. Typically, the depth is not actually visible, but the effect of depth can be seen in a rendered and displayed two-dimensional image. By converting to a two-dimensional image for displaying various viewpoints, the user can perceive the generated depth image on the optical assembly 180A-B or other image display (e.g., a display of a mobile device). The generated depth image is in a three-dimensional spatial domain and may include a vertex matrix on a three-dimensional position coordinate system, which includes an X-axis representing horizontal position (e.g., length), a Y-axis representing vertical position (e.g., height), and a Z-axis representing depth (e.g., distance). Each vertex includes color attributes (e.g., red pixel light value, green pixel light value, and / or blue pixel light value); position attributes (e.g., X position coordinate, Y position coordinate, and Z axis coordinate); texture attributes, and / or reflectivity attributes. Texture attributes can quantify the perceived texture of the depth image, such as the spatial arrangement of color or intensity in the vertex area of ​​the depth image.

[0044] Generally speaking, depth perception comes from the disparity of a given 3D point in the left and right raw images captured by visible light cameras 114A-B. Disparity is the difference in the image position of the same 3D point when projected under the perspective of visible light cameras 114A-B (d = x 左 -x 右 ). For example, the correlation of left and right pixels in each left and right original image can be achieved by semi-global block matching (SGBM). For a system with parallel optical axes, focal length f, baseline b and corresponding imaging point (x 左 ,y 左 ) and (x 右 ,y 右 ) can use triangulation to deduce the position (Z-axis position coordinate) of the 3D point, which determines the depth through parallax. Generally, the depth of the 3D point is inversely proportional to the parallax. A number of other techniques can also be used. The generation of three-dimensional depth images and photo filter (e.g., artistic / stylized painting) light field effect images will be described in more detail below.

[0045] In one example, a photo filter (e.g., artistic / stylized painting) light field effect system includes an eye-mounted device 100. The eye-mounted device 100 includes a frame 105 and a left temple 110A extending from a left side 170A of the frame 105, and a right temple 110B extending from a right side 170B of the frame 105. The eye-mounted device 100 also includes a depth capture camera. The depth capture camera includes: (i) at least two visible light cameras with overlapping fields of view; or (ii) at least one visible light camera 114A-B and a depth sensor ( Figure 2A Element 213 is shown). In one example, the depth capture camera includes a left visible light camera 114A with a left field of view 111A connected to the frame 105 or the left temple 110A to capture a left image of the scene. The eye-mounted device 100 also includes a right visible light camera 114B connected to the frame 105 or the right temple 110B with a right field of view 111B to capture (e.g., simultaneously with the left visible light camera 114A) a right image of the scene partially overlapping the left image.

[0046] Photo Filters (e.g., Artistic / Stylized Paintings) The light field effects system also includes a computing device, such as a host (e.g., Figure 9-10 Mobile device 990 shown). The photo filter (e.g., artistic / stylized painting) light field effect system also includes an image display (eye-mounted device optical components 180A-B; Fig.10 The image display 1080 of the mobile device 990 is shown. The image sequence includes an original image, an original image in two-dimensional space or a processed original image (e.g., after correction), a photo filter (e.g., an artistic / stylized effect) image, and a photo filter (e.g., an artistic / stylized painting) light field effect image. The photo filter (e.g., an artistic / stylized painting) light field effect system also includes an image display driver ( Fig. 9 Component 942 of the eye-mounted device 100 is shown; Fig.10 Element 1090 of mobile device 990 shown) coupled to an image display (optical components 180A-B of the eye-mounted device; Fig.10 The image display 1080 of the mobile device 990 shown in the figure controls the image display to display an image sequence. The image sequence may include original images, original images in two-dimensional space or processed original images (e.g., rectified), photo filter (e.g., artistic / stylized effect) images, and photo filter (e.g., artistic / stylized painting) light field effect images.

[0047] The photo filter (e.g., artistic / stylized effect) light field effect system also includes a user input device for receiving a two-dimensional input selection from a user. Examples of user input devices include a touch sensor ( Fig. 9Component 991 for eye-mounted device 100), touch screen display ( Fig.10 Element 1091 for mobile device 1090 shown), and a computer mouse for a personal computer or laptop. Photo filter (e.g., artistic / stylized painting) The light field effect system also includes a processor ( Fig. 9 Component 932 of the eye-mounted device 100 is shown; Fig.10 Element 1030 of the mobile device 990 shown). The photo filter (e.g., artistic / stylized painting) light field effects system also includes a memory accessible to the processor ( Fig. 9 Component 934 of the eye-mounted device 100 is shown; Fig.10 1040A-B of the mobile device 990 shown), and photo filters (e.g., artistic / stylized paintings) light field effects programming in memory ( Fig. 9 Component 945 of the eye-mounted device 100 is shown; Fig.10 945 of the mobile device 990 shown), such as in the eye-mounted device 100 itself, the mobile device ( Fig. 9 element 990 shown) or a photo filter (e.g., an artistic / stylized painting) or another part of a light field effects system (e.g., Fig. 9 In the server system 998 shown.

[0048] As described below, the photo filter (e.g., artistic / stylized painting) light field effect system takes a left image and a right image as input viewpoints, but no image between the viewpoints. To generate the artistic / stylized painting effect (e.g., a user stylized at different angles to freeze a moment in time while the camera rotates around one face of the image), interpolation is performed between the left and right images captured by the left and right cameras 114A-B. The artistic / stylized painting effect images can be stitched together as a series of images in a video from several different viewpoints to provide spatial motion.

[0049] Two non-original RGB (modified / non-realistic) left and right images are interpolated to generate a photo-filtered (e.g., artistic / stylized painting) light field effect image, and the interpolation is based on a disparity map generated from the two original RGB images. This provides the appearance of a 3D world by rotating even non-realistic images, but only two modified two-dimensional images (frames) are required to produce the stylized painting effect. The disparity map determines how many pixels need to be moved between pixels in the left image to obtain the corresponding pixels in the right image, and vice versa. Disparity is calculated between stereo pixel pairs corresponding to depth to interpolate between the two non-original RGB images. In some examples, the left image can be a black and white mixture, while the right image can be in color. In another example, an artistic / stylized painting style is simulated in one image (such as the left image), and an original RGB image is simulated in another image (such as the right image), and interpolation is performed between the original RGB image and the modified image based on the left-right disparity.

[0050] Figure 1B yes Figure 1A The top cross-sectional view of the right block 110B of the eye-mounted device 100 is shown, showing the visible light camera 114B and the circuit board on the right side of the depth capture camera. Figure 1C 1A is a left view of the example hardware configuration of the eye-mounted device 100, showing the visible light camera 114A to the left of the depth capture camera. Figure 1D yes Figure 1C The left block 110A of the eyewear device is shown in a top cross-sectional view, illustrating the left visible light camera 114A and the circuit board of the depth capture camera. The configuration and installation position of the left visible light camera 114A are basically similar to those of the right visible light camera 114B, and the only difference is that it is connected and coupled to the left side 170A. Figure 1B In the example shown, the eye-mounted device 100 includes a right visible light camera 114B and a circuit board, which can be a flexible printed circuit board (PCB) 140B. The right hinge 126B connects the right block 110B to the right temple 125B of the eye-mounted device 100. In some examples, components of the right visible light camera 114B, the flexible printed circuit board 140B, or other electrical connections or contacts can be located on the right temple 125B or the right hinge 126B.

[0051] The right block 110B includes a block body 211 and a block cap. Figure 1BThe block cap is omitted from the cross-section shown. Arranged within the right block 110B are various interconnected printed circuit boards (PCBs), such as conventional or flexible printed circuit boards, including controller circuitry for the right visible light camera 114B, a microphone, low power wireless circuitry (e.g., for wireless short range network communications via Bluetooth™), high speed wireless circuitry (e.g., for wireless local area network communications via WiFi).

[0052] The right visible light camera 114B is coupled or arranged on the flexible printed circuit board 240 and covered by the visible light camera cover lens, which is aligned by an opening formed in the frame 105. For example, the right frame 107B of the frame 105 is connected to the right block 110B and includes an opening for the visible light camera cover lens. The frame 105 includes a front face configured to face outward away from the user's eyes. The visible light camera cover lens opening is formed on and through the front side. In this example, the right visible light camera 114B has an outward field of view 111B, with the right eye line of sight or viewing angle of the user of the eye-mounted device 100. The visible light camera cover lens can also be adhered to the outer surface of the right block 110B, forming an opening with an outward coverage angle thereon, but in a different outward direction. Indirect coupling via an intermediate component can also be used.

[0053] The left (first) visible light camera 114A is connected to the left image display of the left optical assembly 180A to capture the left eye scene observed by the wearer of the eye-mounted device 100 in the left raw image. The right (second) visible light camera 114B is connected to the right image display of the right optical assembly 180B to capture the right eye scene observed by the wearer of the eye-mounted device 100 in the right raw image. The left raw image and the right raw image partially overlap to present a three-dimensional observable space of the generated depth image.

[0054] The flexible printed circuit board 140B is disposed within the right block 110B and is coupled to one or more other components within the right block 110B. Although shown as being formed on a circuit board of the right block 110B, the right visible light camera 114B may also be formed on a circuit board of the left block 110A, the temples 125A-B, or the frame 105.

[0055] Figure 2A 1 is another example hardware configuration left view of the eye-mounted device 100 used in a photo filter (e.g., artistic / stylized painting) light field effect system. As shown in the figure, the depth capture camera includes a left visible light camera 114A and a depth sensor 213 located on the frame 105 for generating a depth image. Here, instead of using at least two visible light cameras 114A-B, a single visible light camera 114A and a depth sensor 213 are used to generate a depth image. Figure 1A-D example, the photo filter selection from the user should be applied to the original image to create a photo filter image, and then generate a photo filter art / stylized painting effect image. The infrared camera 220 of the depth sensor 213 has an outward field of view that essentially overlaps with the left visible light camera 114A for the user's line of sight. As shown in the figure, the infrared emitter 215 and the infrared camera 220 are co-located with the left visible light camera 114A on the upper part of the left frame 107A.

[0056] exist Figure 2A In the illustrated example, the depth sensor 213 of the eye-mounted device 100 includes an infrared emitter 215 and an infrared camera 220 for capturing infrared images. The visible light cameras 114A-B typically include a blue light filter to block infrared light detection. In one example, the infrared camera 220 is a visible light camera, such as a low-resolution video graphics array (VGA) camera (e.g., 640 x 480 pixels, 0.3 megapixels in total), with its blue filter removed. The infrared emitter 215 and the infrared camera 220 are co-located on the frame 105, for example, both are shown as being connected to the upper portion of the left frame 107A. As further described below, the frame 105 or one or more left and right blocks 110A-B include a circuit board for the infrared emitter 215 and the infrared camera 220. For example, the infrared emitter 215 and the infrared camera 220 can be connected to the circuit board by soldering.

[0057] Other arrangements of the infrared emitter 215 and the infrared camera 220 may be implemented, including an arrangement where both the infrared emitter 215 and the infrared camera 220 are located on the right frame 107A, or an arrangement where they are located at different locations on the frame 105, for example, the infrared emitter 215 is located on the left frame 107B and the infrared camera 220 is located on the right frame 107B. Nevertheless, typically, at least one visible light camera 114A and the depth sensor 213 have substantially overlapping fields of view for generating a three-dimensional depth image. In another example, the infrared emitter 215 is located on the frame 105 and the infrared camera 220 is located on one of the blocks 110A-B, or vice versa. Basically, the infrared emitter 215 can be connected to any location on the frame 105, the left block 110A, or the right block 110B for emitting infrared patterns within the user's eye line of sight. Similarly, infrared camera 220 may be attached to substantially any location on frame 105, left block 110A, or right block 110B to capture at least one reflected variation in the infrared light emission pattern of a three-dimensional scene within the user's eye line of sight.

[0058] The infrared emitter 215 and the infrared camera 220 are arranged outward to pick up an infrared image of the scene with objects or object features observed by the user wearing the eye-mounted device 100. For example, the infrared emitter 215 and the infrared camera 220 are located directly in front of the eye, in the upper part of the frame 105 or in the blocks 110A-B at both ends of the frame 105, with a forward field of view to capture an image of the scene the user is looking at, for measuring object depth and object features.

[0059] In one example, the infrared emitter 215 of the depth sensor 213 emits infrared illumination within a forward field of view of the scene, which may be near infrared light or other short wavelength beams of low energy radiation. Alternatively or additionally, the depth sensor 213 may include other emitters for emitting light of other wavelengths in addition to infrared light, and the depth sensor 213 also includes a camera sensitive to the wavelength for receiving and capturing images having the wavelength. As described above, the eye-mounted device 100 is coupled to a processor and a memory, for example, a processor and a memory in the eye-mounted device 100 body or in other components of the filter (e.g., artistic / stylized painting) light field effect system. The eye-mounted device 100 or the photo filter (e.g., artistic / stylized painting) light field effect system may then process the captured infrared image during the generation of a three-dimensional depth image (such as a depth image).

[0060] Figure 2B -C is a rear view of an example hardware configuration of the eye-mounted device 100, including two different types of image displays. The eye-mounted device 100 is in the form of a configuration for a user to wear, in this example, glasses. The eye-mounted device 100 can take other forms and can be combined with other types of frames, such as headgear, headphones, or helmets.

[0061] In the glasses example, the eyewear device 100 includes a frame 105, wherein a left frame 107A is connected to a right frame 107B via a nose bridge 106 that fits the user's nose. The left and right frames 107A-B include respective openings 175A-B for accommodating respective optical elements 180A-B, such as lenses and display devices. The term "lens" used herein refers to a curved and / or flat transparent or translucent glass or plastic piece for covering, with or without light converging / diverging function.

[0062] Although shown as being equipped with two optical elements 180A-B, the eye-mounted device 100 may also include other arrangements, such as a single optical element, or not include any optical elements 180A-B, depending on the specific use or intended user of the eye-mounted device 100. As further shown, the eye-mounted device 100 includes a left block 110A adjacent to the left side 170A of the frame 105 and a right block 110B adjacent to the right side 170B of the frame 105. The blocks 110A-B may be integrated into the frame 105 on each side 170A-B (as shown), or implemented as separate components connected to the frame 105 on each side 170A-B. Alternatively, the blocks 110A-B may also be integrated into the temples (not shown) connected to the frame 105.

[0063] In one example, the image display of the optical components 180A-B includes an integrated image display. Figure 2B As shown, the optical assembly 180A-B includes a suitable display matrix 170 of any suitable type, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or any other such display. The optical assembly 180A-B also includes one or more optical layers 176, which may include lenses, optical coatings, prisms, reflectors, waveguides, light strips, and other optical components in any combination. The optical layer 176A-N may include a prism, such a prism having a suitable size and configuration, and having a first surface for receiving light from the display matrix and a second surface for emitting light to the user's eyes. The prism of the optical layer 176A-N may extend to all or at least part of the respective openings 175A-B formed in the left and right frames 107A-B, so that the user sees the second surface of the prism when looking through the corresponding left and right frames 107A-B with his eyes. The first surface of the prism of the optical layer 176A-N is arranged to face upward along the frame 105, and the display matrix covers the prism, ensuring that the photons and light emitted from the display matrix are irradiated to the first surface. The size and shape of the prisms can ensure that light is refracted within the prisms and directed toward the user's eyes by the second surfaces of the prisms of the optical layers 176A-N. In this regard, the second surfaces of the prisms of the optical layers 176A-N can be convex to direct light toward the center of the eye. The prism size and shape can be selected to magnify the image projected by the display matrix 170 and to pass light through the prisms so that the image viewed from the second surface is larger than the image emitted from the display matrix 170 in one or more dimensions.

[0064] In another example, the image display device of the optical assembly 180A-B includes Figure 2CThe projected image display shown. The optical assembly 180A-B includes a laser projector 150, which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source such as a laser projector 150 is arranged in or on any temple 125A-B of the eye-mounted device 100. The optical assembly 180A-B includes one or more light bands 155A-N, which are spaced across the lens width of the optical assembly 180A-B or the lens depth between the front and back surfaces of the lens.

[0065] When the photons projected by the laser projector 150 pass through the lenses of the optical components 180A-B, the photons encounter the light bands 155A-N. When a particular photon encounters a particular light band, the photon is either redirected to the user's eye or passed to the next light band. The specific photons or light beams can be controlled by combining the laser projector 150 modulation and the light band modulation. In one example, the processor controls the light bands 155A-N by initiating mechanical, acoustic, or electromagnetic signals. Although shown as having two optical components 180A-B, the eye-mounted device 100 may also adopt other arrangements, such as a single or three optical components, or different optical component 180A-B arrangements may be adopted depending on the application of the eye-mounted device 100 or the specific circumstances of the intended user.

[0066] like Figure 2B As further shown in Figure 1-C, the eye-mounted device 100 includes a left block 110A adjacent to the left side 170A of the frame 105 and a right block 110B adjacent to the right side 170B of the frame 105. Blocks 110A-B can be integrated into the frame 105 on each side 170A-B (as shown in the figure), or implemented as independent components connected to the frame 105 on each side 170A-B. Alternatively, blocks 110A-B can also be integrated into the temples 125 AB connected to the frame 105. Blocks 110A-B used in this article may include a housing to encapsulate a combination of a processing unit, a camera, a sensor, etc. (for example, there are differences for the right and left sides).

[0067] In one example, the image display includes a first (left) image display and a second (right) image display. The eye-mounted device 100 includes first and second openings 175A-B, which respectively accommodate first and second optical components 180A-B. The first optical component 180A includes a first image display (e.g., Figure 2B The display matrix 170A shown; or Figure 2C The second optical assembly 180B includes a second image display, for example, Figure 2B The display matrix 170B is shown; or Figure 2C Light strips 155A-N" and projector 150B are shown).

[0068] Figure 3 Show Figure 2A The rear perspective cross-sectional view of the eye-mounted device shown shows the infrared camera 220, the front portion 330 of the frame, the rear portion 335 of the frame, and the circuit board. It can be seen that the upper portion of the left frame 107A of the frame 105 of the eye-mounted device 100 includes the front portion 330 of the frame and the rear portion 335 of the frame. The front portion 330 of the frame includes a forward side configured to face outward away from the user's eyes. The rear portion 335 of the frame includes a rear side configured to face inward toward the user's eyes. An opening for the infrared camera 220 is formed on the front portion 330 of the frame.

[0069] As shown in the cross-sectional view 4-4 of the upper center portion of the left frame 107A of the frame 105, a circuit board (flexible printed circuit board (PCB) 340) is sandwiched between the frame front 330 and the frame rear 335. In addition, the left block 110A is further shown to be connected to the left temple 325A via the left hinge 126A. In some examples, the components of the depth sensor 213, including the infrared camera 220, the flexible printed circuit board 340 or other electrical connectors or contacts, can be located on the left temple 325A or the left hinge 126A.

[0070] In one example, the left block 110A includes a block body 311, a block cap 312, an inwardly facing surface 391, and an outwardly facing surface 392 (labeled but not visible). Disposed within the left block 110A are various interconnected circuit boards, such as a printed circuit board or a flexible printed circuit board, which include controller circuits for charging the battery, inwardly facing light emitting diodes (LEDs), and outwardly facing (forward) LEDs. Although shown as being formed on the left frame 107A circuit board, the depth sensor 213 (including the infrared emitter 215 and the infrared camera 220) can also be formed on the right frame 107B circuit board for capturing infrared images used in generating a three-dimensional depth image, for example, in conjunction with the right visible light camera 114B.

[0071] Figure 4 is through the infrared camera 220 and corresponds to Figure 3 The eyewear device shown is a cross-sectional view of the frame around the cross-sectional view 4-4. Figure 4The layers of the eye-mounted device 100 can be seen in the cross-section shown. As shown, the flexible printed circuit board 340 is arranged on the rear portion 335 of the frame and connected to the front portion 330 of the frame. The infrared camera 220 is arranged on the flexible printed circuit board 340 and covered by the infrared camera cover lens 445. For example, the infrared camera 220 is connected to the back of the flexible printed circuit board 340 by reflow soldering. The infrared camera 220 is reflow soldered to the electrical contact pads formed on the back of the flexible printed circuit board 340 by subjecting the flexible printed circuit board 340 to controlled heat, and the controlled heat is used to melt the solder paste to achieve a connection between the two components. In one example, reflow soldering is used to surface mount the infrared camera 220 on the flexible printed circuit board 340 and achieve an electrical connection between the two components. However, it should be understood that through holes (for example) can be used to connect the leads from the infrared camera 220 to the flexible printed circuit board 340 by interconnection.

[0072] The frame front portion 330 includes an infrared camera opening 450 for an infrared camera cover lens 445. The infrared camera opening 450 is formed on the front of the frame front portion 330, which is configured to face outwardly away from the user's eyes and toward the scene viewed by the user. In an example, the flexible printed circuit board 340 can be connected to the frame back portion 335 by a flexible printed circuit board adhesive 460. The infrared camera cover lens 445 can be connected to the frame front portion 330 by an infrared camera cover lens adhesive 455. The connection can be made indirectly through an intermediate component.

[0073] Figure 5 Show Figure 2A The eye-mounted device 100 includes an infrared emitter 215, an infrared camera 220, a front portion 330 of the frame, a rear portion 335 of the frame, and a circuit board 340. Figure 3 As shown, you can see Figure 5 The upper portion of the left frame of the eyewear device 100 includes a frame front portion 330 and a frame rear portion 335. An opening for the infrared emitter 215 is formed on the frame front portion 330.

[0074] As shown in the cross-sectional view 6-6 of the upper left frame of the frame, a circuit board (flexible printed circuit board 340) is sandwiched between the frame front 330 and the frame back 335. In addition, the left block 110A is further shown to be connected to the left temple 325A via the left hinge 126A. In some examples, the components of the depth sensor 213, including the infrared emitter 215, the flexible printed circuit board 340 or other electrical connections or contacts, can be located on the left temple 325A or the left hinge 126A.

[0075] Figure 6 is passed through the infrared emitter 215 and corresponds to Figure 5 The eyewear device shown is a cross-sectional view of the frame around the cross-sectional view 6-6. Figure 6 As shown in the cross-section of the multiple layers of the eyewear device 100, the frame 105 includes a frame front portion 330 and a frame rear portion 335. The flexible printed circuit board 340 is arranged on the frame rear portion 335 and connected to the frame front portion 330. The infrared emitter 215 is arranged on the flexible printed circuit board 340 and covered by the infrared emitter cover lens 645. For example, the infrared emitter 215 is connected to the back of the flexible printed circuit board 340 by reflow soldering. The infrared emitter 215 is reflow soldered to the electrical contact pad formed on the back of the flexible printed circuit board 340 by subjecting the flexible printed circuit board 340 to controlled heat, and the controlled heat is used to melt the solder paste to achieve a connection between the two components. In one example, the infrared emitter 215 is surface-mounted on the flexible printed circuit board 340 using reflow soldering, and the two components are electrically connected. However, it should be understood that the leads from the infrared emitter 215 can be connected to the flexible printed circuit board 340 by interconnection using through holes (for example).

[0076] Frame front 330 includes an infrared emitter opening 650 for infrared emitter cover lens 645. Infrared emitter opening 650 is formed on a front face of frame front 330, which is configured to face outwardly away from the user's eyes and toward the scene viewed by the user. In an example, flexible printed circuit board 340 may be coupled to frame back 335 via flexible printed circuit board adhesive 460. Infrared emitter cover lens 645 may be coupled to frame front 330 via infrared emitter cover lens adhesive 655. Indirect coupling via an intermediate component may also be employed.

[0077] Figure 7 An example of an emission pattern of infrared light 781 emitted by the infrared emitter 215 of the depth sensor 213 is shown. As shown in the figure, the reflection change of the emission pattern of infrared light 782 is captured as an infrared image by the infrared camera 220 of the depth sensor 213 of the eye-mounted device 100. The reflection change of the emission pattern of infrared light 782 is used to measure the depth of pixels in the original image (for example, the original image on the left) to generate a three-dimensional depth image including a depth image.

[0078] The depth sensor 213 in this example includes an infrared emitter 215 for projecting an infrared light pattern, and an infrared camera 220 (for capturing an infrared image of the distortion of the projected infrared light by objects or object features in space), as shown in the scene 715 observed by the wearer of the eye-mounted device 100. For example, the infrared emitter 215 can emit infrared light 781, which falls on an object, or on an object feature within the scene 715, like a collection of a large number of points. In some examples, the infrared light is emitted in a pattern such as a line, a spiral, or concentric rings. Infrared light is generally invisible to the naked eye. The infrared camera 220 is similar to a standard red, green, and blue (RGB) camera, except that it receives and captures images of light in the infrared wavelength range. For depth sensing, the infrared camera 220 is coupled to an image processor ( Fig. 9 912 in the system) and photo filter (e.g., artistic / stylized painting) light field effects programming or application (element 945) that determines the time of flight based on the infrared image of the captured infrared light. For example, the distorted dot pattern 782 in the captured infrared image can then be processed by the image processor to determine the depth based on the displacement of the dots. Typically, nearby targets or target features will appear as a pattern with a sparser distribution of dots, while distant objects will appear as a denser pattern of dots. It should be understood that the aforementioned functionality can be embodied in the programming instructions of the photo filter (e.g., artistic / stylized painting) light field effects programming or application (element 945) that can be found in one or more system components.

[0079] Fig. 8A An example of infrared light captured by the infrared camera 220 of the depth sensor 213 using the left infrared camera field of view 812 is shown. The infrared camera 220 captures the reflected changes in the emission pattern of the infrared light 782 in the three-dimensional scene 715 as an infrared image 859. As further shown, the left visible light camera 114A captures visible light using the left visible light camera field of view 111A as a left raw image 858A. Based on the infrared image 859 and the left raw image 858A, a three-dimensional depth image of the three-dimensional scene 715 can be generated.

[0080] Figure 8B An example of visible light captured by the left visible light camera 114A and visible light captured by the right visible light camera 114B is shown. The visible light captured by the left visible light camera 114A using the left visible light camera field of view 111A is used as the left raw image 858A. The visible light captured by the right visible light camera 114B using the right visible light camera field of view 111B is used as the right raw image 858B. Based on the left raw image 858A and the right raw image 858B, a three-dimensional depth image of the three-dimensional scene 715 can be generated.

[0081] Fig. 9is a high-level functional block diagram of an example photo filter (e.g., artistic / stylized painting) light field effect system 900, which includes a wearable device (e.g., eye-mounted device 100), a mobile device 990, and a server system 998 connected via various networks. Eye-mounted device 100 includes a depth capture camera, such as at least one visible light camera 114A-B; and a depth sensor 213, shown as an infrared emitter 215 and an infrared camera 220. The depth capture camera may alternatively include at least two visible light cameras 114A-B (one associated with the left side 170A and the other associated with the right side 170B), in which case a depth sensor is not required. The depth capture camera can generate depth images 961A-N, which are rendered three-dimensional (3D) models, texture mapped images of the imaged scene in red, green, and blue (RGB), for example, derived from the original images 858A-N, and processed (e.g., rectified) images 965A-N.

[0082] The mobile device 990 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the eye-mounted device 100 using both the low-power wireless connection 925 and the high-speed wireless connection 937. The mobile device 990 accesses the server system 998 and the network 995. The network 995 may include any combination of wired and wireless connections.

[0083] The eye-mounted device 100 also includes two image displays of the optical components 180A-B (one associated with the left side 170A and the other associated with the right side 170B). The eye-mounted device 100 also includes an image display driver 942, an image processor 912, a low power circuit 920, and a high speed circuit 930. The image displays of the optical components 180A-B are used to present images, such as original images 957A-N (e.g., original images 858A-N and processed images 965A-N), photo filter (e.g., stylized painting effect) images 963A-N, and photo filter (e.g., stylized painting) light field effect images 964A-N. The image display driver 942 is coupled to the image displays of the optical components 180A-B to control the image presentation of the image displays of the optical components 180A-B. The eye-mounted device 100 further includes a user input device 991 (eg, a touch sensor) for receiving a photo filter (eg, stylized painting effect) selection 962a input (based on the mark 962b input); and may receive a two-dimensional (2D) input selection 973 from the user.

[0084] Fig. 9The components of the eye-mounted device 100 shown, such as a printed circuit board or a flexible printed circuit board, are located on one or more circuit boards in the frame or temples. Alternatively or additionally, the components shown may be located in a block, frame, hinge, or nose bridge of the eye-mounted device 100. The left and right visible light cameras 114A-114B may include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, lenses, or any other corresponding visible light or light capturing elements that can be used to capture data (including images of scenes with unknown objects).

[0085] The eye-mounted device 100 includes a memory 934 containing photo filter (e.g., stylized painting) light field effect programming 945 for performing a subset or all of the functions of the photo filter (e.g., stylized painting) light field effect described herein, wherein a photo filter selection 962A from a user (based on a user tag 962B) is applied to an original image 858A-B or a processed image 965A-B to generate a photo filter (e.g., stylized painting) light field effect image 964A-N. As shown, the memory 934 also includes a left original image 858A captured by the left visible light camera 114A, a right original image 858B captured by the right visible light camera 114B, and an infrared image 859 captured by the infrared camera 220 of the depth sensor 213. The memory 934 also includes a plurality of depth images 961A-N generated via the depth capture camera.

[0086] Fig.11A and 11B 945. The memory 934 also includes a two-dimensional input selection 962a (e.g., initial contact point and final contact point) and a two-dimensional marker 962b received by the user input device 991. The memory 934 also includes: a left image disparity map 960A, a right image disparity map 960B, a photo filter (e.g., stylized painting effect) image 963AN, a horizontal position parameter 966, a left interpolated pixel matrix 967A including left shifted X-axis position coordinates 968A-N, a right interpolated pixel matrix 967B including right shifted X-axis position coordinates 969A-N, and a left processed (e.g., corrected) image 965A and a right processed (e.g., corrected) image 965B (e.g., removing vignetting toward the end of the lens). As further shown, the memory 934 includes a vertex matrix 970 and a rotation matrix 974. Some or all of the information stored in memory 934 may be generated during image processing of raw images 858A-B and used to generate corresponding photo filtered (eg, artistic / stylized painting) light field effect images 964A-N.

[0087] like Fig. 9As shown, the high-speed circuit 930 includes a high-speed processor 932, a memory 934, and a high-speed wireless circuit 936. In the example, an image display driver 942 is coupled to the high-speed circuit 930 and is operated by the high-speed processor 932 to drive the left and right image displays of the optical components 180A-B. The high-speed processor 932 can be any processor capable of managing the high-speed communications required by the eye-mounted device 100 and any general computing system operations. The high-speed processor 932 includes the processing resources required to manage high-speed data transmission to a wireless local area network (WLAN) over a high-speed wireless connection 937 using the high-speed wireless circuit 936. In a particular embodiment, the high-speed processor 932 can execute an operating system of the eye-mounted device 100, such as a LINUX operating system or other such operating system, and the operating system is stored in the memory 934 for execution. In addition to any other tasks, the high-speed processor 932, which executes the software architecture of the eye-mounted device 100, is used to manage data transmission with the high-speed wireless circuit 936. In some embodiments, high-speed wireless circuitry 936 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communications standard, also referred to herein as Wi-Fi. In other embodiments, other high-speed communications standards may be implemented by high-speed wireless circuitry 936.

[0088] The low power wireless circuitry 924 and high speed wireless circuitry 936 of the eye-mounted device 100 may include a short range transceiver (Bluetooth™) and a wireless wide area, local area, or wide area network transceiver (e.g., cellular or WiFi). The mobile device 990 includes a transceiver that communicates via the low power wireless connection 925 and the high speed wireless connection 937 and may be implemented using details of the architecture of the eye-mounted device 100, as may other elements of the network 995.

[0089] The memory 934 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the left and right visible light cameras 114A-B, the infrared camera 220, and the image processor 912, and images generated by the image display driver 942 for display on the optical assembly 180A-B image display. Although the memory 934 is shown as being integrated with the high-speed circuit 930, in other embodiments, it can be a separate component of the eye-mounted device 100. In some such embodiments, electrical routing lines can provide connections to the memory 934 through a chip that includes the high-speed processor 932 or the low-power processor 922 from the image processor 912. In other embodiments, the high-speed processor 932 can manage the addressing of the memory 934 so that whenever a read or write operation involving the memory 934 is required, the low-power processor 922 will guide the high-speed processor 932.

[0090] like Fig. 9As shown, the processor 932 of the eye-mounted device 100 may be coupled to the depth capture camera (visible light camera 114A-B; or visible light camera 114A, infrared emitter 215 and infrared camera 220), image display driver 942, user input device 991 and memory 934. Fig.10 As shown, the processor 1030 of the mobile device 990 can be coupled to the depth capture camera 1070, the image display driver 1090, the user input device 1091, and the memory 1040A. As the processor 932 of the eye-mounted device 100 executes the photo filter (e.g., artistic / stylized painting) light field effect programming 945 in the memory 934, the eye-mounted device 100 can perform all or a subset of any of the following functions. As a result of the processor 1030 of the mobile device 990 executing the photo filter (e.g., artistic / stylized painting) light field effect programming 945 in the memory 1040A, the mobile device 990 can perform all or a subset of any of the following functions. Functionality may be divided in the filter (e.g., artistic / stylized painting) light field effect system 900 so that the eye-mounted device 100 generates the original images 858A-B, while the mobile device 990 performs the remainder of the image processing on the original images 858A-B to generate the photo filter (e.g., artistic / stylized painting) light field effect images 964A-N.

[0091] In one example, the depth capture camera of the eye-mounted device 100 includes at least two visible light cameras, and the visible light camera consists of a left visible light camera 114A with a left field of view 111A and a right visible light camera 114B with a right field of view 111B. The left field of view 111A and the right field of view 111B have overlapping fields of view 813 (see Figure 8B ). The depth capture camera 1070 of the mobile device 990 may adopt a similar structure.

[0092] The photo filter (e.g., artistic / stylized painting) light field effect programming 945 is executed by the processor 932, 1030, and the photo filter (e.g., artistic / stylized painting) light field effect system 900 is configured to perform functions, including a capture function, through the depth capture camera, the left original image 858A and the right original image 858B. The photo filter (e.g., artistic / stylized painting) light field effect system 900 is used to calculate: (i) a left image disparity map 960A between a left pixel matrix and a right pixel matrix, and (ii) a right image disparity map 960B between a right pixel matrix and a left pixel matrix. The left original image 858A or the left processed image 965A includes the left pixel matrix, and the right original image 858B or the right processed image 965B includes the right pixel matrix.

[0093] The photo filter (e.g., artistic / stylized painting) light field effects system 900 presents an original image 957A via an image display 180A-B, 1080. The photo filter (e.g., artistic / stylized painting) light field effects system 900 receives an artistic effect selection 962a (based on a tag 962b from a user) via a user input device 991, 1091 to be applied to the presented original image 957A. The photo filter (e.g., artistic / stylized painting) light field effects system 900 may create at least one stylized painterly effect image having a stylized painterly effect scene by applying the stylized painterly effect selection 962 (based on a tag 962b from a user) to: (i) the left original image 858A or the left processed image 965A to create a left stylized painterly effect image 963A, (ii) the right original image 858B or the right processed image 965B to create a right stylized painterly effect image 963B, or (iii) a combination of the above.

[0094] The photo filter (e.g., art / stylized painting) light field effect system 900 can generate a stylized painterly effect image 964A having a spatially moved or rotated appearance, a stylized painterly effect scene surrounding at least one stylized painterly effect image. This function is achieved by blending a left stylized painterly effect image 963A and a right stylized painterly effect image 963B based on a left image disparity map 960A and a right image disparity map 960B. The photo filter (e.g., art / stylized painting) light field effect system 900 presents the stylized painterly effect image 964A through the image display 180A-B, 1080.

[0095] The function for calculating the left image disparity map 960A and the right image disparity map 960B includes the following functions. First, a left corrected image 965A is created from the left original image 858A as the left processed image 965A, and a right corrected image 965B is created from the right original image 858B as the right processed image 965B to align the left and right original images 858A-B and remove the distortion in the respective lenses of the left and right visible light cameras 114A-B. Second, the left image disparity map 960A and the right image disparity map 960B are extracted by correlating the pixels in the left corrected image 965A and the right corrected image 965B, and vice versa, to calculate the disparity of each correlated pixel.

[0096] The function for generating the stylized painting effect image 964A includes the following functions. First, the horizontal position shift parameter 966 along the X-axis of the left and right pixel matrices is determined. Second, the left interpolation pixel matrix 967A is filled by shifting the pixels in the left pixel matrix along the X-axis based on the horizontal shift parameter 966. Third, the right interpolation pixel matrix 967B is filled by shifting the pixels in the right pixel matrix along the X-axis based on the horizontal shift parameter 966. Finally, the stylized painting effect image 964A is created by mixing the left interpolation pixel matrix 967A and the right interpolation pixel matrix 967B.

[0097] The function for filling the left interpolation pixel matrix 967A includes the following functions. First, the left image disparity corresponding to the left image disparity map 960A from each corresponding pixel in the left pixel matrix is ​​multiplied by the horizontal shift parameter 966 to derive the corresponding left shifted X-axis position coordinate 968A-N. Second, in the left interpolation pixel matrix 967A, each corresponding pixel is shifted to the corresponding left shifted X-axis position coordinate 968A-N.

[0098] The function for filling the right interpolation pixel matrix 967B includes the following functions. First, the right image disparity map 960B corresponding to each corresponding pixel in the right pixel matrix is ​​multiplied by the complement of the horizontal movement parameter 966 to derive the corresponding right side movement X-axis position coordinate 969A-N. For example, the complement of the horizontal movement parameter 966 is the number 1 minus the horizontal movement parameter 966 (i.e., 1-x). Second, each corresponding pixel is moved to the corresponding right side movement X-axis position coordinate 969A-N in the right interpolation pixel matrix 967B.

[0099] The function for generating the stylized painting effect image 964A by blending the left interpolated pixel matrix 967A and the right interpolated pixel matrix 967B can be based on the disparity confidence level, gradient, or a combination of the two in the left image disparity map 960A and the right image disparity map 960B. For example, the disparity confidence level value is based on the correlation between the left and right pixels.

[0100] The function for determining the horizontal position movement parameter 966 includes the following functions. First, a two-dimensional input selection 973 of the presented original image 957A is received from the user through the user input device 991, 1091. Secondly, the movement of the two-dimensional input selection 973 from the initial contact point to the final contact point of the presented original image 957A is tracked through the user input device 991, 1091. Thirdly, a rotation matrix 974 describing the rotation from the initial contact point to the final contact point is determined, and the horizontal position movement parameter 966 is derived. Nevertheless, it should be understood that the rotation matrix 974 is not required in the stylized painting effect unless the data is represented using 3D vertices. In some examples, the horizontal position movement parameter 966 can also be determined by measuring the IMU 972, for example, using the tilt angle of the mobile device 990 or the eye-mounted device 100.

[0101] In one example, the user input device 991, 1091 includes a touch sensor, which is composed of an input interface and a sensor array, and is used to couple to the input interface to at least receive a single finger contact from a user input. The user input device 991, 1091 also includes a sensing circuit integrated or connected to the touch sensor or connected to the processor 932, 1030. The sensing circuit is configured to measure voltage to at least track a single finger contact on the user input interface. The function of receiving a stylized painting effect selection 962a and a mark 962b from a user through the user input device 991, 1091 includes at least receiving a single finger contact from a user input on the input interface of the touch sensor. The function of tracking the movement of the two-dimensional input selection 973 from the initial contact point to the final contact point through the user input device 991, 1091 includes dragging from the initial contact point to the final contact point on the touch sensor input surface via the sensing circuit from the input interface.

[0102] The touch user input device 991 can be integrated into the eye-mounted device 100. As described above, the eye-mounted device 100 includes a block 110A-B integrated into or connected to the frame 105 on the side 170A-B of the eye-mounted device 100. The frame 105, the temples 125A-B, or the block 110A-B includes a circuit board with a touch sensor. The circuit board includes a flexible printed circuit board. The touch sensor is arranged on the flexible printed circuit board. The sensor array is a capacitor array or a resistor array. The capacitor array or the resistor array includes a grid forming a two-dimensional rectangular coordinate system for tracking X-axis and Y-axis position coordinates.

[0103] In one example of the photo filter (e.g., artistic / stylized effect) light field effect system 900, the processor is composed of a first processor 932 and a second processor 1030. The memory is composed of a first memory 934 and a second memory 1040A. The eye-mounted device 100 includes a first network communication interface 924 or 936 for communicating through a network 925 or 937 (e.g., a wireless short-range network or a wireless local area network). The first processor 932 is coupled to the first network communication interface 924 or 936. The first memory 934 can access the first processor 932. The eye-mounted device 100 also includes a photo filter (e.g., artistic / stylized painting) light field effect programming 945 in the first memory 934. The photo filter (e.g., artistic / stylized painting) light field effect programming 945 is executed by the first processor 932 to configure the eye-mounted device 100 to execute a function of capturing a left original image 858A and a right original image 858B through a depth capture camera.

[0104] The photo filter (e.g., artistic / stylized painting) light field effect system 900 also includes a host, such as a mobile device 990, coupled to the eye-mounted device 100 via a network 925 or 937. The host includes a second network communication interface 1010 or 1020 for communicating via the network 925 or 937. The second processor 1030 is coupled to the second network communication interface 1010 or 1020. The second memory 1040A can access the second processor 1030. The host also includes photo filter (e.g., artistic / stylized painting) light field effect programming 945 in the second memory 1040A.

[0105] The photo filter (e.g., artistic / stylized painting) light field effect programming 945 is executed by the second processor 1030, and the host is configured to execute a function of receiving the original image 957A from the eye-mounted device 100 via the second network communication interface 1010 or 1020 via the network 925 or 937. The photo filter (e.g., artistic / stylized painting) light field effect programming 945 is executed by the second processor 1030, and the host is configured to calculate: (i) a left image disparity map 960A, and (ii) a right image disparity map 960B. The photo filter (e.g., artistic / stylized painting) light field effect programming 945 is executed by the second processor 1030, and the host is configured to present the original image 957A through the image display 1080. The photo filter (e.g., artistic / stylized painting) light field effect programming 945 is executed by the second processor 1030 to configure the host to receive a stylized painting effect selection 962a and a tag 962b from a user via a user input device 1091 (e.g., a touch screen or a computer mouse). The photo filter (e.g., stylized painting) light field effect programming 945 is executed by the second processor 1030 to configure the host to create a stylized painting effect image 963 AB by applying the style of the selected image to the tag. The photo filter (e.g., stylized painting) light field effect programming 945 is executed by the second processor 1030 to configure the host to generate a stylized painting effect image 964A having an appearance of rotation or spatial motion. The photo filter (e.g., stylized painting) light field effect programming 945 is executed by the second processor 1030 to configure the host to present the stylized painting effect image 964A via the image display 1080.

[0106] Each depth image 961A-N is formed by a vertex matrix 970. Each pixel of the two-dimensional images 858A-B, 963A-B, 964A-N, 965A-B can be associated with a respective vertex of the vertex matrix 970. Each vertex has a position attribute. The position attribute of each vertex is based on a three-dimensional position coordinate system, including an X position coordinate on the X axis (indicating a horizontal position), a Y position coordinate on the Y axis (indicating a vertical position), and a Z position coordinate on the Z axis (indicating a depth position). Each vertex also includes one or more of a color attribute, a texture attribute, or a reflection attribute. Thus, the stylized painting effect selection 962a (based on the markup 962b from the user and the style transfer to the marked area) is applied to the function of: (i) the left original image 858A or the left processed image 965A to create the left stylized painting effect image 963A, (ii) the right original image 858B or the right processed image 965B to create the right stylized painting effect image 963B, or (iii) a combination of the two, based on the Z position coordinate to change the filter effect strength of the stylized painting effect function 971 for transforming each pixel according to the depth position of each vertex associated with each pixel. On the Z axis, the intensity of the filter effect applied to each vertex with a Z position coordinate having a deeper depth position is stronger than that of a position with a shallower depth.

[0107] The server system 998 can be one or more computing devices as part of a service or network computing system (e.g., including a processor, a memory, and a network communication interface) to communicate with the mobile device 990 and the eye-mounted device 100 through the network 995. The eye-mounted device 100 is connected to the host. For example, the eye-mounted device 100 is paired with the mobile device 990 via the high-speed wireless connection 937, or connected to the server system 998 via the network 995.

[0108] A lenticular print can be made from a generated photo-filtered (e.g., artistic / stylized painting) light field effect image 964A-N by generating multiple (e.g., ten 964A-J) views for a particular moment between the left and right images 858A-B, 965A-B. Multiple views (one view corresponding to a generated photo-filtered stylized painting effect image 964A-J) are printed in strips. The lenticular is made of plastic and is glued to the lenticular printing device with a half-tube, which acts like a lens. When an observer looks at it with both eyes, each eye sees a different set of stripes and thus a different image 964A-J. By forming multiple (ten) views 964A-J and gluing the lenticular on top, a stereoscopic lenticular print can be created to provide a 3D appearance. Moving the lenticular print provides the effect of different viewpoints, thereby obtaining a short animation within the lenticular print. A photo printing service (implemented by a host, such as a server system 998 or a mobile device 990) may receive a plurality of generated photo filter (e.g., artistic / stylized painting) light field effect images 964A-N via a network 925, 937, 995, which may be printed out as a raster print (e.g., using a 3D printer). In some examples, the raster print may stitch the photo filter (e.g., artistic / stylized painting) light field effect images 964A-N together in sequence to form a short video.

[0109] For example, N viewpoints are created, 0.1, 0.2, up to 1, to generate 10 views corresponding to 10 photo-filtered (e.g., artistic / stylized painting) light field effect images 964A-J. The printer sequence is as follows: first column of first view 964A, first column of second view 964B, and so on, up to 964J; followed by the next pixel column, i.e., second column of first view 964A, second column of second view 964B, and so on, up to 964J. A raster is glued to the print, whereby each image up to 964A-J can be viewed at a different angular orientation. When the user rotates the raster print, the ten different views are rotated relative to each other up to 964A-J.

[0110] The output components of the eye-mounted device 100 include visual components such as Figure 2B-C shows the left image display and the right image display of the optical assembly 180A-B (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector or a waveguide, etc.). The image display of the optical assembly 180A-B is driven by the image display driver 942. The output components of the eye-mounted device 100 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components of the eye-mounted device 100, the mobile device 990, and the server system 998 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric inputs, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide touch positions and forces or touch gestures), audio input components (e.g., microphones), etc.

[0111] The eye-mounted device 100 may optionally include additional peripheral device elements. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the eye-mounted device 100. For example, the peripheral device element may include any input / output component, consisting of an output component, a motion component, a position component, an indicator, or any other such component described herein.

[0112] For example, the biometric component includes components that can be used to detect expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measure biological signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identify people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The position component includes a position sensor component for generating position coordinates (e.g., a global positioning system (GPS) receiver component), a WiFi or Bluetooth™ transceiver for generating positioning system coordinates, an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which the altitude can be derived), a direction sensor component (e.g., a magnetometer), etc. Such positioning system coordinates can also be received from the mobile device 990 via the low power wireless circuit 924 or the high-speed wireless circuit 936 via the wireless connections 925 and 937.

[0113] The inertial measurement unit (IMU) 972 is an electronic device that uses a combination of accelerometers and gyroscopes, and sometimes a magnetometer, to measure and report specific forces, angular rates, and sometimes magnetic fields around the body. In the case of a magnetometer, the magnetic field can be used as an input to detect specific postures that rely on the earth's magnetic field or an artificial magnetic field. In this example, the inertial measurement unit is used to determine the rotational acceleration of the eye-mounted device 100. The inertial measurement unit 972 works by detecting linear acceleration using one or more accelerometers and detecting rotation rate using one or more gyroscopes. A typical configuration of an inertial measurement unit includes one accelerometer, gyroscope, and magnetometer for each of the three axes: a horizontal axis (X) for left-right movement, a vertical axis (Y) for top-bottom movement, and a depth axis or distance axis (Z) for up-down movement. The gyroscope detects rotation rate around the 3 axes (X, Y, and Z). The magnetometer is used to detect magnetic fields (e.g., south, north, etc.), similar to a compass, and its principle is to produce a heading reference that is a mixture of the earth's magnetic field and other artificial magnetic fields (such as those generated by power lines). The three accelerometers detect acceleration along the horizontal (X), vertical (Y), and depth or distance (Z) axes defined above, which can be defined relative to the ground, the eye-mounted device 100, the depth capture camera, or the user wearing the eye-mounted device 100. Therefore, the accelerometers are used to detect a three-axis acceleration vector, which can then be used to detect the earth's gravity vector.

[0114] Fig.10 is a high-level functional block diagram of an example mobile device 990 that is configured to Fig. 9 The photo filter (e.g., stylized painting) light field effect system 900 is shown communicating with the mobile device 990. The mobile device 990 includes a user input device 1091 for receiving a photo filter (e.g., stylized painting) selection 962a, a mark 962b, or a two-dimensional input selection 973, which is applied to the original image 957A to generate a photo filter (e.g., stylized painting) light field effect image 964A.

[0115] The mobile device 990 includes flash memory 1040A containing photo filter (e.g., artistic / stylized painting) light field effect programming 945 for performing all or a subset of the functions described herein for generating photo filter (e.g., artistic / stylized painting) light field effects, wherein photo filter selections 962a (based on tags 962b from a user and style transfer of regions) are applied to original images 858A-B or processed images 965A-B to generate photo filter stylized painting effect images 964A-N.

[0116] As shown, the memory 1040A also includes a left raw image 858A captured by the left visible light camera 114A, a right raw image 858B captured by the right visible light camera 114B, and an infrared image 859 captured by the infrared camera 220 of the depth sensor 213. The mobile device 1090 may include a depth capture camera 1070, which is composed of at least two visible light cameras (a first and a second visible light camera with overlapping fields of view), or at least one visible light camera, and a depth sensor with substantially overlapping fields of view (such as the eye-mounted device 100). When the mobile device 990 includes a component such as the eye-mounted device 100 (e.g., a depth capture camera), the left raw image 858A, the right raw image 858B, and the infrared image 859 may be captured by the depth capture camera 1070 of the mobile device 990.

[0117] The memory 1040A also includes a plurality of depth images 961A-N, which are generated by the depth capture camera of the eye-mounted device 100 or by the depth capture camera 1070 of the mobile device 990 itself. Fig.11A and 11B 1040A also includes a two-dimensional input selection 973, such as an initial contact point and a final contact point received by a user input device 1091. Memory 1040A also includes: a left image disparity map 960A, a right image disparity map 960B, a photo filter (e.g., stylized painting effect) image 963 AN, a horizontal position parameter 966, a left interpolated pixel matrix 967A including left shifted X-axis position coordinates 968A-N, a right interpolated pixel matrix 967B including right shifted X-axis position coordinates 969A-N, and a left processed (e.g., corrected) image 965A and a right processed (e.g., corrected) image 965B (e.g., to remove vignetting toward the end of the lens). As further shown, memory 1040A includes a vertex matrix 970 and a rotation matrix 974. Some or all of the information stored in memory 1040A may be generated during image processing of raw images 858A-B to generate corresponding photo filtered (eg, artistic / stylized painting) light field effect images 964A-N.

[0118] As shown in the figure, the mobile device 990 includes an image display 1080, an image display driver 1090 for controlling image display, and a user input device 1091 similar to the eye-mounted device 100. Fig.10 In the example shown, the image display 1080 and user input device 1091 are integrated into a touch screen display.

[0119] Examples of touch screen mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen devices are provided by way of example, and the subject technology described herein is not limited thereto. Therefore, for purposes of this discussion, Fig.10 A block diagram illustration of an exemplary mobile device 990 is provided that is equipped with a touch screen display for displaying content and receiving user input as (or as part of) a user interface.

[0120] The focus activities discussed herein generally involve applying a photo filter selection 962a from a user (based on user tags 962b and style transfer to the tagged regions) to an original image 858A-B or processed image 965A-B to generate data communication portable eye-mounted device 100 or mobile device 990 associated with a photo filter stylized painting effect image 964A-N. Fig.10 As shown, the mobile device 990 includes at least one digital transceiver (XCVR) 1010, which is shown as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 990 also includes additional digital or analog transceivers, such as a short-range XCVR 1020 for short-range network communication, for example, by means of NFC, VLC, DECT, ZigBee, Bluetooth™, or WiFi. Taking the short-range XCVR 1020 as an example, it can take the form of any available two-way wireless local area network (WLAN) transceiver, the type of which is compatible with one or more standard communication protocols implemented in a wireless local area network, such as a Wi-Fi standard according to IEEE 802.11 and WiMAX.

[0121] To generate location coordinates for locating the mobile device 990, the mobile device 990 may include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 990 may utilize one or both of the short-range XCVR 1020 and the WWAN XCVR 1010 to generate location coordinates for positioning. For example, a positioning system based on GPS, a cellular network, WiFi, or Bluetooth™ can generate very accurate location coordinates, especially when used in combination. Such location coordinates can be transmitted to the eye-mounted device via one or more network connections utilizing the XCVRs 1010, 1020.

[0122] Transceivers 1010, 1020 (network communication interface) conform to one or more of the various digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers 1010 include (but are not limited to) transceivers operating in accordance with Code Division Multiple Access (CDMA) and Third Generation Partnership Project (3GPP) network technology configurations, including but not limited to 3GPP2 type (or 3GPP2) and LTE, sometimes referred to as "4G". Taking transceivers 1010, 1020 as examples, they can provide two-way wireless communication of information, including digitized audio signals, still images and video signals, web page information for display and input related to the web page, and various types of mobile message communications between mobile devices 990 for photo filters (e.g., artistic / stylized paintings) light field effects.

[0123] As previously described, several types of communications through the transceivers 1010, 1020 and the network involve protocols and processes that support communications with the eye-mounted device 100 or server system 998 that generates photo-filtered (e.g., artistic / stylized painting) light field effect images 964A-N, such as sending left original image 858A, right original image 858B, infrared image 859, depth image 961A-N, photo-filtered image 963A-B, and processed (e.g., corrected) image 965A-B. Taking these communications as an example, packetized data can be transmitted to or from the eye-mounted device 100 via the short-range XCVR 1020 via the wireless connections 925 and 937, such as Fig. 9 In addition, this type of communication can also be achieved through Fig. 9 The illustrated network (eg, the Internet) 995 transmits data utilizing IP packet data transmission via the WWAN XCVR 1010. Both the WWAN XCVR 1010 and the short range XCVR 1020 are connected to associated antennas (not shown) via radio frequency (RF) transmit and receive amplifiers (not shown).

[0124] The mobile device 990 also includes a microprocessor, shown as CPU 1030, sometimes referred to as a host controller in this article. A processor is a circuit whose elements are constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components can be used, components that constitute a programmable CPU are used in the example. For example, a microprocessor includes one or more integrated circuit (IC) chips that integrate electronic components that perform CPU functions. Taking processor 1030 as an example, it can be based on any known or available microprocessor architecture, such as reduced instruction set computing (RISC) using the ARM architecture, which is very common in today's mobile devices and other portable electronic devices. Other processor circuits can be used to form CPU 1030 or processor hardware in smartphones, laptops, and tablets.

[0125] The microprocessor 1030 acts as a programmable host controller for the mobile device 990 by configuring the mobile device 990 to perform various operations (e.g., according to instructions or programming executable by the processor 1030). For example, such operations may include various general operations of the mobile device, as well as operations related to photo filters (e.g., artistic / stylized paintings), light field effects programming 945, and communication with the eye-mounted device 100 and the server system 998. Although the processor can be configured by employing hard-wired logic, a typical processor in a mobile device is a general processing circuit that is configured by executing programming.

[0126] The mobile device 990 includes a memory or storage device system for storing data and programming. In an example, the memory system may include flash memory 1040A and random access memory (RAM) 1040B. RAM 1040B is used as short-term storage for instructions and data processed by processor 1030, for example, as working data processing memory. Flash memory 1040A generally provides longer-term storage.

[0127] Thus, in the mobile device 990 example, the flash memory 1040A is used to store programming or instructions executed by the processor 1030. Depending on the device type, the mobile device 990 may store and run a mobile operating system, through which specific applications are executed, including the photo filter (e.g., artistic / stylized painting) light field effect programming 945. Taking the photo filter (e.g., artistic / stylized painting) light field effect programming 945 application as an example, it can be a native application, a hybrid application, or a web application (e.g., a dynamic web page executed by a web browser) running on the mobile device 990 to generate the photo filter (e.g., stylized painting) light field effect image 964A-N according to the received photo filter (e.g., stylized painting) selection 962a and the mark 962b. Examples of mobile operating systems include Google Android®, Apple iOS® (iPhone® or iPad® device), Windowsmobile®, Amazon Fire® OS, RIM BlackBerry® operating system, etc.

[0128] The mobile device 990 should be understood to be just one type of host in the photo filter (eg, artistic / stylized painting) light field effect system 900, and other arrangements may also be used. Fig. 9 Taking the server system 998 shown as an example, after generating the original images 858A-B, it can generate a photo filter (e.g., artistic / stylized painting) light field effect image 964A through the depth capture camera of the eye-mounted device 100.

[0129] Fig.11Ais a flow chart of a method with specific steps that can be implemented in a photo filter (e.g., artistic / stylized painting) light field effect system 900 for applying a photo filter selection 962a from a user (based on user tags 962b and style transfer to the tagged areas) to an original image 858A-B or a processed image 965A-B to generate a photo filter (e.g., artistic / stylized painting) light field effect image 964A-N. In an example using a depth capture camera with two visible light cameras, the left and right original images are used. In an example using a depth capture camera with one visible light camera and one infrared camera, the left or right original image from the visible light camera is used (depending on which side the visible light camera is located), and the right or left processed image is generated from the left or right visible original image using an offset based on distance information obtained from the infrared camera, and the process is performed using the original image and the processed image.

[0130] Taking the method shown in box 1100 as an example, it includes capturing the left raw image 858A and the right raw image 858B through the depth capture camera. For example, the left visible light camera 114A and the right visible light camera 114B capture the left raw image 858A and the right raw image 858B respectively. In another example, the method used includes capturing the left raw image 858A or the right raw image 858B through the depth capture camera, and generating the right processed image 965B or the left processed image 965A by offsetting based on the distance information obtained from the infrared camera. According to this example, in the following steps, the processed image of one side is generated from the other side raw image using such an offset to replace the original image.

[0131] Continuing with block 1110, the method also includes calculating: (i) a left image disparity map 960A between a left pixel matrix and a right pixel matrix, and (ii) a right image disparity map 960B between a right pixel matrix and a left pixel matrix. The left pixel matrix is ​​based on the left original image 858A or the left processed image 965A. The right pixel matrix is ​​based on the right original image 858B or the right processed image 965B.

[0132] Calculating the left image disparity map 960A and the right image disparity map 960B includes the following steps. First, a left corrected image 965A is created from the left original image 858A as the left processed image 965A, and a right corrected image 965B is created from the right original image 858B as the right processed image 965B to align the left and right original images 858A-B and remove the distortion in the respective lenses of the left and right visible light cameras 114A-B. Second, the left image disparity map 960A and the right image disparity map 960B are extracted by correlating the pixels in the left corrected image 965A and the right corrected image 965B, and vice versa, to calculate the disparity of each correlated pixel (for example, using SGBM).

[0133] Correction is applied to modify each captured image or video so that corresponding pixels are on the same raster line (row). Once this is done, an image disparity calculation algorithm such as SGBM can be applied. The disparity calculation algorithm finds a corresponding pixel for each pixel in the right image and the left image. For each pixel in the right image, find the corresponding pixel in the left image. For non-occluded pixels (pixels that can be seen from both cameras), the same disparity is usually found in the order from left to right and from right to left; but on the other hand, occluded pixels need to be processed separately, usually using neighboring pixel blending techniques.

[0134] The method shown in block 1120 also includes presenting the original image 957A via the image display 180A-B, 1080. The original image 957A is based on the left original image 858A, the left processed image 965A, the right original image 858B, the right processed image 965B, or a combination thereof.

[0135] The method shown in box 1130 also includes receiving a photo filter selection 962a from the user through the user input device 991, 1091 to be applied to the rendered original image 957A.

[0136] The method shown in block 1135 also includes creating a photo filter (e.g., stylized painting) image having a photo filter effect scene. In one example, a mark on the original image 957A is received via a user input device 991, 1091 (block 1135a; Fig. 11B The tag defines the area of ​​the image to which the image style is applied and can be applied to a blank image and stored in memory. The style of the image is then applied to the tag (block 1135b; Fig. 11B ). The user can select a target image from a list of pre-selected images, an image found on the Internet, an image stored on a mobile device, or an image captured by a mobile device, through the user input device 991, 1091. The style of the image can be transferred to the marker by NST.

[0137] The method shown in block 1140 also includes a photo filter image created in response to a user selection of: (i) the left original image 858A or the left processed image 965A for creating the left photo filter image 963A, (ii) the right original image 858B or the right processed image 965B for creating the right photo filter image 963B, or (iii) a combination of the two. The photo filter image may be overlaid on the original image to replace the marked area (see Fig. 12C ), or can be blended with pixels from the original image in the marked area to allow for passing through features such as shadows (see Fig.13 ).

[0138] Continuing with block 1150, the method also includes generating a photo filter stylized painting effect image 964A with a spatially shifted or rotated appearance, which surrounds at least one photo filter scene in the photo filter stylized painting effect image 963 AB. This can be achieved by blending the left photo filter image 963 A and the right photo filter image 963 A, based on the left image disparity map 960A and the right image disparity map 960B. Generating the photo filter stylized painting effect image 964A includes the following steps. First, determine the horizontal position shift parameter 966 along the X axis of the left and right pixel matrices. Second, fill the left interpolation pixel matrix 967A by shifting the pixels in the left pixel matrix along the X axis based on the horizontal shift parameter 966. Third, fill the right interpolation pixel matrix 967B by shifting the pixels in the right pixel matrix along the X axis based on the horizontal shift parameter 966. Finally, create the photo filter stylized painting effect image 964A by blending the left interpolation pixel matrix 967A and the right interpolation pixel matrix 967B.

[0139] The step of filling the left interpolation pixel matrix 967A includes the following functions. First, the left image disparity corresponding to the left image disparity map 960A from each corresponding pixel in the left pixel matrix is ​​multiplied by the horizontal shift parameter 966 to derive the corresponding left shifted X-axis position coordinate 968A-N. Second, in the left interpolation pixel matrix 967A, each corresponding pixel is shifted to the corresponding left shifted X-axis position coordinate 968A-N.

[0140] The operation of filling the right interpolation pixel matrix 967B includes the following steps. First, the corresponding right image disparity in the right image disparity map 960B of each corresponding pixel in the right pixel matrix is ​​multiplied by the complement of the horizontal motion parameter 966 (e.g., subtracting the horizontal motion parameter 966 from the number 1; i.e., 1-x) to derive the corresponding right moving x-axis position coordinate 969A-N. Second, each corresponding pixel is moved to the corresponding right moving x-axis position coordinate 969A-N in the right interpolation pixel matrix 967B.

[0141] Once two disparity maps (a left image disparity map 960A and a right image disparity map 960B) are created, the horizontal shift parameter 966 will be moved between 0 and 1 to set or tilt the spatial shift or rotation of the generated photo filter (e.g., artistic / stylized painting) light field effect image 964A. Assuming that the horizontal shift parameter 966 is set to 0 to tilt completely toward the left image, the horizontal shift parameter 966 is set to 1 to tilt completely toward the right image. If the horizontal shift parameter 966 is set to 0, the weight is set to output the left image as the photo filter (e.g., artistic / stylized painting) light field effect image 964A. If the horizontal shift parameter 966 is set to 1, the weight is set to output the right image as the photo filter (e.g., artistic / stylized painting) light field effect image 964A. When the photo filter (e.g., artistic / stylized painting) light field effect image 964A is not equal to 0 or 1 (intermediate values), the spatial shift or rotation is between the left and right images. For the horizontal motion parameter 966 set to 0.5, the empty interpolated pixel matrix 967A-B is filled with RGB values ​​to derive the intermediate photo filter (e.g., artistic / stylized painting) light field effect image 964A-N. For the left interpolated pixel matrix 967A, since the horizontal motion parameter 966 is set to 0.5, the pixels in the left image are moved to half of the corresponding pixels in the right image according to the corresponding disparity values ​​from the left image disparity map 960A. For example, each disparity value from the left image disparity map 960A is multiplied by 0.5 and added to the X-axis position coordinate to derive the left shifted X-axis position coordinate 968A. The right interpolated pixel matrix 967B is filled in the same manner by moving the pixels in the right image to half of the corresponding pixels in the left image according to the corresponding disparity values ​​from the right image disparity map 960B. For example, each disparity value from the right image disparity map 960B is multiplied by 0.5 and added to the X-axis position coordinate to derive the right shifted X-axis position coordinate 969A. Therefore, for each pixel, the color value remains the same, but the X-axis position coordinate is shifted by half the disparity value on the X-axis. If a pixel has no value (occluded), but a neighboring pixel has a value, the pixel value of the occluded pixel is calculated based on the weighted neighboring pixels and the disparity confidence level.

[0142] In another example, assume that the horizontal shift parameter 966 is set to 0.1. To fill the left interpolated pixel matrix 967A, the following calculation is used: For each left pixel in the left image, the corresponding disparity value from the left image disparity map 960A is multiplied by 0.1 to derive the corresponding left shifted X-axis position coordinates 968A-N. To fill the right interpolated pixel matrix 967B, the following calculation is used: For each right pixel in the right image, the corresponding disparity value in the right image disparity map 960B is multiplied by 0.9 to obtain the corresponding right shifted X-axis position coordinates 969A-N. This will create a new view between the left image and the right image.

[0143] The step of generating a photo filter (e.g., art / stylized painting) light field effect image 964A is achieved by blending the left interpolated pixel matrix 967A with the right interpolated pixel matrix 967B. The blending is based on the disparity confidence level (e.g., by weighting the contribution of each side), gradient, or a combination thereof in the left image disparity map 960A and the right image disparity map 960B. Taking the disparity confidence level value as an example, it is based on the magnitude of the correlation between the left and right side pixels. Although it may be expected to obtain the same image, due to differences in reflection, lighting, etc. caused by different viewing angles in the left and right images, the combined photo filter (e.g., art / stylized painting) light field effect image 964A is not the same (hence the name stylized painting effect). This will create a photo filter (e.g., art / stylized painting) light field effect image 964A using a new view.

[0144] When generating the photo filter (e.g., artistic / stylized painting) light field effect image 964A, actual distance or depth rotation is not used, and 3D vertices are not employed. Instead, parallax is used, which is related to depth, but parallax is not a direct substitute for depth. In fact, parallax is the movement of pixels, which means that image processing can be done in 2D space to speed up operation and reduce memory requirements. No 3D transformation is required, only interpolation between corresponding pixels and corresponding pixels is provided. Although the correspondence (parallax) can be converted into depth (distance), the photo filter (e.g., artistic / stylized painting) light field effect does not require depth. It does not matter whether the depth on the Z axis is 10 meters or 20 meters, because the pixels are moved to different X-axis position coordinates based on the horizontal movement parameters 966.

[0145] Going to box 1160, the method also includes presenting the photo filter stylized painting effect image 964A through the image display 180A-B, 1080. In some examples, the steps of capturing the left original image 858A and the right original image 858B through the depth capture camera are implemented on the eye-mounted device 100. Calculation steps: (i) left image disparity map 960A, and (ii) right image disparity map 960B; presenting the original image 957A through the image display 1080; receiving the photo filter effect selection 962a through the user input device 1091; creating the photo filter image 963 AB; generating the photo filter stylized painting effect image 964A; and presenting the photo filter stylized painting effect image 964A on the host 990, 998 through the image display 1080.

[0146] Applying the stylized painting effect selection 962a from the user to: (i) the left original image 858A or the left processed image 965A to create the left stylized painting effect image 963a, (ii) the right original image 858B or the right processed image 965B to create the right stylized painting effect image 963B, or (iii) a combination of the two can be based on the Z position coordinate. This can change the filter effect strength of the stylized painting effect function 971 to transform each pixel according to the depth position of each vertex associated with each pixel. On the Z axis, the intensity of the filter effect applied to each vertex with a Z position coordinate of a deeper depth position is stronger than that of a position with a shallower depth.

[0147] Now completing the discussion of block 1170, the illustrated method may also include generating a raster print from views of a plurality of photo-filtered (e.g., artistic / stylized painterly) light field effect images 964A-N. Raster prints are used to create two 3D images from a left image and a right image and all images in between. In one example, 15 different views 964A-0 may be combined together, and as the raster print moves around, a hologram-like image may appear. Using the stylized painterly image, various stylized painterly image views may be printed to provide a hologram (moving image) experience. To achieve the migration of the raster print, for example, fifteen end views 964A-0 are generated and subsequently printed, each pixel of the raster print may be taken as the first pixel of the first stylized painterly image, the first pixel of the second stylized painterly image, and so on to the first pixel of the nth stylized painterly image. Next, the second pixel of the first stylized painterly image, the second pixel of the second stylized painterly image, and so on to the second pixel of the nth stylized painterly image are printed. This provides a stripe of all N images. When a lenticular print is made, a lens shines the light of each stripe directly into the viewer's eyes. When each image is viewed with the lenticular on top, the full stylized, painterly image from that single view is observed. But when the lenticular print is moved, the lens causes each eye to see a different image because the light is pointed in different directions. When the viewer looks at it with both eyes, two different views appear, providing a 3D experience.

[0148] Fig. 12A An example of a first rendered raw image 957A is shown, which is a processed (eg, corrected) post image 965A. The first rendered raw image 957A includes various two-dimensional pixels having X and Y position coordinates on the X axis 1205 and the Y axis 1210.

[0149] Fig. 12B Shown in Fig. 12A 1 shows an example of an image 1202 with a marker 962b on the first rendered original image 957A. The marker 962b identifies the area in the image 1202 where the stylized painting is applied. The user may input the marker (e.g., by drawing on the display with a finger) via the user input device 991. The marker 962b may be recorded on the blank image and stored in the memory.

[0150] Fig. 12C The example shows that from Fig. 12A The first presents the original image 957A and Fig. 12B962b in the first rendered original image 957A. As shown, applying the stylized painting effect selection 962a from the user to the area of ​​the first rendered original image 957A defined by the user mark 962b is based on the first photo filter (e.g., stylized effect) function 971A, which converts each pixel of the first rendered original image 957A in the area defined by the mark 962b to create a stylized painting effect scene. The photo filter (e.g., stylized painting effect) function 971 transfers the style from a painting such as Edvard Munch's 1983 "The Scream" to the area as a stylized painting effect scene.

[0151] Fig.12D The example shows that from Fig. 12C A first photo filter (eg, stylized painting effect) light field effect image 1206 is generated by applying a photo filter (eg, stylized painting effect) image 1204 to the image 1204, wherein the spatial movement or rotation is tilted to the left.

[0152] Fig.12E The example shows that from Fig. 12C A first photo filter (eg, stylized painting effect) light field effect image 1208 generated by applying a photo filter (eg, stylized painting effect) image 1204 to the image 1204, wherein the spatial shift or rotation is tilted to the right.

[0153] The left and right disparity maps are calculated from the original RGB images. To obtain a stylized painting effect that rotates around the stylized painting image to have spatial motion, two modified images can be blended, or a modified and an unmodified RGB image can be blended. When two or more corresponding pixels are modified in the left and right images, the unmodified image disparity, i.e., the disparity pre-computed based on the unmodified image, is used to achieve the stylized painting effect.

[0154] Another example in FIG12 shows a photo filter (e.g., stylized painting effect) image 1300 created from a first rendered original image and a label with stylized painting 963a. The label includes two arrows (a straight arrow and a curved arrow). The photo filter image 1300 is blended with the background so that shadows 103 from, for example, people 1304 and structures 1306 are preserved.

[0155] As previously described, any of the photo filter (e.g., stylized painting) light field effect functions described herein for the eye-mounted device 100, the mobile device 990, and the server system 998 may be embodied in one or more applications. According to some embodiments, "function (one)", "function (multiple)", "application (one)", "application (multiple)", "instruction (one)", "instruction (multiple)", or "programming" is a program that executes the functions defined in the program. Various programming languages ​​may be used to create one or more applications structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or programming languages ​​(e.g., C or assembly language). In a specific example, a third-party application (e.g., an application developed by an entity other than a vendor of a specific platform using an ANDROID™ or IOS™ software development kit (SDK)) may be a mobile software running on a mobile operating system (e.g., IOS) such as IOS, ANDROID™, WINDOWS® Phone, or other mobile operating systems. In this example, the third-party application may call an API call provided by the operating system to facilitate the functions described herein.

[0156] Therefore, machine-readable media can take the form of tangible storage media in many forms. Taking non-volatile storage media as an example, it includes optical or magnetic disks, such as all storage devices in any computer, etc., which can be used to implement the client devices, media gateways, transcoders, etc. shown in the figure. Volatile storage media include dynamic memory, such as the main memory of a certain computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that make up the bus within a computer system. The form of carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punch cards, paper tapes, any other physical storage media with hole patterns, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other media from which a computer can read programming code and / or data. Among other things, computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0157] The scope of protection is defined solely by the appended claims. The scope is intended and should be interpreted to be as broad as it would be when interpreted in accordance with this specification, with the ordinary meaning of the language used in the claims consistent with the following examination history and interpreted to include all structural and functional equivalents. No claim is intended to include subject matter that fails to meet the requirements of sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in such a manner. Any unintended coverage of such subject matter is not claimed herein.

[0158] Except as described above, what is described and shown is not intended and should not be construed to cause any component, step, function, object, benefit, advantage or equivalent to be exclusive to the public regardless of whether it is recited in the claims.

[0159] It should be understood that, unless otherwise specified herein, the terms and expressions used herein have the general meanings assigned to them by their respective fields of investigation and study. Relational terms such as "first" and "second" may be used only to distinguish one entity or action from another entity or action, without requiring or implying any actual such relationship or order between such entities or actions. The terms "comprise", "include", "contain", "included" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising or including a series of elements or steps includes not only these elements or steps, but also other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. Without further limitation, an element preceded by "a" does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0160] Unless otherwise stated, any and all measurements, values, ratings, positions, magnitudes, dimensions, etc., described in this specification (including the claims that follow) are approximate values ​​and not exact values. Such quantities are intended to have a reasonable range consistent with the functions to which they are related and consistent with the practices in the field to which they belong. For example, unless otherwise expressly stated, parameter values, etc. may vary from the specified amount by ±10%.

[0161] Furthermore, in the foregoing detailed description, it can be seen that various examples group various features together in order to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than those expressly recited in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected does not lie in all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separately claimed subject matter.

[0162] Although the foregoing describes what are believed to be the best modes and other examples, it is understood that various modifications may be made in such modes and examples, that the subject matter disclosed herein may be implemented in various forms and examples, and that such modes and examples may be applied to a variety of applications, only some of which are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A stylized painting effect system for creating a stylized painting effect image, the system comprising: An eye-mounted device, comprising: a frame having temples attached to the sides of the frame; and a depth capture camera configured to capture at least one of a left original image or a right original image; An image display for presenting an image, including an original image, wherein the original image is based on a left original image, a left processed image, a right original image, a right processed image, or a combination thereof; an image display driver coupled to the image display to control the image display to present an original image; a user input device for receiving a markup of an original image, a stylized painting effect selection, and a style selection from a user; a memory and a processor coupled to the depth capture camera, the image display driver, the user input device, and the memory; and Programming in the memory, wherein execution of the programming by the processor configures the stylized painting effect system to perform functions, including the following functions: capturing at least one of a left original image or a right original image by a depth capture camera; displaying the original image through an image display; receiving, via a user input device, a markup, a stylized painting effect selection, and a style selection from a user; Creating at least one stylized painting effect image using the stylized painting effect scene; Applying the stylized painting effect image to the following marks: (i) the left original image or the left processed image to create the left stylized painting effect image, (ii) the right original image or the right processed image to create the right stylized painting effect image, or (iii) a combination of the two; By mixing the stylized painting effect image on the left and the stylized painting effect image on the right, a stylized painting effect image having a spatial movement or rotation appearance is generated, and a stylized painting effect scene surrounding at least one stylized painting effect image; and Through image display, a stylized painting effect image is presented.

2. The system of claim 1, wherein the functionality for receiving a style selection consists of: receiving an image selection from a user; and in, The function of creating at least one stylized painting effect image using a stylized painting effect scene is composed of the following: Applying image selection to labeling using neural style transfer.

3. The system of claim 1, wherein the function of creating at least one stylized painting effect image is constituted by: become: Blends the stylized painting effect image with the original image in the marked area.

4. The system of claim 1, wherein the user input device comprises: An inertial measurement unit configured to move or rotate in space about at least one stylized painterly image Painting effect scene.

5. The system according to claim 1, further comprising the following functions: Calculate: (i) a left image disparity map between a left pixel matrix and a right pixel matrix, and (ii) a right image disparity map between a right pixel matrix and a left pixel matrix, for blending a left stylized painting effect image with a right stylized painting effect image.

6. The stylized painting effect system according to claim 5, wherein the function of generating a stylized painting effect image includes include: Determine the horizontal position shift parameters along the X-axis of the left pixel matrix and the right pixel matrix; Filling the left interpolated pixel matrix by shifting pixels in the left pixel matrix along the X-axis based on the horizontal shift parameter; filling the right interpolation pixel matrix by shifting pixels in the right pixel matrix along the X-axis based on the horizontal shift parameter; and Creates a stylized painting effect image by blending the interpolated pixel matrix on the left with the interpolated pixel matrix on the right.

7. The stylized painting effect system according to claim 6, wherein: By mixing the left interpolated pixel matrix and the right interpolated pixel matrix, the function of generating the stylized painting effect image is based on the disparity confidence level, the gradient or a combination of the two in the left image disparity map and the right image disparity map.

8. The stylized painting effect system according to claim 6, wherein: Determine the horizontal position movement parameters include: receiving, via a user input device, a two-dimensional input selection from a user presenting an original image; and Via the user input device, movement of the two-dimensional input selection from an initial contact point to a final contact point where an original image is presented is tracked.

9. The stylized painting effect system according to claim 8, wherein: User input devices include: A touch sensor, the sensor comprising an input interface and a sensor array, is used to couple to the input interface so as to be able to receive at least a single finger contact from a user input; and a sensing circuit integrated in or connected to the touch sensor and connected to the processor, the sensing circuit being configured to measure a voltage capable of tracking at least a single finger contact on the input interface; The functionality of receiving stylized painting effect selections and markings from a user via a user input device includes being capable of receiving at least a single finger contact from a user input on a touch sensor input interface; and The function of tracking the movement of a two-dimensional input selection from an initial contact point to a final contact point through a user input device includes at least being able to track a single finger contact on an input interface, dragging from an initial contact point to a final contact point on a touch sensor input interface via a sensing circuit.

10. A method comprising the following steps: capturing at least one of a left original image or a right original image via a depth capture camera; Presenting an original image via an image display, wherein the original image is based on a left original image, a left processed image, a right original image, a right processed image, or a combination thereof; Receiving, via a user input device, a mark, a stylized painting effect selection, and a style selection from a user; Creating at least one stylized painting effect image using the stylized painting effect scene; Applying the stylized painting effect image to at least the following marks: (i) the left original image or the left processed image to create the left stylized painting effect image, (ii) the right original image or the right processed image to create the right stylized painting effect image, or (iii) a combination of the two; By mixing a stylized painting effect image on the left and a stylized painting effect image on the right, a stylized painting effect image with a spatial movement or rotation appearance is generated, and a stylized painting effect scene surrounding at least one stylized painting effect image is generated; and the stylized painting effect image is presented through image display.

11. The method according to claim 10, wherein the receiving comprises: receiving an image selection from a user; as well as Among them, creation includes: Applying image selection to labeling using neural style transfer.

12. The method according to claim 10, further comprising: An inertial motion of an image display is determined, configured to spatially translate or rotate, a stylized painterly scene around at least one stylized painterly image.

13. The method of claim 10, wherein creating at least one stylized painterly image comprises: Blends the stylized painted image with the original image in the marked areas.

14. The method according to claim 10, further comprising: Calculate: (i) a left image disparity map between a left pixel matrix and a right pixel matrix, and (ii) a right image disparity map between a right pixel matrix and a left pixel matrix, for mixing a left stylized painting effect image with a right stylized painting effect image, wherein the left pixel matrix is ​​based on a left original image or a left processed image, and the right pixel matrix is ​​based on a right original image or a right processed image.

15. The method according to claim 14, wherein the step of generating a photo filter stylized painting effect image comprises: Determine the horizontal position shift parameters along the X-axis of the left pixel matrix and the right pixel matrix; Filling the left interpolated pixel matrix by shifting pixels in the left pixel matrix along the X-axis based on the horizontal shift parameter; Filling the right interpolation pixel matrix by shifting pixels in the right pixel matrix along the X-axis based on the horizontal shift parameter; as well as Creates a photo filter stylized painting effect image by blending the left interpolated pixel matrix with the right interpolated pixel matrix.

16. The method according to claim 15, wherein: The step of generating a photo filter stylized painting effect image by blending the left interpolation pixel matrix and the right interpolation pixel matrix is ​​based on the disparity confidence level, gradient or a combination thereof in the left image disparity map and the right image disparity map.

17. The method of claim 15, wherein determining the horizontal position shift parameter comprises: receiving, via a user input device, a two-dimensional input selection from a user presenting an original image; as well as Via the user input device, movement of the two-dimensional input selection from an initial contact point to a final contact point where an original image is presented is tracked.

18. The method of claim 17, wherein: Each pixel is associated with a respective vertex in the vertex matrix; Each vertex has a position attribute; The position attribute of each vertex is based on a three-dimensional position coordinate system, including the X position coordinate on the X axis (indicating the horizontal position), the Y position coordinate on the Y axis (indicating the vertical position), and the Z position coordinate on the Z axis (indicating the depth position); as well as Applying a stylized painting effect selection from a user to: (i) the original image on the left or the processed image on the left to create a photo filter image on the left, (ii) the original image on the right or the processed image on the right to create a photo filter image on the right, or (iii) a combination of the two, changes the filter effect intensity of the photo filter function based on the Z position coordinate to transform each pixel according to the depth position of each vertex associated with each pixel.

19. A non-transitory computer readable medium storing program code which, when executed, causes an electronic processor to perform the following steps: capturing at least one of a left original image or a right original image via a depth capture camera; Presenting an original image via an image display, wherein the original image is based on a left original image, a left processed image, a right original image, a right processed image, or a combination thereof; Receiving, via a user input device, a mark, a stylized painting effect selection, and a style selection from a user; Creating at least one stylized painting effect image using the stylized painting effect scene; Applying the stylized painting effect image to at least the following marks: (i) the left original image or the left processed image to create the left stylized painting effect image, (ii) the right original image or the right processed image to create the right stylized painting effect image, or (iii) a combination of the two; By mixing the stylized painting effect image on the left and the stylized painting effect image on the right, a stylized painting effect image having a spatial movement or rotation appearance is generated, and a stylized painting effect scene surrounding at least one stylized painting effect image; and Through image display, a stylized painting effect image is presented.

20. The non-transitory computer readable medium of claim 19, wherein the receiving comprises: receiving an image selection from a user; as well as Among them, creation includes: Applying image selection to labeling using neural style transfer.

Images