Video processing with AR effect preview

Abstract

Image enhancement effects are provided on an apparatus including a display and a camera device. The simplified augmented reality effect is applied to an image stream captured by a camera device to generate a preview image stream. A preview image stream is displayed on a display. And storing a second image stream corresponding to the first image stream to the initial video file. A complete augmented reality effect corresponding to the simplified augmented reality effect is then applied to the second image stream to generate a fully augmented image stream, which is saved to another video file. Additional video files may then be played back on the display to show a final complete augmented reality effect applied to the image stream.

Description

BACKGROUND OF THE DISCLOSURE

[0001] Social networking and messaging applications provide tools for sharing user content such as photos or videos. In some instances, photos or videos can be supplemented with augmented reality or other effects that are generated live on a camera device feed and displayed on a display of a mobile device for preview. A user is able to select and manipulate effects to be applied to the live camera device feed and, when satisfied with the result, capture an image including the effect or record a video including the effect. The captured video or photo can then be shared on a social networking platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] In the drawings (which are not necessarily drawn to scale), like reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or act, one or more of the most significant digits in the reference numeral refers to the figure number in which the element is first introduced. Some non-limiting examples are shown in the drawings, in which:

[0003] Figure 1 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, in accordance with some examples.

[0004] Figure 2 is a diagrammatic representation of a messaging system having both client-side functionality and server-side functionality, in accordance with some examples.

[0005] Figure 3 illustrates a recording and display processing flow and a playback processing flow of an AR-enhanced video in a single camera device stream implementation, in accordance with some examples.

[0006] Figure 4 illustrates a recording and display processing flow and a playback processing flow of an AR-enhanced video in a single camera device stream implementation, in accordance with some examples.

[0007] Figure 5 illustrates a recording and display processing flow of an AR-enhanced video in a dual camera device stream implementation, in accordance with some examples.

[0008] Figure 6 illustrates a playback / re-recording processing flow of an AR-enhanced video in a dual camera device stream implementation, in accordance with some examples.

[0009] Figure 7 illustrates a playback / re-recording processing flow of an AR-enhanced video in a dual camera device stream implementation, in accordance with some examples.

[0010] Figure 8 illustrates a playback / re-recording processing flow of an AR-enhanced video in a dual camera device stream implementation, in accordance with some examples.

[0011] Figure 9 Shows the architecture of an AR enhanced video in a dual camera device stream implementation according to some examples.

[0012] Figure 10 Is a block diagram showing a software architecture in which examples may be implemented.

[0013] Figure 11 Is a graphical representation of a machine in the form of a computer system within which a set of instructions may be executed to cause the machine to perform any one or more of the methods discussed herein. Detailed Description

[0014] Discloses systems and methods for providing improved video capture, display, or forwarding in an augmented reality (AR) device.

[0015] As mentioned herein, the term "augmented reality experience" includes or relates to various image processing operations corresponding to image modification, filtering, media overlay, transformation, etc. In some examples, these image processing operations provide an interactive experience of the real-world environment, where objects, surfaces, backgrounds, lighting, etc. in the real world are enhanced by computer-generated perceptual information. The augmented reality experience may also include associated audio, such as a sound track or effect sounds. In this context, an "AR effect" includes a collection of data, parameters, and other assets required to apply a selected augmented reality experience to an image or video feed. In some examples, augmented reality effects are provided by Snap, Inc. under the registered trademark LENSES.

[0016] AR effects are applied to the video stream captured by the camera device in the AR device during use to provide an enhanced user experience. However, the video stream can also be used for many different purposes, including object detection and tracking, AR device position and orientation detection using image processing techniques such as simultaneous localization and mapping, and QR code detection. AR effects can be rendered onto the video stream for display to the user, for recording, and for forwarding to other users.

[0017] The requirements imposed by AR effects on AR devices and the video processing pipeline in AR devices can lead to video stream stuttering, which provides an undesirable user experience. This can negatively impact both the local rendering of the AR-enhanced video stream to the display of the AR device and the resulting AR-enhanced video recorded from the video stream for later viewing or forwarding to other users. In particular, the need to apply AR effects to the camera device stream, render the AR-enhanced stream to the display (or "viewfinder") of the AR device for real-time viewing by the user, and render the AR-enhanced stream for recording can cause stuttering in the camera device stream. This is especially true for AR effects based on or leveraging machine learning (ML) models, which impose higher requirements on the processing capabilities of the AR device compared to conventional AR effects.

[0018] To address this issue, in some examples, two camera device streams are provided. A first stream is provided to the device display (or "viewfinder") for real-time viewing by the user, while a second stream is provided directly to the video encoder for recording. The first stream applies an approximation of the AR effect, which will provide insight into the final AR effect and has lower requirements such that the AR device can reasonably render it in real time. By directly recording the second stream without similarly rendering it for display and without applying the AR effect, the second stream is less likely to include any stuttering. Thus, any stuttering that may occur on the independent first stream will not be reflected in the recorded video file. Subsequently, the full AR effect can be applied to the unenhanced video file without simultaneously rendering it for display. Then the enhanced video file can be played back for the user to view.

[0019] Alternatively, in some examples, a single camera device stream is provided. An approximation of the AR effect is applied only for display purposes. As before, this approximation provides a preview of the final AR effect, which the AR device can reasonably render in real time. However, this enhanced preview version of the single stream is not recorded. The unenhanced stream is recorded. Subsequently, the full AR effect can be applied to the unenhanced video file without simultaneously rendering it for display. Then the enhanced video file can be played back for the user to view.

[0020] In some examples, a method is provided for providing an image enhancement effect on a device including a display and at least one camera device, the method being executed by one or more processors and including: receiving a first image stream captured by the at least one camera device, applying a simplified augmented reality effect to the image stream captured by the at least one camera device to generate a preview image stream, displaying the preview image stream on the display, and saving a second image stream corresponding to the first image stream captured by the at least one camera device to an initial video file.

[0021] The method may further include: retrieving a second image stream from an initial video file, applying a complete augmented reality effect corresponding to a simplified augmented reality effect to the second image stream to generate a fully augmented image stream, and saving the fully augmented image stream to a separate video file. The complete augmented reality effect may be based on a machine learning model.

[0022] The second image stream may be an image stream parallel to the first image stream. The second image stream may have a higher resolution than the first image stream. The second image stream may also be a video-coded version of the first image stream.

[0023] In some examples, retrieving the second image stream from the initial video file may automatically start when the saving of the initial video file is completed, and the method may further include: automatically playing back the separate video file on a display once the separate video file has been saved.

[0024] In some examples, a non-transitory computer-readable storage medium is provided. The computer-readable storage medium includes instructions that, when executed by a computer, cause the computer to perform operations for providing an image enhancement effect on a device including a display and at least one imaging device according to any one of the methods and limitations set forth above. The operations include, but are not limited to: receiving a first image stream captured by the at least one imaging device, applying a simplified augmented reality effect to the image stream captured by the at least one imaging device to generate a preview image stream, displaying the preview image stream on the display, and saving a second image stream corresponding to the first image stream captured by the at least one imaging device to the initial video file.

[0025] In some examples, a computing device is provided, including: at least one imaging device, a display, one or more processors, and a memory storing instructions that, when executed by the one or more processors, configure the device to perform operations for providing an image enhancement effect according to any one of the methods and limitations set forth above. The operations include, but are not limited to: receiving a first image stream captured by the at least one imaging device, applying a simplified augmented reality effect to the image stream captured by the at least one imaging device to generate a preview image stream, displaying the preview image stream on the display, and saving a second image stream corresponding to the first image stream captured by the at least one imaging device to the initial video file.

[0026] Other technical features will be readily apparent to those skilled in the art from the accompanying drawings, description, and claims.

[0027] Figure 1FIG. 0 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages, media, and associated content) over a network. Messaging system 100 includes multiple instances of user devices 102, each of the multiple instances hosting several applications including a messaging client 104 and other applications 106. Each messaging client 104 is communicatively coupled via a network 112 (e.g., the Internet) to other instances of messaging client 104 (e.g., hosted on corresponding other client devices 102), a messaging server system 108, and a third-party server 110. Messaging client 104 may also communicate with locally hosted applications 106 using an application programming interface (API).

[0028] Messaging client 104 is capable of communicating with and exchanging data with other messaging clients 104 and with messaging server system 108 via network 112. The data exchanged between messaging clients 104 and between messaging client 104 and messaging server system 108 includes functionality (e.g., commands for activating functionality) and payload data (e.g., text, audio, video, or other multimedia data).

[0029] Messaging server system 108 provides server-side functionality to a particular messaging client 104 via network 112. While certain functions of messaging system 100 are described herein as being performed by messaging client 104 or by messaging server system 108, whether certain functions are located within messaging client 104 or within messaging server system 108 can be a design choice. For example, it may be technically preferred to initially deploy certain technologies and functionality within messaging server system 108 but later migrate the technology and functionality to messaging client 104 where the user device 102 has sufficient processing power.

[0030] Messaging server system 108 supports various services and operations provided to messaging client 104. Such operations include transmitting data to messaging client 104, receiving data from messaging client 104, and processing data generated by messaging client 104. By way of example, the data may include message content, client device information, geographical location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. Data exchange within messaging system 100 is activated and controlled via functions available through a user interface (UI) of messaging client 104.

[0031] Now turning specifically to the messaging server system 108, an application programming interface (API) server 116 is coupled to the application server 114 and provides a programming interface to the application server 114. The application server 114 is communicatively coupled to a database server 120, which facilitates access to a database 126 that stores data associated with messages processed by the application server 114. Similarly, a web server 128 is coupled to the application server 114 and provides a web-based interface to the application server 114. To this end, the web server 128 processes incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0032] The application programming interface (API) server 116 receives and transmits message data (e.g., commands and message payloads) between the user device 102 and the application server 114. Specifically, the application programming interface (API) server 116 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by the messaging client 104 to activate the functions of the application server 114. The application programming interface (API) server 116 exposes various functions supported by the application server 114, including: account registration; login functionality; sending messages from a particular messaging client 104 to another messaging client 104 via the application server 114; sending media files (e.g., images or videos) from the messaging client 104 to the messaging server 118 for possible access by another messaging client 104; setting a collection of media data (e.g., a story); retrieving a list of friends of the user of the user device 102; retrieving such collections; retrieving messages and content; adding and deleting entities (e.g., friends) in an entity graph (e.g., a social graph); locating friends within the social graph; and opening application events (e.g., related to the messaging client 104).

[0033] The application server 114 hosts several server applications and subsystems, including, for example, a messaging server 118, an image processing server 122, and a social network server 124. The messaging server 118 implements several message processing techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 104. Text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 104. Given the hardware requirements for other processor- and memory-intensive data processing, such processing can also be performed on the server side by the messaging server 118.

[0034] The application server 114 also includes an image processing server 122 that is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at the messaging server 118.

[0035] The social network server 124 supports and makes available to the messaging server 118 a variety of social networking functions and services. To this end, the social network server 124 maintains and accesses an entity graph within the database 126. Examples of functions and services supported by the social network server 124 include identifying other users of the messaging system 100 who are related to or being "followed" by a particular user, as well as identifying the interests and other entities of a particular user.

[0036] Figure 2 is a block diagram showing additional details regarding the messaging system 100 according to some examples. Specifically, the messaging system 100 is shown as including a messaging client 104 and an application server 114. The messaging system 100 includes a number of subsystems that are supported on the client side by the messaging client 104 and on the server side by the application server 114. These subsystems include, for example, a user interface 202, a collection management system 204, an enhancement system 208, a map system 210, and a game system 212.

[0037] The user interface 202 is responsible for providing output to and receiving input from a user of the messaging client 104 on the user device 102. As is known in the art, the user interface provides a user-manipulable display output on a display of the user device 102 (see also the user output component 1126 in Figure 11 and as described below). In one example, the user interface includes a chat interface whereby a user can send messages and associated content and receive messages and associated content from one or more remote users. The user interface 202 also allows a user to manipulate live or captured media, such as by providing an augmented reality effect on a captured photo or video, or on a live video feed from a camera device of the user device, to manipulate live or captured media.

[0038] The collection management system 204 is responsible for managing collections or sets of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, video, text, and audio) can be organized into "event galleries" or "event stories". Such collections can be made available for a specified period of time (such as the duration of an event related to the content). For example, content related to a concert can be made available as a "story" for the duration of that concert. The collection management system 204 can also be responsible for publishing an icon providing a notification of the existence of a particular collection to the user interface of the messaging client 104.

[0039] The collection management system 204 also includes a curation interface 206 that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface 206 enables an event organizer to curate a collection of content related to a particular event (e.g., delete inappropriate content or redundant messages). Additionally, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate the content collections. In some examples, a user may be compensated for including user-generated content in a collection. In such cases, the collection management system 204 operates to automatically pay such users for the use of their content.

[0040] The enhancement system 208 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content associated with a message. For example, the enhancement system 208 provides functions related to generating and publishing a media overlay for a message processed by the messaging system 100. The enhancement system 208 operably provides a media overlay or enhancement (e.g., an image filter) to the messaging client 104 based on the geographical location of the user device 102. In another example, the enhancement system 208 operably provides a media overlay to the messaging client 104 based on other information such as the social network information of the user of the user device 102. The media overlay can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects can be applied to a media content item (e.g., a photo) at the user device 102. For example, the media overlay can include text or an image that can be superimposed over a photo taken by the user device 102. In another example, the media overlay includes a location identification overlay (e.g., Venice Beach), the name of a live event, or a merchant name overlay (e.g., Beach Café). In another example, the enhancement system 208 uses the geographical location of the user device 102 to identify a media overlay that includes the name of a merchant at the geographical location of the user device 102. The media overlay can include other markers associated with the merchant. The media overlay can be stored in the database 126 and accessed via the database server 120.

[0041] The map system 210 provides various geographical location functions and supports the presentation of map-based media content and messages by the messaging client 104. For example, the map system 210 enables a user icon or avatar to be displayed on a map to indicate the current or past location of a user's "friends", and media content (e.g., a collection of messages including photos and videos) generated by such friends to be displayed in the context of the map. For example, a message posted by a user from a specific geographical location to the messaging system 100 can be displayed to the "friends" of the specific user at that specific location in the context of the map on the map interface of the messaging client 104. A user can also share his or her location and status information with other users of the messaging system 100 (e.g., using an appropriate status avatar) via the messaging client 104, where the location and status information is similarly displayed to selected users in the context of the map interface of the messaging client 104.

[0042] The game system 212 provides various game functions in the context of the messaging client 104. The messaging client 104 provides a game interface that presents a list of available games that can be launched by a user within the context of the messaging client 104 and played with other users of the messaging system 100. The messaging system 100 also enables a particular user to invite such other users to participate in playing a particular game by sending an invitation from the messaging client 104 to the other users. The messaging client 104 also supports both voice messaging and text messaging (e.g., chatting) in the context of playing a game, provides a leaderboard for the game, and also supports providing in-game rewards (e.g., game currency and items).

[0043] Figure 3 Shown is a recording and display processing flow 302 and a playback processing flow 304 of an AR-enhanced video in a single camera device stream implementation according to some examples, where the user device 102 is powerful enough or the AR effects are not overly burdensome such that the AR effects can be satisfactorily rendered in real time.

[0044] In the recording and display processing flow 302, the camera device server 308 receives a video stream of the user 306 generated by a camera device on the user device 102. The camera device server 308 passes the video stream to the camera device frame scheduler 310, which provides video frames to the enhancement system 208, which in turn applies augmented reality effects to the video frames and thus to the video stream to generate an enhanced video stream.

[0045] Then, in a rendering operation 314, the enhanced video stream is rendered for display on a viewfinder or display 316. The enhanced video stream is then passed to a presentation operation 318, which records the video stream to an AR-enhanced video file 320.

[0046] For convenience, in the drawings, in Figure 3 the AR-enhanced user 312 depicted in the enhanced video stream or in the AR-enhanced video file 320 is shown as including a dog's tongue, ears, and nose to distinguish it from the unenhanced video of the user 306.

[0047] In the playback processing flow 304, the AR-enhanced video file 320 is retrieved and decoded by a video decoder 322 and played back by a video player 324, which renders the enhanced video stream in a rendering operation 326 for display on the display 316 as before.

[0048] Figure 4Illustrated is the recording and display processing flow 402 of an AR enhanced video in a single camera device stream implementation according to some examples, where the user device 102 is not powerful enough, or the AR effects are too heavy or may be too heavy for the user device 102 to satisfactorily render the AR effects in real time.

[0049] In the recording and display processing flow 402, the camera device server 308 receives the video stream of the user 306 generated by the camera device on the user device. The camera device server 308 passes the video stream to the camera device frame scheduler 310, and the camera device frame scheduler 310 provides video frames to the enhancement system 208. The enhancement system 208 applies a simplified version of the full AR effect to generate a preview enhanced video stream including a depiction of the preview AR enhanced user 410.

[0050] Then in the rendering operation 404, the preview enhanced video stream is rendered for display on the viewfinder or the display 316. However, the preview enhanced video stream is not recorded. Instead, the unenhanced video stream is passed to the presentation operation 406, and the presentation operation 406 encodes and records the original video stream as a second video stream in the unenhanced video file 408. In this serial example, the "first video stream" provided to the enhancement system 208 to apply the simplified version of the AR effect is the same as the "second video stream" recorded, except for being encoded using a video codec.

[0051] For convenience, in the drawings, the preview AR enhanced user 410 depicted in the preview enhanced video stream is shown as including eyelash extensions to distinguish it from the unenhanced video of the user 306. In this example, the full AR enhancement includes a makeover, including makeup, eyelash extensions, and a new hairstyle.

[0052] Figure 5 Illustrated is the recording and display processing flow 502 of an AR enhanced video in a dual camera device stream implementation according to some examples.

[0053] In the recording and display processing flow 502, the camera device server 308 receives the video stream of the user 306 captured by the camera device on the user device 102. The camera device server 308 passes the first video stream 504 to the camera device frame scheduler 310, and the camera device frame scheduler 310 provides video frames to the enhancement system 208. The enhancement system 208 applies a simplified version of the full AR effect to generate a preview enhanced video stream including a depiction of the preview AR enhanced user 410.

[0054] Then in the rendering operation 510, the preview enhanced video stream including a depiction of the preview AR enhanced user 410 is rendered for display on the viewfinder or the display 316.

[0055] However, compared withFigure 3 Different from the display processing flow 302 shown in [description], in the recording and display processing flow 502, the camera device server 308 also provides a separate second video stream 506 for the user 306 in parallel with the first video stream 504. The second video stream 506 does not have any AR effects applied to it and is provided to the video codec 508, which encodes and then saves the initial or unenhanced video file 408 to a local or remote storage device.

[0056] Associated with the unenhanced video file 408 are descriptors that identify the AR effects and any associated parameters that define the complete AR effects. These descriptors and any associated parameters are associated with the unenhanced video file 408 for later use, for example, by saving the descriptors and any associated parameters together with the unenhanced video file 408 as metadata, or by saving the descriptors and any associated parameters in a separate file that has a link or identifier between the separate file and the unenhanced video file 408 to associate them with the unenhanced video file 408.

[0057] Figure 6 Shows the re-recording processing flow 602 for AR-enhanced video according to some examples. This processing flow automatically starts after the recording and display processing flow 302 or the recording and display processing flow 402 is completed, although in some examples, the re-recording processing flow 602 is initiated in response to receiving a user input requesting playback of the (fully) enhanced version of the unenhanced video file 408.

[0058] If the re-recording processing flow 602 starts automatically, an animation "Processing" or "Busy" icon is displayed on the display 316.

[0059] In the re-recording processing flow 602, the video decoder 322 retrieves and decodes the unenhanced video file 408 to generate an unenhanced video stream. The unenhanced video stream is provided to the enhancement system 208, which retrieves and applies the complete AR effects and any associated parameters as originally expected. In some examples, a machine learning (ML) model is used to generate the complete AR effects. The resulting fully enhanced video stream including the representation of the AR-enhanced user 606 is then rendered in the rendering operation 608 and saved to the fully enhanced video file 604 in the operation 610.

[0060] For convenience, in the drawings, the AR-enhanced user 606 depicted in the enhanced video stream or the fully enhanced video file 604 is shown as including an AR hairstyle in addition to the extended eyebrows to distinguish it from the unenhanced user represented in the unenhanced video file 408 and the preview AR-enhanced user 410.

[0061] By providing separate recording and display processing flows 402, 502, and a re-recording processing flow 602 as separate serial processes, the frame rate and amount of stuttering can be improved in terms of the live display of the preview-enhanced video stream and any subsequent display of the enhanced video stream recording.

[0062] The first video stream 504 and the second video stream 506 can be the same. However, in some examples, the first video stream 504 can be a lower-resolution computer vision-level video stream suitable for performing AR-related analysis (such as object detection, tracking, QR code recognition, etc.), while the second video stream 506 is a full-resolution video stream more desirable for messaging and other social media applications. Capturing the first video stream 504 at a lower resolution reduces the overall computational resource usage associated with capturing and processing the first video stream 504.

[0063] However, the first video stream and the second video stream correspond because they reflect the same video object captured simultaneously by the same imaging device in most but not all cases.

[0064] Figure 7 A playback processing flow 702 for an AR-enhanced video according to some examples is shown. This processing flow automatically starts after the re-recording processing flow 602 in Figure 6 is completed, although it can also be initiated in response to receiving user input to request playback of the fully enhanced video file 604.

[0065] In the playback processing flow 702, the fully enhanced video file 604 is retrieved and decoded by the video decoder 322 and played back by the video player 324. In the rendering operation 326, the video player 324 renders the fully enhanced video stream including the AR-enhanced user 606 for display on the display 316 as before.

[0066] Figure 8 A user interface flow 802 reflecting the methods described herein according to some examples is shown. The user interface flow 802 is shown as a sequential first display screen 804, second display screen 806, and third display screen 808 shown on the display 316 of the user device 102. The user has selected an AR effect that adds a wreath to an image of the user's head displayed on the display. This AR effect has been selected from a carousel 810 of available AR effects.

[0067] In the first display screen 804, a representation 812 of the user is shown in a live video stream, enhanced with a simple wreath 814, which is a preview of the final AR effect applied to the live video stream. Applying preview AR effects such as the simple wreath 814 does not unduly burden the user device 102 and provides a smooth frame rate as the user moves relative to the imaging device and the simple wreath 814 is updated accordingly. The user may provide user input to record video from the video stream. The first display screen 804 continues until the user terminates the recording, resulting in an unenhanced video file being saved, as Figure 5 or Figure 6 discussed.

[0068] The user device 102 immediately begins processing the unenhanced video file (as Figure 6 shown), while displaying an in - processing animation icon 818 on the representation 816 of the user. The second display screen 806 may include a still image of the user to reduce the processing requirements on the user device 102. The first display screen 804 is displayed until Figure 6 the processing is complete, resulting in a fully enhanced video file 604 being saved.

[0069] Then, the user device 102 displays a representation 820 of the user rendered from the fully enhanced video file 604, which includes a complete wreath 822. The user can then manipulate the video (pause, play, scroll forward and backward) like any other video, and save, forward, publish or delete the video as known.

[0070] Figure 9 An architecture 900 for an AR - enhanced video in a dual - imaging - device stream implementation according to some examples is shown. The architecture 900 is incorporated in the user device 102, where the imaging device server 902 is a service provided by the operating system, and the application 904 runs on the device.

[0071] It can be seen that the imaging device server 902 provides frames 906 to the application 904 in a first video stream 504 of imaging device frames. The application 904 generates a texture 908 from the frame 906, for example, by applying an augmented reality effect to the frame 906. The application 904 then renders the texture 908 to the frame 912 and displays it to the user on the display 910.

[0072] The camera device server 902 also provides the frame 916 to the application 904 in the second video stream 506 of the camera device frame. The frame 916 may be the same as the frame 906, but may also be different, for example, having a higher resolution. The frame 916 is received by the recorder 914, which includes an encoder 918, a multiplexer 920, and a file system 922. The encoder 918 encodes the stream of the frame 916 into an appropriate video format and passes it to the multiplexer 920, where it is combined with any associated audio received, for example, from a microphone associated with the user device 102. Then, the resulting encoded video stream is passed to the file system 922, where it is saved as a file for future viewing, enhancement, forwarding, etc.

[0073] Figure 10 FIG. 1000 is a block diagram showing a software architecture 1004 that may be installed on any one or more of the devices described herein. The software architecture 1004 is supported by hardware such as a machine 1002 that includes a processor 1020, a memory 1026, and I / O components 1038. In this example, the software architecture 1004 may be conceptually viewed as a stack of layers, where each layer provides a specific function. The software architecture 1004 includes the following layers, such as an operating system 1012, libraries 1010, frameworks 1008, and applications 1006. In operation, the application 1006 activates API calls 1050 through the software stack and receives messages 1052 in response to the API calls 1050.

[0074] The operating system 1012 manages the hardware resources and provides common services. The operating system 1012 includes, for example: a kernel 1014, services 1016, and drivers 1022. The kernel 1014 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1014 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. The services 1016 may provide other common services for other software layers. The drivers 1022 are responsible for controlling or interfacing with the underlying hardware. For example, the drivers 1022 may include a display driver, a camera device driver, or a low-power driver, a flash driver, a serial communication driver (e.g., a USB driver), drivers, an audio driver, a power management driver, etc.

[0075] Library 1010 provides common low-level infrastructure used by application 1006. Library 1010 may include system library 1018 (e.g., C standard library), which provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, library 1010 may include API library 1024, such as media libraries (e.g., libraries for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), High Efficiency Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., OpenGL framework for 2D and 3D presentation in graphical content on a display), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functions), etc. Library 1010 may also include various other libraries 1028 to provide many other APIs to application 1006.

[0076] Framework 1008 provides common high-level infrastructure used by application 1006. For example, framework 1008 provides various Graphical User Interface (GUI) functions, high-level resource management, and high-level location services. Framework 1008 may provide a wide range of other APIs that can be used by application 1006, some of which may be specific to a particular operating system or platform.

[0077] In an example, application 1006 may include a home application 1036, a contacts application 1030, a browser application 1032, a book reader application 1034, a location application 1042, a media application 1044, a messaging application 1046 (e.g., messaging client 104), a game application 1048, and various other applications such as third-party application 1040. Application 1006 is a program that executes functions defined in the program. One or more of the applications 1006 can be created using various programming languages, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C language or assembly language). In a specific example, the third-party application 1040 (e.g., an application developed using the ANDROID TM or IOS TM Software Development Kit (SDK)) can be an application developed on platforms such as IOS TM 、ANDROID TM 、 Mobile software running on the mobile operating system of a Phone or another mobile operating system. In this example, third-party application 1040 can activate API call 1050 provided by operating system 1012 to facilitate the functions described herein.

[0078] Figure 11 is an illustrative representation of a machine 1100 (e.g., user device 102) within which instructions 1110 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 1100 to perform any one or more of the methods discussed herein. For example, instructions 1110 can cause the machine 1100 to perform any one or more of the methods described herein. The instructions 1110 transform a general purpose, unprogrammed machine 1100 into a particular machine 1100 programmed to perform the functions described and shown in the manner described. The machine 1100 can operate as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1100 can operate in a server-client network environment as a server machine or a client machine, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1100 can include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular telephones, smart phones, mobile devices, wearable devices (e.g., smart watches), smart home devices (e.g., smart appliances), other smart devices, web appliances, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing the instructions 1110 specifying the actions to be taken by the machine 1100. Further, while only a single machine 1100 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 1110 to perform any one or more of the methods discussed herein. For example, the machine 1100 can include the user device 102 or any one of a number of server devices forming part of the messaging server system 108. In some examples, the machine 1100 can also include both a client system and a server system, where certain operations of a particular method or algorithm are executed on the server side and certain operations of the particular method or algorithm are executed on the client side.

[0079] Machine 1100 may include a processor 1104, a memory 1106, and input / output (I / O) components 1102, which may be configured to communicate with each other via a bus 1140. In an example, the processor 1104 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1108 and a processor 1112 that execute instructions 1110. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 11 multiple processors 1104 are shown, machine 1100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0080] The memory 1106 includes a main memory 1114, a static memory 1116, and a storage unit 1118, all of which may be accessed by the processor 1104 via the bus 1140. The main memory 1106, the static memory 1116, and the storage unit 1118 store instructions 1110 that implement any one or more of the methods or functions described herein. The instructions 1110 may also reside, completely or partially, within the main memory 1114, within the static memory 1116, within the machine-readable medium 1120 within the storage unit 1118, within at least one of the processors 1104 (e.g., within the cache memory of the processor), or any suitable combination thereof, during execution by the machine 1100.

[0081] The I / O components 1102 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, and so on. The specific I / O components 1102 included in a particular machine will depend on the type of the machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I / O components 1102 may include Figure 11Many other components not shown. In various examples, the I / O component 1102 may include a user output component 1126 and a user input component 1128. The user output component 1126 may include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The user input component 1128 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or another pointing instrument), tactile input components (e.g., physical buttons, a touch screen that provides the location and force of a touch or touch gesture, or other tactile input components), audio input components (e.g., a microphone), etc.

[0082] In additional examples, the I / O component 1102 may include a biometric component 1130, a motion component 1132, an environmental component 1134, or a positioning component 1136 and various other components. For example, the biometric component 1130 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying a person (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 1132 includes acceleration sensor components (e.g., accelerometers), gravity sensor components, rotational sensor components (e.g., gyroscopes).

[0083] The environmental component 1134 includes, for example, one or more camera devices (with still image / photo and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect the ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors that detect the concentration of hazardous gases for safety or measure pollutants in the atmosphere), or other components that can provide an indication, measurement, or signal corresponding to the surrounding physical environment.

[0084] Regarding the imaging device, the user equipment 102 may have an imaging device system, which includes, for example, a front imaging device on the front surface of the user equipment 102 and a rear imaging device on the rear surface of the user equipment 102. The front imaging device may be used, for example, to capture still images and videos of the user of the user equipment 102 (e.g., "selfies"), and these still images and videos can then be enhanced using the above-mentioned enhancement data (e.g., filters). The rear imaging device may be used, for example, to capture still images and videos in a more traditional imaging device mode, and these images are similarly enhanced using the enhancement data. In addition to the front imaging device and the rear imaging device, the user equipment 102 may also include a 360° imaging device for capturing 360° photos and videos.

[0085] In addition, the imaging device system of the user equipment 102 may include dual rear imaging devices (e.g., a main imaging device and a depth sensing imaging device) on the front and rear sides of the user equipment 102, or even a triple rear imaging device, a quadruple rear imaging device, or a quintuple rear imaging device configuration. For example, these multiple imaging device systems may include a wide-angle imaging device, an ultra-wide-angle imaging device, a telephoto imaging device, a macro imaging device, and a depth sensor.

[0086] The positioning component 1136 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, and the altitude can be obtained based on the air pressure), an orientation sensor component (e.g., a magnetometer), etc.

[0087] A variety of techniques can be used to implement communication. The I / O component 1102 also includes a communication component 1138, which is operable to couple the machine 1100 to the network 1122 or the device 1124 via corresponding couplings or connections. For example, the communication component 1138 may include a network interface component for docking with the network 1122 or another suitable device. In another example, the communication component 1138 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, components (e.g., low power consumption), components, and other communication components that provide communication via other modalities. The device 1124 may be another machine or any peripheral device among various peripheral devices (e.g., a peripheral device coupled via USB).

[0088] In addition, the communication component 1138 can detect an identifier or include components operable to detect an identifier. For example, the communication component 1138 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, UltraCode, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying a tagged audio signal). In addition, various information can be obtained via the communication component 1138, such as a location derived via Internet Protocol (IP) geolocation, a location derived via signal triangulation, a location derived via detecting an NFC beacon signal that can indicate a specific location, etc.

[0089] Various memories (e.g., main memory 1114, static memory 1116, and the memory of the processor 1104) and the storage unit 1118 can store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 1110), when executed by the processor 1104, cause the various operations to implement the disclosed examples.

[0090] The instructions 1110 can be transmitted or received over the network 1122 via a network interface device (e.g., the network interface component included in the communication component 1138) using a transmission medium and using any one of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 1110 can be transmitted or received using a transmission medium via an interface with the device 1124 (e.g., a peer-to-peer interface).

[0091] Glossary

[0092] A "carrier signal" refers to any non-transitory medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other non-transitory media to facilitate the communication of such instructions. Instructions can be transmitted or received over a network via a network interface device using a transmission medium.

[0093] "Client device" means any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, desktop computer, laptop computer, portable digital assistant (PDA), smartphone, tablet computer, ultrabook, netbook, laptop, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or any other communication device that a user can use to access a network.

[0094] "Communication network" means one or more portions of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), plain old telephone service (POTS) network, cellular telephone network, wireless network, a network, other types of networks, or a combination of two or more such networks. For example, a network or a portion of a network can include a wireless network or a cellular network, and the coupling can be a code division multiple access (CDMA) connection, global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling can implement any data transfer technology among various types of data transfer technologies, such as single-carrier radio transmission technology (1xRTT), evolved data optimized (EVDO) technology, general packet radio service (GPRS) technology, GSM enhanced data rates for GSM evolution (EDGE) technology, 3rd Generation Partnership Project (3GPP) including 3G, 4th Generation Wireless (4G) network, universal mobile telecommunications system (UMTS), high-speed packet access (HSPA), worldwide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other data transfer technologies defined by various standards-setting organizations, other long-distance protocols, or other data transfer technologies.

[0095] "Component" refers to a device, physical entity, or logic having a boundary defined by a function or subroutine call, a branch point, an API, or other technology that provides partitioning or modularization for a particular processing or control function. Components can be combined with other components via their interfaces to perform machine processing. A component can be an encapsulated functional hardware unit designed to be used with other components and is typically part of a program that performs a specific function among related functions. Components can constitute software components (e.g., code implemented on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or a part of an application) to operate to perform certain operations described herein as a hardware component. A hardware component can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic permanently configured to perform certain operations. A hardware component can be a dedicated processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component can also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a particular component of a machine) uniquely customized to perform the configured function and is no longer a general-purpose processor. It will be appreciated that the decision of whether to implement a hardware component mechanically in dedicated and permanently configured circuitry or in temporarily configured (e.g., software-configured) circuitry can be made for cost and time considerations. Thus, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, i.e., an entity physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in some manner or perform certain operations described herein. Considering an example where a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any given time. For example, in the case where a hardware component includes a general-purpose processor configured by software to become a dedicated processor, the general-purpose processor can be configured as different dedicated processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to, for example, constitute a particular hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Thus, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more of the hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure to which the multiple hardware components have access and retrieving the information from the memory structure. For example, one hardware component can perform an operation and store the output of the operation in a memory device communicatively coupled thereto. Then, other hardware components can access the memory device at a later time to retrieve the stored output and process it. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors temporarily configured (e.g., via software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more of the operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least in part processor-implemented, where a particular one or more processors are examples of hardware. For example, at least some of the operations of the method can be performed by one or more processors 1104 or processor-implemented components. Additionally, one or more processors can also operate to support the execution of relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS) operation. For example, at least some of the operations can be performed by a group of computers (as an example of machines including processors), where the operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of certain operations can be distributed among processors, not residing only within a single machine but being deployed across several machines. In some examples, the processor or processor-implemented components can be located in a single geographical location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processor or processor-implemented components can be distributed across several geographical locations.

[0096] "Computer-readable storage medium" refers to both machine storage media and transmission media. Thus, the term includes both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium", and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0097] "Ephemeral message" refers to a message that is accessible for a limited duration. The ephemeral message can be text, image, video, etc. The access time of the ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is transient.

[0098] "Machine storage medium" refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but not be limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", and "computer storage medium" mean the same thing and can be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium".

[0099] "Non-transitory computer-readable storage medium" refers to a tangible medium that can store, encode, or carry instructions executable by a machine.

[0100] "Signal medium" refers to any intangible medium that can store, encode, or carry instructions executable by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal in which one or more of its characteristics are set or changed in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and can be used interchangeably in this disclosure.

Claims

1. A method, executed by one or more processors, for providing an image enhancement effect on a device including a display and at least one imaging device, the method comprising: Receiving a first image stream captured by the at least one imaging device; Applying a simplified augmented reality effect to the image stream captured by the at least one imaging device to generate a preview image stream; Displaying the preview image stream on the display; And Saving a second image stream corresponding to the first image stream captured by the at least one imaging device to an initial video file.

2. The method according to claim 1, further comprising: Retrieving the second image stream from the initial video file; Applying a complete augmented reality effect corresponding to the simplified augmented reality effect to the second image stream to generate a fully enhanced image stream; and Saving the fully enhanced image stream to another video file.

3. The method according to claim 2, wherein, Automatically starting to retrieve the second image stream from the initial video file when the saving of the initial video file is completed, the method further comprising: Automatically playing back the another video file on the display once the another video file has been saved.

4. The method according to claim 2, wherein, The complete augmented reality effect is based on a machine learning model.

5. The method according to claim 1, wherein The second image stream is an image stream parallel to the first image stream.

6. The method according to claim 5, wherein, The second image stream has a higher resolution than the first image stream.

7. The method according to claim 1, wherein The second image stream includes a video encoded version of the first image stream.

8. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform operations for providing an image enhancement effect on a device including a display and at least one imaging device, the operations comprising: Receiving a first image stream captured by the at least one imaging device; Applying a simplified augmented reality effect to the image stream captured by the at least one imaging device to generate a preview image stream; Displaying the preview image stream on the display; And Saving a second image stream corresponding to the first image stream captured by the at least one imaging device to an initial video file.

9. The non-transitory computer-readable storage medium according to claim 8, wherein, The operations further comprise: Retrieving the second image stream from the initial video file; Applying a complete augmented reality effect corresponding to the simplified augmented reality effect to the second image stream to generate a fully enhanced image stream; and Saving the fully enhanced image stream to another video file.

10. The non-transitory computer-readable storage medium according to claim 9, wherein, Automatically starting to retrieve the second image stream from the initial video file when the saving of the initial video file is completed, the operations further comprising: Automatically playing back the another video file on the display once the another video file has been saved.

11. The non-transitory computer-readable storage medium according to claim 9, wherein, The complete augmented reality effect is based on a machine learning model.

12. The non-transitory computer-readable storage medium according to claim 8, wherein, The second image stream is an image stream parallel to the first image stream.

13. The non-transitory computer-readable storage medium according to claim 12, wherein, The second image stream has a higher resolution than the first image stream.

14. The non-transitory computer-readable storage medium according to claim 12, wherein, The second image stream includes a video encoded version of the first image stream.

15. A computing device, comprising: At least one imaging device; A display; One or more processors; And A memory storing instructions that, when executed by the one or more processors, configure the device to perform operations for providing an image enhancement effect, the operations including: Receiving a first image stream captured by the at least one imaging device; Applying a simplified augmented reality effect to the image stream captured by the at least one imaging device to generate a preview image stream; Displaying the preview image stream on the display; and Saving a second image stream corresponding to the first image stream captured by the at least one imaging device to an initial video file.

16. The computing device according to claim 15, wherein, The operations further include: Retrieving the second image stream from the initial video file; Applying a full augmented reality effect corresponding to the simplified augmented reality effect to the second image stream to generate a fully enhanced image stream; and Saving the fully enhanced image stream to a further video file.

17. The computing device according to claim 16, wherein, Automatically starting to retrieve the second image stream from the initial video file upon completion of the saving of the initial video file, the operations further including: Automatically playing back the further video file on the display once the further video file has been saved.

18. The computing device according to claim 16, wherein, The full augmented reality effect is based on a machine learning model.

19. The computing device according to claim 15, wherein, The second image stream is an image stream parallel to the first image stream.

20. The computing device according to claim 15, wherein, The second image stream has a higher resolution than the first image stream.

Images