Automatic slow motion video recording

Abstract

A system, method, and computer-readable medium for automatic slow motion video recording are provided. An example method may include: storing an ordered sequence of frames captured by an image sensor in a buffer; sending a first frame from the ordered sequence of frames in the buffer for display at a time interval; calculating an amount of motion between the first frame in the buffer and a second frame in the buffer; determining a frame rate for encoding a frame set from the ordered sequence of frames in the buffer based on the amount of motion, the frame set including the second frame in the buffer and one or more frames sequentially between the first frame and the second frame; and encoding the frame set at the determined frame rate.

Description

[0001] priority

[0002] This patent application claims priority to U.S. Non-Provisional Application No. 16 / 672,286, filed on November 1, 2019, entitled “AUTOMATIC SLOW MOTION VIDEORECORDING,” which is assigned to the assignee of the present application and is expressly incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to automatic slow motion video recording. Background Art

[0004] A camera is a device that captures image frames such as still images or video frames using an image sensor behind one or more lenses that can bend and / or focus light passing through one or more lenses. A camera may include a processor such as an image signal processor (ISP) that can receive one or more image frames captured by the camera's image sensor and process the one or more image frames. The images or videos produced by the camera may be stored on a storage device, presented on a display device, and / or sent to a separate or remote device for storage and / or display. In addition, the video produced by the camera may have certain effects that may be of interest to the user. An example video effect is slow motion. Summary of the invention

[0005] In some examples, systems, methods, and computer-readable media for providing automatic slow motion video recording based on an adaptive frame rate are described. According to at least one example, a method for automatic slow motion video recording may include: storing an ordered sequence of frames captured by an image sensor in a buffer; sending a first frame from the ordered sequence of frames in the buffer for display at a time interval; calculating an amount of motion between a second frame in the buffer and the first frame in the buffer; determining a frame rate for encoding a set of frames from the ordered sequence of frames in the buffer based on the amount of motion, the set of frames including the second frame in the buffer and one or more frames sequentially between the first frame and the second frame; and encoding the set of frames at the determined frame rate.

[0006] In another example, a device for automatic slow motion video recording is provided. The device may include a memory and one or more processors coupled to the memory, the one or more processors being configured to: store an ordered sequence of frames captured by an image sensor in a buffer; send a first frame from the ordered sequence of frames in the buffer for display at a time interval; calculate an amount of motion between a second frame in the buffer and the first frame in the buffer; determine a frame rate for encoding a frame set from the ordered sequence of frames in the buffer based on the amount of motion, the frame set including the second frame in the buffer and one or more frames sequentially between the first frame and the second frame; and encode the frame set at the determined frame rate.

[0007] In another example, a non-transitory computer-readable medium for automatic slow motion video recording is provided. The non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause the one or more processors to: store an ordered sequence of frames captured by an image sensor in a buffer; send a first frame from the ordered sequence of frames in the buffer for display at a time interval; calculate an amount of motion between a second frame in the buffer and the first frame in the buffer; determine a frame rate for encoding a set of frames from the ordered sequence of frames in the buffer based on the amount of motion, the set of frames including the second frame in the buffer and one or more frames sequentially between the first frame and the second frame; and encode the set of frames at the determined frame rate.

[0008] In another example, a device including components for automatic slow motion video recording is provided. The device may include components for storing an ordered sequence of frames captured by an image sensor in a buffer; components for sending a first frame from the ordered sequence of frames in the buffer for display at a time interval; components for calculating an amount of motion between a second frame in the buffer and the first frame in the buffer; components for determining a frame rate for encoding a set of frames from the ordered sequence of frames in the buffer based on the amount of motion, the set of frames including the second frame in the buffer and one or more frames sequentially between the first frame and the second frame; and components for encoding the set of frames at the determined frame rate.

[0009] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include displaying a frame set at a playback frame rate that is lower than a frame rate determined for the frame set. In some examples, displaying a frame set at a playback frame rate that is lower than a frame rate determined for the frame set may include generating a slow motion effect for the frame set based on the determined frame rate and the playback frame rate.

[0010] In some aspects, determining a frame rate for encoding a set of frames based on an amount of motion may include: when the calculated amount of motion is greater than a motion threshold, determining a first frame rate for encoding the set of frames; and when the calculated amount of motion is less than the motion threshold, determining a second frame rate for encoding the set of frames, the second frame rate being lower than the first frame rate.

[0011] In some examples, the amount of motion between the first frame in the buffer and the second frame in the buffer is calculated based on a comparison of the first frame and the second frame. In addition, in some examples, the amount of motion between the first frame in the buffer and the second frame in the buffer is calculated based on one of an optical flow technique or a block matching technique.

[0012] In some aspects, the methods, non-transitory computer-readable media, and devices described above may include storing a video comprising a sequence of frames, the video comprising a set of frames encoded at a determined frame rate and at least one additional set of frames encoded at a lower frame rate than the frame rate associated with the set of frames, the lower frame rate based on corresponding motion captured in the at least one additional set of frames.

[0013] In some examples, calculating the amount of motion, determining the frame rate, and encoding the set of frames is performed when one or more new frames are received from the image sensor and / or stored in a frame buffer.

[0014] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include presenting a preview including the first frame and / or the second frame using a display device. In some examples, at least one or more other frames in the ordered sequence of frames are not included in the preview, and the at least one or more other frames are not presented in a different preview.

[0015] In some aspects, determining a frame rate for encoding a set of frames from an ordered sequence of frames in a buffer includes determining a first frame rate when the amount of motion includes a first level of motion, determining a second frame rate when the amount of motion includes a second level of motion that is higher than the first level of motion, and / or determining a third frame rate when the amount of motion includes a third level of motion that is higher than the second level of motion. In some examples, the third frame rate may be higher than the first frame rate and the second frame rate, and the second frame rate may be higher than the first frame rate. Additionally, in some cases, encoding the set of frames may include encoding the set of frames at a first frame rate when the amount of motion includes a first level of motion, encoding the set of frames at a second frame rate when the amount of motion includes a second level of motion, and / or encoding the set of frames at a third frame rate when the amount of motion includes a third level of motion.

[0016] In some aspects, the apparatus described above may include one or more sensors. In some examples, the apparatus described above may include a mobile phone, a wearable device, a mobile computer, a head-mounted display, and / or a camera.

[0017] This summary is neither intended to identify key or essential features of the claimed subject matter nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0018] The foregoing as well as other features and embodiments will become more fully apparent after reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to describe the manner in which the various advantages and features of the present disclosure can be obtained, a more particular description of the above principles will be provided with reference to specific embodiments thereof shown in the accompanying drawings. It is to be understood that these drawings depict only exemplary embodiments of the present disclosure and are not to be considered as limiting the scope thereof. The principles of the present invention will be described and explained with additional particularity and detail through the use of the accompanying drawings, in which:

[0020] Figure 1 shows an example architecture of a video processing system according to some examples;

[0021] Figure 2 A system flow for performing automatic slow motion video recording at an adaptive frame rate according to some examples is shown;

[0022] Figure 3 is a diagram illustrating an example of a system for performing automatic slow motion video recording at an adaptive frame rate according to some examples;

[0023] Figure 4 is a diagram illustrating analysis of frames in a frame buffer to estimate motion in one or more frames according to some examples;

[0024] Figure 5 An example video playback interface of a video player showing indicators identifying portions of a video having detected motion and configured for slow motion playback according to some examples is shown;

[0025] Figure 6 shows an example flow chart for performing automatic slow motion video recording according to some examples;

[0026] Figure 7 An example method for automatic slow motion video recording using adaptive frame rate according to some examples is shown; and

[0027] Figure 8 An example computing device architecture according to some examples is shown. DETAILED DESCRIPTION

[0028] Provided below are some aspects and embodiments of the present disclosure. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that each embodiment may be implemented without these specific details. The accompanying drawings and description are not intended to be restrictive.

[0029] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Specifically, the following description of these exemplary embodiments will provide a feasible description for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0030] Disclosed herein are systems, methods, and computer-readable media for performing automatic slow-motion video recording using adaptive frame rates. Slow-motion video is a video effect in which time appears to slow down at certain portions of a video. Slow-motion video playback can be achieved when video is captured and / or encoded at a higher frame rate and played back at a lower frame rate (and therefore stretched over a longer period of time). Viewers often use slow-motion video playback to more clearly see fast movements or features that might otherwise be difficult or impossible for the human eye to perceive in its original state.

[0031] Typically, a slow-motion video recording system records the entire video at a high frame rate. However, this may impose a significant burden on resources (such as storage devices, power, bandwidth, and computing resources) used to generate, store, and / or stream these slow-motion videos. In addition, watching the entire video in slow motion may be tedious for viewers who may not be interested in watching certain parts of the video in slow motion (such as parts with little (or no) motion and parts that the viewer is not very interested in). Although the user can edit the video after recording the video to modify which parts maintain a high frame rate and maintain slow motion, this post-editing may also be tedious and undesirable for the user. As further discussed herein, the method of this article provides a technical improvement to slow-motion video technology and solves the aforementioned and other challenges.

[0032] The present technology will be described in the following disclosure as follows. The discussion begins with Figures 1 to 5A description of example systems, architectures, and techniques for performing automatic slow motion video recording using an adaptive frame rate is shown. Figure 6 and Figure 7 A description of an example process and method for performing automatic slow motion video recording using an adaptive frame rate is shown. The discussion ends with Figure 8 The present disclosure now turns to a description of an example computing device architecture including example hardware components suitable for performing automatic slow motion video recording. Figure 1 .

[0033] Figure 1 An example architecture of a video processing system 100 is shown. The video processing system 100 can perform various image / video processing tasks and generate various image / video processing effects as described herein. For example, the video processing system 100 can perform automatic slow motion video recording, and can adaptively change the video recording frame rate as needed or in real time (or near real time). In some illustrative examples, the video processing system 100 can adaptively change the video frame rate based on the motion and / or activity amount / intensity in the recording (multiple) scenes. Although the slow motion effect is used in this article as an illustrative example effect that can be produced by the video processing system 100, the video processing system 100 and technology herein can also be used for other image / video processing effects, such as feature extraction effects, image recognition effects, machine vision effects, any combination thereof, and / or for any other image / video processing effects.

[0034] The video processing system 100 may be part of a computing device or multiple computing devices. For example, the video processing system 100 may be part of one or more electronic devices such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a phone system (e.g., a smart phone, a cellular phone, a conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a television, a game console, a video streaming device, a robotic device, an IoT (Internet of Things) device, a smart wearable device, or any other (multiple) suitable electronic devices. In some examples, the video processing system 100 may be part of a camera system capable of capturing and recording video.

[0035] exist Figure 1 In the example shown in , the video processing system 100 includes an image sensor 102, a frame buffer 104, a storage device 106, an output device 108, a computing component 110, an image processing engine 120, a frame grabber 122, an encoder engine 124, a motion detector engine 126, a machine learning engine 128, and a decoder engine 130. Although the video processing system 100 is shown as including certain components, those of ordinary skill in the art will appreciate that the video processing system 100 may include more than one component. Figure 1 For example, in some instances, the video processing system 100 may include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more camera lenses, and / or Figure 1 Any other hardware or processing device not shown in the figure. Figure 1 Illustrative examples of computing devices and hardware components that may be implemented with video processing system 100 are described.

[0036] In some embodiments, the image sensor 102, the frame buffer 104, the storage device 106, the output device 108, the computing component 110, the image processing engine 120, the frame grabber 122, the encoder engine 124, the motion detector engine 126, the machine learning engine 128, and the decoder engine 130 may be part of the same computing device. For example, in some cases, the image sensor 102, the frame buffer 104, the storage device 106, the output device 108, the computing component 110, the image processing engine 120, the frame grabber 122, the encoder engine 124, the motion detector engine 126, the machine learning engine 128, and the decoder engine 130 may be integrated into a camera, a smart phone, a laptop computer, a tablet computer, a smart wearable device, a gaming system, and / or any other computing device. However, in some embodiments, image sensor 102, frame buffer 104, storage device 106, output device 108, computing component 110, image processing engine 120, frame grabber 122, encoder engine 124, motion detector engine 126, machine learning engine 128, and decoder engine 130 may be parts of two or more separate computing devices.

[0037] The image sensor 102 may be any image sensor or capture device, such as a digital or video camera sensor, a smartphone camera sensor, a video capture / recording device on an electronic device such as a television, computer, camera, etc. In some cases, the image sensor 102 may be part of a camera or computing device such as a digital or video camera, an IP camera, a smartphone, a smart television, a gaming system, etc. The image sensor 102 may capture image and video content (e.g., raw image or video data), which may then be processed by the computing component 110, the image processing engine 120, the frame capturer 122, the encoder engine 124, the motion detector engine 126, the machine learning engine 128, and / or the decoder engine 130 as described herein. In some implementations, the image sensor 102 may include a high frame rate image sensor capable of capturing high frame rate frames.

[0038] Frame buffer 104 may provide buffer space that may be used by any of computing components 110. Frame buffer 104 may store frames captured by image sensor 102 and preserve the frames for subsequent processing by computing component 110 (e.g., via image processing engine 120, frame grabber 122, encoder engine 124, motion detector engine 126, machine learning engine 128, and / or decoder engine 130). As further described herein, in some cases, frame buffer 104 may store frames captured by image sensor 102 after they have been processed or pre-processed by image processing engine 120.

[0039] In some embodiments, the frame buffer 104 may be a circular buffer. A circular buffer may be used to store frames and provide a circular or first-in-first-out (FIFO) queue for the frames. In addition, the frame buffer 104 may be implemented by or be part of a circuit board on the video processing system 100, a storage device 106, one or more of the computing components 110, or a separate memory device.

[0040] Storage 106 may be any storage device(s) for storing data such as, for example, image or video data. In addition, storage 106 may store data from any component of video processing system 100. For example, storage 106 may store data from image sensor 102, data from computing component 110 (e.g., processing parameters, output images and / or video, computational results, etc.), and / or data from image processing engine 120, frame grabber 122, encoder engine 124, motion detector engine 126, machine learning engine 128, and / or decoder engine 130 (e.g., output images, processing results, etc.). In some examples, storage 106 may include a buffer for storing data (e.g., image data) for processing by computing component 110.

[0041] Output device 108 may be any device capable of rendering / displaying data, such as images and videos. In some examples, output device 108 may be a display device for presenting images and videos.

[0042] In some embodiments, the computing component 110 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. The computing component 110 may perform various operations such as slow motion video processing and recording, image enhancement, object or image segmentation, computer vision, graphics rendering, augmented reality, image / video processing, sensor processing, recognition (e.g., motion recognition, activity recognition, object recognition, feature recognition, tracking or pattern recognition, scene change recognition, etc.), disparity detection, machine learning, filtering, and any of the various operations described herein. In some examples, the computing component 110 may implement an image processing engine 120, a frame grabber 122, an encoder engine 124, a motion detector engine 126, a machine learning engine 128, and a decoder engine 130. In other examples, the computing component 110 may also implement one or more other processing engines.

[0043] In some cases, the operations of image processing engine 120, frame grabber 122, encoder engine 124, motion detector engine 126, machine learning engine 128, and decoder engine 130 may be implemented by one or more of computing components 110. For example, image processing engine 120, frame grabber 122, encoder engine 124, motion detector engine 126, machine learning engine 128 (and associated operations), and decoder engine 130 may be implemented by CPU 112, GPU 114, DSP 116, and / or ISP 118. In some cases, computing component 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof to perform any of the various operations described herein.

[0044] In some cases, the computing component 110 may receive data (e.g., image / video data, etc.) captured by the image sensor 102 and process the data to generate output frames, some of which may have a slow motion effect. For example, the computing component 110 may receive image / video data (e.g., one or more frames, etc.) captured by the image sensor 102, detect any motion and / or activity in the captured scene (e.g., in the captured frames) based on the image / video data, and generate a video with a slow motion effect, wherein some frames in the video have a higher frame rate adaptively generated based on the motion and / or activity detected in the captured scene. In some examples, the frames may include a red-green-blue (RGB) image or frame having red, green, and blue components per pixel; a luminance, red chrominance, blue chrominance (YCbCr) image or frame having a luminance component and two chrominance (color) components (red chrominance and blue chrominance) per pixel; or any other suitable type of color or monochrome picture.

[0045] The computing component 110 may implement the image processing engine 120, the frame grabber 122, the encoder engine 124, the motion detector engine 126, the machine learning engine 128, and the decoder engine 130 to perform various image processing operations and generate / record a video with a slow motion effect as described herein. For example, the computing component 110 may implement the image processing engine 120, the frame grabber 122, the encoder engine 124, the motion detector engine 126, and / or the machine learning engine 128 to perform automatic slow motion video recording as described herein. The computing component 110 may process the image data or frames captured by the image sensor 102 to perform such slow motion video recording.

[0046] The image processing engine 120 may perform various image processing operations. For example, the image processing engine 120 may perform lens shading correction, downsampling, feature detection, blurring, segmentation, filtering, color correction, noise reduction, scaling, demosaicing, pixel interpolation, image signal processing, image enhancement, color space conversion, etc. In addition, the image processing engine 120 (and associated operations) may be implemented by one or more of the computing components 110. For example, the image processing engine 120 may be implemented by the CPU 112, the DSP 116, and / or the ISP 118, and the GPU 114 may implement operations for rendering image / video data from the image processing engine 220 (and / or the encoder engine 124).

[0047] The frame grabber 122 may obtain frames captured by the image sensor 102 and stored in the frame buffer 104 and provide the frames to the encoder engine 124, the motion detector engine 126, and / or the machine learning engine 128 for further processing.

[0048] The encoder engine 124 may encode the video data to generate an encoded video bitstream. In some examples, the encoded video bitstream (or "video bitstream" or "bitstream") is a series of one or more encoded video sequences. A coded video sequence (CVS) may include a series of AUs starting with an access unit (AU). An access unit (AU) may include one or more coded pictures and control information corresponding to the coded pictures sharing the same output time. Coded slices of a picture may be encapsulated in a bitstream level into a data unit called a network abstraction layer (NAL) unit.

[0049] In some examples, the encoder engine 124 can encode video frames at a specific frame rate that can be used to generate certain effects such as slow motion effects. For example, the encoder engine 124 can encode high frame rate frames that can create slow motion effects when played at a lower frame rate. In some cases, the encoder engine 124 can adaptively encode the frame rate based on the amount / intensity of motion or activity detected in the captured scene (and / or in the associated frames to be encoded). For example, if a frame set captures a large amount of motion or activity, the encoder engine 124 can encode the frame set at a higher frame rate, and if a different frame set captures a small amount of motion or activity, the encoder engine 124 can adaptively change the encoding of these frames to a lower frame rate. In addition, in some cases, the encoder engine 124 can adaptively change the frame rate for different frames as needed or in real time (or near real time).

[0050] The motion detector engine 126 may analyze the image / video data (e.g., frames) to determine the magnitude of motion, if any, in the image / video data. For example, the motion detector engine 126 may perform one or more optical flow techniques to estimate motion in or between image frames. In some examples, the motion detector engine 126 may estimate motion in an ordered sequence of images or frames as image velocity or discrete image displacement.

[0051] The machine learning engine 128 may implement one or more neural networks to process frames and detect actions in the frames. In some examples, the machine learning engine 128 may implement a convolutional neural network (CNN), a time delay neural network (TDNN), or a probabilistic neural network (PNN).

[0052] The neural network implemented by the machine learning engine 128 can be trained to recognize one or more actions in a frame. These actions can be previously defined or previously identified actions. The neural network implemented by the machine learning engine 128 can be previously trained using one or more videos (e.g., video sequences) as input, where each input video includes one or more frames. Labels can be added to the input videos (e.g., added by a person or automatically), which classify whether each input video or one or more frames of each input video include one or more previously defined actions. Using the one or more input videos and the labels, back-propagation techniques can be used to tune the parameters of the neural network (e.g., weights, biases, and / or other parameters), thereby training the neural network to detect similar actions when the input videos are processed by the neural network during inference (or runtime).

[0053] For example, during training, a neural network may receive a video as an input and may provide an output indicating the likelihood (e.g., probability) that the video includes a defined action. The label may be compared to the actual output from the neural network to determine a loss or error associated with the actual output. The loss may be based on any suitable loss function, such as using a mean squared error (MSE) function, a mean absolute error (MAE) function, a Huber loss function, a log cosh loss function, a quantile loss function, a logarithmic loss function, an exponential loss function, a focal loss function, a cross entropy loss function, a hinge loss function, a Kullback-Liebler (KL) divergence loss function, any suitable combination thereof, and / or other loss functions.

[0054] Using the determined loss, back propagation can be used to determine whether the parameters (e.g., weights, biases, etc.) of the neural network should be adjusted to improve accuracy by reducing the error to within a certain threshold error or to within a threshold similarity to the corresponding label. For example, internal machine learning parameters can be adjusted (e.g., by modifying weights and / or other parameters associated with various neurons or layers), which can change the output generated by the neural network when another input video is received and, therefore, will reduce the overall loss.

[0055] Once the parameters (e.g., weights of the hidden layers and / or other parameters) have been adjusted based on the loss, another input video and corresponding label can be input to the neural network. The neural network can then generate another output based on the input video, and the loss associated with the new input video can be determined. The loss associated with the new input video can again be used to determine how much the parameters of the neural network should be adjusted to further improve its output accuracy. The loss determination and backpropagation process can be repeated for multiple iterations (also referred to as epochs) to improve the accuracy of the neural network until a predicted output is generated that is below a threshold error or within a threshold of similarity to the label.

[0056] The defined actions that the neural network implemented by the machine learning engine 128 can be trained to detect or recognize can include, for example, a person or animal running, a person or animal jumping, a person or animal throwing an object (e.g., an athlete throwing a ball), a person or animal colliding with an object (e.g., an athlete catching a ball), a person or animal colliding with one or more other people or animals (e.g., an athlete colliding with another athlete), an object held by a person or animal colliding with another object (e.g., a baseball bat held by a baseball player colliding with a baseball), a specific acrobatic move of the person or animal (e.g., a front flip, a back flip, a barrel roll, a spin, or a combination thereof), a person or animal diving into a liquid, a person or animal or object entering a liquid, a collision of one or more objects (e.g., a car crash, a projectile hitting a wall or other object), another defined action, or some combination thereof.

[0057] The decoder engine 130 may decode the video bitstream data encoded by the encoder engine 124. For example, the decoder engine 130 may receive the video bitstream data encoded by the encoder engine 124 as input, and perform a decoding operation on the input video bitstream data to generate a decoded video. In some examples, the decoder engine 130 may decode the video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more encoded and decoded video sequences that constitute the encoded video data. The decoder engine 130 may then rescale and perform an inverse transform on the encoded video bitstream data. The residual data is then passed to the prediction stage of the decoder engine 130. The decoder engine 130 then predicts a pixel block (e.g., a PU). In some examples, the prediction is added to the output (residual data) of the inverse transform.

[0058] The decoder engine 130 may output the decoded video to a video destination device, which may store, transmit, or display the decoded video data. For example, the decoder engine 130 may output the decoded video to an output device 108, which may display the decoded video data to a consumer of the content.

[0059] Figure 2 A system flow 200 for performing automatic slow motion video recording at an adaptive frame rate is shown. The image sensor 102 may first capture a video frame of a scene. In some cases, the image sensor 102 may be a high frame rate image sensor. Thus, the video frames captured by the image sensor 102 may be captured at a high frame rate such as 480 FPS (frames per second), 960 FPS, and / or other high frame rates.

[0060] The image processing engine 120 can receive captured frames from the image sensor 102 and perform image processing operations such as, but not limited to, lens shading correction, downsampling, segmentation, filtering, color correction, noise reduction, scaling, demosaicing, pixel interpolation, image signal processing, image enhancement, color space conversion, etc.

[0061] Image processing engine 120 may then store the processed frames in frame buffer 104. Frame buffer 104 may be a ring buffer configured to store a certain number of frames for further processing. In some examples, frame buffer 104 may store frames in a circular or FIFO queue for later processing.

[0062] The frame grabber 122 may obtain frames stored in the frame buffer 104 and provide the frames to the output device 108. In some cases, the frame grabber 122 may obtain the latest frame in the frame buffer 104 and provide the latest frame to the output device 108. In addition, in some implementations, the frame grabber 122 may send frames from the frame buffer 104 to the output device 108 at a specific interval or rate. For example, the frame grabber 122 may send a different frame from the frame buffer 104 to the output device 108 every 33 milliseconds (e.g., at 30 FPS). When the output device 108 receives the frames from the frame grabber 122, the output device 108 may render / display the preview 202 based on the received frames.

[0063] The frame grabber 122 may also provide frames from the frame buffer 104 to the motion detector engine 126 for motion estimation. In some cases, the frame grabber 122 may provide the same frames that it provides to the output device 108 to the motion detector engine 126 (e.g., with some delay between providing the frames to the output device 108 and providing the frames to the motion detector engine 126). In other cases, the frame grabber 122 may provide the motion detector engine 126 with frames in the frame buffer 104 that are older than the frames provided to the output device 108. This may provide some time or delay to allow the motion detector engine 126 to perform motion estimation. In some cases, the time or delay may also allow the encoder engine 124 to encode the frames and / or obtain additional past frames in the frame buffer 104 for adaptive higher frame rate encoding as described herein.

[0064] In some cases, motion detector engine 126 may obtain one or more frames in frame buffer 104 from frame grabber 122 at a particular interval or rate. In some examples, motion detector engine 126 may obtain one or more different frames in frame buffer 104 from frame grabber 122 at different intervals / rates, which may depend on the amount of motion and / or activity detected for those frames.

[0065] When the motion detector engine 126 receives a frame from the frame grabber 122, the motion detector engine 126 may perform motion estimation to determine the magnitude of motion in the frame and / or in the scene captured in the frame. In some cases, the motion detector engine 126 may compare the frame to past frames to calculate an image velocity or discrete image displacement for estimating motion between frames.

[0066] In some examples, motion detector engine 126 may perform optical flow or block matching techniques to estimate motion between the received frame and past (e.g., previous) frames, such as frames provided to output device 108 by frame grabber 122. In other examples, motion detector engine 126 may implement any other motion estimation techniques to estimate motion between the received frame and past frames. Other non-limiting examples of motion estimation methods include phase correlation and frequency domain methods, pixel recursive methods, parametric motion estimation, and multi-resolution methods.

[0067] In some cases, instead of or in addition to providing frames to the motion detector engine 126, the frame grabber 122 may provide frames from the frame buffer 104 to the machine learning engine 128. The machine learning engine 128 may then analyze the frames to determine whether the frames depict one or more specific actions. The machine learning engine 128 may detect the motion detector engine 126 as previously described. Figure 1 Describes the action in the frame.

[0068] After performing motion estimation, the motion detector engine 126 may provide a motion vector indicating or describing the amount / magnitude of motion (if any) detected for the frame (e.g., captured by the frame) to the encoder engine 124. The motion detected by the motion detector engine 126 may include motion captured by the frame as well as motion captured by one or more past (e.g., previous) frames in the frame buffer 104. For example, the motion detected by the motion detector engine 126 may include motion between the frame and a previous frame (such as a frame sent by the frame grabber 122 to the output device 108 for rendering / presenting the preview 202). Such motion may include motion captured in the frame as well as in any intermediate frames between the frame and the previous frame used to estimate the motion.

[0069] In some implementations where the machine learning engine 128 is used to estimate actions in a frame, the machine learning engine 128 may provide an indication of the action(s) depicted in the frame and detected by the machine learning engine 128 to the encoder engine 124. In some cases, the machine learning engine 128 may provide one or more estimated probabilities that the frame depicts one or more corresponding actions to the encoder engine 124. In other cases, the machine learning engine 128 may provide the estimated action having the highest probability as determined by the machine learning engine 128 to the encoder engine 124.

[0070] The encoder engine 124 may use the motion vectors from the motion detector engine 126 and / or the activity information from the machine learning engine 128 to adaptively encode one or more frames, including frames provided to the motion detector engine 126 by the frame grabber 122, at a particular frame rate and generate encoded frames 206 having the particular frame rate. For example, if the motion vectors from the motion detector engine 126 indicate that a high amount of motion is detected, the encoder engine 124 may encode one or more past frames 204 (including frames provided to the motion detector engine 126 by the frame grabber 122) at a higher frame rate than if the motion vectors indicate a lower amount of motion.

[0071] Thus, the frame rate applied by the encoder engine 124 when encoding one or more frames to generate the encoded frames 206 may depend on, and / or may be a function of, the amount / magnitude of motion detected by the motion detector engine 126 for those frames (and / or activity detected by the machine learning engine 128). In this way, the encoder engine 124 may adaptively change the frame rate of those frames based on the amount of motion and / or activity in those frames in the video captured by the image sensor 102. And, because the magnitude of the slow-motion effect of the encoded frames 206 may depend on the frame rate of the encoded frames 206 and the frame rate used to play the encoded frames 206, the encoded or recorded video generated by the encoder engine 124 may be automatically configured to provide different slow-motion effects at different portions of the video (e.g., portions having a higher frame rate as opposed to portions having a lower frame rate).

[0072] For example, if the encoded frame 206 has a frame rate of 960 FPS and is played at a frame rate of 30 FPS, the slow motion effect will be greater than if the encoded frame 206 played at 30 FPS has a frame rate of 240 FPS. Similarly, if the encoded frame 206 has a frame rate of 30 FPS and is played at a frame rate of 30 FPS, the encoded frame 206 will not produce a slow motion effect (or will produce a smaller slow motion effect than when encoded at a frame rate higher than 30 FPS). Therefore, by adaptively changing the frame rate of frames in the video captured by the image sensor 102 based on the amount of motion and / or activity of those frames, the encoder engine 124 can adaptively and intelligently change the slow motion effect of the video based on the amount of motion and / or activity in different parts of the video. Thus, instead of generating an entire video in slow motion and requiring the user to consume the entire video in slow motion or perform post-editing to reconfigure the video to provide slow motion effects only at specific portions, the encoder engine 124 can intelligently encode the video to have different frame rates at different portions depending on the motion / activity in different portions, and thereby provide different slow motion effects based on the motion / activity in different portions of the video.

[0073] In some examples, the encoder engine 124 may adaptively encode different frames at a plurality of different frame rates depending on the amount of motion detected for the different frames by the motion detector engine 126. For example, frames with no motion or a small amount of motion (e.g., a first level of motion below a first threshold) may be encoded at a normal or base frame rate (e.g., 20-30 FPS), frames with a higher amount of motion (e.g., above the first level of motion, above a first threshold set for the normal or base frame rate, and / or a second level of motion below a second threshold) may be encoded at a higher frame rate (e.g., at 480 FPS or within a range of 480 FPS), and frames with an even higher amount of motion (e.g., a third level of motion above a second threshold) may be encoded at an even higher frame rate (e.g., at 960 FPS or within a range of 960 FPS). In some cases, the encoder engine 124 may encode frames at different varying frame rates corresponding to additional levels of motion different from the levels of motion described in the previous examples.

[0074] As previously described, the encoded frame 206 may be generated based on the past frame 204. In some examples, the number of past frames 204 used to generate the encoded frame 206 may depend on the frame rate implemented for the encoded frame 206. For example, when generating encoded frames at a lower frame rate, the encoder engine 124 may use fewer frames (which the encoder engine 124 may obtain from the frame buffer 104) than when the encoder engine 124 generates encoded frames at a higher frame rate.

[0075] Figure 3 is a diagram illustrating an example system 300 for performing automatic slow motion video recording at an adaptive frame rate. In this example, image sensor 102 may first capture a video frame of a scene. Image sensor 102 may be a high frame rate image sensor configured to capture frames at a high frame rate. Image processing engine 120 may receive the captured frames from image sensor 102 and perform image processing operations such as, for example, but not limited to, downsampling, segmentation, filtering, color correction, noise reduction, scaling, demosaicing, pixel interpolation, image signal processing, image enhancement, color space conversion, etc.

[0076] The image processing engine 120 may store the processed frames in the frame buffer 104. In this example, the frame buffer 104 is a ring buffer configured to store a certain number of frames from the image processing engine 120 for further processing. As shown, the frame buffer 104 has stored a plurality of frames 302 and 304 for later processing. The frames 302 may include preview frames; representing frames that are sent to the output device 108 for rendering / rendering of the preview 202. In addition, the frames 304 may include intermediate frames; representing frames that are sequentially between the preview frames.

[0077] For example, in some cases, the frame grabber 122 may send new frames (e.g., preview frames 302) from the frame buffer 104 to the output device 108 at a certain interval / rate (e.g., every n milliseconds and / or at nFPS), which the output device 108 may use to render / display the preview 202. The frames sent to the output device 108 for preview are referred to herein as preview frames 302. On the other hand, any frames stored on the frame buffer 104 between two preview frames 302 are referred to herein as intermediate frames 304, and may be used by the encoder engine 124 along with the preview frames 302 to generate the encoded frames 206 as described herein.

[0078] exist Figure 3 In the example, the frame grabber 122 obtains the preview frame 302A from the frame buffer 104 and sends it to the output device 108. The output device 108 may receive the preview frame 302A and render / display the preview 202 based on the preview frame 302A. The preview frame 302 may be a newer frame stored on the frame buffer 104 by the image processing engine 120.

[0079] The frame grabber 122 may also obtain a preview frame 302B from the frame buffer 104 and send it to the motion detection engine 126 for motion estimation. In some cases, the preview frame 302B may be a past preview frame that is older than the preview frame 302A and was previously sent by the frame grabber 122 to the output device 108 for preview. Since the preview frame 302B is older in time than the preview frame 302A, there is a delay 306 in time between the time when the preview frame 302B is captured by the image sensor 102 and stored on the frame buffer 104 and the time when the preview frame 302A is captured by the image sensor 102 and stored on the frame buffer 104. This delay 306 in time may provide the motion detector engine 126 with some time to perform motion estimation before the preview frame 302B (and the intermediate frames 304 between the preview frames 302A and 302B) become less relevant or outdated to the user because the image sensor 102 captures new frames for the frame sequence of the video.

[0080] In some examples, the delay 306 may be based on the interval / rate at which new preview frames 302 are provided to the output device 108 for preview. Further, in some cases, the delay 306 may be based at least in part on or take into account the amount of time it takes for the motion detector engine 126 to perform motion estimation. For example, if the motion detector engine 126 takes (e.g., on average, at most, etc.) 1 second to perform motion estimation, the delay 306 may be greater than or equal to 1 second to give the motion detector engine 126 sufficient time to perform motion estimation before the newer preview frame 302A replaces or catches up in time with the older preview frame 302B and / or another preview frame replaces the preview frame 302A as the newer preview frame.

[0081] During the delay 306, one or more intermediate frames 304 may be present on the frame buffer 104. Such one or more intermediate frames 304 may be used along with the preview frame 304B to generate the encoded frame 206 at a particular frame rate, as further described herein. For example, if the encoder engine 124 requires additional frames (e.g., in addition to the preview frame 302B) to encode a frame at a particular frame rate, the encoder engine 124 may use one or more of the intermediate frames 304 between the preview frame 302A and the preview frame 302B to generate the encoded frame at the particular frame rate.

[0082] When the motion detector engine 126 obtains the preview frame 302B from the frame grabber 122, it may use the preview frame 302B and the preview frame 302A to estimate the motion between the preview frames 302A and 302B. The motion detector engine 126 may then generate a motion vector that describes the motion between the preview frames 302A and 302B. The motion detector engine 126 may then provide the motion vector to the encoder engine 124 to inform the encoder engine 124 about the estimated motion between the preview frames 302A and 302B.

[0083] The encoder engine 124 may use the motion vectors to determine a frame rate for generating the encoded frame 206. The encoder engine 124 may generate the encoded frame 206 by encoding the preview frame 302B and one or more intermediate frames 304A between the preview frame 302A and the preview frame 302B at the determined frame rate. In addition, the encoder engine 124 may select a higher frame rate for the encoded frame 206 when the motion between the preview frames 302A and 302B is high (as determined based on the motion vectors), and select a lower frame rate for the encoded frame 206 when the motion between the preview frames 302A and 302B is low. For example, if the motion vectors indicate little to no motion, the encoder engine 124 may select a lower frame rate such as, for example, 30 FPS, and if the motion vectors indicate a high amount of motion, the encoder engine 124 may select a higher frame rate such as, for example, 240 FPS or 480 FPS.

[0084] Thus, the encoder engine 124 may adaptively or intelligently select a low or lower frame rate when the intensity of motion is low, and select a higher frame rate when the intensity of motion is high. By selecting or changing the frame rate of frames based on the estimated intensity of motion for those frames, the encoder engine 124 may adaptively or intelligently adjust or influence the slow motion effect produced by a portion of the video corresponding to the encoded frames. The encoder engine 124 may change the frame rate used to encode different frames to reflect or account for differences in the intensity of motion in those frames, and thereby produce different slow motion effects for different portions of the video (in a dynamic or automatic manner).

[0085] When generating the encoded frame 206 based on the motion vectors from the motion detector engine 126, the encoder engine 124 may retrieve one or more intermediate frames 304A from the frame buffer 104 (e.g., via the frame grabber 122). The encoder engine 124 may then generate the encoded frame 206 based on the preview frame 302B and the one or more intermediate frames 304A. In some cases, the number of intermediate frames used to generate the encoded frame 206 may depend at least in part on the frame rate used to encode the encoded frame 206. For example, when the frame rate is higher, the encoder engine 124 may use more intermediate frames to generate the encoded frame 206 than when the frame rate is lower.

[0086] Figure 4 Analysis 400 of frames in the frame buffer 104 is shown to estimate motion in one or more frames. In this example, the frame buffer 104 stores frames 402 captured by the image sensor 102 and processed by the image processing engine 120. The frames 402 may include an ordered sequence of frames associated with a video captured by the image sensor 102. The frames 402 may be stored in the frame buffer 104 according to their ordered order.

[0087] To estimate the amount of motion reflected in preview frame 406 (and / or between preview frame 406 and preview frame 404), motion detector engine 126 may compare preview frame 406 to preview frame 404. In some examples, preview frame 404 may be a newer frame used to generate a recent preview (e.g., preview 202), and preview frame 406 may be an older or past frame previously used to generate a preview, as reflected by a time difference 408 between preview frame 404 and preview frame 406. For example, as previously described, output device 108 may receive and render / present preview frames from frame buffer 104 at a particular time interval or rate. Thus, in this example, preview frame 406 may be a frame selected for preview at a first time interval, and preview frame 404 may be a frame selected for preview at a second time interval after the first time interval.

[0088] Frames 402 in frame buffer 104 may include other frames (e.g., intermediate frames) between preview frames 404 and 406 and other frames after preview frame 406. However, preview frames 404 and 406 may coincide with a particular interval for generating a preview and thus may be designated as preview frames (and used to generate a preview as previously described).

[0089] The time delay 408 between preview frames 404 and 406 may provide time for motion detector engine 126 to estimate motion between preview frames 404 and 406. In some cases, the time delay 408 may be set to be greater than or equal to the amount of time it takes motion detector engine 126 to perform motion estimation on preview frames 404 and 406. Furthermore, the time delay 408 may span one or more intermediate frames between preview frame 404 and preview frame 406. In some examples, this may also provide encoder engine 124 with a pool of frames in frame buffer 104 from which encoder engine 124 may draw to generate encoded frames at a particular frame rate.

[0090] To calculate the motion between preview frames 404 and 406 , motion detector engine 126 may employ one or more motion estimation methods, such as, for example, but not limited to, optical flow methods, block matching methods, phase correlation and frequency domain methods, pixel recursive methods, parametric motion estimation methods, multi-resolution methods, etc. In some examples, motion detector engine 126 may compare preview frames 404 and 406 to determine image velocity or discrete image displacement, which may reflect the motion between preview frames 404 and 406 .

[0091] Based on the motion estimation, the motion detector engine 126 may generate a motion vector 410 that describes the characteristics of the motion (e.g., motion amplitude / frequency / intensity, direction, motion pattern, etc.) between the preview frames 404 and 406. The motion detector engine 126 may provide the motion vector 410 to the encoder engine 124, which may use the motion vector 410 to determine a frame rate for encoding the preview frame 406 and one or more intermediate frames (e.g., one or more frames within the delay 408 between the preview frame 404 and the preview frame 406) as previously described.

[0092] Figure 5An example video playback interface of a video player 510 is shown, showing an indicator 530 identifying a portion (e.g., frame) of a video 520 having detected motion and configured for slow motion playback as described herein. The portion of the video 520 having detected motion can be configured for slow motion playback based on the frame rate(s) used by the encoder engine 124 to encode the frames in the portion of the video 520. As previously described, the frame rate(s) can be calculated based on the detected motion. For example, the encoder engine 124 can determine the frame rate of one or more frames based on, for example, the presence of detected motion in one or more frames, the magnitude / intensity of motion in one or more frames, and / or any other characteristic of the detected motion.

[0093] exist Figure 5 , the interface of the video player 510 displays the video 520. The position of the displayed frame in the video 520 is indicated by the white circle along the slider 550. The left side of the black line of the slider 550 indicates the beginning of the video 520, and the right side of the black line of the slider 550 indicates the end of the video 520. Figure 2 and Figure 3 Video 520 is captured and encoded as described.

[0094] The video player 510 also shows frames 540 as a sequence of parallel vertical black lines. Each frame in the frames 540 is shown as a single vertical black line. The video player 510 also shows indicators 530 represented as black and white triangles, where the lower vertices of the triangles point to a particular frame. The white triangles represent the boundaries (e.g., beginning and / or end) of one or more continuous portions (e.g., a sequence of frames) of the video 520 in which motion has been detected.

[0095] In this example, video 520 includes two portions with detected motion, including a first portion with first detected motion 560 and a second portion with second detected motion 570. The first and last frames of the first portion with first detected motion 560 are marked by white triangle indicators 530, which in some examples can be stored in metadata as timestamps and / or frame numbers corresponding to those frames. Similarly, the first and last frames of the second portion with second detected motion 570 are marked by white triangle indicators 530, which in some examples can be stored in metadata as timestamps and / or frame numbers corresponding to those frames.

[0096] Frame 540 as shown shows that portions of video 520 without detected motion (e.g., those portions other than the first and second portions containing detected motion 560 and 570, respectively) are encoded at a lower frame rate (relative to the first and second portions of video 520 containing detected motion 560 and 570 and / or the high capture frame rate at which frames in those portions of video 520 were captured by image sensor 102) as represented by the larger distance between vertical lines representing the frames (relative to the distance in other portions encoded at a higher frame rate as discussed herein). Therefore, when these portions of video 520 are played back at a normal, low, or same playback frame rate, they will not produce a slow motion effect.

[0097] Frame 540 as shown also shows that the first and second portions of video 520 containing detected motions 560 and 570 are encoded at a higher frame rate (relative to the portions of video 520 not containing detected motions 560 and 570) as represented by the smaller distance between vertical lines representing the frames (relative to the distance in the portions without detected motion). Therefore, when these portions of video 520 are played back at a normal or lower playback frame rate, they will produce a slow motion effect. Frame 540 as shown also shows that the second portion of video 520 containing the second detected motion 570 is encoded at a higher frame rate than the portions not containing detected motion, but is encoded at a lower frame rate than the first portion of video 520 containing the first detected motion 560. Therefore, when played back at the same or normal frame rate as the first portion of video 520 containing the first detected motion 560, the second portion of video 520 containing the second detected motion 570 will produce a smaller slow motion effect than the first portion of video 520 containing the first detected motion 560.

[0098] As previously described, the frame rate used to encode different portions of the video 520 can be based on the motion (or lack of motion) and / or the characteristics of the motion in those portions of the video 520, such that portions with higher motion magnitude / intensity are encoded at a higher frame rate than portions with no motion or lower motion magnitude / intensity. Thus, when the video 520 is then played back at a normal or lower frame rate, those portions of the video 520 with higher frame rates (and higher motion magnitude / intensity) will produce a greater slow motion effect than other portions of the video 520 that were encoded at a lower frame rate (and include lower motion magnitude / intensity or no motion).

[0099] The two indicators 530 shown as black triangles correspond to two still frames 580 and 590, respectively, captured and encoded as described herein. Specifically, a first portion of the video 520 having a first detected motion 560 includes a first black triangle indicator 530 from which an arrow extends downward to a first still frame 580 showing a person jumping, where jumping is the first detected motion 560. A second portion of the video 520 having a second detected motion 570 includes a second black triangle indicator 530 from which an arrow extends downward to a second still frame 590 showing the person from the first still frame 580 colliding with another person, where colliding is the second detected motion 570.

[0100] Having described example systems and concepts, the present disclosure now turns to Figure 6 and Figure 7 The example flowchart 600 and method 700 shown in FIG. The steps outlined in flowchart 600 and method 700 are examples and may be implemented in any combination thereof, including combinations that exclude, add, or modify certain steps.

[0101] Figure 6 An example flow chart 600 for performing automatic slow motion video recording is shown. At step 602, the video processing system 100 may receive a frame from the image sensor 102. The frame may include a sequence of frames in a video captured by the image sensor 102.

[0102] At step 604, the video processing system 100 may apply (e.g., via the image processing engine 120) one or more image processing operations to the frames from the image sensor 102. The image processing operations may include, for example, but not limited to, lens shading correction, downsampling, feature detection, blurring, segmentation, filtering, color correction, noise reduction, scaling, demosaicing, pixel interpolation, image signal processing, image enhancement, color space conversion, and / or any other image processing operation.

[0103] At step 606, the video processing system 100 may store the processed frames in the frame buffer 104. The video processing system 100 may temporarily store the processed frames in the frame buffer 104 for further processing. In some cases, the frame buffer 104 may provide or enable a circular or FIFO queue for subsequent processing of the stored frames.

[0104] At step 608, the video processing system 100 may preview the frames in the frame buffer 104 at specific time intervals. For example, the video processing system 100 may send frames to the output device 108 for preview at a specific frame rate, such as every n milliseconds. To illustrate, the video processing system 100 may send a new frame in the frame buffer 104 to the output device 108 for preview. The image processing engine 120 may continue to process additional frames from the image sensor 102 and store them in the frame buffer 104. After n milliseconds, the video processing system 100 may then send another new frame in the frame buffer 104 to the output device 108 for preview. Every n milliseconds, the video processing system 100 may continue to send newer frames in the frame buffer 104 to the output device 108 for preview.

[0105] At step 610, the video processing system 100 may determine (e.g., via the motion detector engine 126) motion characteristics of the past frame in the frame buffer 104. For example, the video processing system 100 may determine whether there is any motion (and if so, how much motion) between a frame previously sent to the output device 108 for preview and a newer frame being sent or recently sent to the output device 108 for preview. The motion characteristics may include the magnitude / intensity of motion detected for the frame, if any. In some cases, the motion characteristics may also describe the magnitude / intensity of motion in any intermediate frames between the past frame and the newer frame compared to the past frame to determine the motion characteristics.

[0106] At step 612, the video processing system 100 may determine (e.g., via the motion detector engine 126) whether any motion was detected in the past frame. In some cases, the video processing system 100 may determine whether any motion was detected, and if so, whether the motion exceeds a configurable motion threshold amount.

[0107] If the video processing system 100 has not detected any motion (or motion exceeding a threshold) in the past frames, the video processing system 100 may return to step 602 , where it may continue to receive frames from the image sensor 102 .

[0108] If the video processing system 100 does detect motion in the past frames (or motion that exceeds a threshold), then at step 614, the video processing system 100 may (e.g., via the encoder engine 124) calculate a frame rate for encoding the past frames. The video processing system 100 may calculate the frame rate based on the amount of motion detected in the past frames. For example, the video processing system 100 may calculate a higher frame rate when the amount of detected motion is high, and a lower frame rate when the amount of motion is low.

[0109] The calculated frame rate can be used to encode the past frame and one or more past or intermediate frames at a specific frame rate to produce a slow motion effect when the frames are played back at a normal or lower frame rate. In addition, the video processing system 100 can dynamically calculate the frame rate as past frames are stored in the frame buffer 104 and / or additional frames are captured by the image sensor 102 and / or added to the frame buffer 104.

[0110] At step 616, the video processing system 100 may retrieve one or more past frames from the frame buffer 104 for encoding with the past frame. The one or more past frames may include one or more intermediate frames between the past frame used to calculate the frame rate and the newer frame in the frame buffer 104. In some cases, the number of past frames retrieved by the video processing system 100 from the frame buffer 104 may depend on the frame rate calculated at step 614. For example, if the video processing system 100 calculates a high frame rate, the video processing system 100 may retrieve more past frames for encoding than if the video processing system 100 calculates a lower frame rate. This may allow the video processing system 100 to achieve a higher or lower frame rate when encoding the past frame and the one or more past frames.

[0111] At step 618, the video processing system 100 may encode a past frame set at the calculated frame rate (e.g., via the encoder engine 124). The past frame set may include the past frame from step 614 and one or more past frames from step 616. The video processing system 100 may encode the past frame set to achieve a slow motion effect when playing back the frames at a normal or lower frame rate. The video processing system 100 may adaptively encode the past frame set based on the detected motion and the calculated frame rate, which may allow the video processing system 100 to encode different frame rates for different frames with different amounts of motion.

[0112] At step 620, the video processing system 100 may play the encoded frames at a lower frame rate than the frame rate used to encode the frames (e.g., via the video player 510). This will produce a slow motion effect of the frames, which will vary based on the amount of motion in the frames as previously explained.

[0113] Figure 7An example method 700 for automatic slow motion video recording using an adaptive frame rate is shown. At step 702, the method 700 may include storing an ordered sequence of frames captured by the image sensor 102 in the buffer 104. In some examples, the video processing system 100 may receive frames captured by the image sensor 102, process the frames via the image processing engine 120, and store the processed frames in the frame buffer 104 for subsequent processing (e.g., preview, encoding, storage, playback, etc.).

[0114] At step 704, method 700 may include sending a first frame in an ordered sequence of frames from buffer 104 at a time interval (e.g., to display device 108) for display. In some examples, video processing system 100 may send different frames from buffer 104 to display device 108 at a particular frame rate, such as every n milliseconds, for display. Video processing system 100 may send different frames to display device 108 for use by display device 108 in rendering the preview. In some examples, each different frame sent to display device 108 may be a newer frame than a previous frame sent to display device 108. Display device 108 may receive each frame from video processing system 100 and use the received frame to render or display the preview.

[0115] At step 706, method 700 may include calculating an amount of motion between the second frame in buffer 104 (e.g., 302B or 406) and the first frame in buffer 104 (e.g., 302A or 404). In some cases, the amount of motion between the first frame in buffer 104 and the second frame in buffer 104 may be calculated based on a comparison of the first frame to the second frame. For example, the amount of motion may be calculated based on an image velocity and / or an image displacement between the first frame and the second frame.

[0116] In some examples, the amount of motion may be calculated based on one or more optical flow or block matching techniques. In other examples, the amount of motion may be calculated based on any other motion estimation technique, such as, for example but not limited to, phase correlation and frequency domain methods, pixel recursive methods, parametric motion estimation, or multi-resolution methods.

[0117] In some cases, the first frame may be a newer frame sent to the display device at a first time, and the second frame may be an older frame sent to the output device at a second time after the first time. In some cases, for example, the first time and the second time may be determined based on a time interval.

[0118] At step 708, method 700 may include determining a frame rate for encoding a set of frames from the ordered sequence of frames in buffer 104 based on the amount of motion, the set of frames including a second frame (e.g., 302B or 406) in buffer 104 and one or more frames (e.g., 304A) sequentially between the first frame and the second frame (e.g., one or more intermediate frames). In addition, the frame rate determined when the calculated amount of motion is large may be higher than the frame rate determined when the calculated amount of motion is small. In other words, the frame rate may be adaptively determined based on the amount of motion, such that a higher amount of motion may result in a higher frame rate, and vice versa.

[0119] At step 710, method 700 may include encoding the frame set at the determined frame rate. Method 700 herein may generate an encoded frame set having a frame rate calculated based on the amount of motion (e.g., 206). In addition, the frame rate used to encode the frame set may be adaptive based on the motion detected as described above.

[0120] In some examples, calculating the amount of motion, determining the frame rate, and encoding the set of frames may be performed as one or more new frames are received from image sensor 102 and / or stored in frame buffer 104. Thus, the calculating, determining, and encoding steps may be performed on demand or in real time (or near real time) as video of the frames is captured by image sensor 102 and recorded by video processing system 100.

[0121] In some cases, method 700 may include playing the encoded frame set at a playback frame rate that is lower than the frame rate determined for the frame set (and used to encode the frame set) (e.g., via video player 510). By playing the frame set at a playback frame rate that is lower than the frame rate determined for the frame set, method 700 may produce a slow motion effect for the frame set based on the determined frame rate and the playback frame rate.

[0122] In some cases, method 700 may include storing a video including a sequence of frames. The video may include a set of frames encoded at the frame rate determined at step 708 and one or more additional sets of frames encoded at a frame rate lower than the frame rate associated with the set of frames. Thus, the stored video may include different sets of frames adaptively encoded at different frame rates to produce different slow motion effects. Furthermore, the smaller frame rate used to encode the one or more additional sets of frames may be based on the corresponding motion captured in the one or more additional sets of frames.

[0123] In some cases, method 700 may include storing additional frames captured by image sensor 102 in buffer 104; sending a different one of the additional frames in buffer 104 to display device 108 at a time interval; calculating an amount of motion between a newer frame from the additional frames and an older frame from the additional frames; determining a different frame rate for encoding a set of additional frames from the additional frames in buffer 104 based on the amount of motion; and encoding the set of additional frames at the determined different frame rate.

[0124] The additional set of frames may include an older frame from the additional frames and one or more frames sequentially between the newer frame and the older frame. Furthermore, in some cases, the amount of motion calculated between the newer frame and the older frame may be higher than the amount of motion calculated between the first frame and the second frame at step 706. Here, the different frame rate determined for encoding the additional set of frames may be higher than the frame rate determined at step 708 for encoding the set of frames from the ordered sequence of frames.

[0125] In other cases, the amount of motion calculated between the newer frame and the older frame may be lower than the amount of motion calculated between the first frame and the second frame at step 706. Here, the different frame rate determined for encoding the additional set of frames may be lower than the frame rate determined for encoding the set of frames from the ordered sequence of frames at step 708. As previously described, such differences in frame rates may be adaptively calculated based on differences in motion detected for associated frames.

[0126] In some examples, determining a frame rate for encoding a set of frames based on the amount of motion may include: determining a first frame rate for encoding the set of frames when the calculated amount of motion is greater than a motion threshold, and determining a second frame rate for encoding the set of frames when the calculated amount of motion is less than the motion threshold. In this example, the second frame rate may be lower than the first frame rate.

[0127] In some examples, determining a frame rate for encoding a set of frames from an ordered sequence of frames in a buffer may include determining a first frame rate when the amount of motion includes a first level of motion, determining a second frame rate when the amount of motion includes a second level of motion that is higher than the first level of motion, and / or determining a third frame rate when the amount of motion includes a third level of motion that is higher than the second level of motion. In some cases, the third frame rate may be higher than the first frame rate and the second frame rate, and the second frame rate may be higher than the first frame rate. Additionally, in some examples, encoding the set of frames may include encoding the set of frames at a first frame rate when the amount of motion includes a first level of motion, encoding the set of frames at a second frame rate when the amount of motion includes a second level of motion, and / or encoding the set of frames at a third frame rate when the amount of motion includes a third level of motion.

[0128] In some examples, flowchart 600 and method 700 may be performed by one or more computing devices or apparatuses. In one illustrative example, flowchart 600 and method 700 may be performed by Figure 1 The video processing system 100 shown and / or having Figure 8 The computing device architecture 800 shown in the figure can be executed by one or more computing devices. In some cases, the computing device or device may include a processor, a microprocessor, a microcomputer, or other components of a device configured to perform the steps of flowchart 600 and method 700. In some examples, the computing device or device may include one or more sensors configured to collect sensor data. For example, the computing device may include a head-mounted display, a mobile device, a camera, or other suitable device. In some examples, the computing device or device may include a camera configured to capture one or more images or videos. In some cases, the computing device may include a display for displaying images and / or videos. In some examples, one or more sensors and / or cameras are separated from the computing device, in which case the computing device receives the sensor data. The computing device may also include a network interface configured to transmit data.

[0129] Flowchart 600 and method 700 are illustrated as a logical flow diagram, the operations of which represent a series of operations that can be implemented using hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the operation. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0130] Additionally, flowchart 600 and method 700 may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, by hardware, or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0131] Figure 8 An example computing device architecture 800 is shown that can implement various techniques described herein. For example, the computing device architecture 800 can implement Figure 1 800 and performs tracking, positioning, mapping, and rendering operations as described herein. The components of the computing device architecture 800 are shown as being in electrical communication with each other using a connection 805 such as a bus. The example computing device architecture 800 includes a processing unit (CPU or processor) 810 and a computing device connection 805 that couples various computing device components including a computing device memory 815 such as a read-only memory (ROM) 820 and a random access memory (RAM) 825 to the processor 810.

[0132] The computing device architecture 800 may include a cache of high-speed memory directly connected to the processor 810, adjacent to the processor 810, or integrated as part of the processor 810. The computing device architecture 800 may copy data from the memory 815 and / or the storage device 830 to the cache 812 for fast access by the processor 810. In this way, the cache can provide a performance improvement that avoids delays in the processor 810 while waiting for data. These and other modules can control or be configured to control the processor 810 to perform various actions. Other computing device memories 815 are also available. The memory 815 may include a variety of different types of memory with different performance characteristics. The processor 810 may include any general-purpose processor and hardware configured to control the processor 810 or software services such as service 1 832, service 2 834, and service 3 836 stored in the storage device 830, as well as a dedicated processor in which software instructions are incorporated into the processor design. The processor 810 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0133] To enable user interaction with the computing device architecture 800, the input device 845 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 835 can also be one or more of a plurality of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some instances, a multimodal computing device can enable a user to provide multiple types of input to communicate with the computing device architecture 800. The communication interface 840 can generally govern and manage user input and computing device output. There is no limitation on operation on any particular hardware arrangement, and therefore the basic features here can be easily replaced with improved hardware or firmware arrangements when developed.

[0134] The storage device 830 is a non-volatile memory and can be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM) 25, a read-only memory (ROM) 820, and a mixture thereof. The storage device 830 may include services 832, 834, 836 for controlling the processor 810. Other hardware or software modules are contemplated. The storage device 830 may be connected to the computing device connection 805. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer-readable medium connected to the necessary hardware components (such as the processor 810, the connection 805, the output device 835, etc.) to perform the function.

[0135] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include the following: carrier waves and / or temporary electronic signals propagated wirelessly or on a wired connection. Examples of non-transitory media may include, but are not limited to: disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. Computer-readable media may have codes and / or machine-executable instructions stored thereon, which may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent via any appropriate means including memory sharing, message passing, token passing, network transmission, etc.

[0136] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0137] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those of ordinary skill in the art that these embodiments can be implemented without these specific details. For clarity of explanation, in some cases, the technology herein can be presented as a separate functional block including the following functional blocks, which include equipment, device components, steps or routines in the method embodied in software, or a combination of hardware and software. In addition to the components shown in the figures and / or described herein, additional components can also be used. For example, circuits, systems, networks, processes, and other components can be shown as components in block diagram form, so as not to obscure these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures, and techniques may be shown as not having unnecessary details, so as to avoid obscuring these embodiments.

[0138] The above text may describe each embodiment as a process or method, and the process or method is depicted as a flow chart, a schematic flow chart, a data flow chart, a structure diagram or a block diagram. Although the flow chart can describe the operation as a sequential process, many of these operations can be performed in parallel or simultaneously. In addition, the order of the operation can be rearranged. The process is terminated after its operation is completed, but may have additional steps that are not included in the figure. Processing can correspond to a method, a function, a process, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0139] The process and method according to the above examples can be implemented using computer executable instructions, which are stored in a computer readable medium or can be obtained from a computer readable medium in other ways. Such instructions may include, for example, instructions or data, which cause a general-purpose computer, a special-purpose computer or a processing device to execute or otherwise be configured to execute a specific function or a specific set of functions. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, etc.

[0140] The equipment implementing the process and method according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof, and may adopt any of a variety of form factors. When implemented with software, firmware, middleware or microcode, the program code or code segment (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc., of laptop computers, smart phones, mobile phones, tablet devices, or other small form factors. The functions described herein may also be embodied in peripheral devices or plug-in cards. By way of additional example, this function may also be implemented on a circuit board between different chips or different processes executed in a single device.

[0141] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0142] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative embodiments of the present application have been described in detail herein, it should be understood that the inventive concept can be embodied and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations, except for variations limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or collectively. In addition, without departing from the broader spirit and scope of this specification, the embodiments can be used in any number of environments and applications other than the environments and applications described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that in an alternative embodiment, the method can be performed in an order different from the described order.

[0143] It will be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0144] When a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing circuits or other hardware to perform the operation, by programming a programmable circuit (e.g., a microprocessor or other appropriate circuit) to perform the operation, or any combination thereof.

[0145] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other appropriate communication interface).

[0146] Claim language or other language that recites "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that recites "at least one of A and B" means A, B, or A and B. In another example, claim language that recites "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that recites "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0147] Various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, frames, modules, circuits and steps have been generally described around their functions above. Whether this function is implemented as hardware or software depends on specific application and the design constraints imposed on the entire system. Technicians can implement the described functions in different ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the scope of the application.

[0148] The technology described herein can also be implemented with electronic hardware, computer software, firmware, or any combination thereof. Such technology can be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device mobile phone, or an integrated circuit device with multiple uses (including applications in wireless communication devices and other devices). Any features described as modules or components can be implemented together in an integrated logic device, or implemented as separate but interoperable logic devices. If implemented with software, the technology can be implemented at least in part by a computer-readable data storage medium, which includes a program code, and the program code includes instructions for executing one or more of the above methods when executed. The computer-readable data storage medium can form a part of a computer program product, and the computer program product can include packaging materials. The computer-readable medium can include a memory or a data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, etc. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium (such as a propagated signal or wave) that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0149] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any technology described in this disclosure. A general-purpose processor can be a microprocessor, but in an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein can refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the technology described herein.

Claims

1. A method for encoding, comprising: storing an ordered sequence of frames captured by an image sensor in a buffer; sending a first frame of the ordered sequence of frames from the buffer for display at a time interval; calculating an amount of motion between a second frame in the buffer and the first frame in the buffer; Determining a frame rate for encoding a set of frames from the ordered sequence of frames in the buffer based on the amount of motion, the set of frames including the second frame in the buffer and a number of frames sequentially between the first frame and the second frame, the number of frames being selected based at least in part on the determined frame rate for encoding the set of frames, wherein determining the frame rate for encoding the set of frames from the ordered sequence of frames in the buffer includes at least one of: determining a first frame rate when the amount of motion includes a first level of motion; determining a second frame rate that is higher than the first frame rate when the amount of motion includes a second level of motion that is higher than the first level of motion; and determining a third frame rate that is higher than the second frame rate when the amount of motion includes a third level of motion that is higher than the second level of motion; and The set of frames is encoded at the determined frame rate.

2. The method of claim 1, further comprising displaying the frame set at a playback frame rate lower than the frame rate determined for the frame set.

3. The method of claim 2, wherein: Displaying the frame set at the playback frame rate that is lower than the frame rate determined for the frame set includes generating a slow motion effect for the frame set based on the determined frame rate and the playback frame rate.

4. The method of claim 1, wherein: The amount of motion between the first frame and the second frame is calculated based on a comparison of the first frame and the second frame.

5. The method of claim 1, wherein: The amount of motion between the first frame and the second frame is calculated based on one of an optical flow technique or a block matching technique.

6. The method of claim 1, further comprising: A video including the sequence of frames is stored, the video including the set of frames encoded at the determined frame rate and at least one additional set of frames encoded at a frame rate lower than the frame rate associated with the set of frames, the lower frame rate based on corresponding motion captured in the at least one additional set of frames.

7. The method of claim 1, wherein: Calculating the amount of motion, determining the frame rate, and encoding the frame set are performed when one or more new frames are subjected to at least one of the following operations: receiving from said image sensor; and is stored in the buffer.

8. The method of claim 1, further comprising presenting a preview including at least one of the first frame and the second frame using a display device.

9. The method of claim 8, wherein: At least one or more other frames in the ordered sequence of frames are not included in the preview, and wherein the at least one or more other frames are not presented in a different preview.

10. The method of claim 1, wherein: Encoding the frame set includes at least one of: encoding the frame set at the first frame rate when the motion amount includes the first level of motion, encoding the frame set at the second frame rate when the motion amount includes the second level of motion, and encoding the frame set at the third frame rate when the motion amount includes the third level of motion.

11. An apparatus for encoding, comprising: Memory; as well as One or more processors configured to: storing an ordered sequence of frames captured by an image sensor in a buffer; sending a first frame of the ordered sequence of frames from the buffer for display at a time interval; calculating an amount of motion between a second frame in the buffer and the first frame in the buffer; Determining a frame rate for encoding a set of frames from the ordered sequence of frames in the buffer based on the amount of motion, the set of frames including the second frame in the buffer and a number of frames sequentially between the first frame and the second frame, the number of frames being selected based at least in part on the determined frame rate for encoding the set of frames, wherein determining the frame rate for encoding the set of frames from the ordered sequence of frames in the buffer includes at least one of: determining a first frame rate when the amount of motion includes a first level of motion; determining a second frame rate that is higher than the first frame rate when the amount of motion includes a second level of motion that is higher than the first level of motion; and determining a third frame rate that is higher than the second frame rate when the amount of motion includes a third level of motion that is higher than the second level of motion; and The set of frames is encoded at the determined frame rate.

12. The device of claim 11, wherein: The one or more processors are configured to display the frame sets at a playback frame rate that is lower than the frame rate determined for the frame sets.

13. The device of claim 12, wherein: Displaying the set of frames at the playback frame rate that is lower than the frame rate determined for the set of frames includes generating a slow motion effect for the set of frames based on the determined frame rate and the playback frame rate.

14. The device of claim 11, wherein: The amount of motion between the first frame and the second frame is calculated based on a comparison of the first frame and the second frame.

15. The device of claim 11, wherein: The amount of motion between the first frame and the second frame is calculated based on one of an optical flow technique or a block matching technique.

16. The device of claim 11, wherein: The one or more processors are configured to: A video including the sequence of frames is stored, the video including the set of frames encoded at the determined frame rate and at least one additional set of frames encoded at a frame rate lower than the frame rate associated with the set of frames, the lower frame rate based on corresponding motion captured in the at least one additional set of frames.

17. The device of claim 11, wherein: Calculating the amount of motion, determining the frame rate, and encoding the frame set are performed when one or more new frames are subjected to at least one of the following operations: receiving from said image sensor; and is stored in the buffer.

18. The device of claim 11, wherein: The one or more processors are configured to present a preview including at least one of the first frame and the second frame using a display device.

19. The device of claim 18, wherein: At least one or more other frames in the ordered sequence of frames are not included in the preview, and wherein the at least one or more other frames are not presented in a different preview.

20. The device of claim 11, wherein: Encoding the frame set includes at least one of: encoding the frame set at the first frame rate when the motion amount includes the first level of motion, encoding the frame set at the second frame rate when the motion amount includes the second level of motion, and encoding the frame set at the third frame rate when the motion amount includes the third level of motion.

21. The apparatus of claim 11, comprising at least one of a mobile phone and a camera system.

22. A non-transitory computer-readable medium comprising: instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: storing an ordered sequence of frames captured by an image sensor in a buffer; sending a first frame of the ordered sequence of frames from the buffer for display at a time interval; calculating an amount of motion between a second frame in the buffer and the first frame in the buffer; Determining a frame rate for encoding a set of frames from the ordered sequence of frames in the buffer based on the amount of motion, the set of frames including the second frame in the buffer and a number of frames sequentially between the first frame and the second frame, the number of frames being selected based at least in part on the determined frame rate for encoding the set of frames, wherein determining the frame rate for encoding the set of frames from the ordered sequence of frames in the buffer includes at least one of: determining a first frame rate when the amount of motion includes a first level of motion; determining a second frame rate that is higher than the first frame rate when the amount of motion includes a second level of motion that is higher than the first level of motion; and determining a third frame rate that is higher than the second frame rate when the amount of motion includes a third level of motion that is higher than the second level of motion; and The set of frames is encoded at the determined frame rate.

23. The non-transitory computer readable medium of claim 22, further comprising instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to: The frame set is displayed at a playback frame rate lower than the frame rate determined for the frame set, wherein displaying the frame set at the playback frame rate lower than the frame rate determined for the frame set includes generating a slow motion effect for the frame set based on the determined frame rate and the playback frame rate.

24. The non-transitory computer readable medium of claim 22, wherein: Encoding the frame set includes at least one of: encoding the frame set at the first frame rate when the motion amount includes the first level of motion, encoding the frame set at the second frame rate when the motion amount includes the second level of motion, and encoding the frame set at the third frame rate when the motion amount includes the third level of motion.

25. An apparatus for encoding, comprising: Components for performing the method of any one of claims 1-10.

Images