Real-time video special effects system and method

Abstract

SOLUTION: To provide a real-time video special effects system 10 for recording a video while applying special effects in real-time before and / or during recording, in which an image processor receives raw video data at a native speed rate from a camera 12 or a remote video stream and an input from an interface 30, determines whether the native speed rate of the raw video data is to be modified, and if so, modifies at least one frame of the raw video data to create modified video data at a modified speed rate different from the native speed rate, generates output video recording data, and writes it to a memory 18.EFFECT: Video from a camera or video feed can be recorded continuously at different speed rates without changing the operation or settings of the camera 12.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This application claims priority from and is a continuation of U.S. Patent Application No. 16,173,033, filed in the U.S. Patent and Trademark Office on October 29, 2018, the entire contents of which are incorporated herein by reference for continuity of disclosure.

[0002] The present technology relates to a real-time video special effects system and method for use in connection with controlling or creating special effects in video recordings while the recording is in progress, and more particularly to a system and method for live or real-time decoding and modifying the speed rate of a video stream from a device before it is recorded and written to long-term memory storage without post editing. [Background technology]

[0003] Modern video formats use a wide variety of frame rates. Film, most of which was shot at 24 frames per second, cannot be displayed at its native frame rate and requires pull-down conversion, often resulting in "judder." For example, to convert 24 frames / second to 60 frames / second, odd frames are doubled and even frames are tripled, resulting in choppy motion. Other conversions result in uneven frame doubling as well. Newer video standards support 120, 240, or 300 frames / second, so frames can be multiplied evenly for common frame rates such as 24 frames per second (fps) film and 30 fps video, as well as 25 and 50 fps video on a 300 fps display. These standards also support video that natively supports higher frame rates and video with interpolated frames between native frames.

[0004] A device's native camera application (app) running on an electronic device such as a smartphone can record video at a fixed time and then process the video data stream to generate slow motion, speed up or time lapse. However, these known systems or methods do not utilize a user interface that allows the user to manually control "time special effects" in the footage in real time while recording. For example, the native camera app on the Samsung Galaxy S9+ has a special effect feature where the camera changes the number of frames per second when the app detects that an object has passed inside a portion of the screen, in this case as indicated by a box outline in the center of the screen.

[0005] Third party apps such as Instagram®, Facebook® and Snapchat® use the mobile device camera, but these apps do not allow the app user to slow down or speed up the recording speed in real time while the app is recording.

[0006] More feature-rich camera apps, such as FILMiC Pro®, allow users to preset capture and playback frame rate speeds independently of each other, allowing for slow-motion and speed-up effects in the final video.

[0007] Optical flow time remapping, as it is known in Premiere Pro CC 2015, allows users to achieve smooth speed and frame rate changes by interpolating missing frames. Optical flow interpolation adjusts the speed of clips that contain objects with no motion blur, moving in front of a mostly stationary background as opposed to the moving object.

[0008] Motion Interpolation or Motion Compensated Frame Interpolation (MCFI) is a form of video processing that creates intermediate animation frames between existing frames by interpolation in order to make animation more fluid and compensate for motion blur on displays.

[0009] It can be understood that motion interpolation can be used to reduce motion blur produced by camera pans and camera shake, making the image clearer. It can also be used to increase the apparent frame rate of video game software for a more realistic feel, although it may have the undesirable side effect of introducing input lag. This "video look" was intentionally created by the VidFIRE (Video Field Interpolation Restoration Effect) technique, which restores archival television programs that survive only as film telerecordings. VidFIRE is a restoration technique that aims to restore video-like movement to footage shot with television cameras that now only exist in telerecording-based format. The main differences between achieving artificially high frame rates and naturally high frame rates (by interpolation versus in-camera) are that the latter do not suffer from the aforementioned artifacts, contain more accurate (or "true-to-life") image data, and require more storage space and bandwidth, since the frames are not generated in real time.

[0010] Motion compensation is an algorithmic technique used to predict frames in a video to give previous and / or future frames by taking into account the movement of the camera and / or objects in the video. It is employed in encoding video data for video compression, e.g. in the generation of MPEG-2 files. Motion compensation represents an image in terms of a transformation from a reference image to the current image. The reference image may be either past in time or future. Compression efficiency can be improved if an image can be accurately synthesized from previously transmitted / stored images.

[0011] Motion compensation takes advantage of the fact that for many frames of a video, the only difference between one frame and another is often the result of either camera movement or movement of objects within the frame. This means that when you look at a video file, most of the information that describes one frame will be the same as the information used for the next frame.

[0012] With motion compensation, the video stream contains a number of complete (reference) frames, during which only the information needed to transform the previous frame into the next is stored.

[0013] Frame blending can be another technique known to those skilled in the art. In some footage, using optical flow to generate smoother motion may not produce the desired results. In these situations, one of the other time interpolation options, frame sampling or frame blending, can be used. Frame sampling repeats and removes frames as necessary until the desired speed is reached. Frame blending repeats frames and also blends between frames as necessary to smooth the motion.

[0014] While the above-mentioned devices meet their respective specific objectives and requirements, the above-mentioned patents are directed to a real-time video special effects system that allows for the creation of special effects in a recording while the recording is in progress. And the method is not described. Summary of the Invention [Means for solving the problem]

[0015] In view of the aforementioned drawbacks inherent in known types of video speed rate change systems and methods currently existing in the prior art, the present technology provides a novel real-time video special effects system and method that overcomes one or more of the aforementioned drawbacks and difficulties of the prior art. Thus, the object of the present technology, as described in more detail below, is to provide a new and novel real-time video special effects system and method having all of the advantages of the prior art discussed above, as well as many novel features that, singly or in any combination thereof, result in a real-time video special effects system and method not anticipated, obvious, suggested, or even implied by the prior art.

[0016] According to one aspect of the present invention, there is provided a video capture and display system for controlling special effects manipulation of live video recording data in real time, the system comprising: a camera configured to capture video of a real world scene, at least one memory, and at least one processing unit operatively connected or connectable to the camera, a graphical user interface, and the at least one memory, the at least one processing unit configured to play the captured video at a normal speed on the graphical user interface, and to change the playback speed of the captured video from the normal playback speed to a modified playback speed in response to user input received by the graphical user interface.

[0017] According to another aspect, a method is provided for controlling special effects manipulation of live video capture data in real time, the method including capturing video of a real world scene with a camera, playing the captured video on a graphical user interface at normal speed, and responsive to user interface input received by the graphical user interface in real time during recording, changing the playback speed of the captured video on the graphical user interface from the normal playback speed displayed on the graphical user device during live recording to a modified playback speed displayed on the graphical user device during recorded playback.

[0018] According to another aspect of the present technology, the present technology can be essentially a video recording system for controlling special effects operation of live video recording data in real time. The video recording system includes at least one camera configured or configurable to provide raw video data at a native speed, at least one memory, and at least one processing unit in operative communication with the camera and the memory. The raw video data can correspond at least in part to images captured by the camera in real time. The processing unit is configured or configurable to receive the raw video data from the camera in real time. Then, determine whether the input is associated with a change in the native speed rate of the raw video data, and if so, modify at least one frame of the raw video data in real time while receiving the raw video data from the camera to create modified video data at one or more modified speed rates different from the native speed rate. And write at least one output video recording data to the memory, where the output video recording data is one of raw video data at a native speed rate, modified video data at a modified speed rate, and a combination of the raw video data and the modified video data.

[0019] According to another aspect of the present technology, the present technology can be a client-side electronic system for controlling special effects manipulation of live video recording data in real time, the system can include a memory and a processor configured to store and execute software instructions, respectively. The instructions comprise a raw data receiving component configured or configurable to receive a request to obtain raw video data from the camera at a native speed rate, the raw video data at least partially corresponding to an image captured by the camera in real time; a speed rate determination component configured or configurable to receive at least one request to change the native speed rate of the raw video data from the camera to at least one modified speed rate; a frame modification component configured or configurable to, upon receiving the request to change the native speed rate, identify at least one frame or location in the raw video data to be modified and modify the frame or location to create modified video data at a modified speed rate that is different from the native speed rate in real time while receiving the raw video data from the camera; and an output video recording data writing component configured or configurable to write output video recording data to one of the memories and a secondary memory in real time while receiving the raw video data from the camera, the output video recording data being selected from one of raw video data at the native speed rate, modified video data at a modified speed rate, and a combination of raw video data and modified video data.

[0020] According to another aspect of the present technology, the technology can be a method of controlling special effects manipulation of live video recording data in real time. The method includes receiving, by at least one processing unit, raw video data at a native speed rate corresponding at least in part to footage captured by a camera in real time, and receiving, by the processing unit, at least one input from at least one interface operatively associated with the processing unit. The method can include determining, by the processing unit, whether the input is associated with changing a native speed of the raw video data, and if so, modifying the raw video data in real time as the raw video data is received from the camera to create modified video data at one or more modified speed rates that are different from the native speed rate. The method can further include writing, by the processing unit, output video recording data to at least one memory, the output video recording data being one of raw video data at a native speed rate, modified video data at a modified speed rate, and a combination of raw video data and modified video data.

[0021] According to another aspect of the present technology, the present technology can be a system for controlling special effects manipulation of live video recording data in real time. The system includes at least one camera configured or configurable to provide raw video data at a native speed rate, at least a portion of which corresponds to images captured by the camera in real time, at least one memory unit, and at least one processing unit in operative communication with the camera and the memory unit. The processing unit is configured or configurable to receive the raw video data from the camera in real time and to receive input from at least one interface operatively associated with the processing unit. The processing unit is further configured or configurable to determine whether the input is associated with changing the native speed of the raw video data, and if so, to modify at least one frame of the raw video data in real time while receiving the raw video data from the camera to create modified video data at one or more modified speed rates that are different from the native speed rate. The processing unit is then configured or can be configured to write the output video recording data to at least one memory, the output video recording data being one of: unprocessed video data at the native speed rate, modified video data at the modified speed rate, and a combination of the unprocessed video data and the modified video data.

[0022] According to yet another aspect of the present technology, the technology can include an interface system including an interface in operative communication with at least one camera configured or configurable to provide raw video data at a native speed rate, at least one memory unit, and a processing unit in operative communication with at least one display, the raw video data corresponding at least in part to images captured by the camera. The interface can be associated with at least one computer readable storage medium in communication with the processing unit, or a computer system including at least one processor. The interface can be a graphical user interface including a portion configured or configurable to generate an input related to a desired speed rate of the raw video data received from the camera. The processing unit can be configured or configured to use the input to determine whether to modify the raw video data in real time while receiving the raw video data from the camera to generate modified video data having a speed different from the native speed of the raw video data. The interface can be configured or configured to display the output video recording data in real time while receiving the raw video data from the camera. The output video recording data is configured or can be configured to include a combination of raw video data and modified video data, with the transition between the raw video data and the modified video data depending on the input.

[0023] According to another aspect of the present technology, the technology may be a computer-implemented method for controlling real-time special effects manipulation of live video recording data. The method may include receiving, by at least one processing unit, raw video data at a native speed rate from a camera at the request of a user input from at least one interface operatively associated with the processing unit, the raw video data corresponding at least in part to images captured by the camera in real time. The method may include determining whether the input is related to changing the native speed rate of the raw video data, and if so, modifying the raw video data in real time as the raw video data is received from the camera to create modified video data having one or more modified speed rates that differ from the native speed rate. The method may further include writing at least one output video recording data to a memory, the output video recording data being one of raw video data at a native speed rate, modified video data at a modified speed rate, and a combination of raw video data and modified video data.

[0024] In some embodiments of the present technology, the method may include preventing unprocessed video data from being written to memory from the camera.

[0025] In some embodiments of the present technology, the input is not associated with a native speed rate change and raw video data is written to memory.

[0026] In some embodiments of the present technology, the method may include determining whether the modified speed rate is less than the native speed rate, and if so, modifying the raw video data includes adding at least one new frame to the raw video data to create the modified video data.

[0027] In some embodiments of the present technology, the method may include: adding the new frame includes copying at least one unprocessed frame to create the new frame and adding the new frame to the unprocessed video data adjacent to the unprocessed frame.

[0028] In some embodiments of the present technology, the new frame can be multiple new frames, each of which is a copy of at least one unprocessed frame from the raw video data.

[0029] In some embodiments of the present technology, each new frame can be added to the raw video data adjacent to the raw frame or adjacent to a second raw frame of the raw video data.

[0030] In some embodiments of the present technology, the method may include determining whether the modified speed rate is greater than the native speed rate, and if so, modifying the raw video data includes removing at least a first unprocessed frame from the raw video data to create the modified video data.

[0031] In some embodiments of the present technology, the method may include: selecting the first unprocessed frame for removal; and removing the first unprocessed frame from the unprocessed video data to create modified video data.

[0032] In some embodiments of the present technology, the interface can be a graphical user interface that includes a portion that is configured or can be configured to generate input associated with a native speed rate or a modified speed rate, and the interface can be configured or can be configured to display the output video recording data.

[0033] In some embodiments of the present technology, the output video recording data is a combination of raw video data and modified video data, where the modified video data is configured or configurable to include multiple subsets, each having a speed rate that is input dependent, with transitions between the raw video data and any one of the subsets or between the subsets. The output video recording data is displayed in a graphical user interface in real time as the raw video data is received from the camera.

[0034] There are, of course, additional features of the technology that will be described hereinafter and which will form the subject matter of the claims appended hereto. [Brief description of the drawings]

[0035] [Figure 1] 1 is a block diagram of one embodiment of a real-time motion special effects system constructed in accordance with the principles of the present technique; [Diagram 2] FIG. 2 is a block diagram of a real-time recording speed control method according to the present technology. [Figure 3A] 1 is a depiction of a sequence of video frames illustrating an example of a native frame rate in accordance with the teachings of the present technology. [Figure 3B] 1 is a depiction of a sequence of video frames showing an example of frame dropping following a fast motion speed of twice the native frame rate. [Figure 3C] 1 is a depiction of a sequence of video frames showing an example of frame dropping following a fast motion speed of three times the native frame rate. [Figure 3D] 11 is a depiction of a sequence of video frames illustrating an example of adding frames using Frame Copy according to a slow-motion speed of -2x the native frame rate. [Figure 3E] 11 is a depiction of a sequence of video frames illustrating an example of adding frames using Frame Copy according to a slow motion speed of -3 times the native frame rate. [Figure 3F]11 is a depiction of a sequence of video frames illustrating an example of using frame blending to add frames according to a slow-motion speed of -2 times the native frame rate. [Figure 3G] 11 is a depiction of a sequence of video frames illustrating an example of adding frames using frame blending at a slow-motion speed of -3 times the native frame rate. [Figure 4] FIG. 1 illustrates an exemplary integrated circuit chip incorporated into an electronic computing device that may be used to implement embodiments of the present technology. [Diagram 5] FIG. 1 illustrates an exemplary electronic computing device that may be used to implement embodiments of the present invention. [Figure 6] 1 is a flow chart illustrating an example of an overall process including available subroutines. [Figure 7] 13 is a flow chart illustrating an example of a main process that first determines whether speed-up and / or slow-down special effects should be utilized in real time. [Figure 8] 13 is a flowchart illustrating an example of a write video stream subroutine associated with the present technology. [Figure 9] 13 is a flowchart showing an example of a special effect application subroutine according to the present technology. [Figure 10] 10 is a flowchart illustrating an example of a speed-up subroutine according to the present technology. [Figure 11] 11 is a flow chart illustrating an example of a frame dropping subroutine for simulating fast motion associated with the present technology. [Figure 12] 13 is a flowchart illustrating an example of an advanced slow motion subroutine associated with the present technology. [Figure 13] 13 is a flowchart illustrating an example of a frame addition subroutine for simulating slow motion according to the present technology. [Figure 14] 13 is a flow chart illustrating an example of a variable high recording fps subroutine (120 fps) for simulating slow motion related to the present technology. [Figure 15] 11 is a flow chart illustrating an example of a constant frame rate slow motion subroutine associated with the present technology. [Figure 16] 13 is a flow chart illustrating an example of a constant high recording fps subroutine (60 fps) for simulating slow motion associated with the present technology. [Figure 17] 13 is a flow chart illustrating an example of a constant high recording fps subroutine (120 fps) for simulating slow motion associated with the present technology. [Figure 18] 13 is a flow chart illustrating an example of a constant high recording fps subroutine (240 fps) for simulating slow motion associated with the present technology. [Figure 19] 13 is a flowchart illustrating an example of an extreme slow motion subroutine related to the present technology. [Figure 20] 1 is a flow chart of an example of a time expansion and compression subroutine for simulating slow motion and fast motion associated with the present technology. [Figure 21] 21 is a depiction of a series of related recorded and played back video segments per time in seconds, illustrating the time compression example of FIG. 20. [Figure 22] 1 is a flow chart of an example of recording with variable playback rate to simulate slow motion and fast motion related to the present technology; [Figure 23] 23 is a flow chart of an example of a playback device playing a video file created by an application using the algorithm of FIG. 22 to simulate slow motion and fast motion associated with the present technology. [Figure 24] 1 is a flowchart illustrating an example of a possible process by a user utilizing a user interface related to the present technology. [Diagram 25] 1 is a sample graphical user interface (GUI) screenshot of an interface system of the present technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The present technology will be better understood, and other objects will become apparent, from a consideration of the detailed description set forth below, in which such description refers to the accompanying drawings in which dashed and dotted lines indicate environmental structures, which do not form a part of the present technology as claimed.

[0037] Identical reference numbers refer to the same parts throughout the various figures.

[0038] In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular embodiments, procedures, techniques, etc., in order to provide a thorough understanding of the present technology. However, it will be apparent to one skilled in the art that the present technology may be practiced in other embodiments that depart from these specific details.

[0039] It is known that a video recording consists of a series of frames or groups of images displayed at a speed rate to create movement. These images or frames of video can be characterized as digital frame data, which can be buffered as the video is played back. The frame rate (expressed in frames per second or fps) is the frequency (rate) at which these successive frames are displayed on a display. This applies equally to film and video cameras, computer graphics, and motion capture systems. The frame rate is also called the frame frequency and is expressed in Hertz.

[0040] Real-time recording and / or playback of video is typically performed at a rate of 30 fps. In some situations, it is desired to speed up or slow down the playback of video. This is typically done while maintaining the frames per second of recording and playback at 30 fps per second to maintain compatibility with existing components such as display devices. For example, if a viewer wishes to speed up the playback of video by a certain percentage from the standard real-time playback speed while still maintaining 30 fps, a certain number of frames of information or data must be played back in a time divided into 30 frames. A scheme to achieve this is to skip one frame from the recorded video for every predetermined number of frames so that the appropriate number of frames of the video are displayed at 30 fps. It is noted that these known systems and methods are provided as post-recording processes that skip frames from a 30 fps recording. The recording is initially written to memory in real time at 30 fps and has no special effects.

[0041] The present technique solves the problem of requiring "post production editing" to insert time-altering special effects, which can be time- and resource-costly, especially for amateur filmmakers.

[0042] Furthermore, the present technology solves the problem that once the motion recording speed is preset to either fast motion or slow motion, the user cannot adjust the motion recording speed in real time during the recording process.

[0043] Furthermore, the present technology solves the problem that when the motion recording speed is preset, the user cannot continuously adjust the motion recording speed in real time during recording, and cannot change from fast motion to slow motion.

[0044] The present technique mitigates and solves the problem of requiring hardware support on all devices: by simulating slow motion using software algorithms, it is device independent and the resulting files are much smaller than hardware supported slow motion videos.

[0045] While the above-mentioned devices meet their respective particular objectives and requirements, none of the above-mentioned devices or systems describes a real-time video special effects system and method that allows for creating special effects in a video recording as it is being recorded. The present technology further overcomes one or more shortcomings associated with the prior art by adding or removing frames in real-time to a frame strip provided by a camera.

[0046] Additionally, there is no known interface that allows a user to change the recording speed or application time of special effects in real time while recording. Additionally, the scene must remain relatively stationary, with the camera not panning or tracking the action. The algorithms associated with this known system use motion sensors while the camera remains steadily fixed on the scene, and the subject must traverse the scene while the rest of the scene remains stationary.

[0047] The present technique can utilize a user interface associated with an electronic device to modify frames from the camera in real time prior to recording or storage.

[0048] There is a need for a new and novel real-time video special effects system and method that can be used to create special effects in video recordings while they are being recorded. In this regard, the present technology substantially meets this need. In this respect, the real-time video special effects system and method according to the present technology departs substantially from the conventional concepts and designs of the prior art, and in so doing provides an apparatus developed primarily for the purpose of creating special effects in video recordings while they are being recorded.

[0049] By using a user interface programmed into the device's apps running on supported operating systems and other embedded devices, users of the technology can create videos in "real-time" that include user-controllable variable time changes, so-called fast-motion or slow-motion special effects. The videos created are filmed in one shot, with all time-change commands entered in real-time during recording.

[0050] For exemplary purposes, the technique may utilize a video frame rate set at 30 fps, resulting in 30 frames per second during recording.

[0051] In some embodiments of the present technology, a user may utilize the technology's fast-forward option, which results in frames being dropped according to a set fast-forward rate (1x, 2x, 3x, etc.). If a user sets a 2x fast-forward video, the technology may cause the writer to add the first frame, skip the second, then write the third, then skip the fourth, etc. As a result, the recorded video will be at the preset fast-forward speed in real time while maintaining 30fps.

[0052] In some embodiments, the user can utilize the slow motion option of the technology, which results in the recorded video being slow-motion by adding the same frame twice and repeating this frame. For example, if the user sets 2x slow motion, the technology can add the first frame at the lighter and then add the same frame at the next time / frame slot. As a result, the recorded video will be at the preset slow motion speed in real time while maintaining 30fps.

[0053] The technology allows the user to control the recording speed and other camera settings of the recording device (and any other recording device) while recording using a custom user interface, and when the user plays back the video immediately after the technology algorithm processes the commands, the playback speed of the scene corresponds to the commands during recording. The technology achieves this through software simulation without increasing the frame rate of the recording device, and is device independent and works on any platform.

[0054] A further aspect of the technology is the possibility to increase the frame rate of the recording device while recording. This requires access to an application programming interface (API) that is limited in the number of supported hardware, and there is no industry standard API that limits the number of supported devices. The display shows the current time recording speed in 3x or -3x speed from normal speed (4x, 5x or more). The user can utilize the interface to control the recording speed.

[0055] This technology has many advantages, including an easy-to-use custom user interface, and allows users to add time-changing special effects to videos in real time while they are being recorded. This has the advantage over existing technologies that users can create videos with special effects (variable fast-motion and slow-motion recording speeds) while they are recording the video. This reduces time and costs as no separate video editing software is required to create videos with such special effects or to have a video editor edit and create an equivalent video. Users can view the video created with the special effects added immediately after recording is complete, and the device can process the addition of the special effects and automatically generate a new video with the special effects implemented in a short processing time.

[0056] Another advantage of having manual real-time user control of special effects is that the user can pan along with the movement of the scene, capturing the peak moment of the action, using continuously varying slow / fast motion as needed at the right time, and then returning to normal speed while the user is recording.

[0057] Yet another advantage is that the technique does not rely on hardware to achieve slow-motion or fast-motion special effects: software algorithms simulate the slow-motion or fast-motion movements.

[0058] Yet another advantage is that the manual user interface does not require the camera to remain stationary while pointing at a still scene for the AI ​​software to determine the "action" to apply a special effect.

[0059] Another advantage is that the technology can accept input from, but is not limited to, remote camera feeds, joysticks, retinal scanners, body suit controllers, subject gestures on a display, and haptic user interfaces.

[0060] In some embodiments, the technology can also be applied to add time correction special effects to existing videos. Using the familiarity and ease of use of a compatible device such as a smartphone or tablet, a user can control the time variable during playback and control and correct the value of the playback time value, in this case from -3x to 4x. It can be seen that the time correction element has additional functionality when digital processing technology advances sufficiently to allow for the interpolation of data and images between frames captured in a video.

[0061] When the user slides to the 4x side, the recording speed will play faster than normal, up to 4x faster. When the user slides to the -3x side, the recording speed will play slower than normal, up to 3x slower.

[0062] In some embodiments, the raw video data may include data such as, but not limited to, streaming video data, video, audio, depth, object identification, histograms, and combinations thereof.

[0063] In some aspects, the processing unit is configured or can be configured to prevent raw video data from being written from the camera to the memory unit such that the techniques can intercept the raw video data.

[0064] In some embodiments, the input can be one or more desired speed rate values ​​on which a modified speed rate is based, where the modified speed rate can be one of a speed rate less than the native speed rate or a speed rate greater than the native speed rate.

[0065] If the modified speed rate is less than the native speed rate, the processing unit is configured or can be configured to add at least one frame to the unprocessed video data to create the modified video data.

[0066] If the modified speed rate is greater than the native speed rate, the processing unit is configured or can be configured to remove at least one frame from the unprocessed video data to create the modified video data.

[0067] If the input is not a request to change native speed, the processing unit is configured or can be configured to retain all frames from the raw video data and write the raw video data to memory.

[0068] In some embodiments, the interface can be a graphical user interface that includes a portion that is configured or configurable to generate inputs related to native speed rates or modified speed rates. The graphical user interface can be configured or configurable to display output video recording data in real-time while receiving raw video data from the camera. The output video recording data includes a combination of raw video data and modified video data, and the transition between the raw video data and the modified video data is configured or can be configured to be dependent on the input. It can be appreciated that the interface can be or utilize a joystick.

[0069] In yet another aspect, the interface may be operatively associated with at least one computer-readable storage medium storing instructions that, when executed by a processing unit or processor of the computer system, send raw video data from the camera to the processing unit and memory unit in real time while receiving the raw video data from the camera, and write the raw video data from the processing unit to the memory unit, or apply at least one algorithm to the raw video data to create modified video data and write the modified video data from the processing unit to the memory unit.

[0070] According to yet another aspect of the present technology, the technology can be a method of recording video in real-time at one or more speed rates while receiving raw video data from a camera. The method can include receiving, by at least one processing unit, raw video data from a camera at a native speed in real-time while capturing images at least partially corresponding to the raw video data from the camera, and receiving input from at least one interface operatively associated with the processing unit. The method can include determining, by the processing unit, whether the input is associated with changing a native speed rate of the raw video data, and if so, modifying the raw video data in real-time while receiving the raw video data from the camera to create modified video data at one or more modified speed rates different from the native speed rate. The method can further include writing, by the processing unit, output video recording data to at least one memory, the output video recording data being one of raw video data at a native speed rate, modified video data at a modified speed rate, and a combination of the raw video data and the modified video data.

[0071] Some embodiments of the present technology may include determining whether the modified speed rate is less than the native speed rate, and if so, modifying the raw video data may include adding at least one new frame to the raw video data to create the modified video data.

[0072] In some embodiments, the method may include adding a new frame by copying at least one unprocessed frame to create the new frame, and adding the new frame to the unprocessed video data adjacent to the unprocessed frame.

[0073] In some embodiments, the new frames added may be multiple new frames, each of which is a copy of at least one unprocessed frame from the unprocessed video data, and the new frames are added to the unprocessed video data adjacent to the copied unprocessed frames.

[0074] In some embodiments, the method may include adding the new frame by frame blending at least two unprocessed frames to create a new frame, and adding the new frame to the unprocessed video data between the two unprocessed frames.

[0075] In some embodiments, the added new frame may be multiple new frames, each of which is a blend of at least two unprocessed frames from the raw video data, and the new frames are added to the raw video data between the blended unprocessed frames.

[0076] In some embodiments, each new frame may be added to the raw video data adjacent to the raw frame or adjacent to a second raw frame of the raw video data.

[0077] Some embodiments may include determining whether the modified speed rate is greater than the native speed rate, and if so, modifying the raw video data may include removing at least a first unprocessed frame from the raw video data to create the modified video data.

[0078] In some embodiments, deleting the first unprocessed frame may include selecting the first unprocessed frame to be deleted and then deleting the first unprocessed frame from the unprocessed video data to create a modified frame.

[0079] In some embodiments, the interface may be a graphical user interface that includes a configurable portion that is configured to generate input related to the native speed rate or the modified speed rate, and the interface is configured or can be configured to display the output video recording data.

[0080] Some embodiments may include the output video recording data being a combination of raw video data and modified video data, the modified video data being configured or configurable to include a plurality of subsets, each having a speed rate that is input dependent, and where the transition between the raw video data and any one of the subsets or between the subsets is input dependent, the output video recording data is displayed in real-time in a graphical user interface as the raw video data is received from the camera.

[0081] In some embodiments, the technology can include an extreme slow-motion subroutine at constant high recording fps, which can be utilized for slow-motion speed ranges of -8x and above by passing the video stream through unaltered, or by making a predetermined number of copies of each frame.

[0082] In some embodiments, the technology can include a segment time compression and expansion subroutine that provides algorithms for slow motion and fast motion by speeding up or slowing down the playback time during video processing after recording has stopped. This subroutine can set the device's recording and / or playback fps and set the video segment playback fps equal to the recording fps using an algorithm that utilizes, in part, the segment playback fps and the recording fps.

[0083] In some embodiments, the techniques can include a variable playback speed recording subroutine that provides an algorithm for slow-motion and fast-motion by speeding up or slowing down the playback frame rate during recording. The algorithm can generate a regular video with fast / slow-motion commands embedded in the video's metadata.

[0084] In some embodiments, the techniques may include a variable playback speed playback subroutine that provides an algorithm for playing video files with slow-motion and fast-motion special effects by speeding up or slowing down the playback frame rate during video playback.

[0085] There has thus been outlined rather broadly the features of the present technology in order to better understand the detailed description of the invention that follows, and to better appreciate the present contribution to the art.

[0086] Numerous objects, features and advantages of the present technology will become readily apparent to those skilled in the art upon reading the following detailed description of the technology, the embodiments of which are nevertheless illustrative when taken in conjunction with the accompanying drawings.

[0087] Thus, those skilled in the art will appreciate that the conception underlying the present invention may be readily utilized as a basis for the designing of other structures, methods and systems for carrying out the several purposes of the present technology, and the appended claims are intended to cover such equivalent constructions insofar as they do not depart from the spirit and scope of the present technology.

[0088] It is yet another object of the present technology to provide a real-time video special effects system and method for generating special effects in a video recording while it is being recorded, which allows a user to control the speed rate of the video while capturing the video from the camera, both before and while the recording is proceeding in real time.

[0089] These, together with other objects of the present technology, as well as the various features of novelty which characterize the present technology, are particularly pointed out in the claims annexed to and forming a part of this disclosure. For a better understanding of the present technology, its operating advantages, and certain objects attained by its use, reference should be made to the accompanying drawings and descriptive matter in which embodiments of the present technology are illustrated. Although several objects of the present technology are identified herein, it will be understood that the present technology as claimed is not limited to meeting most or all of the identified objects, and that some embodiments of the present technology may meet only one or none of such objects.

[0090] Referring now to the drawings, and in particular to Figures 1-25, several embodiments of the real-time video special effects system and method of the present technology are shown and generally designated by the reference numeral 10. In general overview, the system includes a camera configured to capture video of any video remote video feed including real-world scenes or video games, a graphical user interface, at least one memory, and at least one processing unit operatively connected or connectable to the camera, the graphical user interface, and the at least one memory. The at least one processing unit is configured to play the video being captured on the graphical user interface at a normal speed, and to change the video playback speed on the graphical interface of the video being captured from the normal playback speed to a modified playback speed in response to user input received by the graphical user interface.

[0091] Turning now to some embodiments in more detail, a novel and novel real-time video special effects system and method 10 of the present technology for generating special effects in video recordings during recording is shown and described with reference to FIG. 1. In more detail, the real-time video special effects system and method 10 may include a camera 12, an image processor or processing unit 14, a user interface 30 associated with the processing unit, a storage or memory unit 18, and a display unit 20. At least one RAM memory and / or at least one non-volatile long-term memory are operably connected or connectable to the processing unit 14. It should be understood that the camera 12 may be any device capable of taking images and / or video, and may be associated with or integrated with a microphone 16. The image processing unit 14 is in operative communication with the camera 12, the microphone 16, the memory unit 18, and / or the display unit 20. The image processing unit 14 intercepts raw video data from the camera 12 and / or microphone 16, processes the raw video data according to at least one algorithm as quickly as possible in real time, and then records the output / final video recording data in a memory unit 18 and / or displays the output / final video recording data on a display unit 20.

[0092] It should be understood that system 10 can be configured or configurable as a complete video system of an electronic device having one or more video cameras 12, one or more display devices 20, and one or more integrated circuits or processors. Alternatively, it can be understood that image processing unit 14 can be configured or configurable as a module or integrated circuit chip integrated into the electronic device or with components of the electronic device. Further alternatively, system 10 can be configured or configurable as a video data processing device such as, but not limited to, a graphics processing unit (GPU), digital signal processor (DSP), active server page (ASP), central processing unit (CPU), accelerated processing unit (APU), application specific integrated circuit (ASIC), etc. Further alternatively, system 10 can be configured or configurable as software or programming code as part of an operating system or application running on or controlling the electronic device or camera.

[0093] The electronic device including the camera 12, microphone 16 and display unit 20 may be, but is not limited to, a smartphone, a smart watch, a tablet, a notebook computer, a desktop computer, a laptop, a DVD player, a television digital camera (point and shoot, SLR, camcorder, high end audio / visual gear), eyewear, drones, gimbals and other stabilizers, selfie sticks, closed circuit video monitoring systems, car dash cams, endoscopes, microscopes, telescopes, camera and / or display embedded circuitry, wearables, Internet of Things (IoT), and the like.

[0094] 2, processing unit 14 is configured or can be configured to receive input of a user selection of a requested recording speed. Raw video data from camera 12 can be transferred to image processing unit 14 where programs and / or algorithms modify or retain raw frames included in the raw video data from camera 12. The raw frames in the data stream are modified or retained in real time by image processing unit 14 and then passed to memory unit 18 and / or display unit 20.

[0095] Examples of the operation of the image processing unit using frame adding, frame blending, and frame dropping 14 are shown in Figures 3A-3G. At or during operation, the image processing unit 14 intercepts raw video data 22 from the camera 12, which includes a series of frames #1-#n at a native frame rate for appropriate presentation by the display unit 20. For illustrative purposes, the frame rate shown in Figure 3A may be 30 fps. The image processing unit 14 receives the raw frames 22 and may then modify or retain the raw frames depending on one or more input signals received by the image processing unit 14. If the image processing unit 14 does not receive an input signal requesting an adjustment of the frame rate rate, all raw frames included in the raw video data 22 are passed to other components, such as a memory unit of the electronic device, as best shown in Figure 3A.

[0096] In some embodiments, when image processing unit 14 receives a special effects input signal associated with a fast motion recording operation representing a speeded up or fast forward display at twice the native frame rate, image processing unit 14 appropriately modifies raw video data 22. In response, raw frames 22 are processed using an algorithm in which every second frame is dropped, as best shown in FIG. 3B. Raw frame #1 can be added to the writer, raw frame #2 can be skipped / dropped, then raw frame #3 can be written, then raw frame #4 is skipped / dropped, and so on, until modified or output video recording data 24 is generated at twice the fast motion rate. This processing occurs in real time, and fast motion output video is recorded in place of raw video data 22 and / or displayed in real time.

[0097] In some embodiments, when image processing unit 14 receives a special effects input signal associated with a fast motion recording operation representing a speeded up or fast forward display at three times the native frame rate, image processing unit 14 appropriately modifies raw video data 22. In response, raw frames 22 are processed using an algorithm in which every second and third frame is dropped, as best shown in FIG. 3C. Raw frame #1 can be added to the writer, raw frames #2 and #3 can be skipped / dropped, then raw frame #4 can be written, then raw frames #5 and #6 can be skipped / dropped, then raw frame #7 can be written, and so on, until modified or output recording data 24 is generated at three times the fast motion speed. This processing occurs in real time, and fast motion output video is recorded in place of raw video data 22 and / or displayed in real time.

[0098] For example, if the image processing unit 14 receives a special effects input signal associated with a slow motion recording operation, this represents a slowing down or slow motion at -2 times the native frame rate. In response, as best shown in FIG. 3D, the raw frames 22 are processed using an algorithm in which every frame is duplicated / repeated. Raw frame #1 may be added to the writer, and then raw frame #1 is duplicated and written, then raw frame #2 is written, then raw frame #2 is duplicated and written, then raw frame #3 is written, then raw frame #3 is duplicated and written, and so on until the modified or output recording data 24 is generated at -2 times slow motion speed. This processing occurs in real time, and the slow motion output video is recorded in place of the raw video data 22 and / or displayed in real time or immediately after recording is stopped and the post-recording algorithm has completed processing commands entered in the meantime.

[0099] In some embodiments, when image processing unit 14 receives a special effects input signal associated with a slow-motion recording operation, representing a slowing down or slow motion at -3 times the native frame rate, image processing unit 14 appropriately modifies raw video data 22. In so doing, as best shown in FIG. 3E, raw frames are processed using an algorithm in which each frame is duplicated / repeated at least twice. Raw frame #1 may be added to the writer, and then raw frame #1 is duplicated twice and each is written, then raw frame #2 is written, then raw frame #2 is duplicated twice and each is written, then raw frame #3 is written, then raw frame #3 is duplicated twice and each is written, and so on until modified or output recording data 24 is generated at -3 times slow motion speed. This process occurs in real time, and slow-motion output video is recorded.

[0100] In some embodiments, when the image processing unit 14 receives a special effects input signal associated with a slow-motion recording operation, this represents a slowing down or slow motion at -2 times the native frame rate. In response, as best shown in Figure 3F, the raw frames 22 are processed using an algorithm in which a new frame is generated as a result of "blending" two adjacent frames. Raw frame #1 is added to the writer, then raw frame #1 is "blended" with raw frame #2 to generate one new frame #1a and #1a is written, then raw frame #2 is written, then raw frame #2 is "blended" with raw frame #3 to generate one new frame #2a and #2a is written, then raw frame #3 is written, then raw frame #3 is "blended" with raw frame #4 to generate one new frame #3a and #3a is written, and so on until the modified or output recording data 24 is generated at -2 times slow-motion speed. This processing occurs in real time and the slow motion output video is recorded in place of the unprocessed video data 22 and / or displayed either in real time or immediately after recording is stopped and the post-recording algorithm has completed processing any commands entered in the meantime.

[0101] In some embodiments, when image processing unit 14 receives a special effects input signal associated with a slow motion recording operation, which represents a slowing down or slow motion at -3 times the native frame rate, unprocessed frames 22 are then processed using an algorithm in which a new frame is generated as a result of "blending" two adjacent frames, as best shown in Figure 3G. Unprocessed frame #1 can be added to the writer, then unprocessed frame #1 is "blended" with unprocessed frame #2 to create two new frames #1a and #1b, which are written as #1a and #1b, then unprocessed frame #2 is written, then unprocessed frame #2 is "blended" with unprocessed frame #3 to create two new frames #2a and #2b, which are written as #2a and #2b, then unprocessed frame #3 is written, then unprocessed frame #3 is "blended" with unprocessed frame #4 to create two new frames #3a and #3b, which are written as #3a and #3b, and so on until modified or output recording data 24 is generated at -3x slow motion speed. This process occurs in real time, and the slow motion output video is recorded in place of the unprocessed video data 22 and / or displayed in real time or immediately after recording is stopped and the post-recording algorithm has completed processing commands entered in the meantime.

[0102] It is to be understood that additional fast motion and / or slow motion operations can be performed at faster or slower motion rates than those described above. It is further to be understood that a combination of fast motion and slow motion rates can be performed in real time on a single raw video data, thus creating output / final video recording data that includes portions at native speed rates, fast motion rates, slow motion rates, or any combination thereof.

[0103] 4 and 5, the companion software application may be associated with and / or executed by the image processing unit 14 or an electronic device, machine, or system 2 operatively associated with the image processing unit 14. FIG. 4 is a schematic diagram of the image processing unit 14 incorporated in an integrated circuit chip 26, which may be incorporated in an exemplary machine, such as the camera 12, or a component thereof, in forming an electronic device 2, in which a set of instructions may be executed to cause the component or electronic device to perform any one or more of the methods described herein. The integrated circuit chip 26 including the image processing unit 14 is configured or may be configured to include firmware for its operation. It is to be understood that the integrated circuit chip 26 may be embedded with the camera 12, the display unit 20, or other components of the electronic device 2. It is to be understood that a remote control connected to the electronic device or camera via Bluetooth or other protocols may be utilized.

[0104] The integrated circuit chip 26 may include a computer or machine readable medium on which is stored one or more sets of instructions and data structures (e.g., instructions) that perform or utilize any one or more of the methods or functions described herein. The instructions are configured or can be configured for operation of the image processing unit 14, which may receive operational instructions from an interface or GUI.

[0105] Device 2 may further include a number of different input (including simultaneous input from multiple feeds) and / or output (I / O) systems, such as, but not limited to, a touch screen and GUI, sonar or subsonic transmitters, receivers and / or transceivers, voice commands, Bluetooth, a remote controller, gesture commands on a display, or infrared. Device 2 may further record video or images from a video recording device to a memory / storage system, such as internal memory, external memory, an external solid-state drive (SSD) or the cloud.

[0106] FIG. 5 is a schematic diagram of an image processing unit 14 incorporated in an electronic device 2 and capable of executing a series of instructions to cause the electronic device to perform any one or more of the methods described herein.

[0107] In various exemplary embodiments, the electronic device 2 may operate as a standalone device or may be connected (e.g., networked) to other devices. In a networked deployment, the electronic device may operate in the capacity of a server or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The electronic device may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a portable music player (e.g., a portable hard drive audio device such as a Moving Picture Experts Group Audio Layer 3 (MP3) player), a web appliance, a network router, a switch, a bridge, or any device capable of executing a set of instructions (sequential or otherwise) that specify operations to be performed by the device. Furthermore, although only a single electronic device is shown herein, the term "device" should be construed to include any collection of devices that individually or jointly execute a set (or multiple) of instructions to perform any one or more of the methods described herein.

[0108] The illustrated electronic device 2 includes a processor or processors (e.g., a CPU, a GPU, or both) and a main memory and / or static memory that communicate with each other via a bus. In other embodiments, the electronic device 2 may further include a video display (e.g., a liquid crystal display (LCD)). The electronic device 2 may also include an alphanumeric input device (e.g., a keyboard), a cursor control device (e.g., a mouse), a voice recognition or biometric authentication unit (not shown), a drive unit (also referred to as a disk drive unit), a signal generation device (e.g., a speaker), a universal serial bus (USB) and / or other peripheral connections, and a network interface device. In other embodiments, the electronic device 2 may further include a data encryption module (not shown) for encrypting data.

[0109] Image processing unit 14 may be a module operatively associated with a drive unit including a computer or machine-readable medium on which is stored one or more sets of instructions and data structures (e.g., instructions) that perform or utilize any one or more of the methods or functions described herein. Also, the instructions may reside, completely or at least partially, in the memory and / or in the processor during execution by electronic device 2. Also, the memory and the processor may constitute machine-readable media.

[0110] The instructions may further be transmitted and received over a network via a network interface device utilizing any one of many well-known transfer protocols (e.g., Extensible Markup Language (XML)). Although the machine-readable medium is shown to be a single medium in the exemplary embodiment, the term "computer-readable medium" should be interpreted to include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more sets of instructions. The term "computer-readable medium" should be interpreted to include a medium capable of storing, encoding, or carrying a set of instructions for execution by a device, causing the device to perform any one or more of the methodologies of the present application, or a medium capable of storing, encoding, or carrying a data structure utilized by or associated with such set of instructions. Thus, the term "computer-readable medium" should be interpreted to include, but is not limited to, solid-state memories, optical and magnetic media, and carrier wave signals. Such media may also include, but are not limited to, hard disks, floppy disks, flash memory cards, digital video disks, random access memories (RAM), read-only memories (ROM), and the like. The exemplary embodiments described herein may be implemented in an operating environment that includes computer installed software, hardware, or a combination of software and hardware.

[0111] It is understood that the software application is configured or configurable to be stored in any memory of the electronic device 2 or on a remote computer in communication with the electronic device 2. The software application is configured or configurable to include an interface that allows a user to define a custom frame rate rate for the video to be recorded without changing the default frame rate rate provided by the camera 12.

[0112] A method for controlling special effects manipulation of live video capture data in real time will now be described in more detail. In summary, in some embodiments, the method includes capturing video of a real-world scene with a camera, playing the captured video on a graphical user interface at a normal speed, and changing the playback speed of the recorded video on the graphical user interface from the normal playback speed to a modified playback speed in response to user interface input received by the graphical user interface. Reference will now be made in more detail to specific processes according to some embodiments for controlling special effects manipulation of live video capture data in real time.

[0113] A possible process for the software application or interface is shown in Figures 6, 7 and 24. The interface and / or software application allows the user to select a predefined video recording speed rate in real time without changing the raw speed rate provided by the camera. This makes the technology camera and device independent. Figure 6 shows the overall process of the technology including the user interface, the device, and the subroutines associated with the overall process.

[0114] 7, the process of the present technology determines whether any special effects options have been requested for the raw video data stream from the camera. For exemplary purposes, the special effects can be a change in video speed rate by modifying frames in the raw video data. The process is configured or configurable to initiate subroutines and / or subprocesses that aid in the overall process.

[0115] The software application of the present invention is started and a user interface is provided to the user. The first step is for the user to open App 50. Then, step 51 allows the user to enter camera settings and select either to use the electronic device's camera or to use a remote video feed. Processing then proceeds to step 52 where the user initiates the recording process. The camera or electronic device receives a "start" command 53 to begin audio / video recording. It should be understood that the camera "start" command can be initiated by the software application of the present invention, the camera application, any other application associated with the electronic device, or a remote device in communication with the electronic device or camera.

[0116] At step 54 of the process it may be determined whether the user has the appropriate permissions to proceed. At step 56 permission attributes may be obtained from a separate user setting, profile, database, key, account, etc. The permission attributes may be obtained from a user database 58.

[0117] At step 60 it is determined whether the user has the proper permissions and the process stops or terminates if the user does not have the proper permissions (step 94). If the user has the proper permissions, the process proceeds to step 62 to get the settings supported by the device including the maximum recording frame rate in frames per second (fps). The process then sets the recording fps for the local or remote device based on the user permissions and device support at step 64 and then opens an input stream from the device at step 66.

[0118] Once the raw data input stream from the camera is communicated to the image processing unit, the process then determines whether a video data stream from the camera is open in step 68 while receiving information from step 62. This request can be used to verify whether the image processing unit is receiving raw video data from the camera. The raw video data stream can include an integrated or peripheral microphone and can be passed to the image processing unit rather than a memory unit or recording device. If the process determines that an input stream is not open, the process stops or ends (step 94).

[0119] If the input stream is open, processing continues at step 70 to determine whether the raw video data should be saved / recorded. If the raw video data is to be saved, processing continues at step 72 to begin a new parallel process utilizing the write video stream subroutine as shown in example 1 of Figure 8. Additional input data from a recording device (step 74) can be saved along with the raw video data.

[0120] If it is determined at step 70 that the unprocessed video data will not be saved, the process proceeds to step 76 to determine if a video input stream is open, and if so, the process continues to step 76 to determine if a user has entered a special effects command (step 82). The process may allow a special effects command to be entered via the interface. If a user has entered a special effects command, step 84 is initiated and a special effects subroutine is applied, as best shown in Figure 9. Additional input data from a recording device (step 86) may be included in the application of special effects at step 84.

[0121] If the user has not input a request for special effects such as a change in video speed rate in step 82, then a step 88 is initiated to apply other commands such as inputting data from a recording device (step 90) and / or inputting processed video and audio data with special effects (step 92). Step 88 may include other information from step 84.

[0122] If the video input stream is determined to be closed in step 76, processing stops or ends (step 94).

[0123] If desired, step 78 can start a new parallel process utilizing the Write Video Stream subroutine as shown in Example 2 of Figure 8. The additional processed video and audio data (step 80) can be saved along with the video data from step 78. The new parallel process of step 78 can be started separately and independently from steps 84 and / or 88. After step 78, processing returns to step 76.

[0124] This process can involve writing the raw video stream using a write video stream subroutine after the raw video data stream has been processed using an apply special effects subroutine or after the raw video data stream has been held.

[0125] With reference to FIG. 8, a write video stream subroutine will be described that provides a process for writing / saving / recording the video data stream to one or more internal memories, one or more removable memories in communication with the electronic device, one or more external devices, and / or one or more cloud devices or accounts.

[0126] The process of the present technology sequentially determines which device the video data stream is to be written to and determines whether the user has proper permissions for each step associated with the video stream writing subroutine. If the user has proper permissions to write to a particular device, the process writes the video data stream to the particular device according to the user's preferences.

[0127] The subroutine begins in response to a command start from the process of Figure 7 (step 104). The subroutine then proceeds to obtain user preferences and permissions (step 102) from the process or database of Figure 7 (steps 104 and 106). After step 102, the subroutine obtains as input a raw video data stream from the camera (step 108). The raw video data stream can be an audio / video stream from the electronic device, camera and / or microphone, per step 110, and / or an audio / video stream from the device's RAM memory and / or non-volatile long-term memory, per step 112.

[0128] After obtaining the raw video data stream, step 114 of this subroutine is initiated, which determines whether the user has permission to write to the internal memory. If the user has the proper permissions and the user settings allow write / copy operations to the internal memory (step 116), then a new process is initiated in step 118 to write the video data stream to the internal memory.

[0129] If from step 114 the user does not have permission to write to the internal memory, or if the user settings in step 116 do not allow the write / copy operation in step 116, or after initiating processing of step 118, the subroutine continues by determining whether the user has permission to write to the removable memory (step 120). If the user has the appropriate permissions and the user settings allow the write / copy operation to the removable memory (step 122), then a new process is initiated in step 124 to write the video data stream to the removable memory.

[0130] If from step 120 the user does not have permission to write to the removable memory, or if the user settings in step 122 do not allow such action, or after initiating the process of step 124, the subroutine continues by determining whether the user has permission to write to the external device (step 126). If the user has the appropriate permissions and the user settings allow the write / copy action to the external device (step 128), then a new process is initiated in step 130 to write the video data stream to the external device.

[0131] If from step 126 the user does not have write permission to the external device, or if the user settings of step 128 do not allow the write / copy operation of step 128, or after the start of processing of step 130 is completed, the subroutine continues to determine whether the user has write permission to the cloud (step 132). If the user has the appropriate permissions and the user settings allow the write / copy operation to the cloud (step 134), then a new process is started in step 136 to write the video data stream to the cloud.

[0132] If the user does not have permission to write to the cloud from step 132, or if the user settings from step 134 do not allow such action, or after starting processing in step 136, the subroutine stops or ends (step 138).

[0133] With reference to Figure 9, an apply special effects subroutine is described which determines whether a special effects option has been requested and which determines the particular operation of the special effects request. The subroutine begins in response to a start command from the process of Figure 7 (step 140). After beginning, the subroutine takes as input a raw video data stream from a camera (step 142). The raw video data stream may be an audio / video stream from an electronic device, a camera and / or a microphone, as in step 146.

[0134] After obtaining the raw video data stream, step 148 begins to determine if the current speed is equal to the normal or native speed, e.g., whether Recording_fps is greater than Playback_fps. If the user has made a speed change request, step 150 begins the advanced slow motion subroutine, as best shown in Figure 12. After completion of step 150, the subroutine is stopped or terminated (step 168).

[0135] If the user has not made a speed change request and the new speed is not set to normal, for example, if Recording_fps is not greater than Playback_fps or Recording_fps is equal to Playback_fps, the subroutine proceeds to step 152 to determine whether the current speed is equal to the normal or native speed. If the user has made a speed change request or if the user has changed the speed back to normal from a previously modified speed setting, the subroutine proceeds to step 154 ​​to write the video stream to RAM memory and / or non-volatile long-term memory buffer, as shown in FIG. 3A. After step 154 ​​is completed, the subroutine proceeds to step 164 to return the video buffer (RAM memory and / or non-volatile long-term memory) to the calling function, which may be a step to determine whether the video stream is open or a step to determine whether the subroutine is to stop or end (step 168).

[0136] If the user has not made a speed change request so that the new speed is not set to normal, the subroutine proceeds to step 156 to determine whether the speed change request is faster or slower than the normal speed of the raw video data stream. This can be accomplished by determining whether the current speed is faster than normal. If the current speed is faster than normal, the subroutine begins the speed up subroutine (step 158), as best shown in Figure 10. After the speed up subroutine is completed, the subroutine begins step 164 to return the video buffer (RAM memory and / or non-volatile long term memory) to the calling function.

[0137] If the current speed requested is not faster than the normal speed, the subroutine proceeds to step 160 to determine whether to make the current speed slower than normal. If the current speed is slower than normal, the subroutine begins the slow down subroutine (step 162), as best shown in Figure 13. After the slow down subroutine is completed, or if the current speed is not slower than normal, the subroutine begins step 164 to return the video buffer (RAM memory and / or non-volatile long term memory) to the calling function.

[0138] With reference to Figure 10, a speed up subroutine is described which determines whether frame dropping options and / or other plug-ins are required. This subroutine starts (step 170) in response to a command from the apply special effects subroutine (Figure 9, step 158). After starting, the subroutine takes as input a raw video data stream from an input streamed from a camera and / or from a remote video feed (step 172). The raw video data stream may be an audio / video stream from a local electronic device including a camera and / or microphone, from a remote device including a camera and / or microphone, or from another audio / video feed, as in step 174.

[0139] After obtaining the raw video data stream, step 176 of this subroutine begins to determine if a video data input stream from a camera is open. If not, the subroutine proceeds to step 189 to stop or end the subroutine.

[0140] If the input stream is open, the subroutine determines whether frame dropping is requested (step 178), and if so, proceeds to step 180, which begins the frame dropping subroutine, as best shown in FIG. 11.

[0141] If frame dropping has not been requested from step 178 or after the frame dropping subroutine of step 180 has been completed, the subroutine proceeds to step 181 to determine whether the use of time compression or expansion has been requested and, if so, proceeds to step 182 to begin the time compression and expansion subprocess, as best shown in FIG. 20.

[0142] If frame time compression and / or expansion has not been requested from step 181, or after the time compression and / or expansion sub-process of step 182 has been completed, the subroutine proceeds to step 183 to determine whether the use of variable FPS playback has been requested, and if so, proceeds to step 184 to begin the variable FPS playback sub-process, as best shown in FIG. 21.

[0143] If frame variable FPS playback has not been requested from step 183, or after the variable FPS playback subprocess of step 184 is completed, the subroutine proceeds to step 185 to determine whether another plug-in or application is requested.

[0144] If another plug-in or application has been requested, the subroutine proceeds to step 186 to execute the other plug-in or application and apply its functionality to the raw video stream from step 178 or the modified video stream from steps 180, 182 and / or 184. For example, the other plug-in or application may be, but is not limited to, a smoothing technique, etc. These other plug-ins or applications may be integrated with the software application of the present technology or may be separate from the present technology but accessible and operable in the software application of the present technology.

[0145] If the user does not request the use of other plug-ins or applications from step 185, or after other plug-in processing in step 186 is completed, the subroutine proceeds to step 188 to return data to the calling function which loops back to step 176 to determine if a video input stream is open. Step 188 may receive the video / audio stream from RAM memory and / or non-volatile long-term memory (step 187).

[0146] It can be seen that the apply special effects subroutine includes a looped sub-process including steps 178, 180, 185, 186, and 188 until step 176 determines that the input stream is not open.

[0147] With reference to FIG. 11, a frame dropping subroutine is described that determines which frames to drop to simulate the requested fast-motion video. An exemplary case for this subroutine may be when Record_fps is equal to Playback_fps. This subroutine starts in response to a command to start from the speed up subroutine (step 190). After starting, the subroutine takes as input a raw video data stream from a camera (step 192). The raw video data stream may be an audio / video stream from a local electronic device that includes a camera and / or microphone, from a remote device that includes a camera and / or microphone, or from other audio / video feeds, as in step 194.

[0148] After obtaining the raw video data stream, step 196 of this subroutine begins to determine if a video data input stream from a camera is open. If step 196 determines that an input stream is not open, then the subroutine proceeds to step 198, which returns data to the calling function of step 180 of FIG. 10. Step 198 may receive video / audio frame data from RAM memory and / or non-volatile long term memory (step 200). After step 198 is completed, the subroutine stops or ends (step 202).

[0149] While the input stream is open from step 196, the subroutine determines whether the speed is twice as fast as normal (step 204). If so, step 206 is initiated to drop the next frame, as in Figure 3B. The subroutine then proceeds to step 220 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 220, the subroutine returns to step 196.

[0150] If the speed is not twice as fast as normal (step 204), the subroutine determines whether the speed is three times faster than normal (step 208). If so, step 210 is initiated to drop the next two frames, as in Figure 3C. The subroutine then proceeds to step 220 and then returns to step 196.

[0151] If the speed is not three times faster than normal (step 208), the subroutine determines whether the speed is four times faster than normal (step 212). If so, step 214 is initiated to drop the next three frames. The subroutine then proceeds to step 220 and then returns to step 196.

[0152] If the speed is not four times faster than normal (step 212), the subroutine continues to continually determine if the speed is "n" times faster than normal (step 216), and if so, each "nth" step initiates the action of dropping the next (n-1) frame (step 218). The subroutine then proceeds to step 220 and then returns to step 196.

[0153] It can be seen that the frame dropping subroutine determines on a frame by frame basis whether or not to drop a frame, resulting in a modified video stream in which certain frames have been removed to simulate fast motion video at a given speed, and this modified video stream is written / stored in memory in real time.

[0154] It can be seen that the frame drop subroutine includes a looped subprocess including steps 204-220 until step 196 determines that the input stream is not open.

[0155] 12, an advanced slow motion subroutine is described which determines if the add frame option or other plug-in is requested. This subroutine begins in response to a command to start (step 222) from the apply special effects subroutine. After beginning, the subroutine takes as input a raw video data stream from a camera (step 224). The raw video data stream may be an audio / video stream from a local electronic device that includes a camera and / or microphone, from a remote device that includes a camera and / or microphone, or from other audio / video feeds, as in step 246.

[0156] After obtaining the raw video data stream, the subroutine begins at step 248 to determine whether a video data input stream from a camera is open. If at step 248 it is determined that an input stream is not open, the subroutine proceeds to step 270 where the subroutine stops.

[0157] While the input stream is open from step 248, the subroutine determines if adding a frame is requested (step 250) and, if so, proceeds to step 252 which begins the Add Frame subroutine, as best shown in FIG. 13.

[0158] If no frames are requested from step 250, or after the add frame subroutine from step 252 is completed, the subroutine proceeds to step 254 to determine whether an increase in frame rate recording speed has been requested. If so, the subroutine proceeds to step 256 to begin either the variable frame rate subroutine or the increase frame rate subroutine, as best shown in FIG.

[0159] If an increase in frame rate recording speed has not been requested from step 254, or after the variable frame rate subroutine from step 256 has been completed, the subroutine proceeds to step 258 to determine whether a constant high frame rate recording speed should be used. If so, the subroutine proceeds to step 260 to begin the constant high frame rate subroutine, as best shown in FIG.

[0160] If a constant high frame rate recording speed has not been requested from step 258, or after the constant high frame rate recording speed subroutine of step 260 has been completed, the subroutine proceeds to step 261 to determine whether the use of time compression or expansion has been requested, and if so, proceeds to step 262 which begins the time compression and expansion subprocess, as best shown in FIG. 20.

[0161] If frame time compression and / or expansion is not requested from step 261 or after the time compression and / or expansion sub-process of step 262 is completed, the subroutine proceeds to step 263 to determine whether the use of variable FPS playback is requested, and if so, proceeds to step 264 to begin the variable FPS playback sub-process, as best shown in FIG. 22. If frame variable FPS playback is not requested from step 263 or after the variable FPS playback sub-process of step 264 is completed, the subroutine proceeds to step 265 to determine whether other special effects enhancements are requested. If other special effects enhancements are requested, the subroutine proceeds to step 267, where the subroutine can execute other special effects sub-routines and apply their functions to the generated modified video stream. The other special effects sub-routines can be integrated with the software application of the subject technology or can be separate from the subject technology but accessible and operable by the software application of the subject technology.

[0162] If the user has not requested the use of the other special effects enhancements from step 265, or after the other special effects subroutine from step 267 has completed, this subroutine proceeds to step 266 to return data to the calling function which loops back to step 248 to determine if the video input stream is open. It can be appreciated that other processed audio / video data may be part of the data returned to the calling function in accordance with step 268.

[0163] It can be seen that this advanced slow motion subroutine includes a looped subprocess including steps 250-266 until step 248 determines that the input stream is not open.

[0164] 13, a frame addition subroutine associated with the slow down subroutine of FIG. 12 is described which determines whether and which frames are added to simulate the requested slow motion video. This subroutine assumes that record fps=playback fps. This subroutine starts in response to a command to start (step 272) from the slow down subroutine. After starting, the subroutine takes as input a raw video data stream from a camera (step 274). The raw video data stream may be an audio / video stream from a local electronic device including a camera and / or microphone, from a remote device including a camera and / or microphone, or from other audio / video feeds, as in step 276.

[0165] After obtaining the raw video data stream, step 274 of this subroutine begins to determine if a video data input stream from a camera is open. If step 278 determines that an input stream is not open, then this subroutine proceeds to step 298, which returns data to the calling function of step 252 of FIG. 12. Step 298 may receive video / audio frame data from RAM memory and / or non-volatile long term memory (step 300). After step 298 is completed, this subroutine stops or ends (step 302).

[0166] While the input stream is open from step 278, the subroutine determines the type of frame addition to utilize in step 280, either simple frame copy (step 281) or the more CPU-intensive frame blending (step 282). If the user selects frame copy, processing continues to step 281 and the algorithm and its description remain unchanged. However, if the user selects "frame blending" and their hardware supports it, the process continues to step 282 and the algorithm may include new or additional steps.

[0167] It can be understood that if frame copy is selected at step 280, then for each speed "check", processing logically proceeds along the left algorithmic path. It can further be understood that if frame blending is selected at step 280, then for each speed "check", processing logically proceeds along the right algorithmic path.

[0168] The subroutine continues by determining whether the speed is twice as slow as normal (step 283). If so, in the frame copy path, step 284 is initialized to copy the frame once for a total of two identical frames, as in FIG. 3D. The subroutine then proceeds to step 296 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 296, the subroutine returns to step 278. In the frame blend path, step 285 is initialized to blend the current frame with the next frame for a total of one new "blended" frame, as shown in FIG. 3F. The subroutine then proceeds to step 296.

[0169] If the speed is not twice as slow as normal (step 283), the subroutine determines whether the speed is three times slower than normal (step 286). If so, a frame copy path is initialized at step 287 to copy the frame twice for a total of three identical frames, as shown in FIG. 3E. The subroutine then proceeds to step 296 and returns to step 278. A frame blend path is initialized at step 288 to blend the current frame with the next frame for a total of two new "blended" frames, as shown in FIG. 3G. The subroutine then proceeds to step 296.

[0170] If the speed is not three times slower than normal (step 286), the subroutine determines whether the speed is four times slower than normal (step 289). If so, a frame copy pass is initialized at step 290 to copy the frame three times for a total of four identical frames. The subroutine then proceeds to step 296 and returns to step 278. A frame blend pass is initialized at step 291 to blend the current frame with the next frame for a total of three new "blended" frames. The subroutine then proceeds to step 296.

[0171] If the speed is not 4 times slower than normal (step 289), the subroutine continues by determining whether the speed is 'n' times slower than normal (step 292). If so, in the frame copy pass, each 'nth' step copies the frame (n-1) times for a total of 'n' identical frames. The subroutine then proceeds to step 296 and then returns to step 278. In the frame blending pass, step 295 is initiated to blend the current frame with the next frame for a total of (n-1) new 'blended' frames. The subroutine then proceeds to step 296.

[0172] It can be seen that the add frame subroutine includes a looped subprocess including steps 280-296 until step 278 determines that the input stream is not open.

[0173] With reference to Figure 14, an example of a variable high recording fps subroutine (120 FPS) is described which is associated with the variable frame rate subroutine of Figure 12. This variable frame rate subroutine can be used to simulate slow motion, but is limited to, for example, the slow motion range = recording speed / playback fps = 120 fps / 30 fps = 4.

[0174] This subroutine begins in response to a command to start from the slow down subroutine (step 304). After starting, the subroutine takes as input a raw video data stream from a camera (step 306). The raw video data stream may be an audio / video stream from a local electronic device that includes a camera and / or microphone, from a remote device that includes a camera and / or microphone, or from other audio / video feeds, as in step 308.

[0175] After obtaining the raw video data stream, step 310 of this subroutine begins and sets the recording frame rate of the device, for example Recording_Frame_Rate=120 fps. Then, step 312 sets the playback frame rate of the device, for example Playback_Frame_Rate=30 fps.

[0176] Step 314 of the subroutine begins to determine whether a video data input stream from a camera is open. If step 314 determines that an input stream is not open, the subroutine proceeds to step 332, which returns data to the calling function, step 256 of FIG. 12. Step 332 may receive video / audio frame data from RAM memory and / or non-volatile long-term memory (step 334). After step 332 is completed, the subroutine stops or ends (step 336).

[0177] While the input stream is open from step 314, the subroutine determines whether the recording speed is equal to "-4x" (step 316), which may be a slow motion range of 4. If so, step 318 is initialized to set the recording frame rate to 120 fps. The subroutine then proceeds to step 330 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 330, the subroutine returns to step 314.

[0178] If the recording speed is not equal to "-4x" (step 316), the subroutine determines whether the recording speed is equal to "-3x" (step 320). If so, step 322 is initialized to set the recording frame rate to 90 fps. The subroutine then proceeds to step 330 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 330, the subroutine returns to step 314.

[0179] If the recording speed is not equal to "-3x" (step 320), the subroutine determines whether the recording speed is equal to "-2x" (step 324). If so, step 326 is initialized to set the recording frame rate to 60 fps. The subroutine then proceeds to step 330 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 330, the subroutine returns to step 314.

[0180] If the recording speed is not equal to "-2x" (step 324), the subroutine sets the recording frame rate to 30 fps (step 328), which may be a lower than "normal" recording speed. The subroutine then proceeds to step 330 and writes the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 330, the subroutine returns to step 314.

[0181] It can be seen that this variable height recording fps subroutine includes a looped subprocess including steps 316-330 until step 314 determines that no input streams are open.

[0182] Referring now to Figure 15, an example of a constant frame rate slow motion subroutine is described which is associated with the constant high frame rate subroutine of Figure 12. This constant frame rate slow motion subroutine can be used to simulate slow motion.

[0183] This subroutine begins in response to a command to start from the slow down subroutine (step 340). After starting, the subroutine takes as input a raw video data stream from a camera (step 342). The raw video data stream may be an audio / video stream from a local electronic device that includes a camera and / or microphone, from a remote device that includes a camera and / or microphone, or from another audio / video feed, as in step 346.

[0184] After obtaining the raw video data stream, the subroutine begins at step 348 to obtain the recording frame rate (recording fps) of the video stream and then proceeds to step 350 to obtain the playback frame rate (playback fps) of the video stream.

[0185] After obtaining the recording frame rate and playback frame rate, the subroutine determines whether recording_fps=playback_fps*2 (step 352). If so, the subroutine proceeds to step 354, which begins the constant high frame rate subroutine at 60 fps, as best shown in Figure 16. The subroutine then stops or ends (step 368).

[0186] If recording_fps=playback_fps*2, the subroutine proceeds to step 356 where it determines whether recording_fps=playback_fps*4. If so, it proceeds to step 358 where it begins the constant high frame rate subroutine at 120 fps, as best shown in Figure 17. The subroutine then stops or ends (step 368).

[0187] If recording_fps=playback_fps*4 is not the case, the subroutine proceeds to step 360 where it determines whether recording_fps=playback_fps*8. If so, it proceeds to step 362 where it begins the constant high frame rate subroutine at 240 fps, as best shown in Figure 18. The subroutine then stops or ends (step 368).

[0188] If recording_fps=playback_fps*8 is not the case, then the subroutine proceeds to step 364, which is common to all other cases, to begin the constant high frame rate subroutine at a higher fps, after which the subroutine stops or ends (step 368).

[0189] With reference to Figure 16, an example of a constant high record fps subroutine (60 FPS) is described which is related to the constant high frame rate subroutine of Figure 15. This constant high frame rate subroutine can be utilized to simulate slow motion, for example, limited to slow motion range = record speed / playback fps = 60 fps / 30 fps = 2. The "slow motion range" is defined as a factor by which the record and playback fps settings can produce a slow motion effect such that the algorithm does not need to use any type of "frame adding".

[0190] This subroutine begins in response to a command from the slow down subroutine (step 370). After starting, the subroutine takes as input a raw video data stream from a camera (step 372). The raw video data stream may be an audio / video stream from a local electronic device that includes a camera and / or microphone, from a remote device that includes a camera and / or microphone, or from another audio / video feed, as in step 374.

[0191] After obtaining the raw video data stream, step 376 of this subroutine begins and sets the recording frame rate of the device, e.g., Recording_Frame_Rate=60 fps. Thereafter, step 378 sets the playback frame rate of the device, e.g., Playback_Frame_Rate=30 fps.

[0192] The subroutine begins at step 380 to determine whether a video data input stream from a camera is open. If step 380 determines that an input stream is not open, the subroutine proceeds to step 398, which returns data to the calling function of step 354 of FIG. 15. Step 398 may receive video / audio frame data from RAM memory and / or non-volatile long term memory (step 400). After step 398 is completed, the subroutine stops or ends (step 402).

[0193] While the input stream is open from step 380, the subroutine determines whether the recording speed is equal to "-4x" (step 382). If so, step 384 is initiated to copy each frame of the stream twice for a total of three identical frames, as in Figure 3E, or to obtain blended frames, as in Figure 3G. The subroutine then proceeds to step 396, where it writes the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 396, the subroutine returns to step 380.

[0194] If the recording speed is not equal to "-4x" (step 382), the subroutine determines whether the recording speed is equal to "-3x" (step 386). If so, step 388 is initiated to copy each frame of the stream once for a total of two identical frames, as in Figure 3D, or to obtain a blended frame, as in Figure 3F. The subroutine then proceeds to step 396 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 396, the subroutine returns to step 380.

[0195] If the recording speed is not equal to "-3x" (step 386), the subroutine determines whether the recording speed is equal to "-2x" (step 390). If so, step 392 is initialized to pass the video stream through unaltered. The subroutine then proceeds to step 396 to write the frame to a buffer (RAM memory and / or non-volatile long term memory). After step 396, the subroutine returns to step 380.

[0196] If the recording speed is not equal to "-2x" (step 390), the subroutine drops one of two frames (1 / 2) for a recording speed equal to "normal" (step 394). The subroutine then proceeds to step 396 and writes the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 396, the subroutine returns to step 380.

[0197] It can be seen that this constant high recording fps subroutine (60 FPS) includes a looped subprocess including steps 382-396 until step 380 determines that no input streams are open.

[0198] With reference to Figure 17, an example of a constant high record fps subroutine (120 FPS) is described which is related to the constant high frame rate subroutine of Figure 15. This constant high frame rate subroutine can be utilized to simulate slow motion, e.g., limited to a slow motion range = record speed / playback fps = 120 fps / 30 fps = 4.

[0199] This subroutine begins in response to a command to start from the slow down subroutine (step 404). After starting, the subroutine takes as input a raw video data stream from a camera (step 406). The raw video data stream may be an audio / video stream from a local electronic device that includes a camera and / or microphone, from a remote device that includes a camera and / or microphone, or from another audio / video feed, as in step 408.

[0200] After obtaining the raw video data stream, step 410 of this subroutine begins and sets the recording frame rate of the device, for example Recording_Frame_Rate=120 fps. Then, step 412 sets the playback frame rate of the device, for example Playback_Frame_Rate=30 fps.

[0201] The subroutine begins at step 414 to determine if a video data input stream from a camera is open. If step 414 determines that an input stream is not open, the subroutine proceeds to step 448, which returns data to the calling function, step 358 of FIG. 15. Step 448 may receive video / audio frame data from RAM memory and / or non-volatile long term memory (step 450). After step 448 is completed, the subroutine stops or ends (step 452).

[0202] While the input stream is open from step 414, the subroutine determines whether the recording speed is equal to "-8x" (step 416). If so, step 418 is initiated to copy the frame four times for a total of five identical frames or to obtain a blended frame. The subroutine then proceeds to step 446 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 446, the subroutine returns to step 414.

[0203] If the recording speed is not equal to "-8x" (step 416), the subroutine determines whether the recording speed is equal to "-7x" (step 420). If so, step 422 is initiated to copy the frame three times for a total of four identical frames or to obtain a blended frame. The subroutine then proceeds to step 446 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 446, the subroutine returns to step 414.

[0204] If the recording speed is not equal to "-7x" (step 420), the subroutine determines whether the recording speed is equal to "-6x" (step 424). If so, step 426 is initiated to either copy the frame twice for a total of three identical frames, as in FIG. 3E, or to obtain a blended frame, as in FIG. 3G. The subroutine then proceeds to step 446, where it writes the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 446, the subroutine returns to step 414.

[0205] If the recording speed is not equal to "-6x" (step 424), the subroutine determines whether the recording speed is equal to "-5x" (step 428). If so, step 430 is initiated to either copy the frame once for a total of two identical frames, as in FIG. 3D, or to obtain a blended frame, as in FIG. 3F. The subroutine then proceeds to step 446 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 446, the subroutine returns to step 414.

[0206] If the recording speed is not equal to "-5x" (step 428), the subroutine determines whether the recording speed is equal to "-4x" (step 432). If so, step 434 is initialized to pass the video stream through unaltered. The subroutine then proceeds to step 446 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 446, the subroutine returns to step 414.

[0207] If the recording speed is not equal to "-4x" (step 432), the subroutine determines whether the recording speed is equal to "-3x" (step 436). If so, step 438 is initiated to drop one (1 / 4) of the four frames (step 438). The subroutine then proceeds to step 446 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 446, the subroutine returns to step 414.

[0208] If the recording speed is not equal to "-3x" (step 436), the subroutine determines whether the recording speed is equal to "-2x" (step 440). If so, step 442 is initiated to drop two out of four frames (2 / 4) (step 442). The subroutine then proceeds to step 446 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 446, the subroutine returns to step 414.

[0209] If the recording speed is not equal to "-2x" (step 440), the subroutine drops three out of four frames (3 / 4) for a recording speed equal to "normal" (step 444). The subroutine then proceeds to step 446 and writes the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 446, the subroutine returns to step 414.

[0210] It can be seen that this constant high recording fps subroutine (120 FPS) includes a looped sub-process including steps 416-446 until step 414 determines that no input streams are open.

[0211] With reference to Figure 18, an example of a constant high record fps subroutine (240 FPS) is described which is related to the constant high frame rate subroutine of Figure 15. This constant high frame rate subroutine can be utilized to simulate slow motion, e.g., limited to a slow motion range = record speed / playback fps = 240 fps / 30 fps = 8.

[0212] This subroutine begins upon command from the slow down subroutine (step 454). After starting, the subroutine takes as input the raw video data stream from the camera (step 456). The raw video data stream may be an audio / video stream from a local electronic device that includes a camera and / or microphone, from a remote device that includes a camera and / or microphone, or from another audio / video feed, as in step 458.

[0213] After obtaining the raw video data stream, step 460 of this subroutine begins, which sets the recording frame rate of the device, e.g., Recording_Frame_Rate=240 fps, and then step 462 sets the playback frame rate of the device, e.g., Playback_Frame_Rate=30 fps.

[0214] The subroutine begins at step 464 to determine whether a video data input stream from a camera is open. If step 464 determines that an input stream is not open, the subroutine proceeds to step 498, which returns data to the calling function of step 362 of FIG. 15. Step 498 may receive video / audio frame data from RAM memory and / or non-volatile long term memory (step 500). After step 498 is completed, the subroutine stops or ends (step 502).

[0215] While the input stream is open from step 464, the subroutine determines whether the recording speed is equal to "-8x" (step 466). If so, step 468 is initialized to pass the video stream through unaltered. The subroutine then proceeds to step 496 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 496, the subroutine returns to step 464.

[0216] If the recording speed is not equal to "-8x" (step 466), the subroutine determines whether the recording speed is equal to "-7x" (step 470). If so, step 472 is initiated to drop one frame (1 / 8) out of every eighth frames. The subroutine then proceeds to step 496 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 496, the subroutine returns to step 464.

[0217] If the recording speed is not equal to "-7x" (step 470), the subroutine determines whether the recording speed is equal to "-6x" (step 474). If so, step 476 is initiated to drop 1 frame (2 / 8) out of every 4 frames. The subroutine then proceeds to step 496 to write the frames to a buffer (RAM memory and / or non-volatile long term memory). After step 496, the subroutine returns to step 464.

[0218] If the recording speed is not equal to "-6x" (step 474), the subroutine determines whether the recording speed is equal to "-5x" (step 478). If so, step 480 is initiated to drop 3 frames (3 / 8) out of every 8 frames. The subroutine then proceeds to step 496 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 496, the subroutine returns to step 464.

[0219] If the recording speed is not equal to "-5x" (step 478), the subroutine determines whether the recording speed is equal to "-4x" (step 482). If so, step 484 is initiated to drop one frame (4 / 8) out of every two frames. The subroutine then proceeds to step 496 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 496, the subroutine returns to step 464.

[0220] If the recording speed is not equal to "-4x" (step 482), the subroutine determines whether the recording speed is equal to "-3x" (step 486). If so, step 488 is initiated to drop 5 frames out of every 8 frames (5 / 8). The subroutine then proceeds to step 496 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 496, the subroutine returns to step 464.

[0221] If the recording speed is not equal to "-3x" (step 486), the subroutine determines whether the recording speed is equal to "-2x" (step 490). If so, step 492 is initiated to drop 3 frames (6 / 8) out of every 4 frames. The subroutine then proceeds to step 496 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 496, the subroutine returns to step 464.

[0222] If the recording speed is not equal to "-2x" (step 490), the subroutine drops 7 of every 8 frames (7 / 8) (step 494). The subroutine then proceeds to step 496 and writes the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 496, the subroutine returns to step 464.

[0223] It can be seen that this constant high recording fps subroutine (240 FPS) includes a looped subprocess including steps 466-496 until step 464 determines that no input streams are open.

[0224] With reference to FIG. 19, an example of extreme slow motion in the constant high recording fps subroutine (240 FPS) associated with the constant high frame rate subroutine of FIG. 15 is described. This constant high frame rate subroutine can be utilized to simulate extreme slow motion, e.g., limited to a slow motion range of -8x to -128x. The constant high recording FPS with frame addition subroutine of FIG. 19 shows an exemplary flow chart algorithm for a combination of high frames per second recording rate, "normal" playback frames per second, and frame addition to enhance slow motion special effects. This subroutine illustrates the cases where speeds are ≧-8x and whole multiples of 2, with speeds slower than -8x best shown in FIG. 18.

[0225] This subroutine begins in response to a command to start from the slow down subroutine (step 510). After starting, the subroutine takes as input a raw video data stream from a camera (step 512). The raw video data stream may be an audio / video stream from an electronic device, a camera and / or a microphone, as in step 514.

[0226] After obtaining the raw video data stream, step 516 of this subroutine begins and sets the recording frame rate of the device, for example, Recording Frame Rate=240 fps. Then, step 518 sets the playback frame rate of the device, for example, Playback Frame Rate=30 fps.

[0227] The subroutine begins at step 520 to determine if a video data input stream from a camera is open. If step 520 determines that an input stream is not open, the subroutine proceeds to step 544 to return data to the calling function, step 358 of FIG. 15. Step 544 may receive video / audio frame data from RAM memory and / or non-volatile long term memory (step 546). After step 544 is completed, the subroutine stops or ends (step 548).

[0228] While the input stream is open from step 520, the subroutine determines whether the recording speed is equal to "-8x" (step 522). If so, step 524 is initialized to pass the video stream through unaltered. The subroutine then proceeds to step 542, where it stores the frames in a buffer (RAM memory and / or non-volatile memory). After step 542, the subroutine returns to step 520.

[0229] If the recording speed is not equal to "-8x" (step 522), the subroutine determines whether the recording speed is equal to "-16x" (step 526). If so, step 528 is initiated to either copy each frame once for a total of two identical frames, as in FIG. 3D, or to obtain a blended frame, as in FIG. 3F. The subroutine then proceeds to step 542, where it writes the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 542, the subroutine returns to step 520.

[0230] If the recording speed is not equal to "-16x" (step 526), ​​the subroutine determines whether the recording speed is equal to "-32x" (step 530). If so, step 532 is initiated to either copy each frame twice for a total of three identical frames, as in FIG. 3E, or to obtain a blended frame, as in FIG. 3G. The subroutine then proceeds to step 542, where it writes the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 542, the subroutine returns to step 520.

[0231] If the recording speed is not equal to "-32x" (step 530), the subroutine determines whether the recording speed is equal to "-64x" (step 534). If so, step 536 is initiated to copy each frame four times for a total of five identical frames or to obtain a blended frame. The subroutine then proceeds to step 542 to write the frames to a buffer (RAM memory and / or non-volatile long-term memory). After step 542, the subroutine returns to step 520.

[0232] If the recording speed is not equal to "-64x" (step 534), the subroutine determines whether the recording speed is equal to "-128x" (step 538). If so, step 540 is initiated where each frame is copied eight times for a total of nine identical frames or blended frames. The subroutine then proceeds to step 542 where the frames are written to a buffer (RAM memory and / or non-volatile long-term memory). After step 542, the subroutine returns to step 520.

[0233] It can be seen that this constant high recording fps subroutine (240 FPS) includes a looped sub-process including steps 520-542 until step 520 determines that no input streams are open.

[0234] An example of a segment time compression and expansion subroutine is described with reference to Figure 20. This subroutine provides a flow chart algorithm for slow motion and fast motion by speeding up or slowing down the play time during video processing after recording has stopped. Frame adding / dropping can be implemented in the time compression / expansion algorithm to simulate slow motion special effects.

[0235] The video files created by this algorithm / subroutine can be played normally on all video players and do not require metadata. This is an alternative to other video files created with this technology.

[0236] This subroutine begins in response to a command to start from the slow down subroutine (step 550). After starting, the subroutine takes as input a raw video data stream from a camera (step 552). The raw video data stream may be an audio / video stream from a local electronic device that includes a camera and / or microphone, from a remote device that includes a camera and / or microphone, or from another audio / video feed, as in step 554.

[0237] After acquiring the raw video data stream, step 556 of this subroutine begins, which sets the recording FPS of the device. Step 558 then sets the playback FPS to be less than or equal to (≦) the recording FPS.

[0238] The subroutine begins at step 560 by determining whether a video data input stream from a camera is open. If step 560 determines that an input stream is not open, the subroutine proceeds to step 576, which may receive video / audio frame data from RAM memory and / or non-volatile long-term memory (step 578). After step 576 is completed, the subroutine stops or ends (step 580).

[0239] While the input stream is open from step 560, the subroutine determines if the speed is slower than "normal" (step 562). If so, step 564 is initialized to set the video segment fps equal to the record fps divided by the speed (Segment FPS=Record_FPS / Speed). The subroutine then proceeds to step 574 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 574, the subroutine returns to step 560.

[0240] If the speed is not slower than "normal" (step 562), the subroutine determines whether the speed is equal to "normal" (step 566). If so, step 568 initializes to set the video segment fps equal to the recording fps (Segment FPS=Record_FPS). The subroutine then proceeds to step 574 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 574, the subroutine returns to step 560.

[0241] If the record speed is not equal to "normal" (step 566), the subroutine determines whether the speed is faster than "normal" (step 570). If so, step 572 is initiated to set the video segment fps equal to the record fps multiplied by the speed (Segment FPS=Record_FPS*Speed). The subroutine then proceeds to step 574 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 574, the subroutine returns to step 560.

[0242] It can be seen that this segment time compression and expansion subroutine includes a looped subprocess including steps 560-574 until step 560 determines that the input stream is not open.

[0243] An example of the segment time compression and expansion subroutine is best shown in FIG. 21, which shows the results of the algorithm of FIG. 20. The top bar represents the video segments 582 in seconds per video segment in a continuous recording. The video segments 582 are processed by the segment time compression and expansion subroutine on a second-by-second basis. The segments 582 are generated when the user / AI changes the speed variable. Time special effects are applied to the raw video segments and written to the processed video stream RAM. Each segment is not compressed, expanded or modified. As a result, a playback video segment 584 is provided in seconds per video segment, which corresponds to the recording segment time in seconds. With reference to FIG. 22, an example of a variable playback speed recording subroutine is shown, which provides a flow chart algorithm for slow-motion and fast-motion by speeding up or slowing down the playback frame rate during recording. The algorithm can create a normal video using fast / slow-motion commands embedded in the video's metadata. The metadata is data embedded in the video file that is not displayed in the video recording.

[0244] This subroutine begins (step 590) in response to a command to start from the slow down subroutine (step 264) of Figure 12. After starting, the subroutine takes as input a raw video data stream from a camera (step 592). The raw video data stream may be an audio / video stream from a local electronic device that includes a camera and / or microphone, from a remote device that includes a camera and / or microphone, or from other audio / video feeds, as in step 594.

[0245] After acquiring the raw video data stream, step 596 of this subroutine begins, which sets the recording FPS of the device. Step 598 then sets the playback FPS to be less than or equal to (≦) the recording FPS.

[0246] The subroutine begins at step 600 by determining whether a video data input stream from a camera is open. If step 600 determines that an input stream is not open, the subroutine proceeds to step 616, which may receive video / audio frame data from RAM memory and / or non-volatile long-term memory (step 618). After step 616 is completed, the subroutine stops or ends (step 620).

[0247] While the input stream is open from step 600, the subroutine determines if the speed is slower than "normal" (step 602). If so, step 604 is initialized to set the segment playback fps equal to the record fps divided by the speed of that video section (Segment FPS=Record_FPS / Speed). The subroutine then proceeds to step 614 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 614, the subroutine returns to step 600.

[0248] If the speed is not slower than "normal" (step 602), the subroutine determines whether the speed is equal to "normal" (step 606). If so, step 608 is initialized to set the segment playback fps equal to the recording fps for that video section (Segment FPS=Record_FPS). The subroutine then proceeds to step 614 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 614, the subroutine returns to step 600.

[0249] If the record speed is not equal to "normal" (step 606), the subroutine determines whether the speed is faster than "normal" (step 610). If so, step 612 is initiated to set the segment play fps equal to the record fps multiple of the speed of that video section (Segment FPS=Record_FPS*Speed). The subroutine then proceeds to step 614 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 614, the subroutine returns to step 600.

[0250] It can be seen that the segment time compression and expansion subroutine includes a looped subprocess including steps 600-614 until step 600 determines that the input stream is not open.

[0251] With reference to FIG. 23, an example of a variable playback speed playback subroutine is shown, which provides a flow chart algorithm for playing video files with slow motion and fast motion special effects by speeding up or slowing down the playback frame rate during video playback. An application using the algorithm of FIG. 23 is required to play videos created by the algorithms of FIG. 20 and FIG. 21. This application needs to be able to decode the information in the metadata and / or accompanying "video project file" and apply the speed up and slow down commands to the playback frame rate during video playback. In addition to the video file, the video project includes an accompanying file that contains the special effect commands to be executed, which the custom player can decode and apply in real-time playback.

[0252] When the video is played on a non-compatible player, the speed up and slow down special effects commands in the metadata are ignored and the video plays continuously at the same speed.

[0253] This subroutine begins in response to a command to start from the slow down subroutine (step 622). After starting, the subroutine gets as input a raw video data stream from a video project file residing in the device's memory (step 624). The raw video data stream may be an audio / video stream in a video project file from the electronic device or a remote video project file, as in step 626.

[0254] After obtaining the raw video data stream, step 628 of this subroutine begins to obtain the video metadata, recording FPS, playback FPS, and variable playback log. Then, step 630 extracts the playback speed (Speed) for each section of the video from FIG. 20 where time effects have been applied to the section.

[0255] The subroutine begins at step 632 with a determination of whether a video data input stream from a camera is open. If step 632 determines that an input stream is not open, the subroutine proceeds to step 648, which may receive video / audio frame data from RAM memory and / or non-volatile long-term memory (step 950). After step 648 is completed, the subroutine stops or ends (step 652).

[0256] While the input stream is open from step 632, the subroutine determines if the speed is slower than "normal" (step 634). If so, step 636 is initialized to set the segment playback fps equal to the record fps divided by the speed of that video section (Segment FPS=Record_FPS / Speed). The subroutine then proceeds to step 646 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). The subroutine then proceeds to step 647 to display the audio / video feed from the RAM buffer, before proceeding to step 632.

[0257] If the speed is not slower than "normal" (step 634), the subroutine determines whether the speed is equal to "normal" (step 638). If so, step 340 is initialized to set the segment playback fps equal to the recording fps for that video section (Segment FPS=Record_FPS). The subroutine then proceeds to step 646 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 646, the subroutine returns to step 632.

[0258] If the record speed is not equal to "normal" (step 638), the subroutine determines whether the speed is faster than "normal" (step 642). If so, step 644 is initialized to set the segment play fps equal to the record fps multiple of the speed of that video section (Segment FPS=Record_FPS*Speed). The subroutine then proceeds to step 646 to write the frame to a buffer (RAM memory and / or non-volatile long-term memory). After step 646, the subroutine proceeds to step 647 to display the audio / video (A / V) feed from the RAM buffer, before returning to step 632.

[0259] It can be seen that this segment time compression and expansion subroutine includes a looped subprocess including steps 632-647 until step 632 determines that the input stream is not open.

[0260] A possible use of the technology is shown in Figure 24. A user can utilize the user interface of the technology to launch application software (App) on a device capable of running the App. The App can open with an image composition screen. This can be the default setting. A favorite or pre-defined setting can be optionally selected by the user. The device settings are applied and the device goes into a ready state, but optionally remains in the image composition screen.

[0261] A user can begin recording using the device's camera, a remote camera, or a remote video stream by touching or activating a "record" icon associated with the app or user interface. Optionally, the user can continue to touch the record icon or button to continue recording. One embodiment can animate the icon or a button associated with the icon to indicate that live recording is active.

[0262] During recording, the user can input a zoom in or zoom out special effect command, and the video being displayed by the device is configured or can be configured to display the zoom in or zoom out special effect associated with the video in real time.

[0263] While recording, the user can enter special effects commands to create slow motion and / or fast motion. One aspect is that there is no difference in the speed at which live video is displayed on the device.

[0264] The user can end the recording by releasing or releasing the touch of the record icon or button, after which the app will stop recording and display a "Review" screen, completing the processing of special effects and providing the option to save or autosave the processed video.

[0265] After saving, the newly created video can be viewed on the device or by a remote device after processing is complete. The video can be played continuously and can be resumed after it has finished. The app can provide a set of editing tools that can be utilized to further edit or modify the raw or processed video. Optionally, the video can be edited by tweaking slow or fast motion or other custom design elements and the video can be posted. This process can be repeated until the desired video result is achieved.

[0266] The app will then complete the process of adding new elements to the video and play them back to the user after each edit is completed. This process can be repeated until the desired video result is achieved.

[0267] Once the processing of the video and / or additional editing of the video is complete, the app may save or edit the final video. The app may save the final video to the device's internal memory, external memory, and / or the cloud.

[0268] The app may further provide an option to allow the user to post the final video to a social media platform. The app may upload the final video to an additional platform or cloud and display a configuration screen to allow the user to start recording a new video. In some embodiments, the app allows the device user to post the video to a story or social media page within the app or share it externally to other social media apps such as Instagram, Facebook, Twitter, etc.

[0269] At least a portion of the interface 30 will be described with reference to FIG. 25. The interface 30 may be, but is not limited to, a GUI interface that can provide a screen for device optimized parameters or variables. The GUI 30 is configured or can be configured to include a record start / stop controller 32 located anywhere on the screen, and a speed selection area 34 located anywhere on the screen that can provide a default or predefined frame rate speed used to manipulate frames in the raw video data from the camera 12. The speed selection area 34 can include a speed control indicator 35 that can move along the speed selection area 34 via control by the user to indicate a current or selectable speed. The GUI interface 30 can also include any area on the screen for controlling zoom 36, zoom and fast motion speed rate 38, and / or zoom and slow motion speed rate 40. In another example, the GUI interface 30 can provide the user with the ability to use a "pinch-to-zoom" operation, where the user can quickly zoom in and out using multi-touch gestures, with the "zoom factor" adjusted up or down accordingly.

[0270] In some embodiments, a user can touch and / or press and hold the start / stop controller 32 to start, stop, or pause the camera's recording function. Additionally, a user can interact with the speed selection area 34 by touching the area with a finger or a stylus-like device and sliding in any direction along the area. The processing unit is configured or can be configured to interpret this sliding motion as a special effect input command. For example, sliding from the center area of ​​the speed selection area 34 to the right can change the speed rate from the native speed rate to 2x, 3x, 4x, or "n" times the native speed rate, depending on how far to the right the sliding motion moves on the speed selection area 34. Sliding from the center area of ​​the speed selection area 34 to the left can change the native speed rate from -2x, -3x, -4x, or -"n" times the native speed rate, depending on how far to the left the sliding motion moves on the speed selection area 34.

[0271] In some embodiments, the user can control the camera zoom function by performing an up / down sliding motion from the lower region of the GUI to the upper region, and the user can further control the combination of zoom and speed rate by performing a curving up / down sliding motion from the lower region of the GUI to the upper right or upper left region, depending on whether zoom and fast or slow motion effect is desired.

[0272] Alternatively, the GUI interface 30 may be configured or arranged to include regions, icons, or windows that allow control of camera and / or display functions, attributes, operations, settings, and / or characteristics, including, but not limited to, flash, hands-free, timer, mute, selfie, broadcast, share, filters, media, stop / start recording, etc. The GUI interface 30 may be configured or arranged to be used to preset minimum, maximum, or ranges for raw video speed rates.

[0273] Additionally, optionally, GUI interface 30 is configured or can be configured to include regions, icons or windows that provide editing options for the video data stream, which can include, but are not limited to, adding text, adding drawings, adding sound, face filters, adding decorations, creating video loops, adding covers, and the like.

[0274] The GUI interface 30 can be configured or arranged to include a display of output video recording data, which can be raw video data and / or modified video data, or can display an edited video data stream. It can be appreciated that the output video recording data displayed by the GUI interface 30 can be dynamically changed in real time with changes in input. Thus, the present technique can display a seamless transition between raw video data, any number of modified video data or subsets thereof in real time while the camera is capturing video and while recording. The modified video data can include any number of fast to slow motion subsets, which can be combined with subsets of the raw video data. The display of the raw video data and / or any number of modified video data is accomplished live or in real time as the camera is capturing images associated with the raw video data. It can be appreciated that the present technique can render the displayed output video data while the camera is capturing images and the output video is being written to memory. As a result, a user can capture video while moving, panning, zooming the camera, and at the same time, any number of special effects can be applied to the raw video data and displayed.

[0275] It can be appreciated that in use, a user can initiate camera operation using an electronic device that includes or is operably associated with a software application of the present technology, or a user can initiate camera operation using a software application of the present technology that is operably associated with a camera.

[0276] Operation of the software application of the present technology provides the user with a user interface for controlling the functionality of the software application and / or the camera of the present technology.

[0277] A user can use the interface to initiate a recording function of the camera, at which time the software application of the technology receives any raw video data from the camera or a remote video feed, which may be associated with a microphone or a peripheral microphone. During this operation, the raw video data from the camera and / or microphone is transferred to the software application of the technology instead of the memory unit that would normally receive the raw data from the camera.

[0278] The interface provides simple input from a user to control the recording speed rate of raw video data received from the camera. For illustrative purposes, this input by a user on the interface can be a movement across a portion of a touch screen or pressure applied to a portion of a touch screen. It is understood that this input can come in a variety of forms, including, but not limited to, cursor movement, voice commands, icon activation, switch or button operation, on-screen gestures, infrasonic devices, etc.

[0279] If the user does not provide input to change the speed rate, the raw video data from the camera is displayed and written to memory.

[0280] In some embodiments, a "one-touch" mode may be utilized to allow a user to manipulate video time. In this mode, a recording operation may be initiated by touching the screen, and the recording operation may be stopped by removing the finger from the screen. Alternatively, the recording operation may occur while the screen is being touched. Exemplary operations include slowing down the video time, moving the touching finger to the left of the center of the screen, restoring the video time to normal speed, moving the touching finger to the center of the screen, speeding up the video time, moving the touching finger to the left or right of the center of the screen, quickly moving the touching finger from far left to far right (or vice versa), initiating a zoom in (telephoto) operation, moving the touching finger up, initiating a zoom out (wide angle) operation, moving the touching finger down, and keeping one finger on the screen while the other finger is being used to individually adjust other settings live, such as, but not limited to, flash, mute, etc.

[0281] Additionally, some embodiments may include a "multiple touch" mode that allows the user to individually select functions via the user interface while the recorded video is being shown in the user interface.

[0282] In some embodiments, a "hands-free" mode may be utilized in which the user taps the speed marker or indicator 35 or slides the indicator 35 to the desired speed position located on the speed selection area 34.

[0283] Additionally, a "hands-on" mode may be utilized, where, for example, the user holds down the "record" button 32 and slides their finger to the left, in this example, such that the button is directly beneath the user's finger such that the button is positioned vertically above the "-2x" affordance label. It may be appreciated that the speed indicator 35 may automatically move along the speed selection area 34 to follow the movement of the "record" button 32. Thus, it may be appreciated that in the "hands-on" mode, the speed indicator 35 located in the speed selection area 34 moves in response to the movement of the "record" button.

[0284] Alternatively, if a user provides input to change the speed rate, the raw video data is processed in real time using the software application of the present technology and its associated algorithms, the raw video data including one or more frames that are processed to create a final video data stream that corresponds to the speed rate entered by the user.

[0285] This is accomplished by utilizing the software application of the present technology to create a modified video data stream. The modified video data stream can be created by dropping certain frames from the raw video data, by copying certain frames and adding the copied frames adjacent to the original frames, or by "frame blending," which interpolates one or more frames between two reference frames. The number of frames dropped or added is determined by the software application of the present technology and is repeated until the desired speed rate is achieved.

[0286] The software application of the present technology can then write the raw video data or the modified video data stream to memory, thereby providing video to be displayed at normal speed rates, fast motion speed rates, or slow motion speed rates.

[0287] It should be appreciated that the video speed rate is not changed after it is written to memory, thereby allowing video to be recorded in real time, with or without special effects, eliminating the need for post-production editing to change the video speed rate.

[0288] The present technology is configured or can be configured such that the algorithm produces smoother time corrections of the video data stream. For example, the algorithm can fill in video gaps when a user jumps from one speed to another. The algorithm can interpolate data between two or more data points, making it even smoother when going from, for example, -3x speed to 4x speed.

[0289] During playback, video can be very abrupt. The algorithm compensates for this and smooths the video, enhancing the viewer experience with higher resolution as users move around and pan the camera as needed to capture peak moments during active sporting events, such as the transition to the start of each special effect, during each special effect, and the transition from special effects to normal time.

[0290] For example, when a subject being filmed jumps, the moment when their upward momentum has run out but they have not yet started to fall is the "peak moment." Using artificial intelligence (AI), the system can calculate the "peak moment" of an action in a scene being recorded and then execute a predefined desired action, such as using slow motion a little before and a little after the "peak moment."

[0291] The technology can be integrated into any camera equipment, including action cameras like GoPro® on-camera, DSLR cameras, mirrorless cameras, professional level video equipment, gimbals, tripods, remote triggered flashlights on-camera, eyeglass cameras, drones, webcams, etc. The technology can be integrated into remote controls and connected via Bluetooth® or other protocols to existing electronic devices that do not have the technology integrated.

[0292] The user interface of the present technology can be rendered in 3D or 2D. A user can slide a finger or stylus left and right in one plane of motion on the touch screen of the electronic device. In a 3D user interface, the electronic device can sense the change in the depth of the user's controller, the amount of pressure the user is applying, and adjust the special effects appropriately. Joysticks can be used and utilized with the present technology.

[0293] The user interface may be pressure sensitive, so that as the user presses harder or softer on the device, the device interprets these as controls to change the playback speed with fast-forward and slow-motion special effects.

[0294] This technique allows recording at a high enough frames per second that the resulting "raw" unedited video (video recorded without special effects applied) can be edited after recording, and slow motion remains smooth because high recording frame rates are supported against slower playback fps.

[0295] It should be appreciated that brainwave sensing devices, implanted or surface mounted devices, or wireless remote sensing may be utilized in conjunction with the present technology to allow the time speed effect to be controlled directly with thought.

[0296] Compression techniques can be used in conjunction with the present technology to improve recording at higher frame rates, allowing for more scene detail and smaller file sizes. Device performance is improved, thus allowing users to record at higher frame rates, allowing for more scene detail while reducing file sizes.

[0297] Audio processing algorithms can be used with the technology to provide the clearest and most understandable audio for video during segments of fast and slow scenes. Audio processing can be performed using APIs from third-party companies such as Dolby Labs, DTS, Inc., Fraunhofer Institut, Philips, Technicolor, IMAX, Sony, and others.

[0298] Data encryption algorithms can be utilized in conjunction with the present technology to provide secure transmission and storage of video.

[0299] Cryptography and blockchain technology algorithms can be used in conjunction with the technology to create a distributed ledger to record the original content creators of videos created using the technology. To access a video, users must "exchange" encrypted tokens to gain access permission.

[0300] Although embodiments of the real-time video special effects system and method have been described in detail, it will be apparent that modifications and variations thereto are possible, all of which are within the true spirit and scope of the present technology. With respect to the above description, it should be understood that the optimal dimensional relationships of the parts of the present technology, including variations in size, material, shape, form, function and operation, assembly and use, are deemed readily apparent and self-evident to those skilled in the art, and all equivalent relationships as shown in the drawings and described in the specification are intended to be encompassed by the present technology. For example, any suitable durable material can be used instead of those described above. Also, although described with respect to generating special effects on a recording while it is being recorded, it should be understood that the real-time video special effects system and method described herein are also suitable for changing frame attributes, changing the recording frame rate, changing the playback frame rate, and time compression and expansion, as well as other real-time special effects associated with any data stream.

[0301] As such, the foregoing is considered as merely illustrative of the principles of the technology. Moreover, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the technology to the exact construction and operation as illustrated and described, but rather, resort may be made to all suitable modifications and equivalents falling within the scope of the technology.

Claims

1. a camera configured to capture video of a real-world scene; A graphical user interface; at least one processing unit operatively connected or connectable to said camera and said graphical user interface; Equipped with the at least one processing unit plays the captured video on the graphical user interface at normal speed; A video capture and display system for controlling special effects operations on live video recording data in real time, configured to change a video playback speed on a graphical interface of the video being captured from a normal playback speed to a modified playback speed in response to user input received by the graphical user interface.

2. 2. The system of claim 1, wherein the at least one processing unit is configured to transition on a graphical user interface from playing the captured video at a modified playback speed back to playing the captured video at a normal speed.

3. 3. The system of claim 2, wherein the at least one processing unit is configured to transition on the graphical user interface from playing the captured video at a modified playback speed back to playing the captured video at a normal speed in response to user input received by the graphical user interface.

4. 2. The system of claim 1, wherein the at least one processing unit is configured to seamlessly change a playback speed of the recorded video on the graphical interface from the normal playback speed to a modified playback speed.

5. The system of claim 1 , wherein the camera and the graphical user interface are integrated into the same electronic device or the camera is separate from the electronic device.

6. The system of claim 5 , wherein the electronic device comprises a smart phone, a computer, a personal digital assistant (PDA), or a tablet device.

7. at least one RAM memory; at least one non-volatile long-term memory; The system of claim 1 , both of which are operatively connected or connectable to the processing unit.

8. at least one camera configured or configurable to provide raw video data at native speed, at least a portion of which corresponds to images captured by the camera in real time; At least one memory; at least one processing unit in operative communication with the camera and the memory; The processing unit includes: receiving raw video data from said camera in real time; Receive at least one input; determining whether the input is associated with a change in the native speed rate of raw video data, and if so, receiving raw video data from the camera; modifying at least one frame of the raw video data in real time while generating modified video data at one or more modified speed rates that are different from the native speed rate; configured or configurable to write at least one output video recording data to a memory; 11. A video recording system for real-time control of special effects operation of live video recording data, wherein the output video recording data is one of: unprocessed video data at the native speed rate, the modified video data at the modified speed rate, and a combination of the unprocessed video data and the modified video data.

9. a memory and a processor configured to store and execute software instructions, The instruction: a raw data receiving component configured or configurable to receive a request from the camera to obtain raw video data corresponding at least in part to images captured by the camera in real time at a native speed rate; a velocity rate determination component configured or configurable to receive at least one request to change the native velocity rate of the raw video data from the camera to at least one modified velocity rate; a frame modification component configured or configurable to, upon receiving a request to modify the native speed rate, identify at least one frame or location within the raw video data to be modified, and modify the frame or location to create modified video data at a modified speed rate that differs from the native speed rate in real time as the raw video data is received from the camera; an output video recording data writing component configured or configurable to write output video recording data to one of the memories and to a secondary memory in real time while receiving the raw video data from the camera; A client-side electronic system for controlling special effects manipulation of live video recording data in real time, wherein the output video recording data is selected from one of: unprocessed video data at the native speed rate, modified video data at the modified speed rate, and a combination of the unprocessed video data and the modified video data.

10. The system of claim 9 , wherein the processor is further configured or configurable to prevent the raw video data from the camera from being written to the memory.

11. The system of claim 9 , wherein the modified speed rate is less than the native speed rate or greater than the native speed rate.

12. 12. The system of claim 11, wherein if the modified speed rate is less than the native speed rate, the processor is configured or configurable to add at least one new frame to the unprocessed video data to create the modified video data.

13. 12. The system of claim 11, wherein if the modified speed rate is greater than the native speed rate, the processor is configured or configurable to remove the frames from the raw video data to create the modified video data.

14. The speed rate determination component receives at least one input from the interface. The system of claim 9 , configured or configurable to receive a force and determine if the input is associated with changing the native speed rate to the modified speed rate.

15. the interface is a graphical user interface that includes a portion configured or configurable to generate an input associated with either the native speed rate and the modified speed rate; the graphical user interface is configured or configurable to display the output video recording data in real-time as the raw video data is received from the camera; the output video recording data is configured or configurable to include a combination of the unprocessed video data and the modified video data; The system of claim 14 , wherein the transition between the unprocessed video data and the modified video data depends on the input.

16. the interface is operatively associated with the memory or at least one computer-readable storage medium that stores instructions for execution by the processor; The instructions cause the processor to: transmitting the raw video data from the camera to the processor in real time rather than to the memory while receiving the raw video data from the camera; 15. The system of claim 14, further comprising: causing the unprocessed video data to be written from the processor to the memory; or applying at least one algorithm to the unprocessed video data to create the modified video data and causing the modified video data to be written from the processor to the memory.

17. The camera captures a video of the real-world scene, Play the video being shot at normal speed on a graphical user interface, A method for controlling special effects operations of live video recording data in real time, comprising changing a playback speed of the video being shot on the graphical user interface from a normal playback speed to a modified playback speed in response to user interface input received at the graphical user interface.

18. 20. The method of claim 17, further comprising: reverting from playing the captured video on the graphical user interface at the modified speed back to playing the captured video on the graphical user interface at the normal speed.

19. 20. The method of claim 18, wherein switching from playing the captured video at the modified speed on the graphical user interface back to playing the captured video at the normal speed on the graphical user interface is performed in response to user input received at the graphical user interface.

20. 20. The method of claim 17, wherein changing the playback speed of the recorded video on the graphical user interface from the normal playback speed to the modified playback speed is performed seamlessly.

21. The method of claim 17 , wherein the camera and the graphical user interface are integrated into the same electronic device.

22. The method of claim 21 , wherein the electronic device comprises a smartphone, a computer, or a tablet.

Images