Method, storage medium and computing device for generating a modified video file

Abstract

A method for generating a modified video file using one or more processors is disclosed. The method includes: detecting an object displayed in an original video file using computer vision object detection technology; determining object motion characteristics of the detected object; selecting a corresponding audio or visual effect based on that a specific object motion characteristic of a specific detected object meets certain requirements; and applying the corresponding visual effect to the original video file to create a modified video file.

Description

[0001] Declaration of priority

[0002] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 200,539 filed on March 12, 2021 and U.S. Patent Application Serial No. 17 / 479,096 filed on September 20, 2021, each of which is incorporated herein by reference in its entirety. Background Art

[0003] Social networking applications provide tools for sharing user content such as photos or videos. In some instances, the photos or videos can be supplemented by augmented reality or other effects that are generated live on the camera feed and displayed on the display of the mobile device for preview. The user can be able to select and manipulate the effects to be applied to the live camera feed, and when satisfied, capture an image or record a video including the effects. The captured video or photo can then be shared on the social networking platform.

[0004] Photos can be easily edited after capture, but it may be desirable to provide additional options to enhance or edit captured or stored videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the highest digit or digits in the reference numeral refer to the figure number in which the element is first introduced. Some non-limiting examples are shown in the figures of the accompanying drawings, in which:

[0006] Figure 1 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, according to some examples.

[0007] Figure 2 is a diagrammatic representation of a messaging system having both client-side and server-side functionality according to some examples.

[0008] Figure 3A is an example of a graph of displacement versus time for a bouncing ball.

[0009] Figure 3B is an example of a graph of velocity versus time for a bouncing ball.

[0010] Figure 3C is an example of a graph of acceleration of a bouncing ball versus time.

[0011] Figure 4A is the first of three video frames showing a person heading the ball.

[0012] Figure 4B is the second of three video frames showing a person heading the ball.

[0013] Figure 4C is the third of three video frames showing a person heading the ball.

[0014] Figure 5 are frames from a video of a person playing with a ball on his head.

[0015] Figure 6 is a flow chart illustrating video analysis and editing operations according to some examples.

[0016] Figure 7 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein.

[0017] Figure 8 is a block diagram illustrating a software architecture in which examples may be implemented. DETAILED DESCRIPTION

[0018] Systems and methods for providing automated video analysis and editing are disclosed. In one example, a video is analyzed using computer vision object detection techniques to identify objects displayed in the video. Motion characteristics of one or more of the identified objects are determined. Based on the type or nature of the motion, the video can be edited to include additional visual or audio effects. In some examples, sound effects can be paired with events detected in the video, such as collisions between objects displayed in the video file or repeated motion of objects. In other examples, the playback speed of the video can be changed. An audio track with an associated beats per minute can also be added to the video using the detected object's motion frequency so that the detected object's motion is synchronized with the beat of the audio track.

[0019] In one example, a method for generating a modified video file for presentation on a device including a display is provided, performed by one or more processors. The example method includes: detecting an object displayed in an original video file using computer vision object detection techniques; determining an object motion characteristic of the detected object; selecting a corresponding audio or visual effect based on that a specific object motion characteristic of a specific detected object meets a predetermined requirement; and applying the corresponding audio or visual effect to the original video file to create a modified video file.

[0020] In another example, a non-transitory computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform operations to generate a modified video file. The operations may include: detecting an object displayed in an original video file using computer vision object detection techniques; determining object motion characteristics of the detected object; selecting a corresponding audio or visual effect based on a specific object motion characteristic of a specific detected object satisfying a predetermined requirement; and applying the corresponding audio or visual effect to the original video file to create a modified video file.

[0021] In another example, a computing device is provided, including a processor and a memory storing instructions, wherein the instructions, when executed by the processor, configure the device to generate a modified video file by: detecting an object displayed in an original video file using computer vision object detection technology; determining object motion characteristics of the detected object; selecting a corresponding audio or visual effect based on that a specific object motion characteristic of a specific detected object meets a predetermined requirement; and applying the corresponding audio or visual effect to the original video file to create a modified video file.

[0022] The predetermined requirement for the motion characteristics of a specific object can be a change in direction of a specific detection object, a repeated motion of a specific detection object, or a collision of a specific detection object with another object. In one example, the corresponding audio or visual effect is a sound effect applied to correspond to the change in direction of the specific detection object. The corresponding audio or visual effect can also be a sound effect repeatedly applied at the motion frequency of the specific detection object. The corresponding audio or visual effect can also be an adjustment of the playback speed of the original video file by a neighboring collision.

[0023] The corresponding audio or visual effect can be a soundtrack with a characteristic frequency, which is basically a multiple or fractional value of the motion frequency of the specific detection object. The playback speed of the original video file or soundtrack can also be adjusted to match the motion frequency of the specific detection object with the characteristic frequency.

[0024] Figure 1 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages, media, and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each hosting multiple applications, including a messaging client 104 and other applications 106. Each messaging client 104 is communicatively coupled to other instances of messaging clients 104 (e.g., hosted on respective other client devices 102), messaging server systems 108, and third-party servers 110 via a network 112 (e.g., the Internet). The messaging clients 104 may also communicate with the locally hosted applications 106 using an application program interface (API).

[0025] The messaging clients 104 are able to communicate and exchange data with other messaging clients 104 and messaging server systems 108 via the network 112. The data exchanged between the messaging clients 104 and between the messaging clients 104 and messaging server systems 108 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0026] The messaging server system 108 provides server-side functionality to specific messaging clients 104 via the network 112. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client 104 or by the messaging server system 108, the location of certain functions within the messaging client 104 or within the messaging server system 108 may be a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the messaging server system 108, but later migrate the technologies and functions to the messaging client 104 where the client device 102 has sufficient processing power.

[0027] The messaging server system 108 supports various services and operations provided to the messaging clients 104. Such operations include sending data to the messaging clients 104, receiving data from the messaging clients 104, and processing data generated by the messaging clients 104. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The exchange of data within the messaging system 100 is activated and controlled by functions available via a user interface (UI) of the messaging clients 104.

[0028] Turning now specifically to the messaging server system 108, an application program interface (API) server 116 is coupled to the application server 114 and provides a programming interface to the application server 114. The application server 114 is communicatively coupled to a database server 120, which facilitates access to a database 126 that stores data associated with messages processed by the application server 114. Similarly, a web server 128 is coupled to the application server 114 and provides a web-based interface to the application server 114. To this end, the web server 128 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0029] The application program interface (API) server 116 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 114. Specifically, the application program interface (API) server 116 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 104 to activate the functions of the application server 114. The application program interface (API) server 116 exposes various functions supported by the application server 114, including: account registration; login functionality; sending messages from a particular messaging client 104 to another messaging client 104 via the application server 114, sending media files (e.g., images or videos) from the messaging client 104 to the messaging server 118, and possible access by another messaging client 104; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 102; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends within a social graph; and opening application (e.g., messaging client 104-related) events.

[0030] The application server 114 hosts multiple server applications and subsystems, including, for example, a messaging server 118, an image processing server 122, and a social network server 124. The messaging server 118 implements multiple message processing techniques and functions, particularly relating to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 104. Text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 104. Given the hardware requirements for other processor and memory intensive processing of data, such processing can also be performed on the server side by the messaging server 118.

[0031] The application server 114 also includes an image processing server 122 that is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at the messaging server 118 .

[0032] The social network server 124 supports various social networking functions and services and makes these functions and services available to the messaging server 118. To this end, the social network server 124 maintains and accesses an entity graph within the database 126. Examples of functions and services supported by the social network server 124 include identifying other users in the messaging system 100 who have relationships with a particular user or who the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0033] Figure 2 1 is a block diagram showing additional details about the messaging system 100 according to some examples. Specifically, the messaging system 100 is shown as including a messaging client 104 and an application server 114. The messaging system 100 contains a number of subsystems that are supported on the client side by the messaging client 104 and on the server side by the application server 114. These subsystems include, for example, a user interface 202, a collection management system 204, an enhancement system 208, a mapping system 210, a gaming system 212, and a video analysis and editing system 214.

[0034] The user interface 202 is responsible for providing output to and receiving input from a user of the messaging client 104 on the client device 102. As is known in the art, the user interface provides a user-manipulatable display output on a display of the client device 102 (see further Figure 7 202 and described below). In one example, the user interface includes a chat interface whereby a user can send and receive messages and associated content from one or more remote users. The user interface 202 also allows the user to manipulate live or captured media, such as by providing augmented reality effects on captured photos or videos, or on a live feed from a camera device of the user device.

[0035] The collection management system 204 is responsible for managing collections and collections of media (e.g., collections of text, image video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be made available for a specified time period, such as the duration of an event related to the content. For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing an icon that provides notification that a particular collection exists to the user interface of the messaging client 104.

[0036] In addition, the collection management system 204 includes a curation interface 206 that enables a collection manager to manage and curate a particular collection of content. For example, the curation interface 206 enables an event organizer to curate a collection of content related to a particular event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation may be paid to a user for including user-generated content in a collection. In such a case, the collection management system 204 operates to automatically pay such a user for using his or her content.

[0037] The enhancement system 208 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content associated with a message. For example, the enhancement system 208 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The enhancement system 208 is operable to provide media overlays or enhancements (e.g., image filtering) to the messaging client 104 based on the geographic location of the client device 102. In another example, the enhancement system 208 is operable to provide media overlays to the messaging client 104 based on other information such as social network information of the user of the client device 102. The media overlay can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, the media overlay can include text or images that can be superimposed on photos taken by the client device 102. In another example, the media overlay includes a location identification overlay (e.g., Venice Beach), a live event name, or a business name overlay (e.g., Beach Cafe). In another example, the augmentation system 208 uses the geolocation of the client device 102 to identify a media overlay that includes a business name at the geolocation of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 126 and accessed through the database server 120.

[0038] The mapping system 210 provides various geolocation functions and supports the presentation of map-based media content and messages by the messaging client 104. For example, the mapping system 210 enables the display of user icons or avatars on a map to indicate the current or past locations of the user's "friends" and media content (e.g., a collection of messages including photos and videos) generated by such friends in the context of the map. For example, a message posted by a user to the messaging system 100 from a particular geolocation can be displayed to the "friends" of a particular user on the mapping interface of the messaging client 104 within the context of that particular location on the map. The user can also share his or her location and status information with other users of the messaging system 100 (e.g., using appropriate status avatars) via the messaging client 104, where the location and status information is similarly displayed to selected users within the context of the mapping interface of the messaging client 104.

[0039] The gaming system 212 provides various gaming functions within the context of the messaging client 104. The messaging client 104 provides a gaming interface that provides a list of available games that can be launched by a user within the context of the messaging client 104 and played with other users of the messaging system 100. The messaging system 100 also enables a particular user to invite other users to participate in playing a particular game by sending an invitation to such other users from the messaging client 104. The messaging client 104 also supports both voice and text messaging (e.g., chat) within the context of gameplay, provides leaderboards for games, and also supports providing in-game rewards (e.g., coins and items).

[0040] The video analysis and editing system 214 implements a method described in more detail below. In particular, the video analysis and editing system 214 analyzes one or more videos to identify objects or features or events that can be paired with visual effects, sounds, sound effects, music or other audio from the same library. The object can be any identifiable object in the video, and the feature can be, for example, the repetitive motion of such an object or repetitive motion that is otherwise apparent from the video, or a sudden change in motion that can be paired with a sound effect. In the case of repetitive motion, a music track or excerpt can be overlaid or added to the existing audio accompanying the video so that the beat of the soundtrack is synchronized with the repetitive motion or frequency characteristics of the video. It will of course be understood that reference to an "object in a video" refers to the display of the object in the video, not the object itself.

[0041] The music track may be selected based on the beats per minute of the soundtrack that substantially matches the repetitive motion in the video, either by an integer value or a fraction thereof, or by another characteristic determined from the soundtrack or accompanying the soundtrack, such as in the metadata of the accompanying soundtrack. Additionally, the playback speed of the audio or video may be adjusted to provide an integer or fractional match between the soundtrack and the video. In such cases, it may be unlikely to find an exact frequency match between the soundtrack and the repetitive motion, and an exact frequency match in this regard is generally understood to mean a frequency match that will still allow the audio and video to satisfactorily match with or without adjusting the playback speed of either or both of the audio or video.

[0042] In addition to, or in lieu of, matching the beat of a soundtrack to a repetitive motion, the video playback speed may be temporarily adjusted based on the content of the soundtrack. For example, if a song has a section with loud drum beats every few seconds, the video may be stopped (i.e., "frozen"), sped up or slowed down at each drum beat, and then may continue playing at the previous / default speed until the next drum beat, at which time another freeze frame may be displayed. Repeating sounds other than drum beats may be identified by appropriate filtering of the soundtrack.

[0043] The video analysis and editing system 214 may initiate a scan of available videos stored locally on the client device 102 or remotely in the messaging server system 108 or on a third party server 110. The scan may be initiated based on the detection of a new video that has been captured or stored by the user of the client device 102, or based on a user-initiated scan of one or more videos in an existing collection. Additionally, a scan of videos may be initiated from the messaging server system 108 or the messaging client 104 based on new features being offered, new sound effects being available, or any other reason. The video analysis and editing system 214 will typically create a new version of any video that it edits so that the original version is not overwritten.

[0044] In one example, when the video analysis and editing system 214 completes editing, a notification is provided to the user through the user interface 202 of the messaging client 104. The notification indicates that the newly edited video is available and prompts to watch the video or dismiss the notification. In one example, dismissal can provide an option to delete the edited video. Alternatively, if the notification is dismissed, the edited video can be automatically deleted. When viewing the video, user interface controls can be provided to allow the video to be saved, forwarded, published, etc.

[0045] Object detection and tracking is performed using current or yet to be developed computer vision techniques by the video analysis and editing system 214. Motion characteristics of detected objects may then be extracted, which in turn may be analyzed to detect secondary characteristics or events in the operation of the system that may be paired with the audio.

[0046] Figure 3A 302 is an example of a graph of displacement of a bouncing ball versus time, wherein displacement is shown on the y-axis and time is shown on the x-axis. In one example, the graph (or corresponding signal) is determined or generated from the video under analysis by using computer vision object detection techniques to identify a ball displayed in a video, and using computer vision object tracking techniques to determine the vertical position of the ball in subsequent frames of the video. Time is a time or timestamp associated with the video under consideration. As can be seen from the accompanying drawings, the graph 308 of displacement of a bouncing ball versus time rises to a peak 310 between bounces 312. The graph 308 can be analyzed using known digital signal processing techniques to extract characteristic periods or frequencies that represent the time between bounces or the frequency of the bounces, respectively.

[0047] Additionally, a change in direction associated with a bounce 312 or peak 310 can be determined based on a change in the sign of the difference between displacements in adjacent or nearby frames. Changes in direction can be used to identify events that may be paired with a sound effect or soundtrack.

[0048] Figure 3B is corresponding to Figure 3A 304. As can be seen, the ball's velocity goes from a maximum velocity 314 when it leaves the ground (where the velocity at the zero crossing 316 corresponding to the peak 310 is zero) to a continuous maximum negative velocity 320 before the ball hits the ground. Upon hitting the ground at velocity 320 corresponding to the bounce 312, the ball's velocity changes rapidly from negative to positive to a new velocity peak 322, etc.

[0049] The video analysis and editing system 214 may analyze the video using known or future digital signal processing techniques. Figure 3B to extract the characteristic period or frequency representing the time between bounces or the frequency of bounces, respectively.

[0050] In addition, the speed zero crossing 316 and zero crossing 318 associated with the bounce 312 or peak 310 can be used by the video analysis and editing system 214 to detect changes in events or motion that may be paired with sound effects or soundtracks. The rate at which the zero crossing 316 and zero crossing 318 occur can also be used to distinguish changes in direction that occur relatively gradually (e.g., at the peak 310) from changes in direction that occur relatively quickly (e.g., at the bounce 312). The video analysis and editing system 214 then selects appropriate sound effects based on the rapidity of the change. For example, a whistle with increasing pitch and decreasing pitch can be paired with the peak 310, while a bang or impact or explosion sound effect can be paired with the bounce 312.

[0051] Figure 3C is corresponding to Figure 3A and Figure 3B 306. As can be seen from the graph, the acceleration of the ball in free fall is at a constant negative one "g" acceleration 324 (neglecting air resistance) for most of the time, with positive acceleration spikes 326 corresponding to bounces 312. As previously described, the video analysis and editing system 214 can be used to analyze the ball's acceleration over time. Figure 3A The curves shown in can be used to determine characteristic frequencies or periods, or to detect events that may be paired with sound effects or soundtracks. For example, a spike in acceleration can be paired with a transient sound such as a bang, crash, or explosion sound effect.

[0052] In addition, from Figure 3A , Figure 3B and Figure 3C The information collected may be combined or used in conjunction to determine appropriate audio pairing, e.g. Figure 3C The constant negative "g" acceleration 324 in Figure 3B The zero crossing 316 in will indicate a smooth transition representing the peak height.

[0053] Figure 4A , Figure 4B and Figure 4C are three video frames showing a person heading a ball. Figure 4A The illustrated frames show a person 402 with a head 406 and a ball 404 approaching the person's head 406 in the direction of arrow 408. These figures show one example of the type of interaction that can be supplemented by adding sound effects by the video analysis and editing system 214. For example, contact between the ball 404 and the person's head 406 can be supplemented by an associated sound effect, such as an explosion, the firing of a gun, or a "boing" sound of a spring or a vocalization such as "What incredible skill!"

[0054] When analyzing the video, the video analysis and editing system 214 identifies objects that are candidates for identifying features or events that may be paired with appropriate audio. Any computer vision object detection and tracking method can be used, such as based on objects in motion, known identifiable objects such as a ball or bat, body parts identified using gesture recognition methods, etc. Objects such as faces can be identified using facial recognition technology, etc. Figure 4A , the video analysis and editing system 214 will, for example, recognize the face or head 406 of the person 402 and the ball 404.

[0055] Once an object has been identified, its motion is analyzed to determine interactions with other objects, such as described above with reference to FIG. 3A to FIG. 3C As shown, the motion of ball 404 will initially be towards Figure 4A 402, and then will be briefly stopped (relative to Figure 4B 406), and then as Figure 4C 406. As discussed above, such a change in motion can be detected, and then the video analysis and editing system 214 can pair the appropriate sound with the change in direction. In this case, the interaction of the ball 404 with the person's head 406 is inferred based on the movement of the ball, rather than by explicitly determining the interaction of the ball 404 with the head 406.

[0056] In another example, the video analysis and editing system 214 may use object collision detection techniques to determine that an interaction has occurred between two tracked objects. For example, the video analysis and editing system 214 may detect that the boundaries of two tracked objects that were not crossing or overlapping or that were in relative motion now cross or overlap. Figure 4B , the overlap or intersection of the person's head 406 and the ball 404 can be detected. The video analysis and editing system 214 can then pair the appropriate visual effects, sounds, or sound effects with the detection of the intersection or overlap.

[0057] Additionally or alternatively, the video analysis and editing system 214 can change the speed of the video as the ball 404 approaches the person's head 406 for dramatic effect. For example, a portion of the video clip that begins shortly before the impact or change in direction of the ball and ends shortly after the impact or change in direction can be marked as an area in which the video will be played back in slow motion. Other visual effects, as described below, can also be applied to the change in direction of the ball 404 when the ball 404 contacts the person's head.

[0058] Figure 5 504. The ball 506 is a frame from a video of a person 502 playing with a ball 506 on their head 504. In this example, the ball 506 is repeatedly bouncing up and down from the person's head 504, as indicated by arrows 508. This figure shows an example of a repetitive motion that may be supplemented by the addition of sound effects by the video analysis and editing system 214, or may benefit from the addition of a music track or excerpt. For example, each time the ball 506 contacts the user's head 504, the contact between the ball 506 and the person's head 504 may be repeatedly supplemented by an associated sound effect, such as an explosion, the firing of a gun, or a "boom" spring sound or sound clip.

[0059] In another example, the movement of the ball can be paired with a sound that changes in pitch depending on the height of the ball above the ground or above the head 504, such as a sliding whistle effect that increases in pitch when the ball rises and decreases in pitch when the ball falls. Additionally, the frequency at which the ball contacts the user's head 504 can be used to pair and synchronize the movement of the ball with a soundtrack that has a corresponding or related beats per minute, such as a multiple or integer or fractional value of the ball's movement. In situations where an exact match is not available, the video analysis and editing system 214 can change the playback speed of one or both of the soundtrack or video so that the movement of the ball matches the beats per minute of the soundtrack.

[0060] As previously described, when analyzing the video, the video analysis and editing system 214 will identify objects that are candidates for identifying features or events that may be paired with appropriate audio. Any computer vision object detection and tracking method may be used, such as based on objects in motion, known identifiable objects such as a ball or bat, body parts identified using gesture recognition methods, faces that may be identified using facial recognition techniques, etc. Figure 5 , the video analysis and editing system 214 will, for example, recognize the face or head 504 of the person 502 and the ball 506.

[0061] Once an object has been identified, its movement can be analyzed, for example, as described above with reference to FIG. 3A to FIG. 3C Similar to the reference FIG. 3A to FIG. 3CThe bouncing ball discussed, the tracking of the motion of the ball will present a curve with peak value 310, bounce 312, zero crossing 316 and zero crossing 318, which can be used to characterize the motion of the ball and to provide an event with sound effects, or the soundtrack can be paired or synchronized. In addition, the motion frequency of the ball can be extracted. Then, the motion characteristics of the ball detected can be used by video analysis and editing system 214 to pair appropriate sounds with changes in direction as discussed above, with the height of the ball or with the motion frequency of the ball.

[0062] Likewise, as previously described, the video analysis and editing system 214 can use object collision detection techniques to determine that an interaction has occurred between two tracked objects. For example, an overlap or intersection of a person's head 504 and a ball 506 can be detected. The video analysis and editing system 214 can then pair appropriate visual effects, sounds, or sound effects with the detection of an overlap or intersection.

[0063] Additionally or alternatively, the video analysis and editing system 214 may include visual effects, such as changes in video speed as the ball 506 approaches the person's head 504, for dramatic effect. For example, a portion of a video clip that begins shortly before the ball impact or direction change and ends shortly after the impact or direction change may be marked as an area where the video will be played back in slow motion, or the playback speed may vary with the height or inverse of the height of the ball 506 above the person's head 504. Other visual effects may also be applied, such as inserting AR effects or objects, providing image or media overlays or inserts into the video, changing color and brightness, zooming in and out of the overall or interesting features, such as the person's head 504 or the ball 404. Examples of video or image overlays may be cartoon-style sound effect bubbles, such as POW! or WHAM! or BOOM! to coincide with the impact or direction change of the ball 506 and any consistent sound effects, or inserting instantaneous images of famous football players.

[0064] The sound effects selected by the video analysis and editing system 214 may be based, for example, on popularity, may rotate between sound effects of the same general type (e.g., short, sharp sound effects for impacts or collisions), may be determined from contextual information based on the current time and date or location or event (e.g., based on the day or date, such as the sound of fireworks on the 4th of July, the sound of cannons if participating in a Civil War reenactment), or based on time, date, location, or keyword information in metadata accompanying the video. For example, it may be determined that a video was recorded near a concert and at the approximate time of the concert (before, after, during), and therefore the band's latest single or their other music may be selected to accompany the video.

[0065] In addition, the user may be presented with a menu including options for audio and visual effects, which may be indicated as preferred (Y), not used (N), or possible (M), possibly as a default. For example:

[0066] Sound effect: BoyingY / N / M ; Bang Y / N / M ;Impact Y / N / M ; Meow Y / N / M ; Bark Y / N / M

[0067] Visual effect: Change speed Y / N / M ; Slow Motion Y / N / M ; Super fast Y / N / M , zoom in / out Y / N / M

[0068] Figure 6 is a flow chart 600 illustrating video analysis and editing operations according to some examples. Figure 6 The operations shown will typically be performed on the client device 102 in an application, such as a messaging client 104, a video editing application, or a camera application provided with the messaging client 104. Various implementations are of course possible, where some of the operations occur in the application server 114, or where one application calls another application or SDK to obtain the desired functionality. In one example, the operations are performed jointly between the user interface 202 and the video analysis and editing system 214 implemented in the messaging client 104 running on the client device 102. For clarity, the flowchart 600 is discussed herein with reference to such an example.

[0069] Prior to the start of flowchart 600, messaging client 104 is in a state where a video is available for editing. For example, the user may have captured a video using a camera included in client device 102, or there may be a video currently stored in a "camera roll" folder of the client device that stores videos and images.

[0070] The method begins at operation 602, where the messaging client 104 initiates a scan of one or more videos by the video analysis and editing system 214. The scan of videos can be triggered by detecting a newly created video (e.g., when a new video has been captured by a user using the messaging client 104 or using an operating system camera app included with the client device). Alternatively, the messaging client 104 can periodically scan the camera roll folder of the client device 102 to identify any videos that have been saved to the folder since the messaging client 104 last scanned for new videos. In addition, the scan can be prompted by the messaging system 100 based on newly available sound effects, upgrades, or any other reason.

[0071] In operation 604, the video analysis and editing system 214 performs computer vision object detection and tracking to identify and track the object of interest, as discussed above with reference to FIGS. 3 and 4. Then, in operation 606, the motion characteristics (e.g., position, displacement, velocity, acceleration, frequency of motion, etc.) of the tracked object are determined by the video analysis and editing system 214. The motion characteristics of the object may be determined with reference to another object, video frame, or environment, such as, for example, by simultaneous localization and mapping performed by the video analysis and editing system 214 on the video.

[0072] Object interactions are then determined for the tracked objects by the video analysis and editing system 214 in operation 608. Interactions between objects may be performed, for example, using object collision detection techniques, or by inferring interactions based on changes in speed, direction, or other changes in motion characteristics of a particular object as discussed above.

[0073] The video analysis and editing system 214 then determines at operation 610 whether any suitable events or motion characteristics have been detected in the video being scanned. For example, a video with a small amount of motion (e.g., of a person making a speech) or a small number of objects of interest that do not interact may not be a good candidate for adding sound effects or other additional audio. The determination in operation 610 can be based on any characteristics determined from the video, such as overall object motion below a certain threshold, motion speed below a certain threshold, lack of detected interactions between objects, lack of repeated motion, no change or slow change in the direction of the tracked object, etc. The suitable events or characteristics so identified are retained together with appropriate identification and characterization information for further processing. The identification and characterization information can be, for example, the nature of the event (e.g., a collision, a rapid change in speed or direction), the associated motion characteristics (e.g., the frequency of the motion), and the associated video timestamp information, which can be used to synchronize the event with the appropriate sound effects or other audio. For example, the timestamp of the event or the timestamp corresponding to the repeated occurrence of the event can be used to adjust the beat of a music track or for repeatedly applying sound effects.

[0074] In the event that no suitable characteristic or event is determined or insufficient suitable characteristics or events are determined, flowchart 600 ends at operation 612 .

[0075] Where a threshold number of suitable motion characteristics or events are determined in operation 610, the flowchart continues at operation 614 where editing effects are selected for the motion characteristics or events based on the nature of the motion characteristics or events determined in operation 610. For example, a collision or rapid change in direction may be paired with an appropriate transient sound as discussed above, while a repetitive motion may be paired with a looping sound or track having a matching beats per minute or fractions or multiples thereof.

[0076] In this regard, any audio or visual video editing effects may be paired with detected events or characteristics, including changing or altering playback speed, inserting AR effects or objects, providing image or media overlays or inserts into the video, changing color and brightness, zooming in and out on general or interesting features, such as a person's face or a moving object that has been detected in the video, etc. In one example, a media overlay or insert may be an image or video clip that overlays at least a portion of the video, or is inserted into the video itself as additional frames at one or more moments that may coincide with a motion characteristic or event.

[0077] The video analysis and editing system 214 then applies the selected audio and visual effects to the video in operation 616. This will create a new version of the video with the effects applied thereto, which will be stored locally on the client device 102, or remotely on one of the messaging server system 108 or third-party servers 110. As described above, the application of the effects by the video analysis and editing system 214 will be synchronized with the relevant event or motion.

[0078] Then, in operation 618, the messaging client 104 notifies the user of the existence of the edited video, such as by providing a pop-up window or other notification on the display of the client device 102 via the user interface 202. The notification may be in the form of a prompt indicating that an edited version of the existing video is available and asking the user whether he wants to watch, save, edit, discard, or be reminded at a later time. The notification may also be dismissed.

[0079] Then, at operation 620, the user's selection is received by the messaging client 104, and at operation 622, the selected action is performed by the messaging client 104. As discussed above, the selected action can be to watch the edited video, save, edit, discard, or provide a reminder at a later time. In operation 622, appropriate action is taken, and appropriate context is provided according to the selected action. For example, if the user chooses to watch the video, the video is displayed on the client device 102 by the messaging client 104, and the options selectable by the conventional messaging client 104 user are provided. For example, options can be provided to send the video in a message or chat session, publish the video to a social network subscription source, save the video to a "camera film" folder on the client device 102, or edit the video. In the case where the user chooses to eliminate the notification, the edited video can be deleted.

[0080] When the selected action is completed in operation 622, flowchart 600 ends at operation 624. At this point, the messaging client may return to the default user interface.

[0081] Figure 7The illustrative embodiment of the present invention is a diagrammatic representation of a machine 700 (e.g., client device 102) within which instructions 710 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 700 to perform any one or more of the methods discussed herein. For example, the instructions 710 may cause the machine 700 to perform any one or more of the methods described herein. The instructions 710 transform a general purpose, unprogrammed machine 700 into a specific machine 700 that is programmed to perform the functions described and illustrated in the manner described. The machine 700 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 700 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 700 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 710 specifying actions to be taken by the machine 700. In addition, although only a single machine 700 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 710 to perform any one or more of the methods discussed herein. For example, the machine 700 may include any of the client device 102 or a plurality of server devices forming part of the messaging server system 108. In some examples, the machine 700 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side, and certain operations of the particular method or algorithm are performed on the client side.

[0082] The machine 700 may include a processor 704, a memory 706, and an input / output (I / O) component 702 that may be configured to communicate with each other via a bus 740. In an example, the processor 704 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 708 and a processor 712 that execute instructions 710. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 7 Multiple processors 704 are shown, but machine 700 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0083] The memory 706 includes a main memory 714, a static memory 716, and a storage unit 718, all of which are accessible by the processor 704 via the bus 740. The main memory 706, the static memory 716, and the storage unit 718 store instructions 710 that implement any one or more of the methods or functions described herein. The instructions 710 may also reside, completely or partially, within the main memory 714, within the static memory 716, within the machine-readable medium 720 within the storage unit 718, within at least one of the processors 704 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 700.

[0084] The I / O components 702 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 702 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It should be appreciated that the I / O components 702 may include Figure 7720 , and the like. In various examples, the I / O components 702 may include a user output component 726 and a user input component 728. The user output component 726 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 728 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides the location and force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0085] In other examples, the I / O component 702 may include a biometric component 730, a motion component 732, an environmental component 734, or a position component 736, as well as a wide range of other components. For example, the biometric component 730 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 732 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0086] Environmental components 734 include, for example, one or more cameras (with still image / photo and video capabilities), lighting sensor components (e.g., a photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., a barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., an infrared sensor that detects nearby objects), gas sensors (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0087] With respect to cameras, client device 102 may have a camera system including, for example, a front-facing camera on a front surface of client device 102 and a rear-facing camera on a rear surface of client device 102. The front-facing camera may, for example, be used to capture still images and videos of a user of client device 102 (e.g., a "selfie"), which may then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera may be used to capture still images and videos in a more conventional camera mode, which images are similarly enhanced with the enhancement data. In addition to the front-facing camera and the rear-facing camera, client device 102 may also include a 360° camera for capturing 360° photos and videos.

[0088] Additionally, the camera system of the client device 102 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even triple, quad, or quintuple rear camera configurations on the front and back sides of the client device 102. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0089] The location component 736 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0090] Various technologies may be used to achieve communication. The I / O component 702 also includes a communication component 738 that is operable to couple the machine 700 to the network 722 or device 724 via a corresponding coupling or connection. For example, the communication component 738 may include a network interface component or another suitable device that interfaces with the network 722. In other examples, the communication component 738 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Parts (e.g. Low energy consumption), Components, and other communication components for providing communication via other forms. Device 724 can be another machine or any peripheral device among various peripheral devices (e.g., a peripheral device coupled via USB).

[0091] In addition, the communication component 738 can detect the identifier, or include a component that is operable to detect the identifier. For example, the communication component 738 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 738, such as obtaining positioning via Internet Protocol (IP) geolocation, ... Positioning is obtained by signal triangulation, positioning is obtained via detection of NFC beacon signals that can indicate a specific position, etc.

[0092] Various memories (e.g., main memory 714, static memory 716, and memory of processor 704) and storage unit 718 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 710) when executed by processor 704 cause various operations to implement the disclosed examples.

[0093] The instructions 710 may be sent or received via a network interface device (e.g., a network interface component included in the communication component 738) using a transmission medium and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over the network 722. Similarly, the instructions 710 may be sent or received via a coupling (e.g., a peer-to-peer coupling) with the device 724 using a transmission medium.

[0094] Figure 8 800 is a block diagram illustrating a software architecture 804 that may be installed on any one or more of the devices described herein. The software architecture 804 is supported by hardware such as a machine 802 including a processor 820, a memory 826, and an I / O component 838. In this example, the software architecture 804 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 804 includes layers such as an operating system 812, a library 810, a framework 808, and an application 806. In operation, the application 806 invokes an API call 850 through the software stack and receives a message 852 in response to the API call 850.

[0095] The operating system 812 manages hardware resources and provides public services. The operating system 812 includes, for example, a kernel 814, a service 816, and a driver 822. The kernel 814 acts as an abstraction layer between the hardware layer and other software layers. For example, the kernel 814 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. The service 816 can provide other public services to other software layers. The driver 822 is responsible for controlling or interfacing with the underlying hardware. For example, the driver 822 may include a display driver, a camera driver, or Low-power drivers, Flash drivers, Serial communication drivers (e.g., USB drivers), Drivers, audio drivers, power management drivers, etc.

[0096] The library 810 provides a common low-level infrastructure used by the application 806. The library 810 may include a system library 818 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 810 may include an API library 824, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting in two dimensions (2D) and three dimensions (3D) in graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 810 may also include various other libraries 828 to provide many other APIs to the application 806.

[0097] The framework 808 provides a common high-level infrastructure used by the applications 806. For example, the framework 808 provides various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The framework 808 can provide a wide range of other APIs that can be used by the applications 806, some of which can be specific to a particular operating system or platform.

[0098] In an example, applications 806 may include a home application 836, a contacts application 830, a browser application 832, a book reader application 834, a positioning application 842, a media application 844, a messaging application 846 (e.g., a messaging client 104), a game application 848, and a variety of other applications such as third-party applications 840. Applications 806 are programs that perform functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 806 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 840 (e.g., applications created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, third-party applications 840 can activate API calls 850 provided by the operating system 812 to facilitate the functions described herein.

[0099] Glossary

[0100] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0101] "Client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0102] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, including the third generation partnership project (3GPP) of 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0103] "Component" refers to a device, physical entity or logic with the following boundaries: the boundaries are defined by function or subroutine calls, branch points, APIs or other technologies that provide partitioning or modularization for specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be packaged functional hardware units designed for use with other components and can be part of a program that generally performs specific functions in related functions. Components can constitute software components (e.g., codes embodied on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function and is no longer a general-purpose processor. It should be recognized that it can be determined for cost and time considerations whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system to implement the hardware component. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. The software configures a specific one or more processors accordingly, for example, to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be realized by (for example, by appropriate circuits and buses) signal transmission between or among two or more hardware components in these hardware components. In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be realized, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (for example, a collection of information). The various operations of the example methods described in this article can be performed at least in part by one or more processors that are temporarily configured (for example, by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described in this article can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations in the operation of the method can be performed by one or more processors 704 or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or as a "software as a service" (SaaS) operation. For example, at least some operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., API). The execution of certain operations can be distributed between processors, not only resident in a single machine, but also deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographic locations.

[0104] "Computer-readable storage media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0105] "Ephemeral messages" are messages that can be accessed for a limited duration of time. Ephemeral messages can be text, images, videos, etc. The access time of an ephemeral message can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0106] "Machine storage media" refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media", "device storage media", and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media", "computer storage media", and "device storage media" explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal media".

[0107] “Non-transitory computer-readable storage medium” refers to tangible media capable of storing, encoding, or carrying instructions to be executed by a machine.

[0108] "Signal medium" refers to any intangible medium that can store, encode or carry instructions executed by a machine and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal in which one or more of its characteristics is set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

Claims

1. A method executed by one or more processors for generating a modified video file for presentation on a device including a display, comprising: Initiating a scan of an original video file stored in a data storage device; detecting objects displayed in the original video file using computer vision object detection techniques; determining object motion characteristics of the detected object; Ending the method based on the object motion characteristic not satisfying the suitability criterion; Based on the object motion characteristics satisfying the suitability criteria, performing the following steps; selecting a relevant audio or visual effect based on detecting that the object motion characteristics of the object meet predetermined requirements; applying the associated audio or visual effect to the original video file to create the modified video file; as well as A notification of the presence of the modified video file is provided on the display of the device.

2. The method according to claim 1, wherein: The predetermined requirement for the object motion characteristic is a change in direction of the detected object.

3. The method according to claim 2, wherein: The associated audio or visual effect is a sound effect applied to correspond to a change in direction of the detected object.

4. The method according to claim 1, wherein: The predetermined requirement for the object motion characteristic is a repetitive motion of the detection object.

5. The method according to claim 4, wherein: The object motion characteristic includes a motion frequency of the detected object, and the associated audio or visual effect is a sound effect that is repeatedly applied to correspond to the motion frequency of the detected object.

6. The method according to claim 4, wherein: The associated audio or visual effect is a sound track having a characteristic frequency, wherein the characteristic frequency is a multiple or fractional value of the frequency of motion of the detected object.

7. The method according to claim 1, wherein: The suitability criterion is a threshold number of motion characteristics that meet the predetermined requirement.

8. The method according to claim 7, wherein: The predetermined requirement for the motion characteristic is the motion speed of the detection object.

9. The method according to claim 1, wherein: The predetermined requirement for the object motion characteristic is a collision of the detected object with another object.

10. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations to generate a modified video file, the operations comprising: Initiating a scan of an original video file stored in a data storage device; detecting objects displayed in the original video file using computer vision object detection techniques; determining object motion characteristics of the detected object; Ending the method based on the object motion characteristic not satisfying the suitability criterion; Based on the object motion characteristics satisfying the suitability criteria, performing the following steps; selecting a relevant audio or visual effect based on detecting that the object motion characteristics of the object meet predetermined requirements; applying the associated audio or visual effect to the original video file to create the modified video file; as well as A notification is provided on the display that the modified video file exists.

11. The computer-readable storage medium of claim 10, wherein: The associated audio or visual effect is a sound effect applied to correspond to a change in direction of the detected object.

12. The computer-readable storage medium of claim 10, wherein: The predetermined requirement for the object motion characteristic is a repetitive motion of the detection object.

13. The computer-readable storage medium of claim 12, wherein: The object motion characteristic includes a motion frequency of the detected object, and the associated audio or visual effect is a sound effect that is repeatedly applied to correspond to the motion frequency of the detected object.

14. The computer-readable storage medium of claim 12, wherein: The suitability criterion is a threshold number of motion characteristics that meet the predetermined requirement.

15. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the device to generate a modified video file by: Initiating a scan of an original video file stored in a data storage device; detecting objects displayed in the original video file using computer vision object detection techniques; determining object motion characteristics of the detected object; Ending the method based on the object motion characteristic not satisfying the suitability criterion; Based on the object motion characteristics satisfying the suitability criteria, performing the following steps; selecting a relevant audio or visual effect based on detecting that the object motion characteristics of the object meet predetermined requirements; applying the associated audio or visual effect to the original video file to create the modified video file; as well as A notification is provided on the display that the modified video file exists.

16. The computing device of claim 15, wherein: The predetermined requirement for the object motion characteristic is a change in direction of the detected object.

17. The computing device of claim 15, wherein: The predetermined requirement for the object motion characteristic is a collision of the detected object with another object.

18. The computing device of claim 15, wherein: The predetermined requirement for the object motion characteristic is the motion frequency of the detection object.

19. The computing device of claim 15, wherein: The suitability criterion is a threshold number of motion characteristics that meet the predetermined requirement.

20. The computing device of claim 19, wherein: The predetermined requirement for the motion characteristic is the motion speed of the detection object.

Images