Method for generating video montage, computing device and readable storage medium

Abstract

Methods, computing devices, and non-transitory computer-readable storage media performed by one or more processors for generating a video Mongolia from a set of video clips taken from a plurality of video files are disclosed. The method includes: performing face recognition on a video file in a plurality of video files to recognize a portion of the video file including a face; determining a correlation of a face in the portion of the video file; generating a video clip segment from the video file by clipping the portion of the video file including the face, the length of the video clip segment being based on the relevance of the face in the video file; and adding the video clip segment to the video Mongolian.

Description

[0001] This application is a divisional application of the Chinese patent application filed on September 28, 2023, with application number 202280026550.3 and invention name “Synchronization of visual content to audio track”. The international filing date of the PCT international application corresponding to the above parent application is March 22, 2022, and the international application number is PCT / US2022 / 021345.

[0002] Cross-reference to related applications

[0003] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 168,500, filed on March 31, 2021, and the benefit of priority to U.S. Application Serial No. 17 / 307,458, filed on May 4, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present application relates generally to synchronization of visual content to audio tracks, and more particularly to methods, computing devices, and non-transitory computer-readable storage media executed by one or more processors for generating a video montage from a collection of video clip segments taken from multiple video files. Background Art

[0005] Social networking applications provide tools for sharing user content such as photos or videos. In some instances, the photos or videos can be supplemented by augmented reality or other effects that are generated live on the camera feed and displayed on the display of the mobile device for preview. The user can select and manipulate the effects to be applied to the live camera feed, and when satisfied, capture an image or record a video including the effects. The captured video or photo can then be shared on the social networking platform.

[0006] Photos can be easily edited after capture, but it may be desirable to provide additional options for enhancing, editing, or presenting captured or stored videos and photos. Summary of the invention

[0007] One aspect of an embodiment of the present disclosure provides a method for generating a video montage from a collection of video clip segments taken from multiple video files, performed by one or more processors, the method comprising: performing facial recognition on a video file among the multiple video files to identify a portion of the video file that includes a face; determining the relevance of the face in the portion of the video file; generating a video clip segment from the video file by trimming the portion of the video file that includes the face, the length of the video clip segment being based on the relevance of the face in the video file; and adding the video clip segment to the video montage.

[0008] Yet another aspect of an embodiment of the present disclosure provides a computing device comprising: one or more processors; and a memory storing instructions which, when executed by the one or more processors, configure the computing device to perform operations for generating a video montage from a collection of video clip segments taken from a plurality of video files, the operations comprising: performing facial recognition on a video file among the plurality of video files to identify a portion of the video file that includes a face; determining the relevance of a face in the portion of the video file; generating a video clip segment from the video file by trimming the portion of the video file that includes the face, the length of the video clip segment being based on the relevance of the face in the video file; and adding the video clip segment to the video montage.

[0009] Yet another aspect of an embodiment of the present disclosure provides a non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform operations for generating a video montage from a collection of video clip segments taken from a plurality of video files, the operations comprising: performing facial recognition on a video file among the plurality of video files to identify a portion of the video file that includes a face; determining the relevance of the face in the portion of the video file; generating a video clip segment from the video file by trimming the portion of the video file that includes the face, the length of the video clip segment being based on the relevance of the face in the video file; and adding the video clip segment to the video montage. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the highest-order digit or digits in the reference numeral refer to the figure number in which the element was first introduced. Some non-limiting examples are shown in the figures of the accompanying drawings, in which:

[0011] Figure 1 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, according to some examples.

[0012] Figure 2 is a diagrammatic representation of a messaging system having both client-side and server-side functionality, according to some examples.

[0013] Figure 3 is a flow chart illustrating video analysis and editing operations according to some examples.

[0014] Figure 4A is a graph illustrating cumulative time differences between video clips and beat slots according to some examples.

[0015] Figure 4Bis a chart illustrating adjustments for preventing or reducing cumulative time differences between video clips and beat slots according to some examples.

[0016] Figure 5 is a flow chart illustrating a video trimming operation according to some examples.

[0017] Figure 6 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein.

[0018] Figure 7 is a block diagram illustrating a software architecture in which examples may be implemented. DETAILED DESCRIPTION

[0019] Users of messaging and social media platforms often wish to share their experiences with other users, including friends or followers. As the number of photos and videos captured by users of smartphones increases, it can become time consuming for users to select and assemble a collection of media to share. In addition, providers of such platforms also wish to encourage user participation in their platforms and associated applications by offering new or enhanced features and capabilities. Furthermore, the creation of content from different media types, such as video, images, and audio, may be beyond the capabilities or interests of some users.

[0020] This document discloses an example of creating shareable content in the form of a video montage with an audio track. The video montage contains short video clips that are obtained from a longer video captured or stored by a user, or generated from an image captured or stored by a user. Each video clip is assigned to one or more time slots based on a feature (e.g., a beat) of the audio track so that the transitions between the video clips are consistent with the features of the audio track. In one example, the transitions are synchronized to the beat of the audio track. The video montage is created by splicing the video clips together and adding an audio track to the video montage so that the starting feature (e.g., the starting beat) of the audio track is consistent with the first video clip.

[0021] In one aspect, a method for generating a video file from multiple media files performed by one or more processors includes: accessing feature information (e.g., beat information) of an audio track having a feature (e.g., beat), the feature information including a starting feature position and subsequent time slot information related to the feature of the audio track; generating a first video segment for a first time slot from a first media file, and generating a second video segment for a second time slot from a second media file; creating a video montage file by adding the second video segment to the first video segment; and adding the audio track to the video montage file, wherein the starting feature position in the audio track coincides with the start of the first video segment.

[0022] The first media file may be a first video file, and the method may further include extracting a portion of the first video file to generate a first video clip. The method may further include adjusting a playback speed of the portion of the first video file to generate the first video clip, the playback speed being based on a length of the first video file. The second media file may be an image, and the second video clip may be generated by converting the second media file into a video clip having a length corresponding to the second time slot. Alternatively, the first media file may be an image, and the first video clip may be generated by converting the first media file into a video clip having a length corresponding to the first time slot.

[0023] The method may also include: determining a first initial clip point for a specific video segment to be generated from a specific video media file; determining an i-frame in the specific video media file that is closest to the first initial clip point; setting a first actual clip point to the position of the closest i-frame; and creating the specific video segment by trimming the specific video media file at the first actual clip point.

[0024] The method may also include: determining a second actual clipping point according to the first actual clipping point and the length of the associated time slot; and creating a specific video segment by trimming the specific video media file at the second actual clipping point.

[0025] The method may also include: determining a second preliminary clipping point based on the first actual clipping point and the length of the associated time slot; setting the second actual clipping point before the second preliminary clipping point based on the cumulative time offset in the video montage file being greater than zero; setting the second actual clipping point after the second preliminary clipping point based on the cumulative time offset in the video montage file being less than zero; and creating a specific video segment by trimming the specific video media file at the second actual clipping point. Other technical features will be apparent to those skilled in the art from the following drawings, descriptions, and claims.

[0026] In another aspect, a computing device includes a processor and a memory storing instructions that, when executed by the processor, configure the device to perform operations for generating a video file from multiple media files for presentation on a display, the operations including: accessing feature information (e.g., beat information) of an audio track having features (e.g., beats), the feature information including a starting feature position and subsequent time slot information related to the features of the audio track; generating a first video segment for a first time slot from a first media file; generating a second video segment for a second time slot from a second media file; creating a video montage file by adding the second video segment to the first video segment; and adding an audio track to the video montage file, wherein the starting feature position in the audio track coincides with the start of the first video segment.

[0027] The first media file may be a first video file, and the instructions may further configure the device to perform operations including: extracting a portion of the first video file to generate a first video clip; and adjusting a playback speed of a portion of the first video file to generate the first video clip, the playback speed being based on a length of the first video file.

[0028] The instructions may also configure the device to perform operations including: determining a first initial clip point of a specific video segment to be generated from a specific video media file; determining the closest i-frame in the specific video media file to the first initial clip point; setting a first actual clip point to the position of the closest i-frame; and creating a specific video segment by trimming the specific video media file at the first actual clip point. The instructions may also configure the device to perform operations including: determining a second actual clip point based on the first actual clip point and the length of the associated time slot; and creating a specific video segment by trimming the specific video media file at the second actual clip point.

[0029] The instructions may also configure the device to perform operations including: determining a second preliminary clip point based on the first actual clip point and the length of the associated time slot; setting the second actual clip point to be before the second preliminary clip point based on the cumulative time offset in the video montage file being greater than zero; setting the second actual clip point to be after the second preliminary clip point based on the cumulative time offset in the video montage file being less than zero; and creating a specific video segment by trimming the specific video media file at the second actual clip point.

[0030] In another aspect, a non-transitory computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed by a computer, cause the computer to: generate a video file from a plurality of media files by accessing feature information (e.g., beat information) of an audio track having features (e.g., beats), the feature information including a starting feature position and subsequent time slot information related to the features of the audio track; generate a first video segment for a first time slot from a first media file; generate a second video segment for a second time slot from a second media file; create a video montage file by adding the second video segment to the first video segment; and add an audio track to the video montage file, wherein the starting feature position in the audio track coincides with the start of the first video segment.

[0031] The first media file may be a first video file, and the computer-readable storage medium may include instructions that further configure the computer to perform operations including: extracting a portion of the first video file to generate a first video segment; and adjusting a playback speed of the portion of the first video file to generate the first video segment, the playback speed being based on a length of the first video file. Alternatively, the first media file may be an image, and the instructions may further configure the computer to perform operations including: generating the first video segment by converting the first media file into a video segment having a length corresponding to the first time slot.

[0032] The computer-readable storage medium may also include instructions for further configuring the computer to perform operations, the operations including: determining a first initial clip point of a specific video segment to be generated from a specific video media file; determining an i-frame in the specific video media file that is closest to the first initial clip point; setting a first actual clip point to the position of the closest i-frame; and creating a specific video segment by trimming the specific video media file at the first actual clip point.

[0033] The computer-readable storage medium may also include instructions to further configure the computer to perform operations including: determining a second actual clip point based on the first actual clip point and the length of the associated time slot; and creating a specific video segment by trimming the specific video media file at the second actual clip point.

[0034] Alternatively, the computer-readable storage medium may include instructions to further configure the computer to perform operations including: determining a second preliminary clip point based on the first actual clip point and the length of the associated time slot; and setting the second actual clip point to be before the second preliminary clip point based on the cumulative time offset in the video montage file being greater than zero. The second actual clip point may be set to be after the second preliminary clip point based on the cumulative time offset in the video montage file being less than zero. Performing the operations may also include creating a specific video segment by trimming the specific video media file at the second actual clip point.

[0035] Other technical features will be readily apparent to those skilled in the art from the following drawings, descriptions and claims.

[0036] Figure 11 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages, media, and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each of which hosts a number of applications including a messaging client 104 and other applications 106. Each messaging client 104 is communicatively coupled to other instances of the messaging client 104 (e.g., hosted on respective other client devices 102), a messaging server system 108, and a third-party server 110 via a network 112 (e.g., the Internet). The messaging client 104 may also communicate with the locally hosted application 106 using an application program interface (API).

[0037] The messaging clients 104 are able to communicate and exchange data with other messaging clients 104 and with the messaging server system 108 via the network 112. The data exchanged between the messaging clients 104 and between the messaging clients 104 and the messaging server system 108 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0038] The messaging server system 108 provides server-side functionality to certain messaging clients 104 via the network 112. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client 104 or by the messaging server system 108, the location of certain functions within the messaging client 104 or within the messaging server system 108 may be a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the messaging server system 108, but later migrate the technologies and functions to the messaging client 104 where the client device 102 has sufficient processing power.

[0039] The messaging server system 108 supports various services and operations provided to the messaging clients 104. Such operations include sending data to the messaging clients 104, receiving data from the messaging clients 104, and processing data generated by the messaging clients 104. As examples, the data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 100 is activated and controlled by functions available via a user interface (UI) of the messaging clients 104.

[0040] Turning now specifically to the messaging server system 108, an application program interface (API) server 116 is coupled to the application server 114 and provides a programming interface to the application server 114. The application server 114 is communicatively coupled to a database server 120, which facilitates access to a database 126 that stores data associated with messages processed by the application server 114. Similarly, a web server 128 is coupled to the application server 114 and provides a web-based interface to the application server 114. To this end, the web server 128 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0041] The application program interface (API) server 116 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 114. Specifically, the application program interface (API) server 116 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 104 to activate the functionality of the application server 114. An application program interface (API) server 116 exposes various functions supported by application server 114, including: account registration; login functionality; sending messages from a particular messaging client 104 to another messaging client 104 via application server 114; sending media files (e.g., images or videos) from a messaging client 104 to messaging server 118 and for possible access by another messaging client 104; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of client device 102; retrieving such a collection; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends in a social graph; and opening application (e.g., related to messaging client 104) events.

[0042] The application server 114 hosts a number of server applications and subsystems, including, for example, a messaging server 118, an image processing server 122, and a social networking server 124. The messaging server 118 implements a number of message processing techniques and functions, particularly relating to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 104. Text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 104. Given the hardware requirements for other processor and memory intensive processing of data, such processing can also be performed on the server side by the messaging server 118.

[0043] The application server 114 also includes an image processing server 122 that is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at the messaging server 118 .

[0044] The social network server 124 supports various social networking functions and services and makes these functions and services available to the messaging server 118. To this end, the social network server 124 maintains and accesses an entity graph within the database 126. Examples of functions and services supported by the social network server 124 include identifying other users in the messaging system 100 who have relationships with a particular user or who the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0045] Figure 2 1 is a block diagram showing additional details about the messaging system 100 according to some examples. Specifically, the messaging system 100 is shown to include a messaging client 104 and an application server 114. The messaging system 100 contains a number of subsystems that are supported on the client side by the messaging client 104 and on the server side by the application server 114. These subsystems include, for example, a user interface 202, a collection management system 204, an enhancement system 208, a mapping system 210, an audio analysis system 212, and a video analysis and editing system 214.

[0046] The user interface 202 is responsible for providing output to a user of the messaging client 104 on the client device 102 and receiving input from a user of the messaging client 104 on the client device 102. The user interface is displayed on the display of the client device 102 (see Figure 6 The user interface 202 may include a user-controllable display output (e.g., additional user output components 626 in the user interface and described below) as known in the art. In one example, the user interface includes a chat interface, whereby a user may send messages and associated content to one or more remote users and receive messages and associated content from one or more remote users. The user interface 202 may also allow a user to manipulate live or captured media, such as by providing augmented reality effects to captured photos or videos or to a real-time feed from a camera of the user device.

[0047] The collection management system 204 is responsible for managing collections or collections of media (e.g., collections of text, image video, and audio data). Collections of content (e.g., messages, including images, video, text, and audio) may be organized into "event galleries" or "event stories." Such collections may be made available for specified time periods, such as the duration of an event related to the content. For example, content related to a concert may be made available as a "story" for the duration of the concert. The collection management system 204 may also be responsible for publishing an icon to the user interface of the messaging client 104 that provides notification that a particular collection exists.

[0048] In addition, the collection management system 204 also includes a curation interface 206, which enables a collection manager to manage and curate a specific collection of content. For example, the curation interface 206 enables an event organizer to curate a collection of content related to a specific event (e.g., to delete inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation can be paid to users for including user-generated content in a collection. In such a case, the collection management system 204 operates to automatically pay such users for using their content.

[0049] The enhancement system 208 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content associated with a message. For example, the enhancement system 208 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The enhancement system 208 is operable to provide media overlays or enhancements (e.g., image filters) to the messaging client 104 based on the geographic location of the client device 102. In another example, the enhancement system 208 is operable to provide media overlays to the messaging client 104 based on other information such as social network information of the user of the client device 102. The media overlay can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, the media overlay can include text or images that can be superimposed on a photo taken by the client device 102. In another example, the media overlay includes a location identification overlay (e.g., Venice Beach), a live event name, or a business name overlay (e.g., Beach Cafe). The media overlays may be stored in database 126 and accessed via database server 120 .

[0050] The mapping system 210 provides various geolocation functions and supports the presentation of map-based media content and messages by the messaging client 104. For example, the mapping system 210 enables the display of user icons or visualizations on a map to indicate the current or past locations of the user's "friends" and media content (e.g., a collection of messages including photos and videos) generated by such friends within the context of the map. For example, messages posted by a user to the messaging system 100 from a particular geolocation can be displayed to the "friends" of a particular user on the mapping interface of the messaging client 104 within the context of that particular location on the map. The user can also share his or her location and status information with other users of the messaging system 100 (e.g., using an appropriate status visualization) via the messaging client 104, where the location and status information is similarly displayed to selected users within the context of the mapping interface of the messaging client 104.

[0051] In certain cases described herein, the audio analysis system 212 provides analysis of an audio track. In particular, the audio analysis system 212 uses transient detection to identify the timestamp of the start beat for pairing a song with a collection of visual media as described herein, and to determine the beats per minute of the audio track, which can then be used to automatically synchronize the presentation of the visual media with the beats of the audio track as described herein. The beats per minute of the audio track can, for example, be converted into a beat period (i.e., the time in seconds between each beat), which, together with the timestamp of the start beat, can be used to generate a beat grid, which can be used to synchronize the presentation of the visual media with the beats of the audio track. Alternatively, the beats per minute, beat period, or beat grid can be included in metadata associated with the audio track.

[0052] The beat grid defines time slots where videos or photos can be inserted. The time slots are defined in terms of a number of beat cycles, which is usually an integer number of beat cycles but not necessarily an integer number of beat cycles. In the simplest example, the beat grid can include a number of equally sized time slots (e.g., 1 beat or 2 beats per time slot), or the time slot size can vary throughout the beat grid.

[0053] An example of a beat grid is as follows:

[0054] "beat_period":0.588,

[0055] "soundtrack_seek_start_time":"5.883",

[0056] "soundtrack":"patent_rock.mp3",

[0057] "beats_per_slot":[

[0058] 2.0,

[0059] 1.0, 2.0

[0061] ].

[0062] In this example, the beat period of the audio track "patent_rock.mp3" is 0.588 seconds, the start beat is located at 5.883 seconds from the start of the audio track, the first time slot is 2.0 beats, the second time slot is 1.0 beat, and the third time slot is 2.0 beats.

[0063] The time slots identify potential transition points between one photo or video and the next photo or video in the generated video. In some examples, the number of beats per time slot for each time slot in the audio track is selected by a human operator, while in other embodiments, the audio track is analyzed by the audio analysis system 212 to determine the time slot size (or transition points from which the time slot size can be determined) based on chord changes or intensity changes (changes in the harmonic or spectral aspects of the audio) in addition to or instead of the location of the beats, or based on major transients such as off-beat hits (based on detecting transients from the audio content). For example, the size of the time slot can be increased so that the beat at the start of the next time slot will coincide with the start of the sound track or segment, or the size of the time slot can be adjusted so that the start or end of the time slot will coincide with a repeated or characteristic word such as "Go!" or "Party!", Love, or "Kiss". The size of the time slots can vary, but is generally intended to be sufficient time for a viewer of the video montage to receive a reasonable visual impression of the image or video clip while maintaining a dynamic overall presentation. In one example, each time slot can be 0.5 seconds to 1 second, and the number of beats per time slot will vary accordingly based on the beat cycle.

[0064] Although described herein in some examples with reference to starting beats and beat parameters such as beat period and beats per time slot, the audio system can generate a feature grid that includes starting feature positions and subsequent time slot information based on or related to any feature of the audio track. The feature can be any identifiable feature in the audio track, for example, a chord change or intensity change (a change in the harmonic or spectral aspects of the audio) or a major transient such as an off-beat hit (based on detecting transients from the audio content), the start of a sound track or segment, a repeated or characteristic word such as "Go!", "Party!", Love, or "Kiss", etc.

[0065] Furthermore, the feature grid accompanying the song may have been or may be defined or modified by a human operator based on features selected based on preference or aesthetic or other considerations.

[0066] The video analysis and editing system 214 implements methods described in more detail below. In particular, the video analysis and editing system 214 selects, retrieves, and optionally modifies one or more videos or photos for inclusion in a video that is matched to an audio track using beat cycle or other characteristic information determined by the audio analysis system 212 or is provided with an audio track.

[0067] The video analysis and editing system 214 can also initiate a scan of available photos and videos stored locally on the client device 102 or remotely on the messaging server system 108 or on a third-party server 110 to initiate the generation of a beat-matched video montage as described herein. The scan can be initiated based on the detection of a certain number of new photos or videos that have been captured or stored by the user of the client device 102, or based on a user initiating a scan of one or more videos in an existing collection. The scan can also be provided based on a time interval or date (e.g., once a week, once a month, etc.) or based on location information, for example, based on the client device 102 being or having been at a vacation destination. In addition, the scan of videos can be initiated from the messaging server system 108 or the messaging client 104 based on an event or promotion initiated by the host of the messaging service. The user can also select specific photos and videos to be included in the video montage generated herein. Further, the photos and videos can be taken from a timeline or "story" posted by the user on the messaging system 100 or social media platform. Finally, once the video analysis and editing system 214 determines that a sufficient amount of new media has become available since the last time a video montage was created, the creation of a new video can occur.

[0068] The video analysis and editing system 214 assigns videos and photos to time slots based on several rules and, in some cases, edits them. For example, a photo may be assigned to one time slot, but may be assigned to two or more time slots if the video analysis and editing system 214 determines that the photo includes a caption or the user's face or the face of a person with whom the user has a close relationship.

[0069] In addition, the number of time slots to which the video is allocated in the video montage and the speed at which the video will be played in the video montage can depend on the duration of the video and the content of the video. For example, a short video can be allocated to one time slot at normal playback speed, while a medium-length video can be allocated to two time slots at 2 times (2X) playback speed, and a long video can be allocated to up to 4 time slots at 4 times (4X) playback speed.

[0070] Even when more time slots and higher play speeds are allocated, the video may still be much longer than the allocated time slots, in which case the video needs to be trimmed to fit the allocated time slots. For example, a 60-second video with a 4X play speed will be 15 seconds long. If the video is assigned to four time slots with a length of one beat cycle, where the beat cycle is 0.5 seconds, only 2 seconds of the 15 seconds are available. In this case, the video analysis and editing system 214 trims the video into a duration of 2 seconds at a 4X play speed. The video analysis and editing system 214 can select the portion of the video to be used in the video montage based on various criteria. In one example, the video analysis and editing system 214 obtains the middle 2 seconds of a 15-second video under the assumption that more relevant content may be found in the middle of the video. In other examples, the selected portion of the video is the presence of the face or voice in the selected portion determined by the facial recognition performed by the video analysis and editing system 214 or the voice recognition performed by the audio analysis system 212. Preference may also be given to close-up faces or faces identified by the video analysis and editing system 214 as the user or a person related to the user.

[0071] An example using the parameters from the patent_rock.mp3 audio track shown above will now be described. In this example, three media items will be included in the video montage, a video (Media_1) of 5.5 seconds duration, and two images, Media_2 and Media_3. The video analysis and editing system 214 performs the following steps:

[0072] 1. Determine the total slot duration for slot 1:

[0073] a. 2.0 beats at a beat period of 0.588. The total time slot duration is 1.176 seconds.

[0074] 2. Determine the duration of Media_1 at 1 times (1x) speed, 2 times (2x) speed, 3 times (3x) speed, and 4 times (4x) speed.

[0075] a. Media_1 at 1x speed is 5.5 seconds long.

[0076] b. Media_1 at 2x speed is 2.75 seconds long.

[0077] c. Media_1 at 3x speed is 1.833 seconds long.

[0078] d. Media_1 at 4x speed is 1.375 seconds long.

[0079] 3. In the case where the media duration is equal to or greater than the slot duration, the media duration that provides the highest speed is selected.

[0080] a. Media_1 at 4x speed is 1.375 seconds, which is greater than or equal to the time slot duration of 1.176 seconds.

[0081] 4. Trim Media_1 equally at the beginning and end of Media_1 to match the time slot duration. At 4X speed, a total of 0.199 seconds needs to be trimmed. At 1X speed, 0.796 seconds needs to be trimmed from Media_1, or two portions of 0.398 seconds each from each end of Media_1.

[0082] a. Trim the end of the media by 0.398 seconds, that is, from time 5.102 seconds to the end time 5.5 seconds.

[0083] b. Trim the start of the media by 0.398 seconds, that is, from the start time 0 seconds to 0.398 seconds.

[0084] c. The new media duration is 4.704 seconds long.

[0085] d. Increase the media speed by 4 times (4x), giving a new duration of 1.176 seconds or 4.704 / 4 seconds.

[0086] 5. Add the converted media_1 to time slot 1.

[0087] 6. Calculate the total slot duration for slot 2:

[0088] a. 1.0 beat at a beat period of 0.588. The total time slot duration is 0.588 seconds.

[0089] 7. Convert media_2 into a video with a total playback time of 0.588 seconds.

[0090] 8. Add the converted media_2 to time slot 2.

[0091] 9. Calculate the total slot duration for slot 3:

[0092] a. 2.0 beats at a beat period of 0.588. The total time slot duration is 1.176 seconds.

[0093] 10. Convert media_3 into a video with a total playback time of 1.176 seconds.

[0094] 11. Add converted media_3 to time slot 3.

[0095] 12. Combine the converted media from Time Slot 1, Time Slot 2, and Time Slot 3 to form a video montage.

[0096] a. The video montage is 2.94 seconds long and consists of playing 1.176 seconds of sped-up video, followed by 0.588 seconds of static video, and then 1.176 seconds of static video.

[0097] 13. Overlay audio tracks onto video montages.

[0098] a. The audio track start time is 5.883 seconds, which matches the "soundtrack_seek_start_time" parameter of this example.

[0099] b. The audio track ends at 8.823 seconds. (5.883 seconds start time plus 2.94 seconds video montage duration).

[0100] After the creation of the video montage is completed by the video analysis and editing system 214, a notification may be provided to the user through the user interface 202 of the messaging client 104. The notification will indicate that a new video montage is available and prompt to view the video montage or cancel the notification. In addition, cancel may provide an option to delete the video montage. Alternatively, if the notification is canceled, the video montage may be automatically deleted. When viewing the video montage, user interface controls may be provided to allow the video montage to be saved, forwarded, published, etc.

[0101] Figure 3 is a flowchart 300 illustrating a video montage creation operation according to some examples. Figure 3 The operations shown will typically be performed in an application on the client device 102, such as the messaging client 104, a camera application provided by the messaging client 104, or a video editing application. Various implementations are of course possible, where some of the operations are performed in the application server 114, or where one application calls another application or SDK for the desired functionality. In one example, the operations are performed jointly between the user interface 202 embodied in the messaging client 104 running on the client device 102 and the video analysis and editing system 214. For purposes of clarity, the flowchart 300 is discussed herein with reference to such an example.

[0102] The method begins at operation 302, where media is identified by the messaging client 104 for inclusion in the video montage. As described above, the messaging client 104 can scan for photos and videos based on the detection of a certain number of new photos or videos that have been captured or stored by the user of the client device 102, or based on a user initiating a scan of one or more videos in an existing collection. The user can also select specific photos and videos to be included in the video montage generated herein, or the photos and videos can be taken from a timeline or "story" posted by the user on the messaging system 100 or a social media platform. The order in which the media is included in the video montage is chronological based on the capture date, but other orders can be provided, for example, by allowing the user to explicitly specify the order.

[0103] Additionally, an associated audio track is determined or selected at operation 302. The selection of the audio track may be made by the user, whether the track selected by the user or the track suggested or selected by the messaging system 100 based on a newly available track or a newly analyzed track having a characteristic grid (e.g., a beat grid) included therewith as metadata. Additionally, contextual information may be used to select the audio track, such as based on the user's preferences or information from their profile or timeline (e.g., favorite bands, recent concerts attended, etc.).

[0104] If the audio track does not have an accompanying feature grid, then a feature grid is generated for the audio track at operation 304. This is done by the audio analysis system 212 as described above, and in one example, the feature grid is a beat grid.

[0105] A first (ie, "next") media item is then selected at operation 308 and associated with the first (next) time slot length in the feature grid.

[0106] Then at operation 308, the video analysis and editing system 214 determines whether the media item is an image or a video. If the media item is an image, the method proceeds to operation 310, where the image is converted into a video of the appropriate length. As described above, in one example, the image is converted into a video segment of one time slot length, unless the image includes content that may be more interesting, such as a caption or the face of the user or the face of a person who has a close relationship with the user, in which case the image is converted into a video segment of more than one time slot length, such as two time slot lengths. In such an example, as part of operation 308, the video analysis and editing system 214 will perform facial recognition or text recognition on the image.

[0107] Then at operation 312, the image video clip is added to the video montage. The method then continues at operation 322, where the video analysis and editing system 214 determines whether the most recent media item is the final media item. If not, the method returns to operation 306 to process the next media item. If so, the method continues at operation 324.

[0108] At operation 308, if the video analysis and editing system 214 determines that the media item is a video, the method continues at operation 314. At operation 314, the duration of the video at various playback speeds is determined as previously described. Then, at operation 316, the video playback speed is selected to be the highest speed if the video duration is equal to or greater than the associated time slot duration. Then, at operation 318, the video analysis and editing system 214 trims the video to create a video segment that is as long as the time slot length associated with the video segment at the selected playback speed. As described above, this can be accomplished by trimming the video equally at each end of the video or by determining the location of potential relevant content in the video. In addition, reference may be made as follows Figure 4A and Figure 4B The exact location of the clipping point is determined as discussed above.

[0109] Then, at operation 320, the video analysis and editing system 214 adds the video clip to the video montage. The method then continues at operation 322, where the video analysis and editing system 214 determines whether the most recent media item is the final media item. If not, the method returns to operation 306 to process the next media item. If so, the method continues at operation 324.

[0110] At operation 324, the video analysis and editing system 214 adds the selected audio track to the video montage, starting from the start time identified in the feature grid in the audio track. The portion of the added audio track will have the same duration as the video montage. The completed video montage is then stored locally on the client device 102, or remotely on the messaging server system 108 or remotely on one of the third-party servers 110.

[0111] Then at operation 326, the messaging client 104 notifies the user of the completion of the video montage, for example by providing a pop-up window or other notification on the display of the client device 102 via the user interface 202. The notification may be in the form of a prompt stating that the video montage is available and asking the user whether he wishes to view the video montage, save the video montage, discard the video montage, or provide a reminder at a later time. The notification may also be dismissed.

[0112] The user's selection is then received by the messaging client 104 at operation 328, and the selected action is performed by the messaging client 104 at operation 330. As described above, the selected action can be to view the edited video, save the edited video, discard the edited video, or provide a reminder at a later time. Depending on the selected action, appropriate steps are performed and appropriate context is provided. For example, if the user chooses to view the video, the video is displayed on the client device 102 by the messaging client 104, and user-selectable options of a conventional messaging client 104 are provided. For example, the option of sending the video in a message or chat session, the option of publishing the video to a social network feed, the option of saving the video to a "camera album" folder on the client device 102, or the option of editing the video can be provided. In the case where the user chooses to cancel the notification, the edited video can be deleted.

[0113] After completing the action selected in operation 330, flowchart 300 ends at operation 332. At this point, the messaging client may return to the default user interface.

[0114] Conventional video encoding includes spaced key frames or intraframes (i-frames), which are full frames of images in the video. Subsequent frames, incremental frames (b-frames or p-frames) contain only information changed from the previous frame. If the video is clipped at the incremental frame, the video needs to be re-encoded to generate an i-frame for the beginning of the video segment, otherwise an incomplete frame with artifacts may be found at the beginning of the video segment. Re-encoding the pruned video to place the new i-frame at the beginning of the video segment has an associated cost because it is more computationally intensive. This may affect the time spent generating the video montage and will place additional requirements on the client device 102 and its resources. For example, using current encoding software, it takes 1.284 seconds to clip a 4.5-second segment from a longer video and generate an i-frame at the clipping point. Therefore, creating a video montage with many video segments may take more time and use more computing resources than what the client device 102 or the user of the client device 102 expects.

[0115] In one example, the video analysis and editing system 214 identifies two adjacent i-frames as the desired first clip point, selects the closest of the two adjacent i-frames as the first clip point, and adjusts the second clip point accordingly. For example, for the above Media_1 example, the original clip times are 0.398 seconds and 5.102 seconds. If there is an i-frame at 0.3 seconds and another i-frame at 1.5 seconds, the video analysis and editing system 214 will subtract the i-frame time from the desired start time, compare the magnitude of the resulting values, determine that the i-frame at 0.3 seconds is closer to the original start time of 0.398 seconds than the i-frame at 1.5 seconds, and then clip the video at 0.3 seconds and 5.004 seconds to generate the desired 4.704 second video clip. Using current encoding software, clipping a 4.5 second clip at an i-frame only takes 0.002 seconds, which is a 64200% improvement in processing time compared to creating a new i-frame for use at the 0.398 second clip point.

[0116] In some cases, the available clip point locations may not match the duration of the beat per time slot or the duration between features in the feature grid. For example, if the desired video segment length is 4.5 seconds, the first available second clip point that includes the complete video segment may be at 4.58 seconds. The 80 milliseconds of additional duration may not be visually noticeable to a viewer of the video segment, but when many video segments are combined, the video montage may become noticeably off-beat compared to the audio track.

[0117] Figure 4A 4 is a graph 402 showing the cumulative time difference between a video clip 410 and a beat time slot 406 based on a requested clip duration of 4.5 seconds at normal playback speed. As can be seen from the figure, the video clip 410 becomes increasingly off beat 408 (or a feature in the feature grid). As shown in the table below, this can occur, for example, when the desired video segment length is 4.5 seconds and the first available second clip point that includes the complete video segment is at 4.58 seconds.

[0118]

[0119] In one example, to address this issue, the requested clip duration is adjusted by the amount of the accumulated offset and then set to the next available clip duration.

[0120] Figure 4B4 is a chart 404 showing adjustments for preventing or reducing cumulative time differences between a video clip 414 and a feature or beat time slot 412. As shown in the following table, the cumulative offset is subtracted from the requested clip duration to create an updated clip duration that will tend to reduce the magnitude of the cumulative offset at the end of the next time slot and will in any event prevent the cumulative offset from becoming noticeable to the user.

[0121]

[0122]

[0123] In an alternative example, the closest available clip duration will be selected which will tend to reduce the size of the cumulative offset. So for example, as shown in the table above, the two available clip durations closest to 4.5 seconds are 4.47 seconds and 4.58 seconds, which will result in the same received clip duration for timeslots 1 to 3, but for timeslot 4, a clip duration of 4.47 seconds will be selected, which will result in a cumulative offset of -0.02 seconds. Timeslot 5 will then have a received clip duration of 4.58 seconds and a cumulative offset of 0.06 seconds.

[0124] In another alternative example, if the cumulative offset is greater than zero, the closest clip duration is selected earlier, and if the cumulative offset is less than zero, the closest clip duration is selected later. If the cumulative offset is equal to zero, the closest clip duration is selected later.

[0125] In yet another alternative example, the length of a video segment generated from an image is selected to be its associated time slot length minus the accumulated offset.

[0126] The cumulative offset is of course dependent on the playback speed of the video clip. For example, if Video 1 above is edited at a perceived 4X playback speed to create a clip of 1.145 seconds (the received clip duration divided by 4), the offset would be 0.02 seconds. In such an example, the cumulative offset is based on the offset of each edited clip as they are inserted into the video montage based on their playback speed.

[0127] Although described with reference to the beat slots, Figure 4A Methods and Figure 4B The method is also applicable to time slot lengths or requested clip durations in a feature grid based on or including other features.

[0128] Figure 5 is a flowchart 500 showing video trimming operations according to one example. These operations are Figure 3 An example of the steps performed in operation 318 and implemented as described above with reference to Figure 4Aand Figure 4B The technology discussed.

[0129] The method begins at operation 502, where a first requested clip point is received or generated by the video analysis and editing system 214 based on an analysis of the video compared to its assigned time slot. At operation 504, the video analysis and editing system 214 determines the closest i-frame to the requested clip point and sets the starting clip point to the location of the closest i-frame at operation 506. Then, at operation 508, a preliminary location of a second clip point is determined by adding the time of the desired video segment to the starting clip point. Then, at operation 510, the video analysis and editing system 214 determines the closest neighboring clip points to the preliminary second clip point.

[0130] At operation 512, if the current cumulative offset is zero, the later available second clipping point is set as the second clipping point. At operation 514, if the current cumulative offset is not zero, the available second clipping point that tends to reduce the size of the cumulative offset is selected as the actual second clipping point. Figure 4A and Figure 4B An example of how the video analysis and editing system 214 makes this selection is discussed. Note that in some cases, depending on the method used, the actual size of the cumulative offset may not actually decrease in some instances, but the overall operation of the method will tend to reduce the cumulative offset as video clips are added to the video montage.

[0131] Then, in operation 516, the cumulative offset is updated by adding the difference between the time of the selected second clip point and the time of the preliminary second clip point to the current cumulative value, adjusting as needed to make any changes to the playback speed of the video segments in the video montage. Then, in operation 518, the video is trimmed at the selected clip point. Then, the method returns to operation 320 of flowchart 300 in operation 520.

[0132] Figure 6The illustrative embodiment of the present invention is a diagrammatic representation of a machine 600 (e.g., client device 102) within which instructions 610 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 600 to perform any one or more of the methods described herein. For example, the instructions 610 may cause the machine 600 to perform any one or more of the methods described herein. The instructions 610 convert a general purpose, unprogrammed machine 600 into a specific machine 600 that is programmed to perform the functions described and illustrated in the manner described. The machine 600 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 600 may operate as a server machine or a client machine in a server-client network environment or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 600 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 610 specifying actions to be taken by the machine 600. In addition, although only a single machine 600 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 610 to perform any one or more of the methods discussed herein. For example, the machine 600 may include any of the client device 102 or a plurality of server devices forming part of the messaging server system 108. In some examples, the machine 600 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of the particular method or algorithm are performed on the client side.

[0133] The machine 600 may include a processor 604, a memory 606, and an input / output (I / O) component 602 that may be configured to communicate with each other via a bus 640. In an example, the processor 604 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 608 that executes instructions 610 and a processor 612. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 6 Multiple processors 604 are shown, but machine 600 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0134] The memory 606 includes a main memory 614, a static memory 616, and a storage unit 618, all of which are accessible by the processor 604 via the bus 640. The main memory 606, the static memory 616, and the storage unit 618 store instructions 610 that implement any one or more of the methods or functions described herein. The instructions 610 may also reside, completely or partially, within the main memory 614, within the static memory 616, within the machine-readable medium 620 within the storage unit 618, within at least one of the processors 604 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 600.

[0135] The I / O components 602 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 602 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It should be appreciated that the I / O components 602 may include Figure 66. In various examples, the I / O components 602 may include a user output component 626 and a user input component 628. The user output component 626 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 628 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides the location and force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0136] In other examples, the I / O component 602 may include a biometric component 630, a motion component 632, an environmental component 634, or a position component 636, as well as a wide range of other components. For example, the biometric component 630 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 632 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0137] Environmental components 634 include, for example, one or more cameras (with still image / photo and video capabilities), lighting sensor components (e.g., a photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., a barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., an infrared sensor that detects nearby objects), gas sensors (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0138] With respect to cameras, client device 102 may have a camera system including, for example, a front-facing camera on a front surface of client device 102 and a rear-facing camera on a rear surface of client device 102. The front-facing camera may, for example, be used to capture still images and videos of a user of client device 102 (e.g., a "selfie"), which may then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera may be used to capture still images and videos in a more conventional camera mode, which images are similarly enhanced with the enhancement data. In addition to the front-facing camera and the rear-facing camera, client device 102 may also include a 360° camera for capturing 360° photos and videos.

[0139] Additionally, the camera system of the client device 102 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even triple, quad, or quintuple rear camera configurations on the front and back sides of the client device 102. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0140] The location component 636 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0141] Various technologies may be used to implement communications. The I / O components 602 also include a communication component 638 that is operable to couple the machine 600 to the network 622 or device 624 via a corresponding coupling or connection. For example, the communication component 638 may include a network interface component or another suitable device that interfaces with the network 622. In other examples, the communication component 638 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Parts (e.g. Low energy consumption), Components, and other communication components for providing communication via other forms. Device 624 can be another machine or any peripheral device among various peripheral devices (e.g., a peripheral device coupled via USB).

[0142] In addition, the communication component 638 can detect the identifier, or include a component that is operable to detect the identifier. For example, the communication component 638 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF417, UltraCode, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 638, such as obtaining positioning via Internet Protocol (IP) geolocation, ... Positioning is obtained through signal triangulation, positioning is obtained via detection of NFC beacon signals that can indicate a specific position, etc.

[0143] Various memories (e.g., main memory 614, static memory 616, and memory of processor 604) and storage unit 618 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 610) when executed by processor 604 cause various operations to implement the disclosed examples.

[0144] The instructions 610 may be sent or received over the network 622 using a transmission medium via a network interface device (e.g., a network interface component included in the communication component 638) and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 610 may be sent or received via a coupling (e.g., a peer-to-peer coupling) with the device 624 using a transmission medium.

[0145] Figure 7 700 is a block diagram illustrating a software architecture 704 that may be installed on any one or more of the devices described herein. The software architecture 704 is supported by hardware such as a machine 702 including a processor 720, a memory 726, and an I / O component 738. In this example, the software architecture 704 may be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 704 includes layers such as an operating system 712, a library 710, a framework 708, and an application 706. In operation, the application 706 invokes an API call 750 through the software stack and receives a message 752 in response to the API call 750.

[0146] The operating system 712 manages hardware resources and provides public services. The operating system 712 includes, for example, a kernel 714, services 716, and drivers 722. The kernel 714 acts as an abstraction layer between the hardware layer and other software layers. For example, the kernel 714 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 716 can provide other public services to other software layers. Drivers 722 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 722 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communication drivers (e.g., USB drivers), Drivers, audio drivers, power management drivers, etc.

[0147] The library 710 provides a common low-level infrastructure used by the application 706. The library 710 may include a system library 718 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 710 may include an API library 724, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting in two dimensions (2D) and three dimensions (3D) in graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 710 may also include various other libraries 728 to provide many other APIs to the application 706.

[0148] The framework 708 provides a common high-level infrastructure used by the applications 706. For example, the framework 708 provides various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The framework 708 can provide a wide range of other APIs that can be used by the applications 706, some of which can be specific to a particular operating system or platform.

[0149] In an example, applications 706 may include a home application 736, a contacts application 730, a browser application 732, a book reader application 734, a positioning application 742, a media application 744, a messaging application 746 (e.g., a messaging client 104), a game application 748, and a variety of other applications such as third-party applications 740. Applications 706 are programs that perform functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 706 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, third-party applications 740 (e.g., applications created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, third-party applications 740 can activate API calls 750 provided by the operating system 712 to facilitate the functions described herein.

[0150] Glossary

[0151] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0152] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0153] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, including the third generation partnership project (3GPP) of 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0154] "Component" refers to a device, physical entity or logic with the following boundaries: the boundaries are defined by function or subroutine calls, branch points, APIs or other technologies that provide partitioning or modularization for specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be packaged functional hardware units designed for use with other components and can be part of a program that generally performs specific functions in related functions. Components can constitute software components (e.g., codes embodied on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function and is no longer a general-purpose processor. It should be recognized that it can be determined for cost and time considerations whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system to implement the hardware component. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. The software configures a specific one or more processors accordingly, for example, to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be realized by (for example, by appropriate circuits and buses) transmitting signals between or among two or more hardware components in these hardware components. In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be realized, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (for example, a collection of information). The various operations of the example methods described in this article can be performed at least in part by one or more processors that are temporarily configured (for example, by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described in this article can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations of the method can be performed by one or more processors 604 or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or as a "software as a service" (SaaS) operation. For example, at least some operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of certain operations can be distributed between processors, not only resident in a single machine, but also deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographic location (for example, in a home environment, an office environment, or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographic locations.

[0155] "Computer-readable storage media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0156] "Ephemeral messages" are messages that can be accessed for a limited duration of time. Ephemeral messages can be text, images, videos, etc. The access time of an ephemeral message can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0157] "Machine storage media" refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media", "device storage media", and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media", "computer storage media", and "device storage media" explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal media".

[0158] “Non-transitory computer-readable storage medium” refers to tangible media capable of storing, encoding, or carrying instructions to be executed by a machine.

[0159] "Signal medium" refers to any intangible medium that can store, encode or carry instructions executed by a machine and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal in which one or more of its characteristics is set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

Claims

1. A method for generating a video montage from a collection of video clips taken from a plurality of video files, performed by one or more processors, the method comprising: performing facial recognition on a video file of the plurality of video files to identify a portion of the video file that includes a face; determining a relevance of the faces in the portion of the video file; generating a video clip segment from the video file by trimming the portion of the video file including the face, the length of the video clip segment being based on the relevance of the face in the video file; as well as The video clip is added to the video montage.

2. The method according to claim 1, wherein: Trimming the portion of the video file including the face comprises: Based on the face not being a close-up of the face, trimming the portion of the video file to a first length; and Based on the face being a close-up of the face, the portion of the video file is trimmed to a second length that is greater than the first length.

3. The method according to claim 1, wherein: Trimming the portion of the video file including the face comprises: Based on the portion of the video file not including speech, trimming the portion of the video file to a first length; and Based on the portion of the video file including the face and the voice, the portion is trimmed to a second length greater than the first length.

4. The method according to claim 1, wherein: Trimming the portion of the video file including the face comprises: trimming the portion of the video file to a first length based on the portion not including a face of a user of a device having at least one of the one or more processors; and Based on the portion of the video file including the face of the user of the device, the portion of the video file is trimmed to a second length that is greater than the first length.

5. The method according to claim 1, wherein: Trimming the portion of the video file including the face comprises: trimming the portion to a first length based on the portion of the video file not including a face of a person related to a user of a device including at least one of the one or more processors; and Based on the portion of the video file including a face of a person having a relationship with a user of the device, the portion is trimmed to a second length that is greater than the first length.

6. The method according to claim 1, wherein: Trimming the portion of the video file including the face comprises: trimming a portion of the video file including the face but not including the caption to a first length; and A portion of the video file including the face and the caption is trimmed to a second length greater than the first length.

7. The method according to claim 1, further comprising: creating a video montage file by adding together a plurality of video clip segments from the collection of video clip segments; as well as Add an audio track to the video montage file.

8. A computing device, comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, configure the computing device to perform operations for generating a video montage from a collection of video clip segments taken from a plurality of video files, the operations comprising: performing facial recognition on a video file of the plurality of video files to identify a portion of the video file that includes a face; determining a relevance of the faces in the portion of the video file; generating a video clip segment from the video file by trimming the portion of the video file including the face, the length of the video clip segment being based on the relevance of the face in the video file; and The video clip is added to the video montage.

9. The computing device according to claim 8, wherein: Trimming the portion of the video file including the face comprises: Based on the face not being a close-up of the face, trimming the portion of the video file to a first length; and Based on the face being a close-up of the face, the portion of the video file is trimmed to a second length that is greater than the first length.

10. The computing device according to claim 8, wherein: Trimming the portion of the video file including the face comprises: Based on the portion of the video file not including speech, trimming the portion of the video file to a first length; and Based on the portion of the video file including the face and the voice, the portion is trimmed to a second length greater than the first length.

11. The computing device according to claim 8, wherein: Trimming the portion of the video file including the face comprises: Based on the portion of the video file not including a face of a user of the computing device, trimming the portion to a first length; and Based on the portion of the video file including the face of the user of the computing device, the portion of the video file is trimmed to a second length that is greater than the first length.

12. The computing device according to claim 8, wherein: Trimming the portion of the video file including the face comprises: trimming the portion of the video file to a first length based on the portion not including a face of a person related to a user of the computing device; and Based on the portion of the video file including a face of a person having a relationship with a user of the device, the portion is trimmed to a second length that is greater than the first length.

13. The computing device according to claim 8, wherein: The operations also include: creating a video montage file by adding together a plurality of video clip segments from the collection of video clip segments; and Add an audio track to the video montage file.

14. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations for generating a video montage from a collection of video clip segments taken from a plurality of video files, the operations comprising: performing facial recognition on a video file of the plurality of video files to identify a portion of the video file that includes a face; determining a relevance of the faces in the portion of the video file; generating a video clip segment from the video file by trimming the portion of the video file including the face, the length of the video clip segment being based on the relevance of the face in the video file; as well as The video clip is added to the video montage.

15. The computer-readable storage medium of claim 14, wherein: Trimming the portion of the video file including the face comprises: Based on the face not being a close-up of the face, trimming the portion of the video file to a first length; and Based on the face being a close-up of the face, the portion of the video file is trimmed to a second length that is greater than the first length.

16. The computer-readable storage medium of claim 14, wherein: Trimming the portion of the video file including the face comprises: Based on the portion of the video file not including speech, trimming the portion of the video file to a first length; and Based on the portion of the video file including the face and the voice, the portion is trimmed to a second length greater than the first length.

17. The computer-readable storage medium of claim 14, wherein: Trimming the portion of the video file including the face comprises: Based on the portion of the video file not including a face of a user of the computer, trimming the portion to a first length; and Based on the portion of the video file including a face of the user of the computer, the portion of the video file is trimmed to a second length that is greater than the first length.

18. The computer-readable storage medium of claim 14, wherein: Trimming the portion of the video file including the face comprises: trimming the portion of the video file to a first length based on the portion not including a face of a person related to a user of the computer; and Based on the portion of the video file including a face of a person related to the user of the device, the portion is trimmed to a second length that is greater than the first length.

19. The computer-readable storage medium of claim 14, wherein: Trimming the portion of the video file including the face comprises: trimming a portion of the video file including the face but not including the caption to a first length; and A portion of the video file including the face and the caption is trimmed to a second length greater than the first length.

20. The computer-readable storage medium of claim 14, wherein: The operations also include: creating a video montage file by adding together a plurality of video clip segments from the collection of video clip segments; and Add an audio track to the video montage file.

Images