Enhanced clip assistance for video editing environments

Abstract

In an implementation, a software application on a computing device directs the device to extract audio data from a media file that includes audio data and video data, generate clip suggestions based on speech analysis in the audio data, and display a visualization of the clip suggestions. The user may provide an input including a selection of at least a portion of the visualization of the clip suggestion as a response of the device to the clip of the media file.

Description

Technical Field

[0001] Aspects of the present disclosure relate to the field of computer software applications, including tools for creating, editing, and sharing videos. Background Art

[0002] An increasing number of computer software applications and services enable users to create, edit, and share videos with each other. Social video platforms are particularly popular, where users can upload short videos to the service platform for distribution to their friends, classmates, or other contacts in their social network. In fact, short video sharing functions have been incorporated into many other scenarios, including classroom tools, office productivity tools, gaming environments, etc.

[0003] Users often want to edit their videos before sharing them. Many video editing environments provide controls that allow users to add music to a video, introduce background effects, or otherwise enrich its content. Video trimming is a specific feature that allows users to remove portions of a video clip. For example, a user may want to remove filler words or pauses from a video clip. To do this, the user must search the video clip for instances of the undesired words, phrases, or pauses, and then mark the sub-segment for removal.

[0004] Unfortunately, these features are often manual operations, requiring the user to specify range(s) within the segment to be cut. Not only can the task of specifying range(s) become tedious, but even in the context of short video segments, the task of finding the portion of the segment to cut can be equally time consuming. For longer segments, these shortcomings are only exacerbated. Summary of the invention

[0005] Disclosed herein are techniques that can improve user experience with respect to editing features and functions in a video editing environment. In an implementation, a software application on a computing device directs the device to extract audio data from a media file that includes audio data and video data, generate editing suggestions based on speech analysis in the audio data, and display a visualization of the editing suggestions. A user can provide input, including selecting at least a portion of the visualization of the editing suggestions, in response to the device editing the media file.

[0006] This summary introduces selected concepts in a simplified form that are further described in the technology disclosed below. It should be understood that this summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Many aspects of the present disclosure may be better understood by reference to the following drawings. The components in the drawings are not necessarily to scale, but rather the emphasis is placed on clearly illustrating the principles of the present disclosure. In addition, in the drawings, the same reference numerals designate corresponding parts throughout the multiple views. Although several embodiments are described in conjunction with these drawings, the present disclosure is not limited to the embodiments disclosed herein. On the contrary, it is intended to cover all alternatives, modifications, and equivalents.

[0008] Figure 1 The diagram illustrates the operating environment in the implementation.

[0009] Figure 2 The diagram illustrates the editing process in implementation.

[0010] Figure 3 The diagram illustrates the system architecture in implementation.

[0011] Figure 4 The diagram illustrates the operation scenario in the implementation.

[0012] FIG. 5A to FIG. 5C Diagram showing the user experience in implementation

[0013] FIG. 6A to FIG. 6E Diagram showing the user experience in implementation.

[0014] Figure 7 The diagram illustrates a computing system suitable for implementing the various operating environments, architectures, processes, scenarios, and sequences discussed below with respect to the other figures. DETAILED DESCRIPTION

[0015] Various implementations are disclosed herein, relating to user experience with respect to video editing tools in a software application environment, and in particular, functions and features that allow a user to automatically or semi-automatically edit portions of a video. The enhanced editing capabilities disclosed herein improve the ease of use of video tools and improve the overall experience of video capture, production, and sharing. Various technical effects achieved by the implementations disclosed herein include reducing the time and steps required to edit a video, and reducing the size of the video.

[0016] Various implementations described herein employ a clipping process on one or more computing devices that allows for automatic or semi-automatic removal of one or more video segments from a media file. The clipping process may be employed locally relative to the user experience (e.g., on the user's device), remotely relative to the user experience (e.g., on a server), or distributed between or among multiple devices.

[0017] In various implementations, such computing devices include one or more processors operatively coupled to one or more computer-readable storage media. Program instructions stored on the one or more computer-readable storage media, when executed by the one or more processors, instruct a given computing device to perform various steps relative to a representative editing process. For example, the computing device extracts audio data from a media file including audio data and video data, generates editing suggestions indicating video segments that may be edited from the media file, and displays a visualization of the video data and a visualization of the editing suggestions. A user can then provide input via a user interface indicating a selection of at least a portion of the visualization of the editing suggestions, and the computing device responsively edits the media file based on the user input.

[0018] In some implementations, editing suggestions can be generated based on analysis of speech represented in the audio data. The analysis can be performed locally (e.g., on a client device) or remotely (e.g., on a server computer in the cloud) relative to the user experience. If the speech analysis is performed remotely, the computing device provides the audio data to an analysis service in the cloud and receives analysis results from the analysis service. The analysis results may include timestamps that define segments of the audio data determined by the analysis service to represent undesirable portions of speech (e.g., filler words, long pauses, etc.). The computing device identifies segments of the video based at least on the timestamps, including by identifying segments of the video corresponding to segments of the audio defined by the timestamps in the results of the analysis.

[0019] In some implementations, the visualization of the video data is a linear representation of the video data. Examples of visualizations of edit suggestions include visually emphasizing segments of the video data indicated in the edit suggestions relative to other segments of the video not indicated in the edit suggestions. In some cases, the visualization of the edit suggestions is overlaid onto a visualization of the video data. Examples of selecting portions of the visualization of the edit suggestions include one or more selections of one or more of the segments in the visualization of the video data.

[0020] In some implementations, the results of the speech analysis include a transcription of the audio data. The computing device may, in some cases, display a visualization of the transcription, allowing the visualization of the clip suggestions to include visual emphasis on portions of the visualization of the transcription corresponding to segments of the audio defined by timestamps in the results of the analysis. The selection provided by the user input may be a selection of one or more of the portions of the visualization of the transcription (instead of, or in addition to, any selection of portions of the visualization of the video data).

[0021] Clipping a media file based on user input may include removing at least one of the segments of video data from the media file and removing one or more corresponding segments of audio data from the media file. In some cases, the video and audio data may be permanently removed. Alternatively, the clipped portion of the media file may be retained in the file, but substantially removed so that the clipped portion is skipped during playback. In some cases, a new version of the media file may be created, which represents the clipped version that does not include the clipped portion of the media file.

[0022] Figure 1 An operating environment 100 in an implementation is illustrated. Operating environment 100 includes computing device 101 and online service 110. Computing device 101 includes one or more software applications capable of employing editing processes with respect to a media file (e.g., media file 107). Computing device 101 can also drive a user experience by displaying a user interface of the application, receiving user input via the user interface, and responding to the user input for each editing process. Examples of computing device 101 include personal computers, tablet computers, mobile phones, and any other suitable devices. Figure 7 The computing device 701 in is a broad representation of this. Example applications include video editing applications and video sharing applications, as well as any other applications with integrated such functionality. The application can be a locally installed and executed application, a browser-based application, or a mobile application, and can be fully executed in a standalone manner, in the context of another application, or otherwise. It should be understood that in some implementations, some features and functions implemented by applications on computing device 101 can be performed by online service 110.

[0023] The online service 110 provides one or more computing services, including a speech analysis service, for endpoints such as computing device 101. The speech analysis service provided by the online service 110 takes audio data as input and generates a transcription of the speech encoded in the audio data. The analysis service can also be used to verify whether there are filler words, long pauses, and (optionally) sensitive words or phrases in the transcription. The analysis service provides speech analysis as output, wherein only the online service 110 can be returned to the endpoint providing the audio data. The online service 110 can optionally provide various other services, including video sharing, file storage, co-authoring and collaboration support, etc. In some examples, the online service 110 can provide a set of applications and services for various computing workloads (such as social sharing, office productivity tasks, email, chat, voice and video, etc.).

[0024] The online service 110 employs one or more server computers that may be co-located or distributed in one or more data centers connected to or otherwise communicating with the computing device 101. Examples of these servers include web servers, application servers, virtual servers, or physical (bare metal) servers, or any combination or variation of these servers. Figure 7 The computing device 701 in FIG. 1 is broadly representative. The computing device 101 can communicate with the online service 120 via one or more intranets, the Internet, wired and wireless networks, local area networks (LANs), wide area networks (WANs), and any other type of network or combination of networks.

[0025] As mentioned, computing device 101 employs an editing process that improves the video editing capabilities of software applications. Figure 2 The editing process 200 in FIG. 2 is representative. The editing process 200 can be implemented by program instructions in the context of any software application, module, component, or other such element of one or more computing devices. The program instructions direct one or more computing devices to operate in the following manner, and for clarity, the computing device is referred to in the singular.

[0026] In operation, a media file includes audio data and video data, the audio data representing sounds encoded according to an audio coding format, and the video data representing images encoded according to a video coding format. A computing device employing the clipping process 200 extracts the audio data from the media file (step 201) and provides the audio data to an online service. The service analyzes the speech transcription represented in the audio data and responds with a transcription and a timestamp that matches the audio stream.

[0027] The computing device generates editing suggestions indicating segments of the video that may be edited from the media file based on the analysis of the speech provided by the online service (step 203). For example, the computing device can check for filler words, long pauses, or other such content in the transcription so that it may be removed from the video. Alternatively, the transcription itself may have identified filler words, long pauses, and / or other undesirable content. In either case, the computing device uses the timestamps for the audio data to identify the segments of the corresponding video data that may be edited from the media file, which constitute the basic components of the editing suggestions. In other words, the editing suggestions are a list of segments of the video data that correspond in time to the portions of the audio data identified as containing filler words, long pauses, etc.

[0028] Next, the computing device displays a visualization of the video data and a visualization of the editing suggestions (step 205). The visualization of the video data can be, for example, a linear representation of the video, such as a timeline view. The visualization of the editing suggestions can be overlaid relative to the visualization of the video data, allowing the user to easily view the portion of the video that is suggested to be edited.

[0029] The computing device displays the visualization in a user interface through which a user can provide input relative to the visualization of the editing suggestion. In some cases, the user input can be a selection of a button or other such element in the user interface, which selection directs the computing device to edit all portions of the video represented in the editing suggestion. In other cases, the user input can be a selection of only a subset of the video portions represented in the editing suggestion, thereby directing the computing device to edit only those selected portions. In any case, the computing device receives the user input and responsively edits the media file based on the user input (step 207).

[0030] Back to Figure 1 , operating environment 100 includes a brief example of an editing process 200 employed by an application on computing device 101. In operation, a user captures a video of a moment or scene, such as capturing the same or different person speaking. The moment may be, for example, a speech by the user captured using the user's device, a video conference call recorded on the user's device, or any other type of video capture. Here, the video may be captured by computing device 101 or a different device of the user (e.g., a phone, tablet, etc.). The video may be of any length and quality, including short and long clips.

[0031] The captured video is stored in a media file 107 suitable for playback in any number of media applications and includes both audio data and video data. The media file 107 may also be referred to as a video file, but in any case, the generated file includes both audio data and video data. The audio data includes an audio signal encoded in accordance with a suitable audio codec, while the video data includes a video image encoded in accordance with a suitable video codec.

[0032] The computing device 101 (under the control of the software application) extracts audio data from the media file and sends it to the online service 110. The online service 110 generates a transcription of the speech represented in the audio data and provides an analysis of the speech to the computing device 101. In some cases, the online service 110 returns only the transcription. In other cases, the online service 110 also analyzes the speech and annotates descriptions of the speech portions in the transcription. For example, the analysis returned by the online service 110 may include timestamps corresponding to moments in the speech content that are determined to be filler words and long pauses. Optionally, the analysis may also identify other characteristics of the speech, such as the specific content discussed at different points in time in the recording, changes in the speaker's tone at different points in time, and the like. When the online service 110 returns only the transcription, the software on the computing device 101 may perform speech analysis that is otherwise attributed to the online service 110.

[0033] The computing device 101 receives the speech analysis from the online service 110 and proceeds to generate editing suggestions for display to the end user. Generating editing suggestions may include examining annotations in the speech analysis to identify portions of the audio data that contain filler words and long pauses (or other qualifying characteristics). The computing device then identifies segments of video data that correspond in time to segments of the audio data that contain qualifying characteristics. Such metadata may then be converted into a visualization of the editing suggestions, such as a graphic overlaid on a visualization of the video data. For example, when the video data is displayed linearly in time, an editing graphic may be overlaid on portions of the linear representation to indicate that these portions are included in the editing suggestions.

[0034] The user can choose whether to clip all the suggested segments or only some of the segments. The clipping function 105 employed in the software environment running on the computing device 101 takes the user's selection as input and clips the media file 107 accordingly. Clipping the media file includes removing the audio data and video data specified by the user's selection. The result is a smaller version of the media file 107 because some audio and video data have been removed. The user can then upload the clipped version of the media file to share with other users.

[0035] Figure 3 A system architecture 300 is illustrated that may be employed in an implementation to provide the enhanced editing functionality disclosed herein. The system architecture 300 includes an application 301, a media subsystem 303, a user interface subsystem 305, and an analysis service 310. The system architecture 300 may be implemented on a single computing device or distributed among multiple computing devices and is generally applicable to representative computing hardware, software, and / or firmware elements for performing video editing operations.

[0036] Application 301 is representative of any software application or component thereof capable of performing an editing process (e.g., editing process 200). Application 301 interfaces with media subsystem 303, user interface subsystem 305, and analysis service 310. Application 301 includes editing module 302, which represents any component or components of application 301 capable of editing media file 307.

[0037] Media subsystem 303 represents a subsystem on a computing device capable of recording video, examples of which include internal and external cameras and microphones. Media subsystem 303 can be, for example, a laptop camera, a cell phone camera, an external camera, a camera integrated into a display, or any other suitable type of camera, as well as the hardware, software, and firmware layers that connect media subsystem 303 to application 301.

[0038] User interface subsystem 305 represents a subsystem on a computing device that is capable of user interface interaction with application 301 and receiving user input. User interface subsystem 305 may include, for example, a display screen that displays user interface elements, input devices through which a user provides user input, and hardware, software, and firmware layers that connect user interface subsystem 305 to application 301.

[0039] Analysis service 310 represents one or more services capable of analyzing audio data to provide a transcription of the speech contained in the audio and an analysis of the speech. Analysis service 310 can be implemented remotely from application 301 (e.g., on a different computing device), or can be implemented in a co-existent manner (e.g., on the same computing device as application 301).

[0040] Figure 4 An operational scenario 400 is illustrated that depicts the interoperation of elements in the system architecture 300. In operation, a user clicks, touches, or otherwise prompts application 301 to begin capturing content (i.e., recording video) via user interface subsystem 305 engaging application 413. Application 301 responsively interacts with media subsystem 303 to begin recording audio and video of the scene. Here, for exemplary purposes, it is assumed that the scene being captured is a scene of the same user speaking.

[0041] The media subsystem 303 captures the user's audio and video and saves the data to a media file. The application 301 can access the media file for playback on the user interface subsystem 305. Accordingly, the video (combined audio and video data) can be rendered and played on a display screen or other such device, including speakers represented by the output audible audio data.

[0042] The screen displayed by the user interface subsystem 305 includes various menus and features for interacting with the video, including a button for clipping the video. User input including selecting the button causes the user interface subsystem 305 to transmit the selection to the application 301. The application 301 extracts the audio data from the media file in response and sends it to the analysis service 310.

[0043] The analysis service 310 receives the audio data and performs analysis to detect and transcribe any speech that may be represented therein. The analysis service 310 replies to the application 301 with the transcription and analysis information characterizing various aspects of the transcription. For example, the analysis information may identify portions of the audio data that contain filler words, long pauses, or other such content.

[0044] Application 301 receives the results of the analysis from analysis service 310 and proceeds to generate editing suggestions to be displayed by user interface subsystem 305. To generate editing suggestions, application 301 identifies portions of the video data that are flagged by the analysis information as corresponding to portions of the audio data that contain filler words, etc. At a minimum, the editing suggestions include a list of corresponding video segments so identified. Although, in most cases, the editing suggestions include visual representations of the corresponding video segments, e.g., visual representations of the flagged segments displayed in association with a more general visual representation of the video.

[0045] Application 301 provides visual representations of video data and editing suggestions to user interface subsystem 305 for display on a screen or other suitable surface. User interface subsystem 305 displays visual representations of the video and editing suggestions for user consideration. Here, for example purposes, it is assumed that the user decides to edit all or part of a video file.

[0046] The user interface subsystem 305 receives the selection through user input and transmits the selection to the application 301. The application 301 responsively clips the media file according to the selection expressed by the user input. In this example, the function of the clipping module 302 is to remove the audio data and video data portions involved in the user selection. For example, if the user chooses to implement the entire clipping suggestion, all segments of the video data indicated in the clipping suggestion will be removed from the media file 307, and all segments in the corresponding audio data segments will also be removed. However, if the user only selects a subset of the segments indicated in the clipping suggestion, only the subset of the video segments (and the corresponding subset of the audio segments) will be removed from the media file 307.

[0047] Application 301 may then provide a trimmed version of the media file 307 for playback or other display via user interface subsystem 305. The user may preview the trimmed version and may ultimately decide to upload the file to a sharing service or other such destination. Not only will the trimmed version of the media file 307 be of reduced size, but its content will also have fewer filler words and long pauses.

[0048] FIG. 5A to FIG. 5C The user experience 500 in the implementation of the enhanced editing function is illustrated. The user experience 500 can be generated by a software application executed on a suitable computing device. For example, the user experience 500 can be generated by the application 301 executing the editing process 200 on the computing device 101.

[0049] User experience 500 begins by displaying user interface 501 to a video editing application or application component. User interface 501 includes a visual area 503 where a video can be displayed and played by a user. User interface 501 also includes a video timeline 505 that displays the various frames of a video in a linear, time-based manner. The user can fast forward, jump, or otherwise navigate the video clip, where the video clip has a duration of 1 minute and 7 seconds (1:07). Marker 506 is a graphical indication of the location on the timeline where the video is currently paused or playing.

[0050] User interface 501 also includes a feature menu 511 that includes various feature elements that the user can select to activate features of the application. Here, feature menu 511 includes an add music button 513 that, when selected, allows the user to add background music to the video displayed in visual area 503. Feature menu 511 also includes an editing assistance button 515 that, when selected, activates the enhanced editing assistance process described herein. Assuming editing assistance button 515 is selected, user experience 500 will transition to Figure 5B .

[0051] In response to selecting the editing assistance button 515, the application driving the user interface 501 will extract the audio data from the video file, obtain an analysis of any speech in the audio data and generate editing suggestions. The application will then display a visualization of the editing suggestions in the user interface 501.

[0052] exist Figure 5BAt this stage, the feature menu 511 has been transformed to include a delete segment button 517 and a confirmation button 519. The user can select the delete segment button to completely delete the video segment. The confirmation button 519 can be selected to continue the visualization of the editing suggestions in the user interface 501. The visualization of the editing suggestions includes a border 521 and a border 523 around two different segments in the video segment, and the border 521 and the border 523 are overlaid relative to the video timeline 505. The user can select one or two highlighted segments, or even other segments of the video to remove from the segment. Once the user is satisfied with the selected segments, the confirmation button 519 can be selected to continue removing the corresponding audio and video data portions from the video file.

[0053] exist Figure 5C 504. For illustration purposes, it is assumed that the user has confirmed the editing suggestion and the selected video segment has been deleted from the file. Accordingly, the video clip has been shortened to forty seconds (0:40), and the number of video segments displayed in the video timeline 505 has been reduced because there are fewer remaining parts in the video file.

[0054] FIG. 6A to FIG. 6E Another user experience 600 in an enhanced editing function implementation is illustrated. The user experience 600 may also be generated by a software application executed on a suitable computing device. For example, the user experience 600 may be generated by an application 301 executing the editing process 200 on the computing device 101.

[0055] User experience 600 begins by presenting a user interface 601 to a video editing application or application component. User interface 601 includes a viewable area 603 where a video can be displayed and played by a user. User interface 601 also includes a video timeline 605 that displays the frames of a video in a linear, time-based manner. The user can fast forward, jump, or otherwise navigate the video clip, where the video clip has a duration of 1 minute and 7 seconds (1:07). Marker 506 is a graphical indication of where the video is currently paused or playing on the timeline.

[0056] User interface 601 also includes a feature menu 611, which includes various feature elements that the user can select to activate features of the application. Here, feature menu 611 includes an add music button 613, which, when selected, allows the user to add background music to the video displayed in visual area 603. Feature menu 611 also includes an editing assistance button 615, which, when selected, activates the enhanced editing assistance process disclosed herein. Assuming editing assistance button 615 is selected, user experience 600 will transition to Figure 6B .

[0057] In response to selecting the editing assistance button 615, the application driving the user interface 601 will extract the audio data from the video file, obtain an analysis of any speech in the audio data and generate editing suggestions. Here, the analysis includes a transcription of the speech in the audio data. The application will then display a visualization of the editing suggestions in the user interface 601, including a visualization 604 of the transcription.

[0058] exist Figure 6B At this stage, the feature menu 611 has transformed to include a delete segment button 617 and a confirmation button 619. The user can select the delete segment button to completely delete the video segment. The confirmation button 619 can be selected to continue the visual editing suggestion in the user interface 601. The visual representation of the editing suggestion includes a border 621 and a border 623 around two different segments of the video segment, and the border 621 and the border 623 are overlaid relative to the video timeline 605.

[0059] Note that borders 621 and 623 are overlaid relative to segments of the video, which for example purposes are assumed to correspond to the times at which filler words appear in the audio transcription (see: two occurrences of "ummmm" in the transcription of visualization 604). The user can select one or both of the highlighted segments, or even select other segments of the video to remove from the clip.

[0060] However, the user can also select portions of the timeline visualization 604. Figure 6C In , the user selects sentence 625 in the transcription. The software application converts the timestamp of the transcription to a corresponding point in the video data and integrates the selection into the clip suggestion. Fig.6D In C, the editing suggestion includes a new border 627 surrounding one or more segments of the video corresponding to the portion transcribed by selection 625.

[0061] Once the user is satisfied with the selected segments, the user may select the confirmation button 619 to proceed with removing the corresponding audio data portion and video data portion from the video file. Fig. 6E In the example of FIG. 604 , for illustration purposes, it is assumed that the user confirms the updated editing suggestion and the selected video clip has been deleted from the file. Therefore, the video clip has been shortened to thirty-six seconds (0:36), and the number of video segments displayed in the video timeline 605 has been reduced.

[0062] Figure 7A computing device 701 is illustrated, which represents any system or collection of systems that can implement the various processes, programs, services, and scenarios disclosed herein. Examples of computing device 701 include, but are not limited to, desktop and laptop computers, tablet computers, mobile computers, and wearable devices. Examples may also include server computers, network servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machines, containers, and any variations or combinations thereof.

[0063] The computing device 701 may be implemented as a single device, system or apparatus, or may be implemented as multiple devices, systems or apparatuses in a distributed manner. The computing device 701 includes, but is not limited to, a processing system 702, a storage system 703, software 705, a communication interface system 707 and a user interface system 709 (optional). The processing system 702 is operatively coupled with the storage system 703, the communication interface system 707 and the user interface system 709.

[0064] Processing system 702 loads and executes software 705 from storage system 703. Software 705 includes and implements a clipping process 706, which is a representative clipping process discussed with respect to previous diagrams (e.g., the clipping process shown in Figure 200). When executed by processing system 702, software 705 directs processing system 702 to operate in the manner described herein, including at least the various processes, operating scenarios, and sequences discussed in the aforementioned implementations. For the sake of brevity, computing device 701 may optionally include additional devices, features, or functions not discussed.

[0065] Continue to refer to Figure 7 , the processing system 702 may include a microprocessor and other circuits that retrieve and execute software 705 from a storage system 703. The processing system 702 may be implemented within a single processing device, but may also be distributed across multiple processing devices or subsystems that cooperate to execute program instructions. Examples of the processing system 702 include a general-purpose central processing unit, a graphics processing unit, a special-purpose processor, a logic device, and any other type of processing device, combination, or variation thereof.

[0066] Storage system 703 may include any computer-readable storage medium that can be read by processing system 702 and that can store software 705. Storage system 703 may include volatile and nonvolatile, removable and non-removable media that implement any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read-only memory, magnetic disks, optical disks, flash memory, virtual and non-virtual memory, magnetic tape, magnetic storage tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage medium. In any case, a computer-readable storage medium is not a propagated signal.

[0067] In addition to computer-readable storage media, in some implementations, storage system 703 may also include computer-readable communication media through which at least some of software 705 may communicate internally or externally. Storage system 703 may be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems, which may be co-located or distributed relative to each other. Storage system 703 may include additional elements, such as a controller, that is capable of communicating with processing system 702 or possibly other systems.

[0068] The software 705 (including the editing process 706) can be implemented as program instructions, and when these instructions are executed by the processing system 702, they can, among other functions, direct the processing system 702 to operate in accordance with the various operating scenarios, sequences, and processes illustrated herein. For example, the software 705 can include program instructions for implementing the editing process described herein.

[0069] Specifically, the program instructions may include multiple components or modules that cooperate or otherwise interact with each other to perform the various processes and operational scenarios described herein.

[0070] The various components or modules may exist in the form of compiled or interpreted instructions, or in some other variant or combination of instructions. The various components or modules may be executed in a synchronous or asynchronous manner, may be executed serially or in parallel, may be executed in a single-threaded environment or in a multi-threaded environment, or may be executed according to any other suitable execution paradigm, variant, or combination thereof. The software 705 may include additional processes, programs, or components, such as operating system software, virtualization software, or other application software. In addition, the software 705 may also include firmware or some other form of machine-readable processing instructions executable by the processing system 702.

[0071] In general, when software 705 is loaded into processing system 702 and executed, a suitable device, system or apparatus (of which computing device 701 is representative) can be converted as a whole from a general-purpose computing system to a special-purpose computing system customized to support enhanced editing features in an optimized manner. In fact, encoding software 705 onto storage system 703 can transform the physical structure of storage system 703. The specific changes in this physical structure can depend on various factors in different implementations of the present description. Examples of these factors can include, but are not limited to, the technology used to implement the storage system 703 storage medium, and whether the computer storage medium is classified as primary storage or secondary storage, as well as other factors.

[0072] For example, if the computer-readable storage medium is implemented as a semiconductor-based memory, then when program instructions are encoded therein, the software 705 can change the physical state of the semiconductor memory, such as by changing the state of transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory. Similar changes may also occur with respect to magnetic or optical media. Other changes in the physical media are also possible without departing from the scope of the present description, and the foregoing examples are provided only for the convenience of this discussion.

[0073] The communication interface system 707 may include communication connections and devices that allow for communication with other computing systems (not shown) via a communication network (not shown). Examples of connections and devices that allow for communication between systems may include network interface cards, antennas, power amplifiers, radio frequency circuits, transceivers, and other communication circuits. The connections and devices may exchange communications with other computing systems or system networks via a communication medium, such as metal, glass, air, or any other suitable communication medium. The media, connections, and devices described above are well known and do not need to be discussed in detail herein.

[0074] The communication between computing device 701 and other computing systems (not shown) can be carried out through one or more communication networks and in accordance with various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, network combination, or variation thereof. The communication networks and protocols described above are well known and need not be discussed in detail here.

[0075] As will be appreciated by those skilled in the art, various aspects of the present invention may be implemented as a system, method or computer program product. Accordingly, various aspects of the present invention may take the form of a complete hardware embodiment, the form of a complete software embodiment (including firmware, resident software, microcode, etc.), or the form of an embodiment in combination with software and hardware aspects, which may generally be referred to as a "circuit", "module" or "system" in this article. In addition, aspects of the present invention may take the form of a computer program product implemented in one or more computer-readable media, wherein there is a computer-readable program code implemented in.

[0076] It will be appreciated that while the inventive concepts disclosed herein are discussed in the context of such social applications, they are equally applicable to other environments, such as gaming applications, virtual and augmented reality applications, business applications, productivity applications, and other types of software applications.

[0077] In fact, the included description and figures depict specific embodiments to teach those skilled in the art how to make and use the best mode. For the purpose of teaching the principles of the invention, some conventional aspects have been simplified or omitted. Those skilled in the art will appreciate that variations within the scope of these embodiments still fall within the scope of the present disclosure. Those skilled in the art will also appreciate that the features described above can be combined in various ways to form a variety of embodiments. Therefore, the present invention is not limited to the specific embodiments described above, but is only limited to the claims and their equivalents.

Claims

1. A computing device, comprising: one or more computer-readable storage media; one or more processors operatively coupled to the one or more computer-readable storage media; as well as program instructions stored on the one or more computer-readable storage media, which, when executed by the one or more processors, direct the computing device to at least: extracting the audio data from a media file comprising audio data and video data; generating, based at least on the speech analysis represented in the audio data, editing suggestions indicating segments of the video data that can be edited from the media file; displaying a visualization of the video data and a visualization of the editing suggestions; receiving user input, the user input comprising a selection of at least a portion of the visualization of the clip suggestion; as well as The media file is clipped based at least on the user input.

2. The computing device of claim 1 , wherein to generate the editing suggestion, the program instructions direct the computing device to at least: providing the audio data to an analytics service; receiving results of the analysis from the analysis service, wherein the results of the analysis include timestamps defining segments of the audio data determined by the analysis service to represent undesirable portions of the speech; as well as The segment of the video data is identified based at least on the timestamp.

3. A computing device according to claim 2, wherein, in order to identify the segment of the video data based at least on the timestamp, the program instructions direct the computing device to identify the segment of the video data corresponding to the segment of the audio data defined by the timestamp in the results of the analysis.

4. The computing device of claim 3, wherein: The visualization of the video data comprises a linear representation of the video data; The visualization of the cut suggestion includes: a visual emphasis on the segment of the video data indicated in the cut suggestion relative to other segments of the video data not indicated in the cut suggestion; and The selection of at least a portion of the visualization of the editing suggestion comprises one or more selections of one or more of the segments in the visualization of the video data.

5. The computing device of claim 4, wherein the results of the analysis further include a transcription of the audio data, and wherein the program instructions further direct the computing device to display a visualization of the transcription.

6. The computing device of claim 5, the visualization of the editing suggestion further comprising: Visual emphasis of a portion of the visualization of the transcription, the portion corresponding to the segment of the audio data defined by the timestamp in the results of the analysis.

7. The computing device of claim 6, wherein the selection of at least a portion of the visualization of the clip suggestion further comprises: One or more selections of one or more of the portions of the visualization of the transcription.

8. A computing device according to claim 1, wherein in order to trim the media file based at least on the user input, the program instructions direct the computing device to remove at least one of the segments of the video data from the media file and to remove one or more corresponding segments of the audio data from the media file.

9. The computing device of claim 1, wherein the program instructions direct the computing device to overlay the visualization of the editing suggestion onto the visualization of the video data.

10. The computing device of claim 1, wherein segments of the video data correspond to segments of the audio data determined by the analysis of the speech to include undesirable content, the undesirable content comprising filler words.

11. A method comprising: extracting the audio data from a media file comprising audio data and video data; generating, based at least on the speech analysis represented in the audio data, editing suggestions indicating segments of the video that can be edited from the media file; displaying a visualization of the video data and a visualization of the editing suggestions; receiving user input, the user input comprising a selection of at least a portion of the visualization of the clip suggestion; as well as The media file is clipped based at least on the user input.

12. The method of claim 11, wherein generating the editing suggestion comprises: providing the audio data to an analytics service; receiving results of the analysis from the analysis service, wherein the results of the analysis include timestamps defining segments of the audio data determined by the analysis service to represent undesirable portions of the speech; as well as The segment of the video data is identified based at least on the timestamp.

13. The method of claim 11, further comprising overlaying the visualization of the editing suggestion onto the visualization of the video data.

14. The method of claim 11, wherein the segments of the video data correspond to segments of the audio data determined by the analysis of the speech to include undesirable content.

15. One or more computer-readable storage media having program instructions stored thereon, which, when executed by one or more processors of a computing device, direct the computing device to at least: extracting the audio data from a media file comprising audio data and video data; generating, based at least on the speech analysis represented in the audio data, editing suggestions indicating segments of the video data that can be edited from the media file; displaying a visualization of the video data and a visualization of the editing suggestions; receiving user input, the user input comprising a selection of at least a portion of the visualization of the clip suggestion; as well as The media file is clipped based at least on the user input.

Images