Processing of video operations as subtasks of video segments

Abstract

Techniques and solutions for performing video processing tasks are described. A video processing task is received, the video processing task including one or more operations to be performed on a digital video file and an identifier of the digital video file. A video processing task is divided into sub-tasks of operations to be performed on segments of a video, such as segments having a particular duration. The duration may correspond to a duration for video streaming. The disclosed techniques may provide improved fault tolerance as compared to video processing performed as a single task, since only failed tasks need to be reprocessed. Video processing subtasks may be distributed to multiple workers, which may further improve fault tolerance and may increase computing power available for video processing, including allowing the use of heterogeneous or unreliable workers.

Description

Technical Field

[0001] The present disclosure relates generally to video processing. Certain embodiments provide for processing of video-specific operations on segments of the video, rather than performing operations on the entire video file. Background Art

[0002] Since the early 1990s, video streaming has become a very popular use of the Internet and has disrupted entire industries - gone are the days of renting VHS tapes or DVDs from your local rental store. As computers have become smaller and more powerful, video streaming has migrated from desktop and laptop applications to being primarily used on tablet devices and smartphones.

[0003] Despite improvements in general computing hardware and networking technology, video streaming remains highly resource intensive. Thus, there is room for improvement. Summary of the invention

[0004] This summary is intended to introduce a group of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] Techniques and solutions for performing video processing tasks are described. A video processing task is received, the video processing task comprising one or more operations to be performed on a digital video file and an identifier of the digital video file. The video processing task is divided into subtasks of operations to be performed on segments of a video, such as segments having a particular duration. The duration may correspond to a duration used for video streaming. Compared to video processing performed as a single task, the disclosed techniques may provide improved fault tolerance because only failed tasks need to be reprocessed. The video processing subtasks may be distributed to multiple workers, which may further improve fault tolerance and may increase the computing power available for video processing, including allowing the use of heterogeneous or unreliable workers.

[0006] In one aspect, a method for dividing a video processing task into a plurality of subtasks that can be performed by one or more worker computing devices is provided. A video processing task is received, the video processing task including an identifier of a digital video file and one or more operations to be performed on at least a portion of the digital video file. A plurality of subtasks for processing the one or more operations are generated. A given subtask in the plurality of subtasks identifies or includes a segment in the digital video file to be processed as part of the subtask.

[0007] At least a first portion of the plurality of subtasks is scheduled to one or more worker computing devices. Subtasks of the at least a first portion of the plurality of subtasks are sent to respective worker computing devices for processing by the respective worker computing devices, thereby providing improved fault tolerance for the video processing task compared to executing the video processing task as a single task.

[0008] In another aspect, a method for performing a video processing subtask by a worker computing device is provided. A subtask of a video processing task is received from a coordinator computing system. The subtask includes one or more operations of the video processing task or instructions derived at least in part therefrom, and an identifier of a video segment to be processed for the subtask. At least the video segment is requested. At least the video segment is received. The operation or instruction is performed on at least the video segment. After or in conjunction with performing the operation or instruction, at least the video segment is encoded to provide a processed video segment.

[0009] On the other hand, the present disclosure provides an alternative method for performing a video processing subtask by a worker computing device. A subtask of a video processing task for a digital video file is received from a coordinator computing system. The subtask includes one or more operations of the video processing task or instructions derived at least in part from it, and a video segment to be processed for the subtask. The operation or instruction is performed on at least the video segment. After or in conjunction with performing the operation or instruction, at least the video segment is encoded to provide a processed video segment. The processed video segment can be streamed to a streaming client without further processing the processed video segment. The processed video segment is sent to a repository including other processed video segments of the digital video file.

[0010] The present disclosure also includes computing systems and tangible non-transitory computer-readable storage media configured to perform the above-described methods or including instructions for performing the above-described methods. As described herein, various other features and advantages may be incorporated into the technology as desired. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] 1 is a diagram illustrating a computing environment including support for video streaming at different video quality levels.

[0012] FIG. 2 is a diagram showing components of a streaming video, such as a video included in an MP4 container.

[0013] Figure 3 is a block diagram illustrating an exemplary computing environment in which the disclosed technology may be implemented, including a coordinator computing system and a worker computing system.

[0014] Figure 4A video processing method that can be performed by a coordinator computing system.

[0015] Figure 5 A video subtask processing method executable by a worker computing device.

[0016] Figure 6 is a flow chart of an exemplary technique for generating video processing subtasks.

[0017] Figure 7 is a flow chart of an exemplary technique for performing a video processing subtask that does not include video data to be processed by a worker computing device.

[0018] Figure 8 is a flow chart of an exemplary technique for performing a video processing subtask, where the subtask includes video data to be processed by a worker computing device.

[0019] Fig. 9 is a diagram of an exemplary computing system in which some described embodiments may be implemented.

[0020] Fig.10 is an exemplary cloud computing environment that can be used in conjunction with the techniques described herein. DETAILED DESCRIPTION

[0021] Example 1 - Overview

[0022] Since the early 1990s, video streaming has become a very popular use of the Internet and has disrupted entire industries - gone are the days of renting VHS tapes or DVDs from your local rental store. As computers have become smaller and more powerful, video streaming has migrated from desktop and laptop applications to being primarily used on tablet devices and smartphones.

[0023] Despite improvements in general computing hardware and networking technology, video streaming remains highly resource intensive. Thus, there is room for improvement.

[0024] Video processing operations may be performed in response to instructions to trim a video, apply effects to a video, add transitions or text to a video, merge different video files, or reorder portions of a video. Other exemplary operations may result from instructions to rotate all or a portion of a video, change the color profile of a video, apply a filter, or adjust the color levels in a video. Typically, these types of operations are applied to the entire video file, which can create a number of problems.

[0025] One problem that may occur is that the process fails, for example due to a hardware or software problem (e.g., a failure) on the device that is processing the particular instruction. If a failure occurs, the entire video processing operation usually needs to be restarted, which consumes additional time and computing resources.

[0026] Even if failure does not occur, performing video processing on a single computing device, processing the entire video file, may also be suboptimal. Video processing may be resource intensive, potentially leading to lengthy processing times, and reducing computing resources available for other processes. Further problems may arise, for example if the processed video is to be used for streaming. In this case, operations are usually applied to the source file to produce a target file. For streaming, the target file is usually processed to produce smaller video segments that can be sent during streaming. However, having a first encoding step for applying the video processing operation and a second encoding step for generating segments for streaming may reduce video quality. That is, video encoding techniques are usually "lossy", which means that the video quality is reduced due to the encoding operation.

[0027] Although video files are usually segmented in some way, segmentation usually involves having separate data streams for video, images, text or subtitles, and metadata. In particular, since video data forms the majority of video files, even separate processing of these separate data streams is difficult to improve the above-mentioned problem.

[0028] Disclosed innovation can solve the above-mentioned problem by dividing the source video file into separate video segments and then applying operations to realize instructions on individual segments rather than on the whole video as a single operation / task. In some cases, a single computing device can be used to process some or all of these video segments. Compared with processing the whole video file, it is also advantageous to process all video segments at a single computing device. One advantage is fault tolerance. If a computing process or hardware failure occurs, the segment that has been processed can be kept, so only the segment that is actively processed when failure occurs needs to be reprocessed.

[0029] In addition, processing video according to individual segments can allow the computing system to schedule video processing operations more flexibly. If a higher priority task is scheduled, the processing of the video segment can be delayed, or fewer computing resources can be allocated (e.g., less memory is allocated, and the scheduling of segment processing operations on the computer processor is reduced).

[0030] The disclosed techniques include segmenting (or splitting) a source video into segments corresponding to units for streaming, such as segments of a determined duration. By providing the segments in the same form as required for streaming, additional encoding steps are avoided, which reduces both computing resource usage and processing time, and provides improved video quality by avoiding additional encoding operations. However, in other cases, additional processing may be performed on the processed video segments produced using the disclosed techniques, including combining the segments into larger video portions or even complete videos.

[0031] The above advantages also apply to situations where some or all video segments are distributed to multiple worker computing systems or devices. Having multiple worker devices provides improved fault tolerance, as if a device fails, only one or more segments assigned to that device need to be reprocessed, and if the originally assigned worker is unavailable, it can be assigned to a different worker device. Because of this fault tolerance, a wider variety of worker devices can be used to perform video processing operations, including devices that may be unreliable, such as devices that may lose connectivity at a given time (when a subtask is to be performed), have insufficient computing resources to perform the processing operation (including because they are performing higher priority tasks), or otherwise become unavailable. Worker devices that may have sufficient capacity to process video segments can also be used, even if they will not be able to perform operations on the entire video file.

[0032] The nature of the worker devices may also vary, including worker devices with different form factors or levels of computing resources—smart phones, tablet devices, laptops, personal computers, servers, cloud computing systems, or other types of computing systems or devices may all be part of the distributed video processing task. Similarly, the worker computing devices may perform operations in different ways, such as having some devices perform encoding operations in software while other devices perform some or all operations in hardware.

[0033] Therefore, distributing the subtasks of the overall video processing task to multiple computing devices can make a larger amount of computing resources available, which can reduce the processing load on any one worker, and can speed up the processing time. In addition, additional prioritization and flexibility are provided, because the central orchestrator (orchestrator) can flexibly assign or reassign subtasks, including the availability based on the worker device. The orchestrator can also prioritize different distributed video processing tasks. As an example, some video processing tasks produce video segments that are stored and streamed when there is a client request. Other video processing tasks only encode segments when requested, and wherein the segments can be optionally cached for future use. If a "real-time" encoding request is received, the orchestrator can prioritize the task / its component subtasks, including delaying the processing of existing subtasks, occupying more workers, or assigning higher priority tasks to higher performance workers (that is, workers that may be more reliable, have greater throughput, etc.).

[0034] Example 2 - Exemplary Video Streaming Environment

[0035] 1 generally illustrates how a source video file 108 can be provided at various quality levels to different consumer devices, such as a desktop computer 112, a laptop device 114, a tablet computing device 116, and a smart phone 118. The consumer devices 112, 114, 116, 118 can differ in a variety of ways, including screen size, which can also be reflected in the resolution most available on the consumer device. The consumer devices 112, 114, 116, 118 can also differ in available computing resources (e.g., storage, memory, processor specifications, and expected network communication speeds).

[0036] Traditionally, video streaming services are available for a variety of consumer devices and provide different versions of the source video file 108 for streaming. Typically, higher resolutions and bit rates are used for more powerful consumer devices. However, for any device, it may be beneficial to allow the consumer device to switch streams (bit rates) when conditions change.

[0037] As shown, four versions of the source video file 108 are available, either encoded, or become available (e.g., by transcoding) using a real-time encoding technique. For this example, it is assumed that video streams with resolutions of 1080p, 720p, 480p, and 240p will become available, corresponding to streams 120, 122, 124, 126, respectively. Each stream 120, 122, 124, 126 is associated with a granular bitrate, which may be a standard bitrate or a bitrate otherwise selected for a particular stream quality. As shown, streams 120, 122, 124, 126 are associated with bitrates of 6 Mb / s, 2.7 Mb / s, 1.2 Mb / s, and 360 Kb / s, respectively.

[0038] Each of the streams 120, 122, 124, 126 is associated with a corresponding plurality of segments 140 that are encoded using a bitrate associated with the corresponding stream. In some cases, the segments 140 actually have the bitrate of the corresponding stream 120, 122, 124, 126, while in other cases the bitrate may vary, such as if the stream bitrate is expressed as a target bitrate for a video encoder. That is, the video encoder may use the provided bitrate as a target, but may generate segments having a bitrate different from the target based on other considerations used by the video encoder.

[0039] Segments 140 correspond to segments of source video file 108. Segments 140 include multiple frames 148. Frames 148 may correspond to frames used in video playback. For example, if playback is specified at 30 frames per second, and the duration of each segment 140 is 5 seconds, each segment includes 150 frames.

[0040] As explained in Example 1, any modifications to the video file are typically performed on the source video before generating the segments 140. This may result in degradation of the source video, which is decoded and then re-encoded into separate segments, which adds additional encoding operations that will provide some quality degradation in addition to the degradation that occurs during decoding / re-encoding of the source video when processing the modifications. In addition, the process of encoding the modified source video to generate the segments 140 introduces additional delays in making the segments available for streaming and uses additional computing resources.

[0041] Example 3 - Exemplary Video Streaming Components

[0042] FIG. 2 provides a simplified representation of a streaming video file 200, or more specifically, a container such as an MP4 container, and the content included in such a container. The streaming video file 200 typically includes a plurality of segments 208, wherein segments 208a, 208b represent different types of segments, discussed further below. The segments 208 may correspond to the segments 140 of FIG. 1. The segments 208 typically have a determined duration, including a fixed duration, typically a duration of 1-10 seconds, but other durations may be used, and the optional segments may have different durations.

[0043] Segments 208 may optionally be further decomposed into smaller units 212, which may be referred to as chunks. Note that the term "chunk" is sometimes used to describe segments 208. Unless otherwise specified, the term "segment" refers to a segment 208 - a typical streaming unit, including for adaptive streaming (e.g., MPEG-DASH and HLS using fMP4). The term "chunk" refers to a portion 212 of a segment, which is consistent with how the term "chunk" is used in technologies such as MPEG-CMAF (Common Media Application Format).

[0044] To represent these two possibilities, segment 208a is shown with blocks 212, while segment 208b is shown without blocks. Typically, a single video streaming file will be available as having either segment 208a or segment 208b, but not both, although the disclosed technology does not prohibit the use of mixed segment types. When blocks 212 are available, blocks 212 can be used as streaming units in addition to or in place of segments 208, while segments 208 are used as streaming units if the segments are not further divided into blocks. The term "segment" refers to any continuous portion of a video file produced by splitting a source file into continuous portions, where a collection of segments can represent the entire source video, or can represent selected portions of the source video.

[0045] A given segment 208 or its block 212 typically includes a plurality of frames 216 (including frames 216a-216c of different types, as discussed further below). For example, assume that the segment 208 has a duration of five seconds and a frame rate of 24 frames per second is used for the video. In this case, the segment 208 includes 120 frames 216. Now assume that the segment 208 is divided into blocks 212 having a duration of one second. The segment 208 still includes 120 frames 216, but each block 212 includes 24 frames.

[0046] Regardless of whether the frame 216 is considered to be in a segment 208, a block 212, or a video file that has not yet been processed into a segment or a block, the frame is usually (but not required) of different types. That is, common video compression techniques implement video compression by omitting information of certain frames 216. That is, some types of frames 216 can be decoded when there are no other frames, but other types of frames may require additional frames in order to be decoded. Frames 216 that can be decoded independently of other frames are usually referred to as "i frames" (or "key frames") and are represented as frames 216a. Because they do not require other frames 216 to be decoded, i frames 216a are usually the most incompressible frame types.

[0047] Another type of frame 216, shown as frame 216b, is called a "p-frame" (predicted frame), and the decoding of a p-frame requires one or more past frames. Yet another type of frame 216, shown as frame 216c, is called a "b-frame" (for "bi-directional"), which refers to a frame in which either or both of the frames before or after the current b-frame are used to decode the current b-frame. Thus, if the first frame 216 in segment 208 is a p-frame 216a, additional information will be required to decode the frame. Similarly, if a specific subset of segment 208 is to be used (as part of block 212 or otherwise), at least one i-frame will be required if the subset begins with a p-frame 216b. Similar considerations may apply if the first frame 216 is a b-frame 216c.

[0048] Note that while the dependencies of frame 216 have been described for segment 208, the same problem exists in non-segmented video files. Furthermore, the dependencies may be more complex than described above. For example, both p-frame 216b and b-frame 216c may depend on multiple previous frames (or in the case of b-frames, multiple previous frames and / or multiple future frames). Furthermore, the dependencies between frames need not be sequential. For example, the sixth frame in a sequence of frames may depend on the second frame in the sequence of frames, but not on the third frame in the sequence.

[0049] The disclosed technology provides for applying video processing operations to discrete video sub-portions (e.g., segments 208 or blocks 212). Depending on how the distribution of frame types within the video aligns with the boundaries of segments 208 or blocks 212, the frames required to decode a segment or block may be in different segments or blocks. In the case where the video is encoded into segments 208 or blocks 212 on a single computing system or device, dependencies between frames are generally not a problem because any required frames are available to the decoder / encoder. However, when the video processing task is divided into subtasks, the work unit assigned to a particular worker device may not include all of the required frames 216. As will be further described, in at least some embodiments, a worker may request or be sent additional frames 216 beyond the specific frames included in the work unit.

[0050] Many video container formats contain information that can be utilized by the disclosed technology. Specific details are provided for the MP4 container, but similar principles can be applied to other container formats. The MP4 container includes "stsc", "atom" or "box" (wherein atoms / boxes are units into which metadata is organized), which can be used to locate the segment (or block) containing a specific frame / sample. The "stts" atom / box contains information that associates a specific time of a media (e.g., video) file to a specific sample (or set of samples, which can correspond to the granularity of the supported time units, such as a set of 24 frames with a granularity of one second and a frame rate of 24 frames / second). Other useful atoms / boxes include the "stsz" atom, which includes information about the size of each sample. The "stco" box provides the offset of the segment / block in the video data / media stream. Other atoms / boxes include information such as the duration of the video, the number of samples, the data format, the time scale, the display characteristics, etc.

[0051] The coordinator computing system that orchestrates the video processing subtasks or the workers that process these subtasks can use the above information for various purposes, including locating specific portions (frames, segments) of video data required for the subtask. The method of accessing / obtaining video data may vary depending on the type of container or the specific implementation of the container. For example, some containers support random access to video, such as MP4 containers, which may include an "stss" atom that provides a "sync sample" that is a random access frame. If no random access information is provided (or if not all frames are random access by default), the byte offset information can be used to describe and request the portion of the video required by the subtask.

[0052] Example 4 - Exemplary computing environment with coordinator computing system and workers for subtask execution

[0053] Figure 3A computing environment 300 is shown in which the disclosed technology can be implemented. It should be understood that the disclosed technology can be implemented in a variety of computing environments, and thus the disclosed technology need not be used in an environment that includes all of the components of the computing environment 300. In addition, components not shown in the computing environment 300 can be used in conjunction with the disclosed technology. Due to the various ways in which the disclosed technology can be implemented, a number of components, end elements of which are shown as optional (indicated by dashed lines).

[0054] In general, computing environment 300 includes: a client that submits a video processing request that includes (or indicates) one or more video files and one or more instructions / operations to be performed on the one or more video files; an orchestrator that is responsible for generating subtasks and sending the subtasks to workers; and workers that apply instructions / operations to portions (segments) of one or more video files. A given computing device may include a single component or multiple components, or may include all components.

[0055] The computing environment 300 includes a client 304, wherein the client generates a video processing request that includes an indication of one or more video files and one or more operations / instructions to be performed on the one or more video files. The client 304 may include a media application 308, such as a video editor, which may be used to identify one or more video files and specify the operations / instructions to be performed thereon. The request sent by the client 304 may or may not include the actual video data to be processed.

[0056] Although media application 308 is shown as being on client 304, media application 308 may be hosted on another computing device. For example, media application 308 may be a web-based application, in which case a client's web browser, for example, together with a remote media application, may serve as media application 308. Similarly, client 308 is shown as optionally storing video file 312. However, video file 312 may be located external to the client, and when generating a task using media application 308, the client may provide the location of the video file or other means of accessing the video file.

[0057] Coordinator system 320 is shown to include orchestrator 324. Orchestrator 324 receives and processes video processing requests, such as requests received from client 304. Orchestrator 324 analyzes video file 312 and generates video processing subtasks, including segmenting video data into segments to be processed as part of a given subtask. Video file 312 is stored on client 304, coordinator system 320, or at another location, such as on a computing device used as worker 340 (of which workers 340 are shown as workers 340a-340d) or in a media repository (including a location on the Internet).

[0058] The video processing subtasks can be implemented in various ways. A given subtask operates on a segment of the video file 312, such as a portion of the file corresponding to a streaming unit. In a specific embodiment, the portion of the video file is determined as / using a duration, such as generating a subtask for a five-second portion of the video file 312.

[0059] However, the subtasks may alternatively be specified in other ways, such as the number of frames or bytes of video data to be processed, where these specifications may correspond to a particular video duration, or may be a "primary" criterion. That is, the number of bytes in a subtask may be determined by counting the number of bytes in the portion of the video file 312 that corresponds to the specified video duration. Alternatively, the number of bytes may be specified, where the duration is determined based on the number of frames whose data is included in the specified bytes.

[0060] For example, assume that the subtask is specified for 788 kB of video file data, and this corresponds to 48 frames of video. If the video has a frame rate of 24 frames per second, the duration of the video portion for the subtask is two seconds. A different 788 kB portion of the video file 312 may also correspond to a two-second excerpt of the video, or the duration may be longer or shorter (because a video excerpt of higher complexity may have more data, while a video excerpt of lower complexity may have less data).

[0061] Subtasks may be defined with consideration of the overall desired output of the video processing task. For example, if the operation is to trim five seconds from the beginning of a video, a subtask may be to encode a portion (eg, segment or block) of a video file starting at the sixth second of the video file.

[0062] The subtasks are assigned to one or more workers 340. However, the coordinator system 320 may also serve as a worker 340 and may process some or all of the subtasks. Similarly, although useful implementations of the disclosed techniques involve multiple workers 340, the disclosed techniques may provide advantages even if a single worker performs a subtask, whether that worker is the coordinator system 320 or a worker external to the coordinator system.

[0063] Typically, the assigned subtasks are sent to the workers 340 by the coordinator system 320, but the disclosed techniques can be used in situations where the workers check the assigned subtasks with the coordinator system. Similarly, the data to be used to perform the subtasks can be sent to the workers 340 by the coordinator system 320, can be requested from the coordinator system by the workers 340, or a combination thereof. For example, the orchestrator 324 can send instructions to the workers 340 to be performed on a segment of the video file 312, and the workers 340 can then request the appropriate data for the video file 312, regardless of where it is stored, including the video data of the video file or metadata data describing the video file 312. These embodiments may be particularly useful when the protocol used to communicate between the orchestrator 324 and the workers 340 does not support a payload size sufficient to contain all the data required to process the subtasks. Even for metadata of video file 312, orchestrator 324 may send no metadata, all metadata, or a portion of the metadata associated with the video file to a given worker 340, where the worker may independently request any additional metadata needed to perform a subtask.

[0064] As discussed in conjunction with Example 4, it is possible that a portion of video file 312 required to perform a particular subtask may not be sent to worker 340, or may be included in a segment of a video file that is explicitly identified as part of a subtask. That is, a frame (e.g., a p-frame) that appears in the beginning portion of a video file segment may refer to a frame other than the frames included within the subtask (e.g., an i-frame or other p-frame). Alternatively, a frame (e.g., a b-frame) that appears in the ending portion of a video segment may refer to a frame other than the frames of the subtask (e.g., an i-frame or other b-frame).

[0065] If it is determined that additional frames are needed, the codec 360 of the worker 340 can determine that additional frames are needed and can request such additional frames from the orchestrator 234, or can obtain the frames directly from the video file 312. In some cases, a determined number of frames can be requested, wherein if the first set of additional frames still does not include all of the frames required for the frames of the decoding or encoding subtask, additional requests can be made. In particular embodiments, the worker (e.g., the worker's codec) can request a specific byte range of the video file corresponding to the desired frame.

[0066] These operations of workers 340 may optionally be performed in conjunction with a given worker's media processor 344, where the media processor provides appropriate data and operating instructions to the codec 348 and optionally acts on feedback provided from the codec (including feedback regarding additional frames that may be required for the decoding or encoding operation).

[0067] In distributing subtasks to workers 340, the orchestrator 324 may work with, or include functionality of, a scheduler 328. The scheduler 328 may perform actions such as assigning subtasks to workers 340, monitoring the completion of subtasks, or adjusting the assignment of subtasks or the priority of individual subtasks of an overall task (and therefore its constituent subtasks). Monitoring the completion of subtasks may include determining whether a worker 340 has failed, or whether a result has not been provided within an expected timeframe. For example, a worker 340 may send a confirmation to the coordinator system 320 when a subtask has been completed, the coordinator system may poll the workers (to determine whether they are still "alive," or to determine the status of one or more subtasks assigned to a given worker), or the coordinator system may otherwise determine when a task has been completed, such as by monitoring a repository for the presence of video units (e.g., segments or blocks) for a particular subtask.

[0068] The threshold time for task completion can be a fixed time, or can be varied, including for a specific task, a specific subtask of a task, or depending on the properties of the worker 340 to which the subtask is assigned. For example, a task for a video unit of a specified duration of ten seconds can be associated with a longer "timeout" threshold value than a task for a video unit of a specified duration of five seconds. Alternatively, a video with relatively high complexity can have a longer timeout threshold setting than for a video of lower complexity. As another option, a worker can periodically renew the lease of a subtask (by communicating with the coordinator computing system), or can write a checkpoint for a subtask. The signaling can indicate to the coordinator node that the worker is active / ongoing. In some cases, lack of lease renewal or failure to write a checkpoint can be used as a timeout criterion that can cause the coordinator computing system to request a subtask from another worker.

[0069] The timeout threshold may also vary depending on the nature of one or more operations applied during the processing of the subtask. For example, an operation for trimming a video may be associated with a shorter timeout value than an operation for applying an effect to all frames of the video.

[0070] With respect to timeout variations for different subtasks within a task, the timeout thresholds may vary based on the complexity of the portion of the video being processed during the subtask, in a manner similar to that described above for processing different video files 312, or based on different operations performed as part of different subtasks. Assume that a transformation is being applied, and only one or a limited number of subtasks include portions of the video associated with the transformation. The timeout value for a subtask associated with a transformation may be set to a longer value than the timeout value for other changes in which portions of the video are encoded in discrete video units, but no source video is specified.

[0071] Workers 340 can be of different types, which can affect the timeout values ​​that are set, and can also be considered when scheduling specific subtasks of one or more tasks. That is, although not required, the disclosed technology supports heterogeneous computing devices used as workers 340. A given implementation of the computing environment 300 can include one or more workers 340 of each of one or more types, including where there is a single worker, and where the worker can be a worker of the coordinator system 320 or a computing device that is separate from the coordinator system but in communication with the coordinator system.

[0072] Worker 340 may include a codec 348 implemented in software or hardware. However, even workers 340 with the same type of implementation (software or hardware) for codec 348 may be different in various aspects. For example, worker 340 may vary in form factor, where different form factors may be associated with the category or availability of computing resources. For example, worker 340 may be in the shape of a mobile device or tablet computer, which is typically limited to communicating via WIFI or cellular connections. Since these devices are mobile, it may be somewhat unpredictable when a given device will be available for scheduler 324 or scheduler 328. On the other hand, a laptop may be more likely to be available and may be connected using WIFI or a hardwired internet connection. Similarly, desktop computers are typically connected using WIFI or a hardwired connection, and are typically more regularly available than even laptops. In some cases, it may be assumed that a desktop computer is always available as a worker 340 unless the computer is unexpectedly shut down, loses connection, or fails. Dedicated servers or cloud computing systems may also serve as workers 340 , and are typically less mobile, more available, and have greater connectivity to the coordinator computing system 320 .

[0073] In addition to having different expected availability and connectivity, different worker devices 340 may generally be categorized into different classes or types, where one class or type is expected to have more or less processing resources (memory, processing power) than another class or type of device. For example, it may generally be assumed that a dedicated server or cloud computing system will have more resources than a personal computer, which in turn will have more resources than a laptop, which may be expected to have more resources than a tablet, which may have more resources than a smartphone.

[0074] Even within a class or type of worker 340, there may be significant differences in available processing resources. For example, a high-end gaming computer may have significantly more processing power and memory than a budget computer, and potentially a powerful GPU that may not be present on a budget computer. Similarly, a smartphone may have processing resources that vary significantly based on the manufacturer, location within a given product line, or version.

[0075] Different types of workers 340 Figure 3 340a , 340b , and 340c , where the workers 340a , 340b , 340c all include a software-based codec 348 . Figure 3 Also shown are one or more workers 340d of a fourth type having a hardware-based codec 348. Although only a single type of worker 340 having a hardware-based codec 348 is shown, it should be understood that different types of workers having hardware-based codecs may exist, or the computing environment 300 may not include any workers having hardware-based codecs. Similarly, the computing environment 300 may only include workers 340 having hardware-based codecs.

[0076] Information about workers 340 may be stored as resource information 332 of coordinator system 320. Resource information 332 may be stored as a list, repository, database, etc., separate from orchestrator 324 or scheduler 328, or may be part of the coordinator or scheduler. Resource information 332 may be stored in one or more instances of data types including abstract or composite data types (e.g., classes, including vector classes) or in one or more instances of data structures (e.g., stacks, queues, heaps, graphs, trees, lists, etc.). In particular embodiments, resource information 332 may be stored in a database, including in one or more tables or views of a relational database system.

[0077] In any case, resource information 332 may identify workers 340 that are currently or potentially available to the coordinator system 320, and may provide details about such workers, such as addresses (e.g., IP addresses or MAC addresses). Resource information may also include details about workers 340, such as availability or reliability rankings or descriptive information, or information about computing resources available on workers, which may be described in detail for a particular worker, or may simply indicate the general type or category to which the worker belongs. Optionally, resource information 332 may include information about subtasks currently assigned to a worker 340, or a log of subtasks previously performed by a worker (which may be in the form of performance information that can be used when assigning subtasks to workers).

[0078] Scheduler 328, optionally in conjunction with orchestrator 324, can be responsible for assigning subtasks to workers, including using resource information 332. That is, orchestrator 324 can generate multiple subtasks for a given video processing task. Scheduler 328 can then schedule the subtasks to individual workers 340. Subtasks can be scheduled simultaneously, or can be scheduled over time, including based on the execution of earlier scheduled subtasks. As an example, consider a video processing task that is divided into one hundred subtasks. One hundred subtasks can be scheduled to workers 340 immediately, where a given worker can be assigned a single subtask or multiple subtasks. Alternatively, a subset of subtasks can be initially scheduled (which can be, but not necessarily, in chronological order), and additional subtasks can be scheduled later, including scheduling additional subtasks to workers 340 after workers 340 complete processing of earlier assigned subtasks.

[0079] Different tasks or subtasks can be associated with different priorities, which can be fixed or can be adjusted over time. In the case of just-in-time encoding, for example, a subtask corresponding to an initial segment required for video streaming can be prioritized over other subtasks of the same task or subtasks of a different task. Prioritization can include preempting a subtask of another task, or scheduling a subtask on a more reliable or higher throughput (or otherwise higher performance) worker 340. As another example, a higher priority task (e.g., a task associated with a particular user or usage scenario (e.g., just-in-time encoding)) can cause the processing of a subtask of a lower priority task to be delayed, or scheduled to a different worker 340.

[0080] The coordinator system 320 may include a user interface (UI) or an application program interface (UUI) or both 336, which may be used for various purposes, including submitting video processing tasks to the orchestrator 324. The UI / API 336 may also be used for other purposes, such as defining / modifying resource information 332 or setting properties of the scheduler 328, including stopping a task / subtask or changing the priority of a task / subtask. The UI / API 336 may receive commands from a user / program interface 352 (where the program interface is a component of a program that calls the API 336 of the coordinator system 320 and may perform other operations, such as providing an API to service, for example, a streaming request from a streaming client 380). Although not shown, the client 304 may include a UI / PI 352, which may be part of the media application 308 or may communicate with the media application 308.

[0081] The processing of the subtasks produces processed video segments 360. The processed video segments 360 can be stored in a single location or in multiple locations, including storing all of the video segments for a given task at each of the multiple locations, or storing a portion of the video segments for a given task at each of the multiple locations. The locations where the processed video segments 360 can be stored can include storing the processed video segments at the client 304, at one of the workers 340 (see worker 340c), at the coordinator system 320, or at various external locations 370 (here, "external location" refers to a location that is not a client 304, a coordinator system 320, or a worker 340).

[0082] In particular embodiments, external location 370 may be a repository for streaming purposes, including for a plurality of video files, as represented by external location 370a. External location 370a may optionally include server functionality 374 for processing streaming requests from streaming clients 380 having at least one media playback component 384 (e.g., a web browser). A given external location 370 may optionally include additional functionality, such as including one or both of video files 312 or UI / PI 352.

[0083] Example 5 - Exemplary Coordinator Computing System Operation

[0084] Figure 4 In certain embodiments of the disclosed technology, a coordinator computing system (e.g. Figure 3 The operations in the method 400 performed by the coordinator computing system 320 of the coordinator computing system. It should be understood that the method performed by the coordinator computing system is not limited to the operations in the method 400, because, for example, certain operations may be added, removed, or reordered compared to the method shown.

[0085] At 404, a coordinator computing system (e.g., having a coordinator component) receives an indicator of one or more video files and an indicator of one or more operations to be performed on the one or more video files. The operations may include instructions to be performed on the video files and may optionally include data to be included in the output (one or more processed video segments produced by the workers, which may include data to be included in all processed video segments).

[0086] The video processing subtask is generated by the coordinator computing system at 408, such as by the orchestration component. Generating the subtask may include determining a video segment (e.g., a segment or block) in the desired output and then generating the subtask. The subtask may include operations, video data (or information sufficient to retrieve such data), and optionally other information required to perform the subtask, such that the subtask will produce the desired output. In particular, generating the subtask may include operations such as determining the size / duration of the processed video segment, such that the subtask will produce a processed video segment having a duration corresponding to a streaming unit, such that the processed video segment can be used for streaming without further processing (although computing elements required for streaming, such as a manifest, may need to be generated before streaming can begin).

[0087] Video segments that are available for streaming without further processing may be referred to as "streamable". "Streamable" may indicate processed video segments having a single / common bitrate, such as a high-resolution bitrate (although non-high-resolution bitrates may be used), or may refer to processed video segments that are available in multiple bitrates (where typically each processed video segment will be available at a given bitrate in the multiple bitrates), such as for adaptive streaming (which may provide a "bitrate ladder"). In the case of using a single bitrate (particularly a high-resolution bitrate), the high-bitrate segments may be encoded (e.g. using just-in-time encoding techniques) to reduce the bitrate, if appropriate.

[0088] In addition to having an appropriate duration for streaming, the processed video segments may also have other properties that make them suitable for streaming without further processing, including media formatting, settings, or structuring. These properties include known properties of streaming video segments, and include properties applicable to streaming protocols such as RTP, RTCP, RTSP, SIP, and RSVP, as well as MPEG-DASH and HLS. For example, individual processed video segments may be delivered in packets or segments that contain header information in addition to the video data. Specific examples of segment formats include ISO Base Media File Format (ISOBMFF, ".m4s" files) and MPEG-TS.

[0089] A subtask may include some information common to all subtasks, while other information may be specific to a subset of the subtasks, including for individual subtasks. As an example, a subtask may include a bitrate to be used when encoding all processed video segments, or instructions for calculating the bitrate, where the same instructions may be applied to each subtask in the same manner, even though the resulting bitrate may be different. The bitrate may correspond to a desired bitrate for streaming purposes. In addition, if the desired video segment is available at multiple bitrates, this information may be included in the subtasks of a single task (i.e., "create processed video segments at multiple bitrates"), or a separate task may be created for each desired bitrate. Alternatively, a single task may specify multiple bitrates to be used, and the coordinator computing system may generate different subtasks for each desired bitrate.

[0090] Subtasks may also include information specific to a particular subtask or a particular set of subtasks. Take filters as an example. If the filter is to be applied to the entire video, all subtasks may have instructions for applying the filter. If the filter is only applied to certain video clips, the coordinator computing system may include filtering instructions only for subtasks that process the video clips to which the filter is to be applied. In another embodiment, all operations / instructions of a task are included in a subtask, and the worker may determine which operations / instructions apply to its subtasks, or adapt the operations / instructions to its subtasks.

[0091] The nature of the information included in a subtask may also depend on whether / what information is "pushed" to a worker, and what information the worker "pulls" when processing a subtask. Some protocols may allow video files or specific portions of a video file (those portions required to process the video fragment associated with the subtask) to be sent to a worker, while other protocols may not support payloads large enough to include such information. Similarly, operations or metadata (including header information) may be included in the subtask to be sent to the worker, or the worker may be provided with information sufficient to request and receive the required information (e.g., an identifier for the video file and information sufficient to determine which portion of the video is to be processed by the subtask).

[0092] At 412, the subtasks are scheduled to the workers. When scheduling the subtasks, the coordinator computing system may consult resource information to determine available / appropriate workers. The coordinator computing system may consider factors such as performance information (reliability, available computing resources) of the workers or the priority of the video processing task. Logic may also be configured to determine whether to schedule all subtasks simultaneously (assuming sufficient resources are available), or whether to schedule subtasks in batches, or to schedule subtasks using a specific number of workers, with additional subtasks being scheduled as workers complete subtasks.

[0093] At 416 , the subtask is sent to the worker. Although shown as occurring as a single process step, sending the subtask to the worker may be performed iteratively / periodically, as described in the discussion of 412 .

[0094] Optionally, it is determined at 420 whether any subtasks have been completed. Subtask completion may be determined in a variety of ways, such as by receiving a completion notification from a worker, by polling a worker, or by monitoring a location where processed file fragments are stored. If a subtask has been completed, it may be determined at 424 whether there are any additional subtasks to be processed. If so, method 400 may return to 412. If not, the method may continue at 428, or may optionally end at 432.

[0095] If no subtasks are completed at 420, if step 420 is omitted, the method may optionally determine at 428 whether any subtasks have failed. Subtask failure may be determined in a variety of ways, such as receiving a failure notification from a worker, polling a worker, not receiving a "survival" message from a worker, or exceeding a timeout value. If it has been determined that a subtask has failed, method 400 may return to 412, where the subtask may be restarted at the same worker, or dispatched to a different worker. Otherwise, method 400 may optionally proceed to 436.

[0096] At 436, it is determined whether any priority changes have been made to the task (which in turn may affect some or all of its subtasks) or to a particular set of one or more subtasks. If no priority changes have occurred, the method 400 may continue to an earlier operation in the method 400, such as to 420, where it may be determined whether all subtasks have been completed.

[0097] Otherwise, if a priority change has occurred at 436, the priority change can be to increase the priority of the task or a portion of its subtasks, or to decrease the priority of the task or a portion of its subtasks. If it is determined that the priority has been increased, the method can proceed to 440. At 440, the existing subtasks can be rescheduled to a higher performance worker, or all or a portion of any unscheduled / unsent subtasks can be scheduled / sent to a worker. Although shown as a separate operation at 440, in some cases, method 400 can return to 412, and the operations described for 440 can be performed as part of the operations at 412. Method 400 can then proceed to an earlier operation in method 400, such as to 420, where it can be determined whether all subtasks have been completed.

[0098] If it is determined that the priority has been reduced, the method can proceed to 444. At 444, the existing subtask can be delayed, canceled, or reassigned to a lower performance worker. As with the operation at 440, the operation at 444 can also be performed as part of operation 412. After processing at 444 (or 412), method 440 can return to an earlier operation, such as returning to 420. Note that the priority can change multiple times during task processing. For example, the priority can be lowered, but if a higher priority task is completed, the priority is increased, and vice versa.

[0099] Various decisions 420, 428, 436 have been described as optional. In addition, it should be understood that decisions 420, 428, 436 can occur in an order different from that shown, and can optionally be performed simultaneously.

[0100] Example 6 - Exemplary Worker Operations

[0101] As already described, the disclosed technology involves splitting a video processing task for a video file into multiple video processing subtasks that can be performed by one or more worker devices, where the device performing the splitting can also serve as a worker or the worker. The information sent by the coordinator computing system to the worker can be implemented in a variety of ways, including based on implementation preferences or based on constraints provided by the underlying technology. For example, the disclosed technology can be implemented with video data being "pushed" to the worker or having the worker "pull" the required video data from another location. However, the messaging protocol used with particular video processing technologies may not support payloads of a size sufficient to push video data associated with the subtasks to the worker.

[0102] Similarly, while some information about the subtask (including the operation or information about the video file / segment to be processed) may be included in the message sent to the worker, other information about the operation or the video file may need to be retrieved by the worker. For example, in order to determine which part of the video file the worker should request, the worker may need header information of the container that stores the file, such as an MP4 header. The worker can receive information about the subtask, such as the operation to be performed and information identifying the segment of the video it is to process. The worker may then need to request header information for the video to determine how to format its request for data for the video segment (e.g., a byte range request). The worker can then request the data for the video segment.

[0103] Figure 5 A flow chart of a method 500 is shown, which includes operations that may be included in the context of "data pull" as well as "data push", but it should be understood that the disclosed technology may be implemented in other ways. Figure 5 The operations shown and described can be found in Figure 3 The computing environment 300 includes the use of a coordinator computing system 320 and one or more workers 340 (where the coordinator computing system can be used as a worker or the worker).

[0104] At 504, the worker receives a subtask of a video processing request, for example from a coordinator computing system, for example in response to Figure 4 The subtask is sent at 416 of method 400. Optionally, at 508, it is determined whether the information received for the subtask is sufficient to process the subtask. If the information is sufficient, or if the implementation is such that all required information is pushed to the worker, method 500 can proceed to 520. Otherwise, method 500 proceeds to 512, where the worker requests the information required to process the subtask. Alternatively, if the implementation is such that it is known that the worker will need to request additional information to process the subtask, method 500 can proceed directly from 504 to 512.

[0105] As already described, when processing a subtask, a worker typically needs access to at least video data associated with the video clip associated with its subtask, operational information, and at least some metadata typically associated with the video file. Thus, if the worker does not receive video data when receiving the subtask at 504, the information received by the worker may have information sufficient to enable the worker to request appropriate video data at 512. Such information may be a starting frame number and number of frames to be processed, a starting frame and an ending frame, a starting frame and duration of the video to be processed, a starting and ending duration, a starting duration and a time window to be processed, a starting byte and byte size to be requested, a byte range to be requested, or other information or information that may be used to determine any of the previously listed types of information.

[0106] Details about the instructions or operations to be performed as part of the subtask may also optionally be received along with the subtask at 504, or may be affirmatively requested by the worker at 512. For example, a worker may be provided with information about the portion of the video to be processed and that it has been assigned to process the video portion, but it may request the actual instructions / operations to be performed, for example, from the coordinator computing system. In some cases, the information received at 504 may simply be an indicator that the subtask has been assigned to the worker, where further details about the subtask (video file, portion to be processed, instructions / operations to be performed) may be requested by the worker at 512.

[0107] All or a portion of the metadata associated with the video file or a portion thereof may also be sent in the subtask received by the worker at 504, or may be requested at 512. In particular, including depending on the protocol used to communicate with the worker, the worker may require metadata such as an MP4 header to process the video file, and metadata such as an MP4 header may or may not be included in the subtask received at 504. In particular, the worker may require header information in order to formulate a request for a portion of the video to be processed as part of the subtask. Such information may be requested at 512.

[0108] The information / data requested by the worker at 512 is received at 516. Although not shown, it is understood that the operations at 512 and 516 may be performed multiple times before method 500 continues to other operations. In particular, the worker may make a first request for header information, determine which portion of the video file it needs to process the subtask, and then issue a separate request for the video data.

[0109] As indicated in the above discussion, a request for information may be sent to one or more recipients. For example, some data (e.g., MP4 headers) may be requested from the coordinator computing system, while other data, such as video file data, may be requested from a repository storing video files. Alternatively, information about operations to be performed as part of a subtask may be requested from the coordinator computing system (or received at 504), and then metadata (including container headers) and video data may be requested from a location storing video data other than the coordinator computing system.

[0110] After receiving the requested information at 516, or if it is determined at 508 that the information received at 504 is sufficient to process the subtask, method 500 proceeds to 520, where the worker applies the operation to the portion of the video data being processed by the worker for the subtask. The operation may include instructions for encoding the video in some manner or for modifying the video content in some manner, such as applying a filter, a transformation, or adding text to the video content. At 524, the operation is provided to the video codec, which may decode the data of the video file, apply the operation, and re-encode the data.

[0111] As part of the re-encoding at 524 (or more generally, as part of an encoding operation), it may be determined that the worker does not have sufficient information. For example, as described with respect to FIG. 2 , portions of the video data obtained by the worker may require frames that are not included in such video data. Thus, it may be determined at 528 whether the video data is sufficient to be processed by the encoder / decoder. If not, additional data may be requested at 532, and the method may continue at 524. If the information is sufficient, or if insufficient information is not determined (i.e., 528 may determine that the data is insufficient, rather than determining that it is sufficient), the method may proceed to 536.

[0112] At 536, the encoded (processed) video segment is sent to a destination, where the destination may store all or a portion of the processed video segment for the task, and optionally, the location may be a location from which such video segment may be streamed. Optionally, at 540, the worker notifies the coordinator computing system that the subtask has been completed.

[0113] Example 7 - Exemplary operations to implement the disclosed technology

[0114] Figure 6 A flow chart of a method 600 for dividing a video processing task into a plurality of subtasks that can be executed by one or more worker computing devices is shown. The method 600 may be performed in Figure 3 The computing environment 300 is implemented and may include or represent Figure 4In particular, the method 600 may be performed by Figure 3 Executed by the coordinator computing system 320.

[0115] At 610, a video processing task is received that includes an identifier of a digital video file and one or more operations to be performed on at least a portion of the digital video file. At 620, a plurality of subtasks are generated for processing the one or more operations. A given subtask in the plurality of subtasks identifies or includes a segment of the digital video file to be processed as part of the subtask.

[0116] At 630, at least a first portion of the plurality of subtasks are scheduled to one or more worker computing devices. At 640, subtasks of the at least a first portion of the plurality of subtasks are sent to respective worker computing devices for processing by the respective worker computing devices, thereby providing improved fault tolerance for the video processing task compared to executing the video processing task as a single task.

[0117] Figure 7 A flowchart of a method 700 for performing a video processing subtask by a worker computing device is shown. The method 700 may be performed in Figure 3 The computing environment 300 is implemented and may include or represent Figure 5 In particular, the method 700 may be performed by Figure 3 The worker 340 executes.

[0118] At 710, a subtask of a video processing task is received from a coordinator computing system. The subtask includes one or more operations of the video processing task or instructions derived at least in part therefrom, and an identifier of a video segment to be processed for the subtask. At 720, at least the video segment is requested. At 730, at least the video segment is received. At 740, the operations or instructions are performed on at least the video segment. At 750, after or in conjunction with performing the operations or instructions, at least the video segment is encoded to provide a processed video segment.

[0119] Figure 8 FIG. 8 is a flowchart of an alternative method 800 for performing a video processing subtask by a worker computing device. As with method 700, method 800 may be performed in Figure 3 The computing environment 300 is implemented and may include or represent Figure 5 In particular, the method 700 may be performed by Figure 3 The worker 340 executes.

[0120] At 810, a subtask of a video processing task is received from a coordinator computing system. The subtask includes one or more operations of the video processing task or instructions derived at least in part therefrom, and a video segment to be processed for the subtask. At 820, the operation or instruction is performed on at least the video segment. At 830, after or in conjunction with performing the operation or instruction, at least the video segment is encoded to provide a processed video segment. The processed video segment can be streamed to a streaming client without further processing the processed video segment. At 840, the processed video segment is sent to a repository of other processed video segments of the digital video file.

[0121] Example 8 - Additional Examples

[0122] Example 1 is a computing system comprising at least one memory and at least one hardware processor coupled to the at least one memory. The computing system also includes one or more computer-readable storage media storing computer-executable instructions that, when executed, cause the computing system to perform various operations. The operations include receiving a video processing task that includes an identifier of a digital video file and one or more operations to be performed on at least a portion of the digital video file. A plurality of subtasks are generated for processing the one or more operations. A given subtask in the plurality of subtasks identifies or includes a segment in the digital video file to be processed as part of the subtask.

[0123] At least a first portion of the plurality of subtasks is scheduled to one or more worker computing devices. Subtasks of the at least a first portion of the plurality of subtasks are sent to respective worker computing devices for processing by the respective worker computing devices, thereby providing improved fault tolerance for the video processing task compared to executing the video processing task as a single task.

[0124] Example 2 includes the subject matter of Example 1, and further specifies that generating a portion of a subtask comprises generating a subtask that will produce a processed video segment having a fixed duration.

[0125] Example 3 includes the subject matter of Example 2, and further specifies that the fixed duration is a duration corresponding to a duration used for video streaming.

[0126] Example 4 includes the subject matter of any of Examples 1-3, and further specifies that generating a portion of a subtask comprises generating a subtask that will produce a streamable processed video segment without further processing the processed video segment.

[0127] Example 5 includes the subject matter of any of Examples 1-4, and further specifies that scheduling at least a first portion of the plurality of subtasks comprises scheduling the at least a first portion of the plurality of subtasks to a plurality of workers.

[0128] Example 6 includes the subject matter of Example 5, and further specifies that workers in the plurality of workers have heterogeneous processing capabilities.

[0129] Example 7 includes the subject matter of any of Examples 1-4, and further specifies that scheduling at least a first portion of the plurality of subtasks comprises scheduling the at least a first portion of the plurality of subtasks to a single worker.

[0130] Example 8 includes the subject matter of any of Examples 1-7, and further includes: determining that execution of a subtask by a worker has failed; and restarting the subtask at the worker.

[0131] Example 9 includes the subject matter of any of Examples 1-6 or 8, and further includes: determining that execution of a subtask by a worker has failed; scheduling the subtask to another worker; and sending the subtask to the other worker.

[0132] Example 10 includes the subject matter of any of Examples 1-6, 8, or 9, and further specifies: receiving a request to increase the priority of the video processing task. Reassigning a subtask from the plurality of subtasks to a worker computing device with higher performance than the worker computing device to which the subtask was originally assigned. Alternatively, sending an additional subtask from the plurality of subtasks to an additional worker computing device.

[0133] Example 11 includes the subject matter of any of Examples 1-6 or 8-10, and further specifies: receiving a request to reduce the priority of the video processing task. Delaying or canceling the execution of one or more of the plurality of subtasks. Or, reassigning a subtask of the plurality of subtasks to a worker computing device with lower performance than the worker computing device to which the subtask was originally assigned.

[0134] Example 12 includes the subject matter of any of Examples 1-11, and further specifies: determining that a first subtask in the at least a portion of the plurality of subtasks has been completed. Sending a second subtask that is not included in the at least a portion of the plurality of subtasks to a worker computing device.

[0135] Example 13 includes the subject matter of any of Examples 1-12, and further specifies that sending the subtask includes sending subtask information, the subtask information including the one or more operations or instructions at least partially derived therefrom, and video data to be processed by the corresponding worker computing device for a corresponding subtask in the at least first portion of the multiple subtasks.

[0136] Example 14 includes the subject matter of any of Examples 1-13, and further specifies that sending the subtask includes sending subtask information, the subtask information including the one or more operations or instructions derived at least in part therefrom, and a time associated with the start of a corresponding segment of the digital video file.

[0137] Example 15 includes the subject matter of any of Examples 1-14, and further specifies that sending the subtask includes sending subtask information, the subtask information including the one or more operations or instructions derived at least in part therefrom, and a frame associated with the beginning of a corresponding segment of the digital video file.

[0138] Example 16 includes the subject matter of any of Examples 1-15, and further specifies that sending the subtask comprises sending subtask information, the subtask information comprising the one or more operations or instructions derived at least in part therefrom, and a header of the digital video file.

[0139] Example 17 includes the subject matter of any of Examples 1-16, and further specifies: receiving a request for streaming at least one processed video segment generated by a worker computing device from a subtask. In response to the request, sending the at least one processed video segment to a streaming client.

[0140] Example 18 is a method that can be implemented in hardware, software, or a combination thereof. A subtask of a video processing task is received from a coordinator computing system, the subtask comprising one or more operations of the video processing task or instructions derived at least in part therefrom, and an identifier of a video segment to be processed for the subtask. At least the video segment is requested and received. The operation or instruction is performed on at least the video segment. After or in conjunction with performing the operation or instruction, at least the video segment is encoded to provide a processed video segment.

[0141] Example 19 includes the subject matter of Example 18, and further specifies that the processed video segments may be streamed to a streaming client without further processing the processed video segments.

[0142] Example 20 is one or more computer-readable media storing computer-executable instructions that, when executed, cause the computing system to perform various operations. The operations include receiving a subtask of a video processing task for a digital video file from a coordinator computing system. The subtask includes one or more operations or instructions of the video processing task, or instructions derived at least in part from it, and a video segment to be processed for the subtask. The operations or instructions are performed on at least the video segment. After or in conjunction with performing the operations or instructions, at least the video segment is encoded to provide a processed video segment. The processed video segment can be streamed to a streaming client without further processing the processed video segment. The processed video segment is sent to a repository including other processed video segments of the digital video file.

[0143] Example 9 - Calculation System

[0144] Fig. 9 A generalized example of a suitable computing system 900 is depicted in which the described innovations may be implemented. The computing system 900 is not intended to suggest any limitation as to the scope of use or functionality of the present disclosure, as the described innovations may be implemented in various general-purpose or special-purpose computing systems.

[0145] refer to Fig. 9 , the computing system 900 includes one or more processing units 910, 915 and memories 920, 925. Fig. 9 , the basic configuration 930 is included within the dashed line. Processing units 910, 915 execute computer executable instructions, for example, to implement the features described in Examples 1-9. The processing unit can be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing power. For example, Fig. 9 A central processing unit 910 and a graphics processing unit or co-processing unit 915 are shown. Tangible memory 920, 925 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two accessible by the processing units 910, 915. The memory 920, 925 stores software 980 implementing one or more innovations described herein in the form of computer-executable instructions suitable for execution by the processing units 910, 915.

[0146] The computing system 900 may have additional features. For example, the computing system 900 includes storage 940, one or more input devices 950, one or more output devices 960, and one or more communication connections 970, including input devices, output devices, and communication connections for interacting with a user. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system 900. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing system 900 and coordinates the activities of the components of the computing system 900.

[0147] Tangible storage 940 may be removable or non-removable and include disks, tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-transitory manner and that can be accessed within computing system 900. Storage 940 stores instructions for software 980 implementing one or more innovations described herein.

[0148] The input device 950 may be a touch input device such as a keyboard, a mouse, a pen or a trackball, a voice input device, a scanning device, or another device that provides input to the computing system 900. The output device 960 may be a display, a printer, a speaker, a CD writer, or another device that provides output from the computing system 900.

[0149] Communication connection 970 enables communication with another computing entity via a communication medium. The communication medium transmits information, such as computer executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal whose one or more characteristics are set or changed in a manner that encodes the information in the signal. By way of example and not limitation, the communication medium may use electrical, optical, RF or other carrier waves.

[0150] The innovation may be described in the general context of computer executable instructions (e.g., instructions included in program modules) executed in a computing system on a target real or virtual processor. Typically, program modules or components include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules may be combined or split between program modules as needed. Computer executable instructions for program modules may be executed within a local or distributed computing system.

[0151] The terms "system" and "device" are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. In general, a computing system or computing device may be local or distributed, and may include any combination of special-purpose hardware and / or general-purpose hardware with software that implements the functionality described herein.

[0152] In various examples described herein, a module (e.g., a component or engine) may be "coded" to perform certain operations or provide certain functionality, indicating that computer executable instructions of the module may be executed to perform such operations, cause such operations to be performed, or otherwise provide such functionality. Although the functionality described with respect to a software component, module, or engine may be performed as discrete software units (e.g., procedures, functions, class methods), it need not be implemented as discrete units. That is, the functionality may be incorporated into a larger or more general program, such as one or more lines of code in a larger or general program.

[0153] For purposes of presentation, the detailed description uses terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer and should not be confused with actions performed by a human. The actual computer operations corresponding to these terms vary depending on the implementation.

[0154] Example 10 - Cloud computing environment

[0155] Fig.10 An exemplary cloud computing environment 1000 is depicted in which the described techniques may be implemented. Cloud computing environment 1000 includes cloud computing services 1010. Cloud computing services 1010 may include various types of cloud computing resources, such as computer servers, data repositories, networking resources, etc. Cloud computing services 1010 may be centrally located (e.g., provided by a data center of an enterprise or organization) or distributed (e.g., provided by various computing resources located in different locations, such as different data centers and / or located in different cities or countries).

[0156] The cloud computing service 1010 is utilized by various types of computing devices (e.g., client computing devices), such as computing devices 1020, 1022, and 1024. For example, the computing devices (e.g., 1020, 1022, and 1024) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smart phones), or other types of computing devices. For example, the computing devices (e.g., 1020, 1022, and 1024) can utilize the cloud computing service 1010 to perform computing operations (e.g., data processing, data storage, etc.).

[0157] Example 11 - Implementation

[0158] Although the operations of some disclosed methods are described in a particular order for ease of presentation, it should be understood that such description includes rearrangement unless the specific language set forth herein requires a particular ordering. For example, in some cases, the operations described in sequence may be rearranged or performed simultaneously. In addition, for simplicity, the accompanying drawings may not show the various ways in which the disclosed methods can be used in conjunction with other methods.

[0159] Any disclosed method may be implemented as computer executable instructions or computer program products stored on one or more computer readable storage media and executed on a computing device (e.g., any available computing device, including a smartphone or other mobile device that includes computing hardware). Tangible computer readable storage media are any available tangible media that can be accessed within a computing environment (e.g., one or more optical media disks such as DVDs or CDs, volatile memory components (e.g., DRAM or SRAM), or non-volatile memory components (e.g., flash memory or hard drives)). As an example and with reference to Fig. 9 , computer-readable storage media include memories 920 and 925 and storage device 940. The term computer-readable storage medium does not include signals and carrier waves. In addition, the term computer-readable storage medium does not include communication connections (such as 970).

[0160] Any computer executable instructions for implementing the disclosed techniques and any data created and used during implementation of the disclosed embodiments may be stored on one or more computer readable storage media. The computer executable instructions may be, for example, a dedicated software application or part of a software application accessed or downloaded via a web browser or other software application (e.g., a remote computing application). Such software may be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (e.g., a cloud computing network or other such network) using one or more network computers.

[0161] For clarity, only certain selected aspects of software-based implementations are described. It should be understood that the disclosed techniques are not limited to any particular computer language or program. For example, the disclosed techniques can be implemented by software written in C++, Java, Perl, JavaScript, Python, Ruby, ABAP, SQL, Adobe Flash, or any other suitable programming language, or in some examples, by a markup language such as html or XML, or a combination of suitable programming languages ​​and markup languages. Likewise, the disclosed techniques are not limited to any particular computer or hardware type.

[0162] Furthermore, any software-based embodiment (including, for example, computer-executable instructions for causing a computer to perform any disclosed method) may be uploaded, downloaded, or remotely accessed via a suitable communication unit. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communications units.

[0163] The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Instead, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, both individually and in various combinations and sub-combinations with each other. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, nor do the disclosed embodiments require the presence of any one or more specific advantages or problems to be solved.

[0164] The technology from any example can be combined with the technology described in any one or more other examples. In view of the many possible embodiments to which the principles of the disclosed technology can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology. On the contrary, the scope of the disclosed technology includes what is covered by the scope and spirit of the appended claims.

Claims

1. A computing system comprising: at least one memory; at least one hardware processor coupled to the at least one memory; as well as One or more computer-readable storage media storing computer-executable instructions that, when executed, cause the computing system to perform video processing operations that provide improved computational flexibility and fault tolerance, the operations comprising: receiving a video processing task, the video processing task comprising an identifier of a digital video file and one or more operations to be performed on at least a portion of the digital video file; generating a plurality of subtasks for processing the one or more operations, a given subtask of the plurality of subtasks identifying or including a segment of the digital video file to be processed as part of the subtask; scheduling at least a first portion of the plurality of subtasks to one or more worker computing devices; and Sending the at least a first portion of the plurality of subtasks to respective worker computing devices for processing by the respective worker computing devices provides improved fault tolerance for the video processing task compared to executing the video processing task as a single task.

2. The computing system of claim 1, wherein: A portion of generating subtasks includes generating subtasks that will produce processed video segments having a fixed duration.

3. The computing system of claim 2, wherein: The fixed duration is a duration corresponding to a duration used for video streaming.

4. The computing system according to any one of claims 1 to 3, wherein: A portion of generating subtasks includes generating subtasks that will produce processed video segments that can be streamed without further processing the processed video segments.

5. The computing system according to any one of claims 1 to 4, wherein: Scheduling at least a first portion of the plurality of subtasks includes scheduling the at least a first portion of the plurality of subtasks to a plurality of workers.

6. The computing system of claim 5, wherein: The workers among the plurality of workers have heterogeneous processing capabilities.

7. The computing system according to any one of claims 1 to 4, wherein: Scheduling at least a first portion of the plurality of subtasks includes scheduling the at least a first portion of the plurality of subtasks to a single worker.

8. The computing system of any one of claims 1-7, wherein the operations further comprise: Determine that the worker's execution of the subtask has failed; as well as Restarting the subtask at the worker.

9. The computing system of any one of claims 1-6 or 8, wherein the operations further comprise: Determining that execution of the subtask by the worker computing device has failed; dispatching the subtask to another worker computing device; as well as The subtask is sent to the other worker computing device.

10. The computing system of any one of claims 1-6, 8 or 9, wherein the operations further comprise: receiving a request for increasing the priority of the video processing task; as well as (1) reassigning a subtask from the plurality of subtasks to a worker computing device with higher performance than the worker computing device to which the subtask was originally assigned; or (2) Sending additional subtasks from the plurality of subtasks to additional worker computing devices.

11. The computing system according to any one of claims 1-6 or 8-10, wherein the operations further comprise: receiving a request to reduce the priority of the video processing task; as well as (1) delaying or canceling the execution of one or more subtasks among the multiple subtasks; or (2) reassigning a subtask from the plurality of subtasks to a worker computing device having lower performance than the worker computing device to which the subtask was originally assigned.

12. The computing system of any one of claims 1-11, the operations further comprising: determining that a first subtask in the at least a portion of the plurality of subtasks has been completed; as well as A second subtask that is not included in the at least a portion of the plurality of subtasks is sent to the worker computing device.

13. The computing system according to any one of claims 1 to 12, wherein: Sending the subtask includes sending subtask information, the subtask information including the one or more operations or instructions at least partially derived therefrom, and video data to be processed by the corresponding worker computing device for a corresponding subtask in the at least first part of the multiple subtasks.

14. The computing system according to any one of claims 1 to 13, wherein: Sending the subtask includes sending subtask information including the one or more operations or instructions derived at least in part therefrom, and a time associated with a start of a corresponding segment of the digital video file.

15. The computing system according to any one of claims 1 to 14, wherein: Sending the subtask includes sending subtask information including the one or more operations or instructions derived at least in part therefrom, and a frame associated with a start of a corresponding segment of the digital video file.

Images