Content-Adaptive Microcoding Optimization for Video

Abstract

To optimize a coding parameter for a part of which a coding is identified for an instance of a content.SOLUTION: A method includes steps of: analyzing a location to which a flag is attached from a plurality of locations in a coding of a video image in order to form a cluster of the location; determining a draft micro chunk boundary to the cluster by searching a first start location and a first termination location in a coding; forming the micro chunk within a first searching range before the first start location and a second searching range after the first termination location; searching a second start location in the first searching range and a second termination location in a second searching range; determining a coding parameter on the basis of the micro chunk to be formed on the basis of a content characteristic of the micro chunk; and using the micro chunk for an insertion in a coding of the video image for coding.SELECTED DRAWING: None

Description

Background Art

[0001]

[0001] A video delivery system can provide a number of instances of content (e.g., video) that can be delivered to client devices. Instances of the content are encoded for delivery to the client. The video delivery system can use an encoding pipeline that includes an encoder, which uses a predefined set of encoding parameters to encode the content. That is, the values of the encoding parameters can be the same for all instances of the content encoded by the encoding pipeline. Typically, the encoding parameters are optimized for characteristics from the most well-known common content that can be encoded by the pipeline. For example, the encoding parameters can be based on the characteristics of action movies if action movies are considered the most common content being encoded. However, the video delivery system can have a vast library of content, which can include action movies, animated movies, television shows, nature shows, etc. The problem experienced is that for some content, or more specifically, for some parts of the same instance of content, the use of the same set of encoding parameters can create a sub-optimal encoding of the instance of content or a portion of the instance of content.

[0002]

[0002] The accompanying drawings are for purposes of illustration only and serve only to provide examples of possible structures and operations for the systems, devices, methods, and computer program products of the disclosed invention. These drawings in no way limit any form and detail changes that may be made by those skilled in the art without departing from the spirit and scope of the disclosed embodiments.

Brief Description of the Drawings

[0003] [Figure 1]

[0003] A simplified system for a video encoding pipeline according to some embodiments is illustrated. [Figure 2]

[0004] Examples of video coding with good frames and flagged frames according to several embodiments are illustrated. [Figure 3]

[0005] Simplified flowcharts illustrating methods for determining draft microchunk boundaries according to several embodiments are shown. [Figure 4]

[0006] A second part of determining the microchunk boundary according to several embodiments is illustrated. [Figure 5]

[0007] Examples of search ranges for adjusting microchunk boundaries in several embodiments are illustrated. [Figure 6]

[0008] The following diagram illustrates an example of video buffering verifier (VBV) usage-based filtering using several embodiments. [Figure 7]

[0009] Examples of scene change-based filtering in several embodiments are illustrated. [Figure 8]

[0010] A first process for selecting a set of coding parameters, according to several embodiments, is illustrated. [Figure 9]

[0011] Examples of predefined coding parameter sets for different labels, according to several embodiments, are illustrated. [Figure 10]

[0012] Examples of systems for implementing a reinforcement learning process, according to several embodiments, are illustrated. [Figure 11]

[0013] This diagram illustrates a video streaming system, according to one embodiment, that communicates with multiple client devices via one or more communication networks. [Figure 12]

[0014] A diagram illustrating a device for viewing video content and advertisements is provided. [Modes for carrying out the invention]

[0004]

[0015] This specification describes techniques for video coding systems. For illustrative purposes, numerous examples and specific details are provided in the following description to provide a complete understanding of several embodiments. Some embodiments, as defined by the claims, may include some or all of the features in these examples, either individually or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0005]

[0016] System Overview

[0006]

[0017] In some embodiments, the system may optimize the encoding parameters for identified portions of the encoding for instances of content. Below, we use video as an example of an instance of content being encoded, but other types of content, such as audio or other content, may also be accepted. The quality control process may identify problematic portions of encoding that do not meet quality metric thresholds. These frames may be referred to as flagged frames or “bad” frames in the video. The number of flagged frames may be small compared to the number of good frames that meet the quality metric thresholds, but there is no guarantee about where flagged frames may occur before encoding, or how flagged frames may cluster throughout the video. For example, a flagged frame may last for several seconds, or it may appear for a fraction of a second. One solution to address problematic portions of encoding might be to re-encode the entire video using different encoding parameters. However, this solution may have drawbacks. For example, re-encoding the entire video can waste computational resources. Since the encoding may include the majority of good frames, re-encoding frames that were good may not be necessary and would waste computational resources. Furthermore, the quality control process may also need to be re-executed for the entire encoding, which would waste additional computational resources. Additionally, re-encoding the entire video may use the same set of revised encoding parameters, which may not address the issues with all local parts where the frames were flagged. For example, flagged frames may still fail the quality control process because these frames contain outlier characteristics of the content compared to the average characteristics of the rest of the frame. In this case, if the revised encoding parameters are based on the average characteristics of the video (as opposed to a general set of encoding parameters initially used for all video), this would not necessarily guarantee a better quality encoding of the flagged frames.

[0007]

[0018] To improve the above, the system may automatically generate optimized encoding parameters for portions of video flagged with quality control issues or other problems. The system may form clusters from the flagged frames. If multiple clusters of video are formed, the system may optimize the encoding parameters for each cluster. This improves the encoding of the video because the system targets the encoding parameters for the characteristics of each cluster. In this process, the system looks up locations such as frame indices or numbers of the flagged video frames. The microchunk optimizer system generates microchunk boundaries, such as start and end frames in the video, based on the clusters of flagged frames. The encoding parameter tuner may analyze the content characteristics of the content associated with the microchunk boundaries and generate an optimized encoding parameter set for the microchunks. The system may also use feedback linking the microchunk optimizer system and the encoding parameter tuner to optimize the microchunk boundary placement and encoding parameter set generation for video segments. The output may be an optimized encoding parameter set for each microchunk in the video. The microchunks can then be re-encoded using their respective optimized encoding parameter sets. The resulting re-encoded microchunks can be inserted into previously generated encodings using microchunk boundaries to form the final encoding of the video.

[0008]

[0019] This discussion uses the re-encoding of frames identified by a quality control process for encoding, but this process can be used for other processes. For example, this process can be used to insert frames into video in an interoperable master format (IMF) workflow for creating content for distribution. This process can encode the flagged portions of video using optimized encoding parameters for those portions. These portions may not fail the quality control process; rather, optimized encoding parameters are desirable for the encoding process. Therefore, the following process can be used for any process that requires the insertion of portions of content during encoding.

[0009]

[0020] system

[0010]

[0021] Figure 1 illustrates a simplified system 100 for a video encoding pipeline in several embodiments. System 100 includes a quality control system 102, a microchunk optimizer system 104, an encoding parameter tuner 106, and a microencoder system 108. Entities in system 100 may run on one or more computing devices, such as servers.

[0011]

[0022] The quality control system 102 may receive video encoding. For example, an encoder may encode video, and the quality control system 102 may analyze frames of the video encoding using quality metrics. The quality control system 102 may be an automated process that automatically analyzes the characteristics of the encoding, or a manual quality control process in which a user analyzes the resulting encoding. The quality control system 102 may determine flagged (e.g., problematic) frames in the encoding, such as frames that do not meet a threshold (e.g., the peak signal-to-noise ratio (PSNR) does not meet the threshold). Flagged frames are those that do not meet the threshold, while other frames that do meet the threshold are considered good frames and are not flagged.

[0012]

[0023] The quality control system 102 outputs the flagged frames to the microchunk optimizer system 104. The flagged frames may be identified by a list of frame locations for encoding. The flagged frames may be distributed or scattered throughout the entire duration of the video. The quality control system 102 may also provide information that can be used to optimize the encoding parameter set; for example, the quality control system 102 may indicate which quality control metrics failed for each frame, or the value of the metrics for each frame. While frames are flagged, the use of frames may include other parts of the encoding, such as blocks of frames, multiple frames, segments, etc.

[0013]

[0024] The microchunk optimizer system 104 receives flagged frames as input. The microchunk optimizer system 104 may also receive other information related to the video, such as metadata from the original encoding statistics used to encode the video, or other metadata. This metadata includes the original encoding statistics of the frame in the encoding, such as bitrate, resolution, frame rate, etc. Other metadata may include information that can help optimize encoding parameters, such as content type, segment duration configuration, group of pictures (GOP) structure, microchunk duration limits, etc. The microchunk optimizer system 104 uses the input to generate microchunk boundaries for clusters of flagged frames. For example, the microchunk optimizer system 104 may cluster a set of flagged frames and create a microchunk boundary for each cluster. Other optional information that the microchunk optimizer system 104 may output may include the target bitrate for each microchunk boundary and a list of instantaneous decoder refresh (IDR) frames used to generate the microchunk boundaries. This information may be used to adjust encoding parameters or to re-encode microchunks. The generation of microchunk boundaries is described in more detail below.

[0014]

[0025] The coding parameter tuner 106 receives a microchunk boundary, which may be a start frame index and an end frame index in video or other location information. The coding parameter tuner 106 also receives source video and uses the frames associated with the microchunk to generate an optimal coding parameter set for the microchunks found within the microchunk boundary. The output may be an optimized coding parameter set for each microchunk. In some embodiments, the coding parameter tuner 106 may generate coding parameter sets using different modes, such as a predefined mode and a continuous learning mode. In the predefined mode, the coding parameter tuner 106 may select a predefined set of characteristics from a plurality of predefined sets of characteristics using a content analyzer process that determines the characteristics of the microchunks. The selected predefined set of characteristics is then used to determine the corresponding coding parameters from a predefined list of coding parameter sets. In the continuous learning process, the coding parameter tuner 106 may use a learning-based process, such as a reinforcement learning-based process, which can automatically discover and learn the optimal coding parameter set for the microchunks. When a revised or optimal set of encoding parameters is found, the values ​​for that set may be revised. For example, the strength of the deblocking filter is an encoding parameter that can be revised based on the content type (for example, the parameter may need to be increased for content such as cartoons and anime compared to live-action content). The encoding parameter tuner 106 may also optimize the bitrate, optimize the encoding frame type and quantization parameters used for flagged frames in microchunks, or perform other optimizations.

[0015]

[0026] When an update regarding the microchunk boundaries is required, the update can be sent to the microchunk optimizer system 104, and the microchunk optimizer system 104 can then generate new microchunk start and end frames. The new microchunk start and end frames are then provided to the encoding parameter tuner 106, and the encoding parameter tuner 106 can then optimize the set of encoding parameters for the new microchunk. This iterative process may require multiple iterations until the optimal microchunk boundaries and the best set of encoding parameter sets are determined.

[0016]

[0027] Once the set of encoding parameters for each microchunk is determined, the microencoder system 108 can re-encode the microchunks according to their respective sets of encoding parameters. For example, the microencoder system 108 can receive the source video and encode a portion of the video according to the microchunk start and end frames and in each respective set of encoding parameters. The encoded microchunks are then inserted during encoding using the microchunk boundaries.

[0017]

[0028] The microchunk optimizer system 104 and the encoding parameter tuner 106 are described in more detail below.

[0018]

[0029] Microchunk Optimizer System

[0019]

[0030] The quality control process may output flagged frames that may be scattered throughout the entire video. The microchunk optimizer system 104 receives the flagged frames and may group them into clusters. The microchunk optimizer system 104 then determines a start frame and an end frame for each cluster of flagged frames. The start frame and end frame are locations such as frame indices in the video and may form microchunks.

[0020]

[0031] Figure 2 illustrates examples of video encoding 200 having flagged frames according to several embodiments. Video encoding 202 includes good frames in 202 and flagged frames in 204. Flagged frames are indicated by diagonal lines. The number of flagged frames in each instance can be different, such as one frame, two or more frames, etc. Also, each instance of flagged frames can be different, although the sizes shown in Figure 2 may be somewhat similar. For example, a flagged frame could be a fraction of a second, a few seconds, etc. As shown, the flagged frames may be scattered throughout the video encoding 200 and may be fewer in number than the number of good frames. However, the number of flagged frames may be greater than the number of good frames.

[0021]

[0032] The microchunk optimizer system 104 can generate microchunks by grouping together portions of flagged frames. Microchunks can be as long as necessary, without exceeding a threshold length. That is, microchunks can be limited in length to avoid the need to re-encode a large portion of the video together. If a large portion is contained within a single microchunk, the microchunk may contain content or video frames that do not share similar video properties, such as spatial and temporal properties. Smaller microchunks may have a higher probability of sharing similar properties compared to larger chunks.

[0022]

[0033] Figures 3 and 4 illustrate examples of determining microchunk boundaries according to several embodiments. Figure 3 illustrates a simplified flowchart 300 of a method for determining draft microchunk boundaries according to several embodiments. In 302, the microchunk optimizer system 104 receives a list of flagged frames. The microchunk optimizer system 104 may process the flagged frames to group scattered but otherwise close frames into clusters. Different methods may be used for clustering the flagged frames. For example, in 304, the microchunk optimizer system 104 determines discontinuities in the list of flagged frames to form groups. In some embodiments, the microchunk optimizer system 104 receives discontinuity thresholds used to determine whether flagged frames belong to the same group. In this process, the microchunk optimizer system 104 may start from the current flagged frame and use discontinuity thresholds around the identified frames to form boundaries. The current frame may be selected as the first flagged frame from the beginning of the video, randomly selected, or user-selected. A neighboring flagged frame may be included in the cluster if it satisfies a discontinuity threshold in terms of distance from the current flagged frame. For example, if the current flagged frame is at index 100 and the threshold is 20 frames, the microchunk optimizer system 104 may determine whether there are any neighboring flagged frames within a boundary that may be forward or backward. For example, any flagged frame between indices 80 and 120 may be included in the group. The boundary may be expanded as frames are added. For example, if a frame is added at index 120, the boundary may be expanded to index 140 by adding the discontinuity threshold to the index of the newly added frame.This process continues until no further frames meet the threshold or the microchunk length is reached. Once a cluster is formed, another flagged frame may be selected, such as a flagged frame that is just after the selected cluster. While the process described above is used to form clusters of flagged frames, other processes may be used, such as an automated clustering process that receives flagged frame indices and automatically forms clusters based on a different algorithm.

[0023]

[0034] Once clusters are found, in 306, the microchunk optimizer system 104 iterates across the clusters. For example, in 308, for each cluster, the microchunk optimizer system 104 determines a draft microchunk boundary. The draft microchunk boundary may be based on different requirements. For example, a minimum microchunk duration may be required, which could be a minimum number of frames or a period of time such as seconds. The minimum microchunk duration may be required to stabilize or efficiently control the encoder's rate. Another requirement may be a forced IDR frame requirement. For example, an IDR frame requirement may stipulate that the start and end frames of a chunk must be at a shot boundary, which is a point of transition or change between consecutive shots in a video sequence. This requirement may be based on different encoding specification requirements. Other requirements may also be used to identify the draft microchunk boundary. Once the draft microchunk boundary is determined, the process continues in Figure 4.

[0024]

[0035] Figure 4 illustrates a second part for determining microchunk boundaries in several embodiments. The second part may analyze the draft microchunk boundaries to determine whether the draft microchunk boundaries should be modified. In 402, the microchunk optimizer system 104 determines whether the draft microchunk is too long. This requirement may be used because it is desirable to optimize the encoding parameter set for smaller portions of the video that may have common characteristics. If the draft microchunk is too long, the characteristics associated with the long microchunk may change, and the initial problem associated with using the same encoding parameter set for the entire video may arise within the microchunk that is considered too long. Therefore, if the draft microchunk is too long, in 403, the microchunk optimizer system 104 determines the division points within the microchunk. Different methods can be used to determine the division points. For example, video buffer verifier (VBV) levels, scene change statistics, fixed cadence / threshold-based methods, etc., may be used to divide the microchunk into multiple microchunks. Feedback from the content characteristics of 408 can also be used to find splitting points within a microchunk. In some embodiments, VBV buffer level data from the original encoding can be used to determine locations for splitting long microchunks. The VBV level is used by the encoder to control the bitrate and buffer usage of the compressed video stream during encoding and decoding. Areas with low VBV usage can be used as points for splitting microchunks. Scene statistics can be used to split long microchunks at scene changes occurring within them. Other methods can also be used to split long microchunks into multiple microchunks, such as a set threshold that splits the microchunks at a set number of frames.

[0025]

[0036] Once an appropriate microchunk size is determined, the process proceeds to complete the microchunk start and end frames. The following steps may be performed sequentially, in parallel, or in different orders. The search range before the start frame and after the end frame of the draft microchunk boundary is used to determine whether a new microchunk boundary should be determined. In 404, the microchunk optimizer system 104 performs VBV buffer usage-based filtering. The microchunk optimizer system 104 may adjust the start or end frame of the microchunk boundary using VBV usage from the original coding statistics. As illustrated in Figures 5 and 6, the microchunk optimizer system 104 may attempt to determine a microchunk boundary with low VBV usage. A threshold may be used to determine when usage is low (e.g., the VBV level falls below the threshold). The VBV usage statistics are measurements used during video coding to prevent buffer underflow or buffer overflow. When the VBV buffer level of the original encoding is already high, any additional bits spent on microchunk encoding can lead to a violation of the VBV buffer model, which can result in playback failure on the client. Therefore, it may be desirable to have a low boundary for VBV usage in order to allow overhead that can increase the bitrate in re-encoding without violating VBV usage requirements. The output of VBV buffer usage-based filtering may be a revised search range. Figures 5 and 6 illustrate in more detail the determination of the revised search range using VBV usage-based filtering.

[0026]

[0037] In 406, the microchunk optimizer system 104 performs scene change-based filtering. For example, the microchunk optimizer system 104 may search for scene changes within the revised search range to find where a microchunk can start or end in a new frame. Another filtering technique may be to ensure that the selected start or end frame is not in a busy action portion of the expected footage where many short scenes are present. This may be measured by calculating the average distance between scene changes and the variance of the durations of these scenes. Selecting a start or end frame at the boundary of a new scene may improve the re-encoding of the microchunk, since the content before the scene change and the content after the scene change may be different. Thus, an optimized set of encoding parameters for the content before the scene change may be different for the content after the scene change. The output of scene-based filtering may be a ranked list of possible start and end frames from the revised search range, or just one start and one end frame. Scene change-based filtering is described in more detail in Figure 7.

[0027]

[0038] In 408, the microchunk optimizer system 104 performs content characteristic-based filtering. This process may be optional or required. In this case, the content of the microchunk may be analyzed to determine whether the start or end frame should be changed. If there is any change in content characteristics based on the microchunk boundary, the microchunk optimizer system 104 may adjust the microchunk boundary accordingly. For example, the microchunk optimizer system 104 may determine whether a draft microchunk contains more than one characteristic. If so, the microchunk optimizer system 104 determines whether the microchunk can be isolated and still be greater than the minimum chunk duration threshold. The microchunk optimizer system 104 may then isolate the microchunks where the characteristic change occurs, and feedback is sent to perform VBV buffer usage-based filtering in 404 and scene change-based filtering in 406 using the isolated chunks again. Furthermore, as will be discussed below, content-characteristic filtering can be used to select one of a ranked list of start and end frames.

[0028]

[0039] In step 410, the microchunk optimizer system 104 adjusts and completes the microchunk start and end frames based on the analysis of steps 404, 406, and 408. In step 412, the microchunk optimizer system 104 outputs the microchunk start and end frames, which can be used by the coding parameter tuner 106.

[0029]

[0040] Before discussing the encoding parameter tuner 106, VBV buffer usage-based filtering and content-characteristic-based filtering will be discussed in more detail.

[0030]

[0041] Figure 5 illustrates examples of search ranges for adjusting microchunk boundaries according to several embodiments. A draft microchunk is shown at 502. For a microchunk, the start frame boundary is shown at 504 and the end frame boundary is shown at 506. This forms the draft microchunk duration. A first search range 508-1 is found before the start frame, and a second search range 508-2 is shown after the end frame. The microchunk optimizer system 104 may search for different start or end frames for microchunks within the search range. The search range may be determined using different methods. For example, a threshold may be used to form the search range, for example, the search range may be 100 frames before the start frame and 100 frames after the end frame. Other methods may be used to determine the search range, such as using a percentage of the draft microchunk duration. For example, if the draft microchunk duration is 200 frames, 20% of the number of frames in the draft chunk duration would be taken as the search range, which could be 40 frames.

[0031]

[0042] Once the search range is determined, VBV usage-based filtering is performed. Figure 6 illustrates an example of VBV usage-based filtering being performed according to several embodiments. The microchunk optimizer system 104 may use the original encoded statistics to determine areas of lower VBV usage. For example, areas of low VBV usage are shown in 602-1 before the start frame at 504 and in 602-2 after the end frame at 506. In some embodiments, the microchunk optimizer system 104 defines lower VBV usage via a moving average with a window size of approximately half the buffer size length. In the example shown in Figure 6, a particular window size may be set to 1 second (or 24 frames for 24Hz video). The microchunk optimizer system 104 then specifies levels of VBV usage for each consecutive second in categories such as low, medium, and high, based on thresholds such as less than 0.5 being high, 0.5 to 0.85 being medium, and greater than 0.85 being low. The VBV buffer level here refers to how much of the buffer is still available. In this case, only 15% of the buffer is occupied. The microchunk optimizer system 104 may then find areas where the longest streak of sustained low VBV usage is observed, for example, one area before the start frame and another area after the end frame. Two areas are described, but more than two areas may be found. Furthermore, in the unlikely event that there is no single area of ​​low VBV usage within the search range, the microchunk optimizer system 104 may expand the search range and perform the search within the expanded search range.

[0032]

[0043] A graph of VBV usage is shown in 606. The Y-axis is the VBV buffer level from the original encoding. The X-axis corresponds to the encoding of the video. Line 608 shows the VBV usage values ​​and draft chunk durations in response to frames within the search range. At the time corresponding to the frame in 602-1, the VBV usage satisfies a low threshold. Also, within the frame found in 602-2, the VBV usage also satisfies a low threshold. In some embodiments, the microchunk optimizer system 104 determines two areas containing the longest streak of persistent low VBV usage: one area before the start frame 504 and one area after the end frame 506. That is, there may be other areas within the search range that contain low VBV usage but use the longest streak. In other embodiments, multiple areas of low VBV usage may be used. A streak can be determined in different ways, such as the VBV usage value having to meet a threshold, the average VBV usage meeting a threshold, and so on. The microchunk optimizer system 104 can then search for boundaries within areas with low VBV usage. For example, the microchunk optimizer system 104 can use scene change-based filtering to search for start frame boundaries within search range 602-1 and end frame boundaries within search range 602-2.

[0033]

[0044] Figure 7 illustrates examples of scene change-based filtering according to several embodiments. The microchunk optimizer system 104 attempts to search for scene changes within the adjusted search ranges 602-1 and 602-2. It may be preferable to either set microchunk boundaries around new scene changes or not place microchunk boundaries around multiple short scene changes.

[0034]

[0045] The microchunk optimizer system 104 may define metrics as indicators of which scene change points can be selected based on statistics (mean, variance) of inter-scene distance and intra-scene duration. In some embodiments, the microchunk optimizer system 104 uses the scene change associated with the longest scene (e.g., looking forward in time, but backward in time may also be used) and at the same time has the inter-scene distance furthest from other scenes. For a given scene change, the intra-scene duration is defined as the length of forward time between this scene change and the next scene change. When this is the last scene before the end of the search range, the microchunk optimizer system 104 uses the end of the search range to calculate the scene duration. For a given scene change, the inter-scene distance may be defined as the average distance to two nearby scene change frames (e.g., backward and forward). For example, if a scene change is detected at frame 100 and two nearby scene changes are detected at frames 70 and 115, respectively, the average distance is calculated to be 27.5 frames. If one of the nearby scene changes cannot be found in any direction until the end of the search range, the microchunk optimizer system 104 may use the distance to other scene changes directly (e.g., not the average). If there is only one scene change within the search range, the microchunk optimizer system 104 uses this scene change directly.

[0035]

[0046] In Figure 7, scene changes are inserted within search ranges 602-1 and 602-2 by the lines indicated at 702. Search range 602-2 is then extended to show the scene changes. Any scene change detection algorithm can be used, such as frame, pixel, and histogram differences, or any other content analysis algorithm. Note that due to the commonly used short ranges, there may be only one scene change detected within search range 602-2, or there may be no scene changes at all. In this case, if there are no scene changes, the microchunk optimizer system 104 may return the frame closest to the start or end frame of the draft microchunk period as the selected start or end frame. If there is one scene change, frames surrounding the scene change may be returned.

[0036]

[0047] In the search range 602-2, a scene change at 704 may be selected. The microchunk optimizer system 104 selects the scene change at 704 because it is further away from the multiple short scene changes found at 706. Furthermore, the scene change at 704 may maximize the in-scene duration, which is from this scene change to the end of the search range 602-2. A scene change at 708 may not be selected, even if it has a longer in-scene duration, because this scene change at 708 is close to multiple other scene changes indicating that a large number of short scenes are expected, and may be a busy area of ​​the video. A busy area of ​​the video may not be a good area to select a scene change because the content may change more drastically and frequently compared to the scene change at 704. A threshold distance from multiple scene changes (e.g., two or more scene changes within a period) may be required to select a scene change. Alternatively, instead of selecting start and end frames, a ranked list of preferred start and end frames may be provided for content-characteristic-based filtering. If content-based filtering is not performed, scene-change-based filtering is used to output the selected start and end frames for the microchunk boundaries.

[0037]

[0048] As discussed above, content-characteristic-based filtering can adjust the start and end frames based on an analysis of the content characteristics. For example, content-characteristic-based filtering may first analyze the original draft microchunk duration. The microchunk optimizer system 104 may determine the content characteristics of the original microchunk. If the microchunk contains multiple content characteristics and the minimum duration can still be met even if the microchunk is separated, the microchunk optimizer system 104 may separate the microchunk. The microchunk can then be re-analyzed using VBV usage-based filtering and scene change-based filtering.

[0038]

[0049] If microchunks are not separated, the microchunk optimizer system 104 analyzes the content before and after the original start frame. For example, 100 frames before the original start frame and 100 frames after the original start frame may be used. If it is determined that the content characteristics are different (e.g., the difference between the two parts before and after the start frame does not meet the threshold), the microchunk optimizer system 104 selects the highest-ranked scene change point before the start frame. The same process is performed for the original end frame, where the microchunk optimizer system 104 analyzes the content before and after the original end frame. If it is determined that the content characteristics are different (e.g., the difference does not meet the threshold), the microchunk optimizer system 104 selects the highest-ranked scene change point after the end frame. The newly formed microchunks may be iteratively analyzed using the content characteristics again, and scene changes in the ranked list are selected until it is determined that the content characteristics before and after the boundary are similar. The output consists of start and end frames based on content characteristics analysis. Content analysis checks whether draft microchunks contain homogeneous characteristics. This may mean that the majority of frames in a draft microchunk (e.g., above a threshold) are identified with the same content label. If content characteristics vary within a microchunk, the process may attempt to separate and / or shorten the microchunk to contain only primarily homogeneous characteristics.

[0039]

[0050] As discussed above, the order of VBV buffer usage-based filtering, content-characteristic filtering, and content-characteristic filtering can vary. In some embodiments, filtering may be performed independently, and then a process may use the results to determine the start and end frames of a microchunk. For example, three processes determine three different start and end frames. Then, the intersection of the three start frames may be used to determine the start frame, and the intersection of the three end frames may be used to determine the end frame. The three start frames may also be analyzed to select one or a different start frame. The same process may be performed on the end frames.

[0040]

[0051] The following describes the operation of the coding parameter tuner after the start and end frames have been selected.

[0041]

[0052] Encoding parameter tuner

[0042]

[0053] The coding parameter tuner 106 may use different methods to select coding parameters. For example, methods using machine learning or deep learning can directly predict the set of coding parameters, or these methods can also first extract features (e.g., low-level or high-level features) and then map those features to coding parameters. Two different methods will be discussed below, but other methods may also be accepted. Figure 8 illustrates a first process for selecting a set of coding parameters according to several embodiments. The coding parameter tuner 106 may use multiple neural networks, such as two neural networks, to determine the set of coding parameters. The use of two neural networks is for illustrative purposes only, and methods for classifying into different labels may differ, such as using a different number of neural networks. In 802, the coding parameter tuner 106 receives source video and the start and end frames of a microchunk. Then, in 802, the following process may be performed for the frames in the microchunk from the start frame to the end frame. In 804, the first neural network #1 may be trained to identify whether a frame falls within a predefined content category, such as hand-drawn animation, natural content, or other categories. The categories are indicated by the labels Content Category #1, Content Category #2, and Other Categories. If the content cannot be classified into a predefined category, Other Categories are used. In some embodiments, the neural network may analyze the content of a frame and classify the content into one of the categories. In other embodiments, the first neural network may output a confidence score that predicts whether the content falls into each of the respective categories.

[0043]

[0054] A second neural network #2 may be trained to identify different aspects of the frame's content characteristics, such as film grain (e.g., noise). If film grain is used, aspects of film grain may be labeled as film grain category #1, film grain category #2, or other film grain categories. Film grain categories may be based on describing different film grain aspects, such as space-like film grain, TV sitcom-like film grain, etc.

[0044]

[0055] In some embodiments, neural networks may be designed to work together. For example, if neural network #1 does not determine the content category (e.g., another category is ranked highest), neural network #2 may be run to determine the film grain category. In other embodiments, neural network #1 and neural network #2 may both be run regardless of classification.

[0045]

[0056] Categories can be associated with a predefined set of coding parameters. The pooling frame results system 808 can pool results for frames in a microchunk to determine a list of labels for the microchunk. For example, neural network #1 or neural network #2 returns predictions of content characteristics one frame at a time. To obtain dominant characteristics for the microchunk, the pooling frame results system 808 can use a pooling method, such as a consensus-based pooling method. In some embodiments, for each frame, the pooling frame results system 808 constructs a dynamic list of predicted labels. The pooling frame results system 808 can then determine a list of labels for the microchunk based on a confidence level, for example. For example, the pooling frame results system 808 may process labels frame by frame, and when the cumulative confidence of the labels exceeds a threshold such as 0.8, the pooling frame results system 808 outputs a list of labels. The pooling frame results system 808 then selects one or more labels from the list.

[0046]

[0057] In some embodiments, labels for microchunks may be selected based on weighted votes, where the weights are determined by reliability. For example, if a frame contains more than one label, the votes may be separated by reliability. That is, if two labels are selected for a frame, one with a reliability of 0.6 and the other with a reliability of 0.2, the first label receives 3 / 4 of the votes and the other category receives 1 / 4 of the votes. Predicted labels for chunks may be selected based on a majority vote. For example, labels could be Content Category #1 (score 0.5), Film Grain Category #1 (score 0.6), Content Category #2 (score 0.1), Film Grain Category #2 (score 0.2), etc. The pooling frame results system 808 may select the label Film Grain Category #1 because it has the highest reliability score.

[0047]

[0058] The coding parameter analyzer 810 then selects a coding parameter set based on the pooling frame results. For example, if one category is selected, such as film grain category #1, the coding parameter analyzer 810 selects a predefined coding parameter set for film grain category #1.

[0048]

[0059] Figure 9 illustrates examples of predefined coding parameter sets for different labels according to several embodiments. Column 902 shows the categorical labels, and column 904 shows the coding parameter sets.

[0049]

[0060] The categories that can be selected include Category #1, Category #2, Category #3, Category #4, and others. Each category label may be associated with an encoding parameter set such as encoding parameter set #1, encoding parameter set #2, encoding parameter set #3, encoding parameter set #4, and the default set. Other categories are associated with a default set of parameters because the content cannot be classified into a predefined category. The encoding parameter set can be a different combination of parameter values ​​based on the content characteristics of each category. Different combinations may include cutree on and off, high VBV rate, low VBV rate, quantization parameter values, specific block segmentation modes, specific motion estimation modes, etc.

[0050]

[0061] The coding parameter tuner 106 may also use a method of continuously learning to determine the optimal coding parameter set. In some embodiments, a learning process (e.g., deep reinforcement learning) may be used. Figure 10 illustrates an example of a system 1000 for implementing a reinforcement learning process according to some embodiments. This process includes an agent 1002 and a microencoder system 108. In particular, for some coding parameters, the coding process may be modeled as a reinforcement learning problem in which the encoder runtime is modeled as an environment, agent 1002 is fed by the encoder state, and a reward is calculated based on immediate actions (or sets of actions that constitute an episode) taken by agent 1002. In some embodiments, the reward may be modeled by the quality of coding of microchunks according to a bitrate target. Actions taken by agent 1002 may be specific coding parameters to be used, such as turning cuttree on / off, quantization parameter values, VBV utilization, a specific block partitioning mode, a specific motion estimation mode, etc. In some embodiments, agent 1002 uses mathematical rewards to optimize quality. Different quality metrics (e.g., peak signal-to-noise ratio, structural similarity, or ideally, more perceptually-based metrics) may be used to optimize the coding parameters. An example of a possible reward function is as follows:

[0051]

number

[0052]

[0062] Here, overshoot can be defined by the delta between the actual bitrate and the target bitrate. Based on the reward value, a neural network trained in a reinforcement learning setting can then ideally learn the best encoding parameters for the microencoder system 108. In some embodiments, an asynchronous advantage actor critic architecture may be used, which allows multiple agents to interact with the reinforcement learning environment. This allows the system to control and optimize multiple encoding parameters simultaneously and reach an optimal state more quickly.

[0053]

[0063] The output of the reinforcement learning process may be a list of frame quantization parameters or other encoding parameters for a microchunk. The microencoder system 108 can then use the values ​​to encode frames for the microchunk. The reinforcement learning process can continuously learn the most optimal parameters based on the encodings encountered. That is, agent 1002 can continuously adjust the parameters in the encoding parameter set based on feedback from the microencoder system 108.

[0054]

[0064] The method described above applies reinforcement learning at the frame level, but the reinforcement learning process can be extended to the subframe or block level, or to multiple frames.

[0055]

[0065] conclusion

[0056]

[0066] Therefore, the process of re-encoding a portion of a video can be performed more efficiently and optimally. For example, the chunk boundaries for microchunks can be optimally determined. Furthermore, the parameters for microchunks can also be optimally determined, resulting in a more resource-efficient process for re-encoding portions of the video using more optimal parameters.

[0057]

[0067] system

[0058]

[0068] Features and embodiments disclosed herein may be implemented in conjunction with a video streaming system 1100 communicating with multiple client devices over one or more communication networks, as shown in Figure 11. Embodiments of the video streaming system 1100 are described solely to provide examples of applications for enabling the distribution and delivery of content prepared in accordance with this disclosure. It should be recognized that this technology is not limited to streaming video applications and may be adapted to other applications and distribution mechanisms.

[0059]

[0069] In one embodiment, a media program provider may include a library of media programs. For example, media programs may be aggregated and delivered through a site (e.g., a website), an application, or a browser. Users can access the media program provider's site or application and request media programs. Users may be limited to requesting only media programs provided by the media program provider.

[0060]

[0070] In system 1100, video data can be obtained from one or more sources, for example, from video source 1110, for use as input to video content server 1102. The input video data may consist of raw or edited frame-based video data in any suitable digital format, such as MPEG (Moving Pictures Experts Group)-1, MPEG-2, MPEG-4, VC-1, H.264 / AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), or other formats. Alternatively, the video may be provided in a non-digital format and converted to a digital format using a scanner or transcoder. The input video data may consist of various types of video clips or programs, such as television episodes, movies, and other content produced as primary content of consumer interest. The video data may also include audio, or only audio may be used.

[0061]

[0071] The video streaming system 1100 may include one or more computer servers or modules 1102, 1104, and 1107 distributed across one or more computers. Each server 1102, 1104, and 1107 may include or be operably coupled to one or more data stores 1109, such as databases, indexes, files, or other data structures. The video content server 1102 may access data stores (not shown) of various video segments. The video content server 1102 may supply video segments as directed by a user interface controller communicating with client devices. As used herein, a video segment refers to a distinct portion of frame-based video data, such as a television episode, a movie, a recorded live performance, or other video content, which may be used in a streaming video session for viewing.

[0062]

[0072] In some embodiments, the video advertising server 1104 may access a data store of relatively short videos (e.g., 10-second, 30-second, or 60-second video advertisements) configured as advertisements for specific advertisers or messages. The advertisements may be provided on behalf of advertisers in exchange for some form of payment, or they may contain promotional messages, public service messages, or any other information for system 1100. The video advertising server 1104 may supply video advertising segments as directed by a user interface controller (not shown).

[0063]

[0073] The video streaming system 1100 may also include system 100.

[0064]

[0074] The video streaming system 1100 may further include integration and streaming components 1107 that integrate video content and video advertisements into streaming video segments. For example, the streaming component 1107 may be a content server or a streaming media server. A controller (not shown) may determine the selection or configuration of advertisements in the streaming video based on any appropriate algorithm or process. The video streaming system 1100 may include other modules or units not shown in Figure 11, such as a management server, a transaction server, network infrastructure, an advertisement selection engine, etc.

[0065]

[0075] The video streaming system 1100 may be connected to a data communication network 1112. The data communication network 1112 may comprise a local area network (LAN), a wide area network (WAN), such as the Internet, a telephone network, a wireless network 1114 (such as a wireless cellular telecommunications network (WCS)), or any combination thereof or similar networks.

[0066]

[0076] One or more client devices 1120 may communicate with the video streaming system 1100 via a data communication network 1112, a wireless network 1114, or another network. Such client devices may include, for example, one or more laptop computers 1120-1, desktop computers 1120-2, "smart" mobile phones 1120-3, tablet devices 1120-4, network-enabled televisions 1120-5, or a combination thereof, via a router 1118 for the LAN, a base station 1117 for the wireless network 1114, or some other connection. During operation, such client devices 1120 may send and receive data or commands to the system 1100 in response to user input received from a user input device or other input. In response, the system 1100 may supply video segments and metadata from the data store 1109 to the client device 1120 in response to the selection of media programs. The client device 1120 can output video content from a streaming video segment in a media player using a display screen, projector, or other video output device, and can receive user input for interacting with the video content.

[0067]

[0077] The distribution of audio-video data can be carried out from a streaming component 1107 to a remote client device through computer networks, telecommunications networks, and combinations of such networks, using various methods, such as streaming. In streaming, the content server continuously streams audio-video data to a media player component that operates at least partially on the client device, and the media player component can play the audio-video data as it receives the streaming data from the server. Although streaming has been discussed, other delivery methods may be used. The media player component may begin playing the video data immediately after receiving the first part of the data from the content provider. Conventional streaming techniques use a single provider to deliver a stream of data to a set of end users. High bandwidth and processing power may be required to deliver a single stream to a large number of viewers, and the bandwidth required of the provider may increase as the number of end users increases.

[0068]

[0078] Streaming media can be delivered on demand or live. Streaming allows for instant playback at any point in a file. End users can skip media files to begin playback or change playback to any point in the media file. Therefore, end users do not have to wait for files to be downloaded gradually. Typically, streaming media is delivered via dedicated devices that accept requests for video files from a small number of dedicated servers with high bandwidth capabilities and deliver only the amount of data necessary to play the video, at the speed required for playback, along with information about the format, bandwidth, and structure of those files. The streaming media server may also consider the transmission bandwidth and capabilities of the media player on the destination client. The streaming component 1107 can communicate with the client device 1120 using control messages and data messages to adapt to changing network conditions as the video is played. These control messages may include commands to enable control functions such as fast forward, rewind, pause, or seek to a specific part of the file on the client.

[0069]

[0079] The streaming component 1107 transmits video data only as needed and at the required speed, thus maintaining precise control over the number of streams supplied. Viewers would not be able to view high-data-rate video through a lower-data-rate transmission medium. However, the streaming media server allows (1) users to have random access to video files, (2) to monitor who is watching which video programs and for how long those programs are being watched, (3) to use transmission bandwidth more efficiently because only the amount of data required to support the viewing experience is transmitted, and (4) video files are not stored on the viewer's computer and are discarded by the media player, thus allowing for more control over the content.

[0070]

[0080] The streaming component 1107 can use TCP-based protocols such as HTTP (HyperText Transfer Protocol) and RTMP (Real Time Messaging Protocol). The streaming component 1107 can also deliver live webcasts and perform multicast, which allows more than one client to tune into a single stream, thereby saving bandwidth. A streaming media player may not rely on buffering the entire video to provide random access to any point in a media program. Instead, this is achieved using control messages sent from the media player to the streaming media server. Other protocols used for streaming are HLS (HTTP live streaming) or DASH (Dynamic Adaptive Streaming over HTTP). The HLS and DASH protocols deliver video over HTTP via a playlist of small segments that are typically made available at various bitrates from one or more content delivery networks (CDNs). This allows the media player to switch both the bitrate and content source on a segment-by-segment basis. Switching helps compensate for network bandwidth dispersion and infrastructure failures that may occur during video playback.

[0071]

[0081] The distribution of video content via streaming can be achieved under various models. In one model, users pay for viewing video programs, for example, by paying a fee for access to a library of media programs or a limited portion of a media program, or by using a pay-per-view service. In another model, widely adopted by television broadcasting shortly after its inception, sponsors pay for the presentation of media programs in exchange for the right to display advertisements during or adjacent to the presentation of the program. In some models, advertisements are inserted at predetermined times within the video program, which may be called "ad slots" or "ad breaks." With streaming video, media players may be configured so that client devices cannot play video without also playing designated advertisements during specified ad slots.

[0072]

[0082] Referring to Figure 12, an illustrative diagram of a device 1200 for viewing video content and advertisements is shown. In a selected embodiment, the device 1200 may include a processor (CPU) 1202 operably coupled to a processor memory 1204, the processor memory 1204 holding functional modules binary-coded for execution by the processor 1202. Such functional modules may include an operating system 1206 for handling system functions such as input / output and memory access, a browser 1208 for displaying web pages, and a media player 1210 for playing video. The memory 1204 may hold additional modules not shown in Figure 12, for example, modules for performing other operations described elsewhere in this specification.

[0073]

[0083] Bus 1214 or other communication components may support the communication of information within the device 1200. Processor 1202 may be a specialized or dedicated microprocessor configured or operable to perform a particular task in accordance with the features and aspects disclosed herein by executing machine-readable software code that defines the particular task. Processor memory 1204 (e.g., random access memory (RAM) or other dynamic storage device) may be connected to bus 1214 or directly to processor 1202 and may store information and instructions to be executed by processor 1202. Memory 1204 may also store temporary variables or other intermediate information during the execution of such instructions.

[0074]

[0084] The computer-readable medium in the storage device 1224 is connected to the bus 1214 and may store static information and instructions for the processor 1202. For example, the storage device (CRM) 1224 may store modules for the operating system 1206, browser 1208, and media player 1210 when the device 1200 is powered off, and when the device 1200 is powered on, the modules may be loaded from there into the processor memory 1204. The storage device 1224 may include a non-temporary computer-readable storage medium that holds information, instructions, or any combination thereof, for example, instructions that, when executed by the processor 1202, configure or enable the device 1200 to perform one or more operations in the manner described herein.

[0075]

[0085] A network communication (comm.) interface 1216 may also be connected to bus 1214. The network communication interface 1216 may optionally provide or support bidirectional data communication between the device 1200 and one or more external devices, such as a streaming system 1100, via a router / modem 1226 and a wired or wireless connection 1225. Alternatively, or in addition, the device 1200 may include a transceiver 1218 connected to antenna 1229, through which the device 1200 may wirelessly communicate with a base station for a wireless communication system or with the router / modem 1226. Alternatively, the device 1200 may communicate with the video streaming system 1100 via a local area network, a virtual private network, or other network. Another alternative is that the device 1200 may be incorporated as a module or component of system 1100 and communicate with other components via bus 1214 or by some other modality.

[0076]

[0086] The device 1200 may be connected to the display unit 1228 (for example, via bus 1214 and graphics processing unit 1220). The display 1228 may include any suitable configuration for displaying information to the operator of the device 1200. For example, the display 1228 may include or utilize a liquid crystal display (LCD), a touchscreen LCD (e.g., a capacitive display), a light-emitting diode (LED) display, a projector, or other display device for presenting information to the user of the device 1200 in a visual display.

[0077]

[0087] One or more input devices 1230 (e.g., an alphanumeric keyboard, microphone, keypad, remote control, game controller, camera, or camera array) may be connected to the bus 1214 via the user input port 1222 to communicate information and commands to the device 1200. In a selected embodiment, the input devices 1230 may provide or support control over cursor positioning. Such cursor control devices, also called pointing devices, may be configured as a mouse, trackball, trackpad, touchscreen, cursor directional keys, or other device for receiving or tracking physical movement and converting that movement into electrical signals indicating cursor movement. The cursor control device may be incorporated into the display unit 1228, for example, using a touch-sensitive screen. The cursor control device may communicate directional information and command selections to the processor 1202 and control cursor movement on the display 1228. The cursor control device may have two or more degrees of freedom, for example, allowing the device to specify the cursor position in a plane or three-dimensional space.

[0078]

[0088] Some embodiments may be implemented by an instruction execution system, apparatus, system, or machine, or in a non-temporary computer-readable storage medium for use in connection therewith. The computer-readable storage medium contains instructions for controlling a computer system to perform the methods described in some embodiments. The computer system may include one or more computing devices. The instructions may be configured or operable to perform what is described in some embodiments when executed by one or more computer processors.

[0079]

[0089] In this description and throughout the following claims, “a,” “an,” and “the” include multiple references unless the context explicitly indicates otherwise. Furthermore, in this description and throughout the following claims, the meaning of “in” includes “in” and “on” unless the context explicitly indicates otherwise.

[0080]

[0090] The above description illustrates various embodiments, along with examples of how aspects of several embodiments may be implemented. The above examples and embodiments should not be considered sole embodiments, but are presented to illustrate the flexibility and advantages of several embodiments as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations, and equivalents may be used without departing from the scope of the invention as defined by the claims. The following is a direct reproduction of the claims as originally filed. [C1] A method, To form a cluster of locations, the method involves analyzing flagged locations from multiple locations in the video encoding, wherein the cluster includes at least one flagged location. Based on searching for a first start location and a first end location in the encoding that includes the cluster, a draft microchunk boundary for the cluster is determined. The process involves searching for a second start location within the first search range and a second end location within the second search range, wherein the second start location and the second end location form a microchunk. Based on the content characteristics of the microchunk, a set of encoding parameters for the microchunk formed by the second start location and the second end location is determined; To encode the microchunks for insertion in the encoding of the video, the encoding parameter set is used. A method that includes [a certain feature]. [C2] The method according to C1, wherein the flagged locations are flagged based on the quality metrics of the flagged locations meeting a threshold. [C3] The method of C1, wherein a quality control process analyzes the encoding of the video based on quality metrics and outputs locations for the flagged locations in the encoding of the video. [C4] Analyzing the flagged locations means The method of C1, comprising grouping the portion of flagged locations in the cluster when the distance between the portions of flagged locations in the encoding of the video satisfies a threshold. [C5] To generate the first search range and the second search range based on a predetermined distance from the first start location or the first end location or the length of the cluster. The method according to C1, further comprising the features described above. [C6] Determining the draft microchunk boundary is The method according to C1, comprising determining the draft microchunk boundary based on the minimum microchunk duration. [C7] Searching within the first search range and the second search range is: Determining a first usage area of ​​video buffer verifier usage that satisfies a threshold within the first search range, wherein the video buffer verifier usage is determined during the encoding of the video. To determine a second usage area of ​​video buffer verifier usage that satisfies the threshold within the second search range, and A method of C1 comprising: [C8] The method according to C7, wherein the first usage area and the second usage area have lower video buffer verifier usage than another area of ​​the first search range or the second search range. [C9] Searching within the first search range and the second search range is: The method of C1, comprising analyzing scene changes to select the second starting location or the second ending location. [C10] Analyzing scene changes is In response to a scene change, the in-scene distance for each scene is calculated based on the scene change and the inter-scene distance between scenes. Based on the distance within the scene and the distance between scenes, the scene change is selected as the second start location or the second end location. A method of C9 comprising the same equipment. [C11] Analyzing scene changes is The method of C9, comprising selecting the scene change as the second start location or second end location based on the distance from a plurality of scene changes that are not within a threshold distance from the aforementioned scene change. [C12] Determining the coding parameter set means that The method of C1, comprising selecting a predefined category based on the content of the microchunk, wherein the set of coding parameters associated with the predefined category is used for the microchunk. [C13] Determining the coding parameter set means The method of C1, comprising selecting a predefined category based on an aspect of the characteristics from a plurality of aspects of the characteristics of the microchunk, wherein the set of coding parameters associated with the predefined category is used for the microchunk. [C14] Determining the coding parameter set means Selecting a first predefined category based on the content of the microchunk, Selecting a second predefined category based on a set of characteristics from among multiple sets of characteristics of the microchunk. The method according to C1, wherein the set of coding parameters associated with the first predefined category or the second predefined category is used for the microchunk. [C15] Determining the coding parameter set means Selecting a number of predefined categories for locations within the microchunk, and associating each of the predefined categories with a predefined set of coding parameters, In order to select a predefined category from the aforementioned multiple predefined categories, the multiple predefined categories are combined, The predefined coding parameter set is used as the coding parameter set for the microchunk. A method of C1 comprising: [C16] Determining the coding parameter set means The method according to C1, comprising using a continuous learning process to select the coding parameter set, wherein the continuous learning process continuously learns the coding parameter set for use when coding microchunks. [C17] The continuous learning process described above is Receiving the state of the encoder encoding the microchunk, To generate a reward used to adjust the aforementioned coding parameter set. A method for C16 comprising the same method. [C18] The method of C1, wherein the flagged location comprises a flagged frame. [C19] A non-temporary computer-readable storage medium storing computer-executable instructions, wherein, when the computer-executable instructions are executed by a computing device, the computing device, To form a cluster of locations, the method involves analyzing flagged locations from multiple locations in the video encoding, wherein the cluster includes at least one flagged location. Based on searching for a first start location and a first end location in the encoding that includes the cluster, a draft microchunk boundary for the cluster is determined. The process involves searching for a second start location within the first search range and a second end location within the second search range, wherein the second start location and the second end location form a microchunk. Based on the content characteristics of the microchunk, a set of encoding parameters for the microchunk formed by the second start location and the second end location is determined; To encode the microchunks for insertion in the encoding of the video, the encoding parameter set is used. A non-temporary, computer-readable storage medium that enables operation to perform the following actions. [C20] A device, One or more computer processors, Computer-readable storage media and The computer-readable storage medium is equipped with, To form a cluster of locations, the method involves analyzing flagged locations from multiple locations in the video encoding, wherein the cluster includes at least one flagged location. Based on searching for a first start location and a first end location in the encoding that includes the cluster, a draft microchunk boundary for the cluster is determined. The process involves searching for a second start location within the first search range and a second end location within the second search range, wherein the second start location and the second end location form a microchunk. Based on the content characteristics of the microchunk, a set of encoding parameters for the microchunk formed by the second start location and the second end location is determined; To encode the microchunks for insertion in the encoding of the video, the encoding parameter set is used. A device comprising instructions for controlling one or more computer processors so that they can operate in order to perform a certain action.

Claims

1. It is a method, To form a cluster of frames, the flagged frames from multiple frames in the video encoding are analyzed, and the cluster comprises multiple flagged frames. Determining the draft microchunk boundaries for the cluster of the first start frame and the first end frame in the encoding including the cluster, The first search range before the first start frame and the second search range after the first end frame are searched for within the first search range and the second end frame within the second search range, wherein the second start frame and the second end frame form a microchunk. In order to determine the content characteristics of the microchunk, a plurality of frames are analyzed based on the second start frame and the second end frame, Based on mapping the content characteristics of the microchunk to an encoding parameter set, the encoding parameter set for the microchunk formed by the second start frame and the second end frame is determined, To encode the multiple frames in the microchunk for insertion in the encoding of the video, the encoding parameter set is used. A method that includes [a certain feature].

2. The method according to claim 1, wherein the flagged frames are flagged based on the quality metric of the flagged frames meeting a threshold.

3. The method according to claim 1, wherein the quality control process analyzes the encoding of the video based on quality metrics and outputs frames for the flagged frames in the encoding of the video.

4. Analyzing the flagged frames mentioned above means The method according to claim 1, further comprising grouping the portion of flagged frames in the cluster when the distance between the frames of the portion of flagged frames in the encoding of the video satisfies a threshold.

5. The first and second search ranges are generated based on a predetermined distance from the first start frame or the first end frame, or the length of the cluster. The method according to claim 1, further comprising:

6. Determining the aforementioned draft microchunk boundaries is The method according to claim 1, comprising determining the draft microchunk boundary based on the minimum microchunk duration.

7. Searching within the first search range and the second search range is To determine the second starting frame, a first usage area of ​​video buffer verifier usage that satisfies a threshold within the first search range is determined, wherein the video buffer verifier usage is determined during the encoding of the video. In order to determine the second termination frame, a second usage area of ​​video buffer verifier usage that satisfies a threshold within the second search range is determined. The method according to claim 1, comprising:

8. The method according to claim 7, wherein the first usage area and the second usage area have lower video buffer verifier usage than another area of ​​the first search range or the second search range.

9. Searching within the first search range and the second search range is The method according to claim 1, further comprising analyzing a scene change to select the second start frame or the second end frame.

10. Analyzing scene changes is In response to a scene change, the in-scene distance for each scene is calculated based on the scene change and the inter-scene distance between scenes. Based on the distance within the scene and the distance between scenes, the scene change is selected as the second start frame or the second end frame. The method according to claim 9, comprising:

11. Analyzing scene changes is The method according to claim 9, further comprising selecting the scene change as the second start frame or the second end frame based on the distance from a plurality of scene changes that are not within a threshold distance from the aforementioned scene change.

12. Determining the aforementioned coding parameter set means The method according to claim 1, comprising analyzing the content of the microchunks in order to classify the microchunks into predefined categories, wherein the set of encoding parameters associated with the predefined categories is used for the microchunks.

13. Determining the aforementioned coding parameter set means The method according to claim 1, comprising analyzing the content of a microchunk to classify the microchunk into a predefined category based on a plurality of aspects of the content characteristics of the microchunk, wherein the set of encoding parameters associated with the predefined category is used for the microchunk.

14. Determining the aforementioned coding parameter set means To classify the microchunks into a first predefined category, the content of the microchunks is analyzed, In order to classify the microchunks into a second predefined category based on the content characteristics of the microchunks from multiple aspects of the content characteristics of the microchunks, the content of the microchunks is analyzed. The method according to claim 1, wherein the set of coding parameters associated with the first predefined category or the second predefined category is used for the microchunk.

15. Determining the aforementioned coding parameter set means To classify the microchunk into a plurality of predefined categories, the frames of the microchunk are analyzed, and each of the plurality of predefined categories is associated with a predefined set of coding parameters. Selecting a predefined category from the aforementioned multiple predefined categories, The predefined coding parameter set is used as the coding parameter set for the microchunk. The method according to claim 1, comprising:

16. Determining the aforementioned coding parameter set means The method according to claim 1, comprising using a continuous learning process to select the encoding parameter set, wherein the continuous learning process continuously learns the encoding parameter set for use when encoding microchunks.

17. The aforementioned continuous learning process is Receiving the state of the encoder encoding the microchunk, To generate a reward used to adjust the aforementioned coding parameter set. The method according to claim 16, comprising:

18. A non-temporary computer-readable storage medium storing computer-executable instructions, wherein when the computer-executable instructions are executed by a computing device, the computing device... To form a cluster of frames, the flagged frames from multiple frames in the video encoding are analyzed, and the cluster comprises multiple flagged frames. Determining the draft microchunk boundaries for the cluster of the first start frame and the first end frame in the encoding including the cluster, The first search range before the first start frame and the second search range after the first end frame are searched for within the first search range and the second end frame within the second search range, wherein the second start frame and the second end frame form a microchunk. In order to determine the content characteristics of the microchunk, a plurality of frames are analyzed based on the second start frame and the second end frame, Based on mapping the content characteristics of the microchunk to an encoding parameter set, the encoding parameter set for the microchunk formed by the second start frame and the second end frame is determined, To encode the multiple frames in the microchunk for insertion in the encoding of the video, the encoding parameter set is used. A non-temporary, computer-readable storage medium that enables operation to perform the following actions.

19. It is a device, One or more computer processors, Computer-readable storage media and The computer-readable storage medium is equipped with, To form a cluster of frames, the flagged frames from multiple frames in the video encoding are analyzed, and the cluster comprises multiple flagged frames. Determining the draft microchunk boundaries for the cluster of the first start frame and the first end frame in the encoding including the cluster, The first search range before the first start frame and the second search range after the first end frame are searched for within the first search range and the second end frame within the second search range, wherein the second start frame and the second end frame form a microchunk. In order to determine the content characteristics of the microchunk, a plurality of frames are analyzed based on the second start frame and the second end frame, Based on mapping the content characteristics of the microchunk to an encoding parameter set, the encoding parameter set for the microchunk formed by the second start frame and the second end frame is determined, To encode the multiple frames in the microchunk for insertion in the encoding of the video, the encoding parameter set is used. A device comprising instructions for controlling one or more computer processors so that they can be operated to perform the following.

Images