Pre-analysis bit budgeting for context and feature-based video encoding

Abstract

A system, apparatus, and method for bit budgeting in pre-analysis of context- and feature-based video encoding are disclosed. A pre-encoder receives a video frame and evaluates each block of the frame for the presence of several context indicators. Context indicators may include memory color, text, depth of field, and other specific objects. A coefficient is generated for each detected context indicator and summed with other coefficients to generate a final importance value for the block. The coefficients can be adjusted so that only defined portions of the image are considered important. The final importance value for the block is used to determine the bit budget for the block. The block bit budget is provided to the encoder and used to influence, for example, the quantization parameters used to encode the block.

Description

[Background technology]

[0001] (Explanation of related technologies) Various applications perform the encoding and decoding of image or video content. For example, video transcoding, desktop sharing, cloud gaming, and gaming spectating are some applications that include support for encoding and decoding content. Encoders typically have a target bitrate that they aim to achieve when encoding a given video stream. The target bitrate is roughly translated into the target bit size of each frame in the encoded version of a given video stream. For example, in one embodiment, the target bitrate is specified in bits per second (e.g., 3 megabits per second (Mbps)) and the frame rate of the video sequence is specified in frames per second (fps) (e.g., 60fps, 24fps). In this example, the preferred bit size of the encoded video frame is calculated by dividing the preferred bitrate by the frame rate. Here, we assume that the bitrate trajectory is linear. If it is not linear, a similar method can be used to estimate the preferred bit size of the encoded frame.

[0002] Each video frame is typically divided into a plurality of blocks. Examples of blocks include Coding Tree Blocks (CTBs) for use with the High Efficiency Video Coding (HEVC) standard or macro blocks for use with the H.264 standard. Other types of blocks for use with other video and image compression standards are also conceivable. The encoder can adjust how each block of the frame is encoded based on the measured properties of the encoded blocks (e.g., level of detail, contrast, etc.). However, if the content of the frame is substantially uniform, it is not preferable to apply the adjustment used for one block to the entire frame. For example, the encoder can determine to allocate a high bit budget to very detailed blocks. However, if most of the blocks are very detailed, the encoder will quickly exhaust the available bits within the budget. Natural scenes (forests, grass, etc.) and video games are typical examples where the entire image or most of the image is detailed and / or uniform.

[0003] The advantages of the methods and mechanisms described herein can be better understood by referring to the following description in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0004] [Figure 1] FIG. is a block diagram of one embodiment of a system for encoding and decoding content [Figure 2] FIG. is a block diagram of one embodiment of a server [Figure 3] FIG. is a block diagram of one embodiment of a pre-encoder coupled to a table [Figure 4] FIG. is a block diagram of one embodiment of a pre-encoder with a plurality of detectors [Figure 5] FIG. is a block diagram of one embodiment of a pre-encoder that generates an importance table [Figure 6] FIG. is a block diagram of one embodiment of a pre-encoder that generates a bit budget table [Figure 7]This figure shows one possible example of a frame being analyzed by a preencoder. [Figure 8] This figure shows one possible example of a frame being analyzed by a preencoder. [Figure 9] This is a generalized flowchart illustrating one embodiment of a method for generating a block-by-block bit budget for a preencoder. [Figure 10] This is a generalized flowchart illustrating one embodiment of a method for adjusting the context indicator coefficients of a frame block based on the scarcity of context indicators across the entire frame. [Modes for carrying out the invention]

[0005] The following description includes numerous specific details to provide a full understanding of the methods and mechanisms presented herein. However, those skilled in the art will recognize that various embodiments can be carried out without these specific details. In some examples, well-known structures, components, signals, computer program instructions, and techniques are not shown in detail to avoid obscuring the methods described herein. For the sake of brevity and clarity, please understand that the elements shown in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to others.

[0006] A system, apparatus, and method for bit budgeting in pre-analysis of video encoding based on context and features is disclosed herein. In one embodiment, the system includes a pre-encoder and an encoder for encoding a video stream. In one embodiment, the pre-encoder receives a video frame and evaluates each block of the frame to check for the presence of several contextual indicators. The graduality of the block may be predefined or adaptively set. For each block, the pre-encoder determines whether any of several different types of contextual indicators are present in the block. Contextual indicators may include, but are not limited to, memory color, text, depth of field, and other specific objects. A coefficient is generated for each detected contextual indicator and is added with other coefficients to generate a final importance value for the block. The coefficients are adjusted so that only defined portions of the image are considered important, and blocks considered important are allocated a specific percentage of the total available bit budget. The final importance value of a block is used to influence the block's bit budget, taking into account the final importance values ​​of other blocks as well. The block bit budget is provided to the encoder and used to influence the quantization parameters, etc., used for encoding the block. It should be noted that context-indicator-based bit buzzing can be combined with one or more other techniques to determine the overall bit budget used for a block of frames. In one embodiment, the encoder selects the quantization intensity (e.g., quantization parameter (QP)) to use when encoding each block based on the bit budget allocated to the block by the preencoder.

[0007] Referring to Figure 1, a block diagram of one embodiment of a system 100 for encoding and decoding content is shown. The system 100 includes a server 105, a network 110, a client 115, and a display 120. In other embodiments, the system 100 may include multiple clients connected to the server 105 via the network 110, and the multiple clients receive the same or different bitstreams generated by the server 105. The system 100 may also include one or more servers 105 that generate multiple bitstreams for the multiple clients.

[0008] In one embodiment, system 100 encodes and decodes video content. In various embodiments, various applications such as video game applications, cloud game applications, virtual desktop infrastructure applications, autonomous vehicle applications, online streaming applications, screen sharing applications, or other types of applications are run by system 100. In one embodiment, server 105 renders video or image frames and then encodes the frames into encoded bitstreams. In one embodiment, server 105 includes a preencoder and an encoder for managing the encoding process. The preencoder may also be referred to herein as a “pre-analysis unit”.

[0009] In one embodiment, the preencoder analyzes a block of frames to detect context indicators. As used herein, “context indicators” are defined as features that are considered perceptually important to a particular application being run by the system. In one embodiment, context indicators may include features such as signs, text, faces, bodies, everyday objects (cars, streets, streetlights, etc.), and memory colors. As used herein, “memory colors” are defined as familiar colors associated with the scene presented in the frame. An example of a “memory color” is skin tone. In other embodiments, other applications may have other types of context indicators.

[0010] For each block, a significance value is generated based on the context indicators detected within the block and the scores assigned to those indicators. The significance value is adjusted as appropriate based on one or more other variables, such as the rarity of the detected context indicators. The preencoder assigns a bit budget to each block based on the significance value generated for the block. In one embodiment, the significance value of a block is scaled so that the total number of bits assigned to all blocks is within a bit size range calculated based on the encoded bitstream that satisfies the target bitrate. The encoder then encodes the block to match the bit budget assigned to it by the preencoder. In one embodiment, the encoder adjusts the quantization parameter (QP) used to encode the block so that the encoded block falls within a threshold amount of the bit budget to which it is assigned. After the encoded bitstream is generated, the server 105 transmits the encoded bitstream to the client 115 via the network 110. The client 115 decodes the encoded bitstream and generates video or image frames to send to the display 120 or display compositor.

[0011] Network 110 represents any type of network or combination of networks, including wireless connections, direct local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), intranets, the Internet, cable networks, packet-switched networks, fiber optic networks, routers, storage area networks, or other types of networks. Examples of LANs include Ethernet® networks, fiber-distributed data interface (FDDI) networks, and token ring networks. In various embodiments, network 110 includes remote direct memory access (RDMA) hardware and / or software, transmission control protocol / Internet protocol (TCP / IP) hardware and / or TCP / IP software, routers, repeaters, switches, grids, and / or other components.

[0012] Server 105 includes any combination of software and / or hardware for rendering video / image frames and encoding these frames into a bitstream. In one embodiment, Server 105 includes one or more software applications running on one or more processors of one or more servers. Server 105 also includes network communication functions, one or more input / output devices, and / or other components. The processor(s) of Server 105 may include any number and type of processors (such as graphics processing units (GPUs), central processing units (CPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc.). The processor(s) are coupled to one or more memory devices that store program instructions executable by the processor(s). Similarly, Client 115 includes any combination of software and / or hardware for decoding the bitstream and sending the frames to Display 120. In one embodiment, Client 115 includes one or more software applications running on one or more processors of one or more computing devices. In various embodiments, the client 115 is a computing device, a game console, a mobile device, a streaming media player, or other type of device.

[0013] Referring to Figure 2, a block diagram of one embodiment of the components of a server 200 for encoding video frames is shown. A new frame 210 of video is received by the server 200 and provided to the preencoder 220 and encoder 230. Each of the preencoder 220 and encoder 230 is implemented using any suitable combination of hardware and / or software. In various embodiments, software instructions for implementing the preencoder 220 and / or encoder 230 are stored in memory 240. Memory 240 represents any number and type of memory devices. In one embodiment, the preencoder 220 generates a block bit budget 225 for the blocks of the new frame 210 based on an analysis of the blocks of the new frame 210. In one embodiment, the rate controller 232 adjusts the block bit budget 225 generated by the preencoder based on the current budget condition (i.e., the budget trajectory on the trajectory). The encoder 230 then adjusts the encoding parameters so that the encoded blocks satisfy or closely approximate the allocated block bit budget 225 when generating the encoded bitstream 235. In one embodiment, the components of server 200 are contained within server 105 (in Figure 1). It should be noted that in other embodiments, server 200 includes other components and / or is configured in other suitable ways other than those shown in Figure 2.

[0014] In one embodiment, the preencoder 220 processes the new frame 210 in blocks. For each block, the preencoder 220 determines which context indicators are present in the block. In one embodiment, the context indicators include signs, text, and memory colors. In another embodiment, the preencoder 220 searches for other types of context indicators. Based on the context indicators found within the block, the preencoder 220 generates a relative importance value for the block. The importance value is then used to generate a block bit budget 225, which is transmitted to the encoder 230. In yet another embodiment, the preencoder 220 provides the proposed bit budget to the rate controller 232, which then adjusts the proposed bit budget to generate a final bit budget used to encode the blocks of the new frame 210. In a further embodiment, the preencoder 220 transmits the importance values ​​to the rate controller 232, which then allocates the bit budget to the blocks based on those importance values.

[0015] In various embodiments, the preencoder 220 and rate controller 232 work together in various different ways to determine the final bit budget used to encode blocks of new frames 210. In one embodiment, the preencoder 220 assigns block bit budgets 225 to blocks without the rate controller 232 performing any adjustments to the block bit budgets 225. In another embodiment, the preencoder 220 assigns block bit budgets 225 to blocks, and these block bit budgets 225 are then refined by the rate controller 232 based on the current budget conditions. In a further embodiment, the rate controller 232 generates the bit budgets, and the preencoder 220 provides guidance on adjusting to the bit budgets of specific individual blocks. In other embodiments, other technologies may influence the bit budget based on effects that the human visual system cannot properly perceive in certain situations, such as high-speed motion or regions with different motion vectors.

[0016] In one embodiment, the encoder 230 determines the quantization intensity used to encode each block of the new frame 210 based on the bit budget allocated to the block. In other embodiments, the encoder 230 adjusts other settings used when encoding each block of the new frame 210 based on the bit budget allocated to the block. In one embodiment, quantization intensity refers to the quantization parameter (QP). Where the term QP is used herein, it should be understood that the term is intended to apply to other types of quantization intensity settings used with any type of coding standard. When encoding a given block, the encoder 230 selects a QP, and as a result, the bit size of the block closely matches the bit budget calculated by the pre-encoder 220. By matching the bit budget allocated to the block by the pre-encoder 220, the encoder 230 can satisfy the desired bitrate of the encoded bitstream 235.

[0017] Referring to Figure 3, a block diagram of one embodiment of the preencoder 305 coupled to table 310 is shown. The preencoder 305 is coupled to any number of tables 310 that specify various settings that affect the generation of the bit budget for individual blocks of the frame being encoded. For example, context indicator table 310A specifies several context indicators that have high values ​​and increase the relative importance of blocks of frames that contain any of these context indicators. Context indicator table 310A also includes a score to apply to each context indicator. Alternatively, in another embodiment, a separate scoring table is included in table 310 to specify a score to apply to each context indicator detected within the frame.

[0018] The memory color table 310B identifies certain memory colors that increase the relative importance of a block of a frame containing an amount exceeding a threshold for one of these memory colors. For example, in one embodiment, skin tone is identified as a memory color in one version of the memory color table 310B. In another embodiment, blue sky is identified as a memory color in another version of the memory color table 310B. Other types of memory colors may be specified in other embodiments as well. The memory color table 310B may also specify that a memory color may have a higher value in the presence of a first attribute, and / or a lower value in the presence of a second attribute. For example, in one embodiment, skin tone has a higher value in the brighter areas of the frame. Also in one embodiment, skin tone has a lower value in areas with fast movement. In another embodiment, grass and trees are less important when affected by depth of field. Other examples of attributes that affect the relative value of a memory color are possible and conceivable.

[0019] In one embodiment, each application running on the host computing system (e.g., system 100 in Figure 1) loads a new set of tables 310 for the preencoder 305. For example, an autonomous vehicle application loads a first set of tables 310 used by the preencoder 305 when analyzing frames captured during the execution of the autonomous vehicle application. A video game application loads a second set of tables used by the preencoder 305 when analyzing frames rendered by the video game application. A video game streaming application loads a third set of tables used by the preencoder 305 when analyzing frames streamed by the video game streaming application. Other types of applications may also load specific sets of tables 310 for the preencoder 305 optimized for the type of frames being generated and / or captured. Furthermore, a single application may load different sets of tables 310 for different phases of the application as the video content changes phase by phase.

[0020] For example, in a video game application or movie, a first table is loaded for the first scene of the video game or movie. The first scene may have a particular type of contextual indicator that is considered more important than other types of contextual indicators. Then, in the second scene, a new set of contextual indicators is loaded into the second table. This reloading of the table can continue in subsequent scenes of the video game or movie. In another embodiment, an autonomous vehicle application loads different tables depending on the current situation. For example, when the vehicle is traveling on a highway at a relatively high speed, a first table of contextual indicators is loaded. Or, when it is near a residential area or a school, a second table of contextual indicators is loaded. Other tables may be loaded when the autonomous vehicle is in other situations (e.g., a parking lot, highway ramp, refueling station, charging station, toll booth, etc.). Known methods can be used to detect use case scenarios (e.g., detection of a parking lot or highway). This method can be a combination of different analyses, such as analyzing GPS data and data from video analysis.

[0021] Referring to FIG. 4, a block diagram of an embodiment of a pre-encoder 410 with a plurality of detectors 415A-415N is shown. In one embodiment, the pre-encoder 410 receives a frame 405 and performs a pre-analysis process on the frame 405. As shown in FIG. 4, the pre-encoder 410 includes a plurality of detectors 415A-415N. It should be noted that the detectors 415A-415N are the logical representations of the detectors, and the detectors 415A-415N are implemented using any suitable combination of software and / or hardware. For example, in one embodiment, each of the detectors 415A-415N is a trained neural network, and each trained neural network is designed to detect a specific type of context indicator. Also, in some embodiments, it should be understood that a single detector can perform the functions of the plurality of detectors 415A-415N. For example, in another embodiment, a single trained neural network is designed to detect a plurality of different types of context indicators.

[0022] In one embodiment, each of the detectors 415A - ··· - 415N serves to analyze a block of the frame 405 to determine whether the block contains the corresponding context indicator. For example, the first detector 415A searches for a label within a block of the frame 405, the second detector 415B searches for text within a block of the frame 405, and the third detector 415N searches for a memory color within a block of the frame 405. In other embodiments, the detectors 415A - ··· - 415N can search for other types of context indicators within a block of the frame 405. After pre-analyzing the frame 405 using the detectors 415A - ··· - 415N, the pre-encoder 410 generates a result table 420 to record which context indicators were found in which blocks of the frame 405. For example, in one embodiment, the result table 420 includes rows for each block of the frame 405, and each column of the result table 420 corresponds to a specific detector of 415A - ··· - 415N.

[0023] The result table 420 represents an example of the results of the pre-analysis phase of the frame 405. As shown in the result table 420, block 405A has "No" in columns 415A and 415N and "Yes" in column 415B. This indicates that block 405A of frame 405 contains the context indicator corresponding to detector 415B, but it was not discovered to contain the context indicators corresponding to detectors 415A and 415N. Also, the entries for blocks 405B - 405C are shown in result table 420. It should be understood that result table 420 simply shows an example of a result table. In other embodiments, result table 420 can be structured in other suitable ways. For example, in another embodiment, result table 420 can include a value of importance or a metric in each field instead of Yes or No values. Also, another table or matrix can be applied to result table 420 to convert the Yes and No values into values of importance. Next, the value of importance can be converted into a corresponding bit budget by pre-encoder 410 or an encoder (not shown). In some cases, pre-encoder 410 correlates the columns of result table 420 on a case-by-case basis to increase or decrease the value of importance when the presence of a certain context indicator is discovered in the presence of another context indicator.

[0024] Referring to FIG. 5, a block diagram of an embodiment of a pre-encoder 520 that generates an importance table 530 for blocks of a frame is shown. In one embodiment, the pre-encoder generates a result table 505 for blocks of a frame as described above in connection with FIG. 4. Next, pre-encoder 520 generates importance table 530 by combining the values retrieved from scoring table(s) 510 with the values of result table 505. For example, in one embodiment, each row of result table 505 has a plurality of fields, and each field contains "Yes" or "No" to indicate the presence or absence of the corresponding context indicator respectively. An example of this type of result table is shown as result table 420 (of FIG. 4).

[0025] In one embodiment, the scoring table(s) 510 includes a score field 545 to be applied to each "Yes" value in the entry column of a given block. The scores are then summed to generate importance values ​​for blocks 500A to 500C of the importance table 530. An example of the scoring table 510 according to one embodiment is shown in an extended form at the bottom of Figure 5. In one embodiment, there is one row for each context indicator, with the entry field 535, context indicator field 540, and score field 545 in separate columns, where the score increases if this context indicator field 550 is present, and decreases if this context indicator field 555 is present.

[0026] In one embodiment, if this context indicator is present in the presence of the context indicator specified in field 550, the score applied to the context indicator specified in field 540 should be increased. For example, if a memory color is found in a bright area of ​​the frame, the score in field 545 should be increased. The amount by which field 545 is increased may be a fixed amount (e.g., 10%), or in another embodiment, the amount by which field 545 is increased may be specified in a column of table 510. Also, if the context indicator specified in field 540 is present in the presence of the context indicator specified in field 550, the score should be decreased. For example, if a memory color is found in an area of ​​the frame that exceeds a motion threshold, the score in field 545 for the memory color should be decreased. The amount by which the score is decreased may be a fixed amount specified in table 510, or an amount specified in another way.

[0027] Blocks 500A to 500C represent blocks of the frame being analyzed by the pre-encoder 520. In one embodiment, a priority table 530 is provided to the encoder (e.g., encoder 230 in Figure 2). The encoder assigns a bit budget to each block of the frame based on the values ​​in the priority table 530 corresponding to the blocks. For example, the higher the value in the priority table 530 for a given block, the higher the bit budget assigned to that block.

[0028] Referring to Figure 6, a block diagram of one embodiment of a preencoder 620 that generates a bit budget table 630 is shown. In one embodiment, the preencoder 620 generates the bit budget table 630 based on the results table 605 and the scoring table(s) 610, rather than generating a importance table (e.g., importance table 530). The bit budget table 630 is then provided to an encoder (e.g., encoder 230 in Figure 2) that encodes blocks to satisfy the per-block bit budget when encoding the corresponding frames. The bit budget table 630 is generated in the same manner as the importance table 530 described in Figure 5. As shown, each of blocks 600A to 600C is assigned a corresponding number of bits based on its importance value. In some cases, the preencoder 620 first generates an importance table and then uses the importance table to generate the bit budget table 630. In another embodiment, the preencoder 620 provides the encoder with an importance table, and the encoder then generates the bit budget table 630 based on the importance table.

[0029] Referring to Figure 7, an example of frame 700 being analyzed by a preencoder according to one embodiment is shown. Frame 700 represents an example of a video frame being analyzed by a preencoder. For the purposes of this explanation, it is assumed that the preencoder is searching for contextual indicators such as signs and text. In one embodiment, the host computing system is running an autonomous vehicle application. In another embodiment, the host computing system is running a video game application. In yet another embodiment, other types of host applications capable of generating frames similar to frame 700 may be running.

[0030] As shown in Figure 7, frame 700 includes labels 705, 710, and 715. In one embodiment, in the first step of preencoder analysis, labels 705, 710, and 715 are identified as labels and also as having text, and are therefore given higher importance. However, rarity analysis of frame 700 would result in a decrease in the importance of labels 705, 710, and 715 due to the large number of blocks containing labels and / or text. Further analysis assigns the highest importance to label 705 based on the fact that it is in focus. If the other labels 710 and 715 are out of focus, their importance scores will decrease.

[0031] The area outside of markers 705–715 includes trees and roads. The area of ​​frame 700 containing trees is considered busy (i.e., has a relatively high spatial frequency), and therefore, conventional algorithms assign relatively high importance and a relatively large bit budget to blocks in these areas. However, in terms of where the user is likely to focus their attention in frame 700, the trees are not as important as markers 705–715. Therefore, because fewer bits are used in the bit budget to encode blocks containing trees, the detail of the trees is low, and the user is unlikely to recognize or notice them. Thus, it is a more appropriate technique to use more bits for blocks containing markers 705–715. However, since the importance of markers 705–715 is not the same, the technique described in Figures 4–6 is used to assign a higher importance to the block containing marker 705 than to the block containing markers 710–715. This higher importance translates to a larger bit budget allocation for the block containing marker 705 when these blocks are encoded by the encoder.

[0032] Referring to Figure 8, an example of frame 800 being analyzed by a preencoder according to one embodiment is shown. Frame 800 represents an example of a video frame being analyzed by a preencoder. As shown, most of frame 800 consists of trees and leaves. These features tend to be detected as "busy" because of the frequent color changes of leaves and branches. As a result, a typical encoder would allocate a large number of bits to the block containing the trees and leaves. However, in the case of frame 800, this would be a misallocation of bits based on where the user is likely to be looking. The user's attention is likely to be focused on marker 805. In one embodiment, marker 805 meets the criteria of three separate contextual indicators: the first criterion is that it is a marker, the second criterion is that it contains text, and the third criterion is that it is in focus. Therefore, when a preencoder (e.g., preencoder 220 in Figure 2) analyzes frame 800 using the techniques described herein, the block containing marker 805 is assigned a relatively high importance and receives a relatively high share of the bit budget allocated to frame 800.

[0033] Referring to Figure 9, one embodiment of Method 900 for a preencoder that generates a block-by-block bit budget is shown. For illustrative purposes, the steps of this embodiment and the steps of Figure 10 are shown in order. However, it should be noted that in various embodiments of the method described, one or more of the elements described may be performed simultaneously, in an order different from that shown, or may be omitted entirely. Other additional elements may also be performed as needed. Any of the various systems or devices described herein are configured to carry out Method 900.

[0034] The preencoder receives the frame to be encoded (block 905). The preencoder analyzes the frame in blocks to determine the context indicators present in each block (block 910). The context indicators vary depending on the embodiment. In one embodiment, the preencoder uses one or more trained neural networks to detect different context indicators within a block. The preencoder then generates a relative importance value for each block based on the presence or absence of a given context indicator and the score assigned to each context indicator (block 915). In one embodiment, the context indicators are identified in a first table specific to the particular application running on the host system. In one embodiment, the score associated with each context indicator is stored in a second table specific to the particular application running on the host system. The preencoder then adjusts the importance value based on one or more other variables (block 920). For example, the other variables may include the rarity of a particular context indicator within the entire frame, the depth of field of the block, and / or other factors.

[0035] Next, the preencoder affects the bit budget allocated to a block based on the importance value of each block (block 925). In one embodiment, the preencoder calculates the bit budget based on the importance value of each block and allocates it to the block. In another embodiment, the preencoder provides importance values ​​to the rate controller and / or encoder, and the rate controller and / or encoder calculates the bit budget based on the importance value of each block and allocates it to the block. The encoder then encodes the block to satisfy the bit budget allocated to the block (block 930). In one embodiment, the encoder adjusts the quantization parameter (QP) used to encode a given block based on the bit budget allocated to that block. In other embodiments, the encoder adjusts other parameters to encode a given block with a number of bits that matches or approximates the bit budget allocated to that particular block. After block 930, method 900 ends. It should be noted that method 900 can be used in combination with one or more other techniques for generating bit budget or importance values ​​that affect how blocks of a frame are encoded.

[0036] Referring to Figure 10, one embodiment of method 1000 for adjusting the context indicator coefficient of a block in a frame based on the rarity of context indicators across the entire frame is shown. A preencoder detects a first context indicator in a given block of the frame (block 1005). Depending on the embodiment, the first context indicator is a sign, text, memory color, or other context indicator. Next, the preencoder determines whether the first context indicator has been detected in other blocks of the frame (block 1010). Next, the preencoder generates a coefficient for the given block that is proportional or inversely proportional to the number of times the first context indicator occurs in other blocks of the frame (block 1015). In the case of inverse proportionality, the less frequently the first context indicator is detected in the frame, the higher the coefficient value generated for the given block. Next, the preencoder adds the coefficient to the total score generated for the given block (block 1020). In one embodiment, the total score is used as a value for the importance of the given block. Next, the total score is used to calculate the bit budget for the given block (block 1025). After block 1025, method 1000 is terminated. It should be noted that method 1000 can be performed not only on each block of a frame, but also on each context indicator of multiple context indicators.

[0037] In various embodiments, program instructions for a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions that can be executed by a general-purpose or dedicated processor are intended. In various embodiments, such program instructions are expressed in a high-level programming language. In other embodiments, program instructions are compiled from a high-level programming language into binary, intermediate, or other forms. Alternatively, program instructions are written to describe the operation or design of hardware. Such program instructions are expressed in a high-level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various embodiments, program instructions are stored in one of various non-temporary computer-readable storage media. The storage media are accessible by the computing system in use to provide the computing system with program instructions for program execution. Generally, such a computing system includes at least one memory and one or more processors configured to execute program instructions.

[0038] It is important to emphasize that the embodiments described above are merely non-limiting examples of embodiments. Many variations and modifications will become apparent to those skilled in the art once the above disclosure is fully understood. The following claims are intended to be construed to encompass all such variations and modifications.

Claims

1. 1. An apparatus comprising: A pre-encoder; an encoder; The pre-encoder receiving a frame to be encoded, the frame including a plurality of blocks; generating an importance value for each block based on the presence or absence of specified context indicators within each block; assigning a bit budget to each block based at least in part on the importance values ​​generated for the blocks; and The encoder comprises: configured to encode each block to meet the bit budget allocated to the block; Device.

2. The pre-encoder determining a rarity of each context indicator throughout the frame; adjusting the importance value of each block based on the rarity of each detected context indicator; configured to:

10. The apparatus of claim 1.

3. the plurality of context indicators include memory colors, text, and signs; 10. The apparatus of claim 1.

4. the pre-encoder is configured to generate an importance value for each block based on a score assigned to each context indicator detected within the block.

10. The apparatus of claim 1.

5. the pre-encoder is configured to adjust a first score assigned to a first context indicator in response to detecting a second context indicator in the block.

5. The apparatus of claim 4.

6. the pre-encoder is configured to access a table specifying a plurality of context indicators to be searched for in the plurality of blocks of the frame.

10. The apparatus of claim 1.

7. The pre-encoder receiving a first table specifying a first plurality of context indicators corresponding to a first application; In response to the first application generating a first frame to be encoded, searching the first plurality of context indicators in the first frame; receiving a second table specifying a second plurality of context indicators corresponding to a second application; In response to the second application generating a second frame to be encoded, searching for the second plurality of context indicators in the second frame; configured to:

10. The apparatus of claim 1.

8. 1. A method comprising: a pre-encoder receiving a frame to be encoded, the frame including a plurality of blocks; generating an importance value for each block based on the presence or absence of specified context indicators within each block; assigning a bit budget to each block based at least in part on the importance values ​​generated for the blocks; an encoder encoding each block to meet the bit budget allocated to the block; method.

9. determining a rarity of each context indicator throughout the frame; and adjusting the importance value of each block based on the rarity of each detected context indicator.

9. The method of claim 8.

10. the plurality of context indicators include memory colors, text, and signs; 9. The method of claim 8.

11. generating an importance value for each block based on a score assigned to each context indicator detected within the block; 9. The method of claim 8.

12. adjusting a first score assigned to the first context indicator in response to detecting a second context indicator within the block. The method of claim 11.

13. accessing a table specifying a plurality of context indicators to be searched for in the plurality of blocks of the frame; 9. The method of claim 8.

14. receiving a first table specifying a first plurality of context indicators corresponding to a first application; In response to the first application generating a first frame to be encoded, searching the first plurality of context indicators in the first frame; receiving a second table specifying a second plurality of context indicators corresponding to a second application; and in response to the second application generating a second frame to be encoded, searching for the second plurality of context indicators in the second frame.

9. The method of claim 8.

15. 1. A system comprising: Memory and a pre-encoder coupled to the memory; The pre-encoder receiving a frame to be encoded, the frame including a plurality of blocks; generating an importance value for each block based on the presence or absence of specified context indicators within each block; assigning a bit budget to each block based at least in part on the importance values ​​generated for the blocks; communicating the bit budget to an encoder to encode each block to satisfy the bit budget allocated to the block; configured to: system.

16. The pre-encoder determining a rarity of each context indicator throughout the frame; adjusting the importance value of each block based on the rarity of each detected context indicator; configured to:

16. The system of claim 15.

17. the plurality of context indicators include memory colors, text, and signs; 16. The system of claim 15.

18. the pre-encoder is configured to generate an importance value for each block based on a score assigned to each context indicator detected within the block.

16. The system of claim 15.

19. the pre-encoder is configured to access a table specifying a plurality of context indicators to be searched for in the plurality of blocks of the frame.

16. The system of claim 15.

20. The pre-encoder receiving a first table specifying a first plurality of context indicators corresponding to a first application; In response to the first application generating a first frame to be encoded, searching the first plurality of context indicators in the first frame; receiving a second table specifying a second plurality of context indicators corresponding to a second application; In response to the second application generating a second frame to be encoded, searching for the second plurality of context indicators in the second frame; configured to:

16. The system of claim 15.

Images