Information processing device, game system, and information processing method

The information processing device calculates and controls the number of method executions for CUs to achieve optimal calculation costs and quality in VVC-based game systems, addressing inefficiencies in existing technologies by normalizing and quantifying calculation costs across environments.

WO2024247722A1PCT designated stage expired Publication Date: 2024-12-05SONY GROUP CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017893
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2024-05-15
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Conventional image encoding technologies face challenges in obtaining an appropriate number of method executions to achieve optimal calculation results, leading to inefficiencies in calculation cost and quality trade-offs, particularly in real-time encoding for remote game play using VVC encoders.

Method used

An information processing device and method that calculates a normalized total calculation cost for CUs, controlling the number of method executions to meet a target cost, by measuring execution time and number of executions for each CU, and using a conversion coefficient to quantify and control calculation costs across environments.

Benefits of technology

This approach allows for flexible control of calculation costs and quality trade-offs, enabling real-time encoding with optimal efficiency and quality in VVC-based game systems, even with varying hardware performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017893_05122024_PF_FP_ABST
    Figure JP2024017893_05122024_PF_FP_ABST
Patent Text Reader

Abstract

This information processing device comprises: a computing cost calculation unit that calculates a total computing cost normalized for a plurality of CUs into which a CTU is divided, on the basis of an execution time and a computing execution count of a method measured for each of the plurality of CUs; a target computing cost calculation unit that calculates a target total computing cost of the plurality of CUs; and an execution count control unit that controls the computing execution count of the plurality of CUs so that the total computing cost calculated by the computing cost calculation unit satisfies the target total computing cost calculated by the target computing cost calculation unit.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, game system, and information processing method

[0001] The present disclosure relates to an information processing device, a game system, and an information processing method.

[0002] In an image information processing device that performs image encoding and decoding processes, calculation results are obtained by repeatedly calculating a method (function). For example, calculation processing for motion vector search is realized by repeatedly calculating a method, and the prediction accuracy of the motion vector, which is the calculation result, is improved in proportion to the number of times the method is executed according to the search range.

[0003] Japanese Patent Application Laid-Open No. 2003-125413

[0004] However, although the calculation result improves depending on the number of times a method is executed, conventional techniques have had the problem of not being able to obtain an appropriate number of times a method is executed in relation to the calculation result.

[0005] The present disclosure has been made in consideration of the above-described circumstances, and provides an information processing device, a game system, and an information processing method that are capable of obtaining an appropriate number of executions of a method in relation to a calculation result.

[0006] The information processing device of the present disclosure includes an operation cost calculation unit that calculates a normalized total operation cost for multiple CUs based on the execution time and number of operation executions of a method measured for each of the multiple CUs into which a CTU is divided, a target operation cost calculation unit that calculates a target total operation cost for the multiple CUs, and an execution count control unit that controls the number of operation executions for the multiple CUs so that the total operation cost calculated by the operation cost calculation unit satisfies the target total operation cost calculated by the target operation cost calculation unit.

[0007] The information processing method disclosed herein calculates a normalized total operation cost for multiple CUs based on the execution time and number of operation executions of a method measured for each of the multiple CUs into which a CTU is divided, calculates a target total operation cost for the multiple CUs, and controls the number of operation executions for the multiple CUs so that the total operation cost satisfies the target total operation cost.

[0008] 1 is a diagram showing an example of processing in which a plurality of types of methods are executed a plurality of times. FIG. 2 is a diagram showing an example of execution time for each type of method in processing in which a plurality of types of methods shown in FIG. 1 are executed a plurality of times. FIG. 3 is a diagram showing an example of processing in which a method is executed a number of times when the maximum quality of output can be obtained. FIG. 4 is a diagram showing an example of processing in which a method is executed a number of times when a certain degree of reduction in calculation cost and medium quality is achieved. FIG. 5 is a diagram showing an example of processing in which a method is executed a number of times when a significant reduction in calculation cost but a significantly low quality is achieved. FIG. 6 is a diagram showing a game server and a game client in a game system according to an embodiment. FIG. 7 is a diagram showing three selectable CU sizes. FIG. 8 is a diagram showing an example of a search result for an optimal CTU splitting pattern. FIG. 9 is a diagram showing the number of executions of an RD cost calculation method for each CU size. FIG. 10 is a diagram showing measured values ​​of calculation execution time for each CU size. FIG. 11 is a diagram showing CU size, method for RD cost calculation, normalized calculation execution time, number of executions, normalized calculation cost, and total value of normalized calculation cost according to an embodiment. FIG. 12 is a diagram showing an example of reduction in normalized calculation cost according to an embodiment. FIG. 13 is a block diagram showing an example of the configuration of an image encoding device which is one aspect of an image processing device according to an embodiment. FIG. 14 is a block diagram showing an example of the main configuration of a transform quantization unit according to an embodiment. 1 is a flowchart illustrating an example of the flow of encoding processing executed by an image encoding device according to an embodiment. 2 is a flowchart illustrating an example of the flow of image encoding processing according to an embodiment. 3 is a flowchart illustrating an example of the flow of image encoding processing according to an embodiment. 4 is a functional block diagram of a mode decision module in a VVC encoder according to an embodiment, which searches for an optimal division pattern of a CTU. 5 is a diagram illustrating a calculation cost management table according to an embodiment. 6 is a functional block diagram illustrating a calculation cost control module that controls the calculation cost of a divided CTU according to an embodiment. 7 is a flowchart illustrating a CTU optimal division pattern search process according to an embodiment. 8 is a flowchart illustrating a block division determination process according to an embodiment. 9 is a diagram illustrating blocks that are the targets of determination in the block division determination process according to an embodiment. 10 is a flowchart illustrating an RD cost calculation process according to an embodiment.10 is a flowchart illustrating an operation of controlling the number of calculation execution times of an optimal division pattern of a CTU according to an embodiment. FIG. 11 is a hardware configuration diagram illustrating an example of a computer that realizes an arithmetic unit of an information processing device equipped with a VVC encoder according to an embodiment.

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. The description will be given in the following order.

[0010] 1. Background of the Problems Relating to the Embodiments 2. Details of the Background of the Problems Relating to the Embodiments 3. Solution Means of the Embodiments When the Preconditions are Satisfied 4. Embodiments 4.1. Overview of the Embodiments 4.2. Assumed Use Cases of the Embodiments 4.3. Configuration of the Embodiments 4.4. Determination of the Optimal CTU Partition Pattern 4.5. Normalization of Operation Cost 4.6. Quantitative Control of Operation Cost 4.7. Block Diagram of an Image Processing Device 4.8. Block Diagram for Searching for the Optimal CTU Partition Pattern 4.9. Functional Block Diagram for Controlling the Number of Executions of the Optimal CTU Partition Pattern 5. Explanation of Operation 5.1. Operation for Searching for the Optimal CTU Partition Pattern 5.2. Operation for Controlling the Number of Executions of the Optimal CTU Partition Pattern 6. Hardware Configuration

[0011] 1. Background of the Problems of the Embodiments The embodiments provide a method for quantitatively controlling computation costs in a computer program that performs processing by executing a method (function) multiple times, with a method that ensures reproducibility of execution results (computation results). "Quantitativeness" and "reproducibility" will be explained below.

[0012] Quantitativeness: Generally, it is possible to measure the number of CPU clocks required to execute a certain method by using a tool called a profiler. However, no technology has been disclosed that quantitatively estimates and controls the computation cost (specifically, the time required to encode an image frame) based on the profiling results.

[0013] For example, in modern operating systems, programs run in parallel using a multi-threading mechanism, so simply adding up the CPU clock speeds of the methods being executed does not allow for an accurate estimation of the overall processing time.

[0014] Quantitative control of computational costs is a function that allows users to proactively select the trade-off between the quality of program execution results and computational costs (execution time).

[0015] - Regarding reproducibility, an easily conceivable method is to quantify the computational cost by measuring the execution time of the controlled section, but this has problems.

[0016] For example, it is conceivable to measure the execution time of the part to be controlled by using a hardware timer function provided by the operating system on which the program runs, but this has the following problems.

[0017] - Reproducibility of execution results (calculation results) When execution time is directly measured and controlled, the measured value changes each time, resulting in no reproducibility. This method is not desirable because maintaining reproducibility of execution results is important for purposes such as program performance evaluation and debugging.

[0018] <2. Detailed Background of the Problems Relating to the Embodiments> Next, detailed background of the problems relating to the embodiments will be described. The embodiments are based on the premise that the following conditions (1) to (3) are satisfied: (1) Processing of a program to be controlled is realized by multiple calls of one or more types of methods; (2) The time required to execute an operation differs depending on the type of method; and (3) There is a trade-off between the number of times a method is executed and the quality of the result achieved by the execution result of the program.

[0019] Explanation of Prerequisite (1) Figure 1 shows an example of a process in which multiple types of methods are executed multiple times. In Figure 1, the process of the program to be controlled is realized by calling method A three times, method B three times, and method C twice.

[0020] Furthermore, it is assumed that the processing results in the output of an image of a cat as shown in Fig. 1. The output does not need to be limited to an image, and other output formats may also be used.

[0021] Explanation of Precondition (2) Figure 2 is a diagram showing an example of the execution time for each type of method in a process in which multiple types of methods shown in Figure 1 are executed multiple times. Figure 2 shows a case in which the execution time of method A is 10, the execution time of method B is 5, and the execution time of method C is 2. As shown in Figure 2, the execution times of methods A to C are different from each other.

[0022] Explanation of Precondition (3) Figure 3 is a diagram showing an example of processing in which methods are executed when the highest possible quality of output can be obtained. As shown in Figure 3, when the highest possible quality of output can be obtained, method A is called three times, method B is called three times, and method C is called twice.

[0023] 4 is a diagram showing an example of processing in which methods are executed with a certain degree of reduction in computational cost and medium quality. As shown in FIG. 4, when a certain degree of reduction in computational cost and medium quality is achieved, method A is called twice, method B is called twice, and method C is called once. In other words, one call to method A, one call to method B, and one call to method C are skipped. In the processing shown in FIG. 4, computational cost is reduced, but the quality of the obtained results is reduced.

[0024] 5 is a diagram showing an example of a process in which a method is executed when the computational cost is significantly reduced but the quality is significantly reduced. As shown in FIG. 5, when the computational cost is significantly reduced but the quality is significantly reduced, method A is called once, method B is called once, and method C is called once. In other words, the calls to method A and method B are skipped twice, and the call to method C is skipped once. In the process shown in FIG. 5, the amount of reduction in computational cost is large, but the quality of the obtained results is significantly reduced.

[0025] As shown in FIGS. 3 to 5, the number of times a method is executed is in a trade-off relationship with the quality of the output achieved as a result of the execution of the program.

[0026] 3. Solutions of the embodiment when the preconditions are met The following solutions are adopted in the embodiment. The number of executions of the method that executes the calculation is the object of control, rather than the time required for calculation execution. The average execution time of each method type is measured, and the total execution time is expressed as a ratio of these values ​​(called the normalized calculation cost). The conversion coefficient that converts the normalized calculation cost into real time is measured for each environment in which encoding is performed, thereby quantitatively estimating and controlling the calculation cost in any execution environment.

[0027] 4. Embodiments 4.1. Overview of the embodiments Next, an overview of the embodiments will be described. In the embodiments, a search process (mode decision) for an optimal partitioning pattern for the block partitioning structure of a VVC (Versatile Video Coding) encoder will be described using a partially simplified version.

[0028] The method to be measured and controlled is called mode decision, which is a method executed to calculate the RD cost for a CU of a certain size in order to search for the optimal encoding mode.

[0029] Here, CU is a coding unit. A fixed-size block formed by combining several CUs is called a CTU, and a set of one or more CTUs is called a slice. The RD cost is a parameter calculated after encoding and indicates the degree of encoding. The RD cost is calculated, for example, from the encoding distortion calculated from the squared error between an actually observed image and a predicted image and the encoding cost. The lower the RD cost, the smaller the difference between the actually observed image and the predicted image, i.e., indicates that encoding is performed efficiently.

[0030] This mode decision process accounts for 90% or more of the computational cost of the encoder. In this embodiment, the computational cost can be quantitatively controlled by omitting the execution of methods through a trade-off between coding efficiency and encoding speed.

[0031] 4.2. Use Cases Assumed by the Embodiment The search for the optimal CTU division pattern according to the embodiment is used in real-time encoding for remote play of games.

[0032] With VVC, the latest generation of codec, it is not possible to encode in real time while maintaining the highest level of coding efficiency (image quality), even on server hardware used for remote game play.

[0033] Therefore, a trade-off between encoding efficiency (image quality) and encoding speed is necessary. It is also necessary to be able to flexibly select the optimal computational cost according to the hardware performance of the server that executes the encoding process.

[0034] The search for the optimal CTU division pattern according to the embodiment satisfies the above two requirements: the user can proactively select the trade-off between coding efficiency (image quality) and encoding speed in accordance with the hardware performance, and flexibly control the computational cost.

[0035] Fig. 6 is a diagram showing a game server 11 and a game client 12 in a game system 10 according to an embodiment. Fig. 6 shows a block diagram focusing on a video encoder and decoder. As shown in Fig. 6, the game server 11 is connected to the game client 12 via a network. The game server 11 has a real-time video encoder 13 according to an embodiment, and the game client 12 has a real-time video decoder 14 that performs decoding corresponding to the real-time video encoder 13.

[0036] The game server 11 encodes game images in real time, and transmits the encoded images and audio to the game client 12 via the network. The real-time video decoder 14 of the game client 12 decodes the images transmitted from the real-time video encoder 13 and presents the decoded images together with audio to the game player. The game client 12 also transmits player operation information to the game server 11.

[0037] The game server 11 progresses the game based on the player's operation information sent from the game client 12, and a real-time video encoder 13 encodes the game video according to the progress, and transmits the encoded video and audio to the game client 12.

[0038] In a game system 10 using VVC, real-time operation is required for the real-time video encoder 13 of the game server 11 and the real-time video decoder 14 of the game client 12. The real-time video encoder 13 and real-time video decoder 14 using the technology of the embodiment can meet this requirement.

[0039] 4.3. Configuration of the embodiment The embodiment assumes a software encoder that runs on an arbitrary operating system or a hardware encoder with similar functionality. The software encoder or the hardware encoder with similar functionality measures the execution time and number of methods on a measurement computer.

[0040] Furthermore, the relationship between the normalized computation cost and the actual encoding time is measured on an arbitrary computer to be controlled, which may be the same as the measurement computer.

[0041] <4.4. Determining Optimal CTU Partition Pattern> In the embodiment, the control target is not the time required for execution of an operation, but the number of executions of the method that has been executed.

[0042] To achieve optimal coding efficiency, it is necessary to calculate and compare the RD costs for all CU sizes. Fig. 7 is a diagram showing three selectable CU sizes. When the RD cost calculation methods required to determine the optimal split pattern are RDC64x64(), RDC32x32(), and RDC16x16(), if the cost of RDC64x64() is greater than the total RD costs of four RDC32x32(), selecting splitting will result in better coding efficiency.

[0043] Similarly, the RD costs of four RDCs 32x32() and four RDCs 16x16() obtained by dividing the CU are compared to determine the division pattern.

[0044] The number of times the method is executed, CCostNxN, is CCost64x64=1, CCost32x32=4, and CCost16x16=16, respectively.

[0045] Fig. 8 is a diagram showing an example of the search results for the optimal CTU division pattern. The optimal CTU division pattern is determined by comparing the RD costs for each CU size to find a division pattern that maximizes coding efficiency. Fig. 9 is a diagram showing the number of executions of the RD cost calculation method for each CU size. As shown in Fig. 9, when the CU size is 64x64, the number of executions is one, when the CU size is 32x32, the number of executions is four, and when the CU size is 16x16, the number of executions is 16.

[0046] <4.5. Normalization of Computational Cost> In the embodiment, taking advantage of the fact that the time required for computation execution mainly depends on the CU size, the average execution time of the method for each CU size is measured, and the total execution time is expressed as a ratio (called the normalized computational cost).

[0047] In Figure 7, the execution time of RDC64x64(), RDC32x32(), and RDC16x16() differs depending on the CU size. Even for the same CU size, there is some variation in execution time. Therefore, the calculation execution time for each CU size is measured, the average value is calculated, and then the average execution time is divided by the smallest value among the average execution times for each CU size for normalization, resulting in the TCost. 64x64 , TCost 32x32 , TCost 16x16 get.

[0048] For example, when the measured values ​​of the calculation execution time are the values ​​shown in FIG. 10, TCost NxN is the average normalized value TCost 64x64 = 200 = 20TCost 32x32 = 50 = 5 TCost 16x16 = 10 = 1.

[0049] By multiplying the normalized average execution time TCostNxN of the method RDCNxN() by the number of executions CCostNxN and summing them, the calculation cost required to search for the optimal partitioning pattern of a certain CTU can be obtained in normalized form.

[0050] 11 is a diagram showing the CU size, the method for calculating the RD cost, the normalized operation execution time (TCost), the number of executions (CCcost), the normalized operation cost (TCost×Ccost), and the total value of the normalized operation cost according to the embodiment. 64x64 , TCost 32x32 , TCost 16x16 Using the values ​​of (a) and (b), the total normalized operation cost can be calculated as shown in Fig. 11. That is, the total value of TCost x CCost = 56 is the total normalized operation cost.

[0051] <4.6. Quantitative Control of Computational Cost> In the embodiment, the conversion coefficient for converting the normalized computational cost into real-time data is measured for each environment in which encoding is performed, thereby enabling quantitative control of the computational cost in any execution environment.

[0052] Determining FCost The sum of TCost x CCost (56) calculated in Figure 11 corresponds to the encoding time for one CTU. The execution time of the method corresponding to TCost x CCost is defined as FCost. The ratio of the normalized calculation cost TCost x CCost to FCost is defined as RCost.

[0053] Measure FCost on any computer to be controlled. The computer to be controlled and the computer used for measurement can be the same. For example, assume that FCost on the computer to be controlled is 84 μsec. In this case, RCost = FCost / (TCost × CCost) = 84 / 56 = 1.5.

[0054] Determining the Control Target: For example, suppose that the number of CTUs in one frame is 1000 and the encoding time for one frame is to be controlled to 60 msec or less. In the above case, it is necessary to execute the calculation process for an average of 60 μsec per CTU.

[0055] Assuming that the mode decision process takes up more than 90% of the encoding time for one frame, the control target is set to FCost = 54 μsec, with some leeway. From (TCost × CCost) = FCost / RCost, it can be seen that if the normalized computation cost can be controlled (computation cost reduced) so that 54 / 1.5 = 36, encoding one frame can be completed within the target computation time.

[0056] Since the control target value is FCost=54 μsec, FCost / RCost=TCost×CCost=54 / 1.5=36. It is desired to control (reduce) the calculation cost so that the total value of the normalized calculation cost becomes 36 or less.

[0057] To reduce computational costs, it is possible to omit the execution of a method. For example, it is possible to use any value for the RD cost of a certain CU without calculating it. For example, if the maximum RD cost is used, the cost of the entire division pattern including that CU increases in the RD cost comparison, making it less likely to be selected.

[0058] It can be seen from FIG. 11 that TCostxCCost can be adjusted by combining the method and the number of times to omit execution.

[0059] 12 is a diagram showing an example of reduction in normalized operation cost according to an embodiment. As shown in FIG. 12, operation cost reduction example (1) shows an example in which the number of executions for a 64×64 CU size is set to 0, thereby reducing the total normalized operation cost to 36. Operation cost reduction example (2) shows an example in which the number of executions for a 32×32 CU size is set to 3, and the number of executions for a 16×16 CU size is set to 1, thereby reducing the total normalized operation cost to 36.

[0060] Therefore, according to the embodiment, any method can be selected as the method for which the computational cost is to be reduced. Furthermore, if the execution of a method is omitted, any value can be used as the RD cost, which is the return value of the method.

[0061] In this case, the value of the RD cost is the maximum or minimum value that the variable of the programming language that holds the RD cost can take, or a value that is simply estimated (with very low computational cost) from image features held by the encoder.

[0062] 13 is a block diagram showing an example of the configuration of an image coding device, which is one aspect of an image processing device according to an embodiment. The image coding device 200 shown in FIG. 13 is a device that codes image data of a moving image. For example, the image coding device 200 codes the image data of a moving image using a coding method such as VVC (Versatile Video Coding), AVC (Advanced Video Coding), or HEVC (High Efficiency Video Coding).

[0063] Note that Fig. 13 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 13. In other words, in the image coding device 200, there may be processing units that are not shown as blocks in Fig. 13, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 13. This also applies to other figures that explain processing units, etc. within the image coding device 200.

[0064] 13 , the image coding device 200 includes a control unit 201, a rearrangement buffer 211, a first operation unit 212, a transform / quantization unit 213, an encoding unit 214, and an accumulation buffer 215. The image coding device 200 also includes a dequantization / inverse transform unit 216, a second operation unit 217, an in-loop filter unit 218, a frame memory 219, and a prediction unit 220.

[0065] <Control Unit> The control unit 201 divides the video data held in the rearrangement buffer 211 into blocks (CU, PU, ​​TU, etc.) based on an externally or pre-specified block size of the processing unit. The control unit 201 also determines the coding parameters (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) to be supplied to each block based on, for example, RDO (Rate-Distortion Optimization).

[0066] For example, the control unit 201 sets the quantization value of a frame, sets the quantization value of a CTU (CTU baseQp), sets the coding mode (e.g., mts_idx, lfnst_idx, etc.), and corrects the quantization value for each CU based on the subjective image quality (e.g., sets CU Qp).

[0067] Among the control units that determine the coding parameters (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) to be supplied to each block, the mode decision module searches for the optimal mode, mainly from the perspective of maximizing coding efficiency. Specifically, it searches for a CTU division pattern and an optimal coding mode for the divided CTUs.

[0068] After determining the above-described coding parameters, the control unit 201 supplies them to each block. For example, the header information Hinfo is supplied to each block. The prediction mode information Pinfo is supplied to the coding unit 214 and the prediction unit 220. The transformation information Tinfo is supplied to the coding unit 214, the transform / quantization unit 213, and the inverse quantization / inverse transformation unit 216. The filter information Finfo is supplied to the coding unit 214 and the in-loop filter unit 218. Of course, the destination to which each coding parameter is supplied is arbitrary and is not limited to this example.

[0069] For example, the control unit 201 supplies the set quantization values ​​(for example, CTU baseQp and CU Qp) and control parameters for coefficient transformation (for example, mts_idx and lfnst_idx) to the transform / quantization unit 213, the encoding unit 214, and the like.

[0070] <Sorting Buffer> Each field (input image) of video data is input to the image coding device 200 in its playback order (display order). The sorting buffer 211 acquires and holds (stores) each input image in its playback order (display order). Under the control of the control unit 201, the sorting buffer 211 sorts the input images in coding order (decoding order) and divides them into blocks, which are processing units. The sorting buffer 211 supplies each processed input image to the first calculation unit 212.

[0071] <First Calculation Unit> The first calculation unit 212 subtracts the predicted image P supplied from the prediction unit 220 from the image corresponding to the block of processing units supplied from the sorting buffer 211 to derive residual data D, and supplies the residual data D to the transform and quantization unit 213.

[0072] <Transform / Quantization Unit> The transform / quantization unit 213 acquires residual data D supplied from the first calculation unit 212. The transform / quantization unit 213 also acquires prediction mode information Pinfo and transformation information Tinfo supplied from the control unit 201. The transform / quantization unit 213 performs transform / quantization processing on the residual data D based on the prediction mode information Pinfo and the transformation information Tinfo, and derives a quantization coefficient level. The transform / quantization processing includes, for example, coefficient transformation (primary transformation or secondary transformation) and quantization. Of course, any processing may be included in the transform / quantization processing, and some of the above-described processing may be omitted, or processing other than the above-described processing may be included. The transform / quantization unit 213 supplies the derived quantization coefficient level to the encoding unit 214 and the inverse quantization / inverse transform unit 216.

[0073] <Encoding Unit> The encoding unit 214 acquires the quantization coefficient level supplied from the transform / quantization unit 213. The encoding unit 214 also acquires various encoding parameters (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) supplied from the control unit 201. The encoding unit 214 also acquires information related to filters, such as filter coefficients, supplied from the in-loop filter unit 218. The encoding unit 214 also acquires information related to the optimal prediction mode supplied from the prediction unit 220.

[0074] The encoding unit 214 performs entropy encoding (lossless encoding) on ​​the quantization coefficient level to generate a bit string (encoded data). The encoding unit 214 may apply, for example, CABAC (Context-based Adaptive Binary Arithmetic Code) as this entropy encoding. The encoding unit 214 may apply, for example, CAVLC (Context-based Adaptive Variable Length Code) as this entropy encoding. Of course, the content of this entropy encoding is arbitrary and is not limited to these examples.

[0075] Furthermore, the encoding unit 214 derives residual information Rinfo from the quantized coefficient levels, encodes the residual information Rinfo, and generates a bit string.

[0076] Furthermore, the encoding unit 214 includes information about the filter supplied from the in-loop filter unit 218 in filter information Finfo, and includes information about the optimal prediction mode supplied from the prediction unit 220 in prediction mode information Pinfo. Then, the encoding unit 214 encodes the various encoding parameters described above (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) to generate a bit string.

[0077] The encoding unit 214 multiplexes the bit strings of the various types of information generated as described above to generate encoded data, and supplies the encoded data to the accumulation buffer 215.

[0078] <Storage Buffer> The storage buffer 215 temporarily stores the encoded data obtained by the encoding unit 214. The storage buffer 215 outputs the stored encoded data, for example, as a bit stream, to the outside of the image encoding device 200 at a predetermined timing. For example, this encoded data is transmitted to the decoding side via any recording medium, any transmission medium, any information processing device, etc. In other words, the storage buffer 215 also functions as a transmission unit that transmits the encoded data (bit stream).

[0079] Information on the code amount of the encoded data stored in the storage buffer 215 is supplied to the control unit 201. Based on this information, the control unit 201 controls the rate of the quantization operation of the transform and quantization unit 213 so as to prevent overflow or underflow.

[0080] <Inverse Quantization and Inverse Transformation Unit> The inverse quantization and inverse transformation unit 216 acquires the quantization coefficient level supplied from the transform and quantization unit 213. The inverse quantization and inverse transformation unit 216 also acquires the transformation information Tinfo supplied from the control unit 201.

[0081] The inverse quantization and inverse transform unit 216 performs inverse quantization and inverse transform processing on the quantized coefficient level based on the transform information Tinfo, and derives residual data D'. This inverse quantization and inverse transform processing is the inverse processing of the transform and quantization processing performed in the transform and quantization unit 213. That is, in the inverse quantization and inverse transform processing, for example, processing such as inverse quantization and inverse coefficient transform is performed. This inverse quantization is the inverse processing of the quantization performed in the transform and quantization unit 213. Furthermore, inverse coefficient transform is the inverse processing of the coefficient transform performed in the transform and quantization unit 213.

[0082] Of course, any processing may be included in this inverse quantization and inverse transform processing, and some of the processing described above may be omitted, or processing other than that described above may be included. The inverse quantization and inverse transform unit 216 supplies the derived residual data D' to the second calculation unit 217.

[0083] <Second Calculation Unit> The second calculation unit 217 obtains the residual data D' supplied from the inverse quantization and inverse transform unit 216 and the predicted image P supplied from the prediction unit 220. The second calculation unit 217 adds the residual data D' to the predicted image P corresponding to the residual data D' to derive a locally decoded image. The second calculation unit 217 supplies the derived locally decoded image to the in-loop filter unit 218 and the frame memory 219.

[0084] <In-loop filter unit> The in-loop filter unit 218 acquires a locally decoded image supplied from the second calculation unit 217. The in-loop filter unit 218 also acquires filter information Finfo supplied from the control unit 201. The in-loop filter unit 218 also acquires an input image (original image) supplied from the rearrangement buffer 211. Note that any information may be input to the in-loop filter unit 218, and information other than the above information may also be input. For example, as necessary, information such as a prediction mode, motion information, a code amount target value, a quantization parameter Qp, a picture type, and a block (CU, CTU, etc.) may be input to the in-loop filter unit 218.

[0085] The in-loop filter unit 218 performs appropriate filtering on the locally decoded image based on the filter information Finfo. The in-loop filter unit 218 also uses the input image (original image) and other input information for the filtering, as necessary.

[0086] For example, the in-loop filter unit 218 may apply a bilateral filter as its filtering process. For example, the in-loop filter unit 218 may apply a deblocking filter (DBF (DeBlocking Filter)) as its filtering process. For example, the in-loop filter unit 218 may apply an adaptive offset filter (SAO (Sample Adaptive Offset)) as its filtering process.

[0087] For example, the in-loop filter unit 218 may apply an adaptive loop filter (ALF (Adaptive Loop Filter)) as its filtering process. Furthermore, the in-loop filter unit 218 may apply a combination of multiple of these filters as its filtering process. Note that which filters to apply and in what order to apply them are arbitrary and can be selected as appropriate. For example, the in-loop filter unit 218 applies four in-loop filters, namely, a bilateral filter, a deblocking filter, an adaptive offset filter, and an adaptive loop filter, in this order as its filtering process.

[0088] Of course, the filtering process performed by the in-loop filter unit 218 is arbitrary and is not limited to the above example. For example, the in-loop filter unit 218 may apply a Wiener filter or the like.

[0089] The in-loop filter unit 218 supplies the filtered locally decoded image to the frame memory 219. When transmitting information about the filter, such as a filter coefficient, to the decoding side, the in-loop filter unit 218 supplies the information about the filter to the encoding unit 214.

[0090] <Frame Memory> The frame memory 219 performs processing related to the storage of image data. For example, the frame memory 219 acquires and holds (stores) locally decoded images supplied from the second calculation unit 217 and locally decoded images that have been filtered and supplied from the in-loop filter unit 218. The frame memory 219 also uses the locally decoded images to reconstruct and hold decoded images for each picture (storing them in a buffer within the frame memory 219). The frame memory 219 supplies the decoded images (or portions thereof) to the prediction unit 220 in response to a request from the prediction unit 220.

[0091] <Prediction Unit> The prediction unit 220 performs processing related to generation of a predicted image. For example, the prediction unit 220 obtains prediction mode information Pinfo supplied from the control unit 201. For example, the prediction unit 220 obtains an input image (original image) supplied from the reordering buffer 211. For example, the prediction unit 220 obtains a decoded image (or a part thereof) read from the frame memory 219.

[0092] The prediction unit 220 performs prediction processing such as inter prediction or intra prediction using the prediction mode information Pinfo and the input image (original image). That is, the prediction unit 220 performs prediction and motion compensation by referring to the decoded image as a reference image, and generates a predicted image P. The prediction unit 220 supplies the generated predicted image P to the first calculation unit 212 and the second calculation unit 217. Furthermore, the prediction unit 220 supplies information on the prediction mode selected by the above processing, i.e., the optimal prediction mode, to the encoding unit 214 as necessary.

[0093] <Transformation and quantization unit> Fig. 14 is a block diagram showing an example of the main configuration of the transformation and quantization unit 213 in Fig. 13. As shown in Fig. 14, the transformation and quantization unit 213 has a coefficient conversion unit 231, a quantization unit 232, and a violation avoidance processing unit 233.

[0094] The coefficient conversion unit 231 acquires the prediction residual (residual data D) supplied from the first calculation unit 212, and performs coefficient conversion on the prediction residual to generate conversion coefficients. The coefficient conversion unit 231 has a primary conversion unit 241 and a secondary conversion unit 242.

[0095] The primary transform unit 241 acquires the prediction residual (residual data D) supplied from the first calculation unit 212 and performs primary transform on the prediction residual to generate primary transform coefficients. The primary transform unit 241 performs primary transform in a mode according to the set mts_idx. This mts_idx is set by the control unit 201 or a mode setting unit 252 (described later). For example, if mts_idx > 1, the primary transform unit 241 performs primary transform by applying the MTS mode. On the other hand, if mts_idx = 0, the primary transform unit 241 performs primary transform without applying the MTS mode. The primary transform unit 241 supplies the generated primary transform coefficients to the secondary transform unit 242.

[0096] The secondary transform unit 242 acquires the primary transform coefficients supplied from the primary transform unit 241 and performs secondary transform on the primary transform coefficients to generate transform coefficients. The secondary transform unit 242 performs the secondary transform in a mode according to the set lfnst_idx. This lfnst_idx is set by the control unit 201 or a mode setting unit 252 (described later). For example, if lfnst_idx > 0, the secondary transform unit 242 performs the secondary transform by applying the LFNST mode. On the other hand, if lfnst_idx = 0, the secondary transform unit 242 performs the secondary transform without applying the LFNST mode. The secondary transform unit 242 supplies the generated transform coefficients to the quantization unit 232.

[0097] The quantization unit 232 acquires transform coefficients supplied from the coefficient conversion unit 231 (secondary conversion unit 242) and performs quantization on the transform coefficients to generate quantized coefficients. The quantization unit 232 performs quantization using a quantization value (CTU baseQp, CU Qp, etc.) set by the control unit 201. For example, the quantization unit 232 performs quantization using CTU baseQp when the control unit 201 sets the coding mode. Furthermore, when the quantization value is corrected at the CU level, the quantization unit 232 performs quantization using CU Qp.

[0098] The quantization unit 232 supplies the generated quantization coefficients to the violation avoidance processing unit 233. The violation avoidance processing unit 233 acquires the quantization coefficients supplied from the quantization unit 232, and if the quantization coefficients violate a constraint related to the coding mode, executes processing to avoid the violation of the constraint. The violation avoidance processing unit 233 has a coding mode violation confirmation unit 251 and a mode setting unit 252.

[0099] The coding mode violation check unit 251 obtains quantized coefficients supplied from the quantization unit 232 and determines whether the quantized coefficients violate constraints related to the coding mode. For example, the coding mode violation check unit 251 determines whether the MTS mode is applied in the primary transform or the LFNST mode is applied in the secondary transform, and the only non-zero quantized coefficients are DC coefficients. For example, if it is determined that the LFNST mode is applied in the secondary transform and the only non-zero quantized coefficients are DC coefficients, the coding mode violation check unit 251 determines that a coding mode violation has occurred. Furthermore, if it is determined that the MTS mode is applied in the primary transform and the only non-zero quantized coefficients are DC coefficients, the coding mode violation check unit 251 determines that a coding mode violation has occurred.

[0100] If it is determined that the constraint is violated, the coding mode violation checking unit 251 supplies information notifying that fact to the mode setting unit 252. On the other hand, if it is determined that the constraint is not violated, the coding mode violation checking unit 251 supplies the quantized coefficients to the coding unit 214 and the inverse quantization and inverse transform unit 216.

[0101] The mode setting unit 252 acquires information indicating the result of the coding mode violation confirmation by the coding mode violation confirmation unit 251. The mode setting unit 252 sets (updates) the coding mode based on the information (i.e., based on the result of the coding mode violation confirmation by the coding mode violation confirmation unit 251). For example, the mode setting unit 252 controls whether to apply the MTS mode or the LFNST mode. That is, the mode setting unit 252 sets (updates) mts_idx and lfnst_idx. The mode setting unit 252 supplies the set (updated) control parameters (mts_idx and lfnst_idx) to the coefficient conversion unit 231 to perform coefficient conversion again. The coefficient conversion unit 231 performs coefficient conversion in accordance with the set (updated) control parameters. That is, coefficient conversion and quantization are performed again (these processes are redone).

[0102] For example, if the coding mode violation checking unit 251 determines that the LFNST mode is applied in the secondary transform and that the only non-zero quantized coefficients are DC coefficients, the mode setting unit 252 sets the control parameter (lfnst_idx) that specifies the mode of the secondary transform to a value (0) that indicates that the LFNST mode is not applied in the secondary transform. Then, the mode setting unit 252 supplies the control parameter (lfnst_idx = 0) to the secondary transform unit 242. The secondary transform unit 242 performs the secondary transform again in accordance with the control parameter (lfnst_idx = 0). That is, the secondary transform and quantization are redone.

[0103] For example, if the coding mode violation checking unit 251 determines that MTS mode is applied in the primary transform and that only DC coefficients are non-zero quantized coefficients, the mode setting unit 252 sets the control parameter (mts_idx) that specifies the mode of the primary transform to a value (0) that indicates that MTS mode is not applied in the primary transform. Then, the mode setting unit 252 supplies the control parameter (mts_idx = 0) to the primary transform unit 241. The primary transform unit 241 performs primary transform again in accordance with the control parameter (mts_idx = 0). That is, the primary transform, secondary transform, and quantization are redone.

[0104] With this configuration, the image coding device 200 can suppress violations of restrictions on coding modes.

[0105] <Flow of Encoding Process> Next, an example of the flow of encoding process executed by the image encoding device 200 described above will be described with reference to the flowchart of FIG.

[0106] When the encoding process starts, the control unit 201 of the image encoding device 200 sets a quantization value for a frame in step S201. The control unit 201 allocates a coding amount for each frame (picture) based on coding amount prediction and feedback control so as to satisfy the constraints of the target bit rate, and determines a quantization value at the frame (picture) level.

[0107] In step S202, the control unit 201 sets a quantization value for a CTU. The control unit 201 divides a frame into CTUs and determines a quantization value CTU BaseQp for each CTU. The control unit 201 predicts the amount of code generated for the entire frame from the amount of code generated for each CTU, and sets CTU BaseQp so as to satisfy the constraints of the target bit rate.

[0108] In step S203, the control unit 201 sets a coding mode and saves the processing result. The control unit 201 determines the CU division pattern (QT / BT / TT), the prediction mode (intra or inter), the coefficient transformation mode of the prediction residual (MTS, LFNST, etc.), etc., so as to obtain optimal coding efficiency under the constraint of the given quantization value (CTU BaseQp). The processing on a CTU-by-CTU basis can be performed sequentially or in parallel.

[0109] In step S204, the control unit 201 corrects the quantization value based on the subjective image quality. To improve the subjective image quality, the control unit 201 corrects the quantization value based on a feature amount of the original image, such as activity, for each CU block divided at the CTU level or below. That is, the control unit 201 derives a quantization value CU Qp for each CU.

[0110] In step S205, each processing unit of the image coding device 200 executes an image coding process, codes the image input to the image coding device 200, and generates a bitstream.

[0111] In step S206, the control unit 201 determines whether the bitstream obtained in step S205 satisfies the rate constraint. If it is determined that the rate constraint is not satisfied, the process returns to step S202, and the subsequent processes are repeated. If it is determined in step S206 that the bitstream satisfies the rate constraint, the process proceeds to step S207.

[0112] In step S207, the accumulation buffer 215 outputs the bitstream it holds to the outside of the image encoding device 200. The output bitstream may be transmitted to a decoding device via an arbitrary transmission path or another device, for example. The output bitstream may also be recorded on an arbitrary recording medium. When the processing of step S207 ends, the encoding process ends.

[0113] <Flow of Image Encoding Process> Next, an example of the flow of the image encoding process executed in step S205 of FIG. 15 will be described with reference to the flowcharts of FIGS. 16 and 17.

[0114] When the image encoding process starts, in step S231 of FIG. 16, the prediction unit 220 generates a predicted image in the optimal prediction mode (intra prediction or inter prediction) set in step S203.

[0115] In step S232, the first calculation unit 212 derives the difference (prediction residual) between the input image and the predicted image generated in step S231. That is, the first calculation unit 212 derives residual data D between the input image and the predicted image. The residual data D derived in this manner has a reduced data volume compared to the original image data. Therefore, the data volume can be compressed compared to when the image is encoded as is.

[0116] In step S233, the primary transform unit 241 performs primary transform on the prediction residuals in the coefficient transform mode (mode indicated by mts_idx) set in step S203, and generates primary transform coefficients.

[0117] In step S234, the secondary transform unit 242 performs secondary transform on the primary transform coefficients in the coefficient transform mode set in step S203 (the mode indicated by lfnst_idx) to generate transform coefficients.

[0118] In step S235, the quantization unit 232 performs quantization on the transform coefficients generated in step S234 using the quantization values ​​(CU Qp) corrected in step S204, to generate quantized coefficients.

[0119] In step S236, the coding mode violation confirmation unit 251 determines whether the quantization coefficients generated in step S235 violate the constraints on the coding mode. If it is determined that the quantization coefficients violate the coding mode constraints, the process proceeds to step S237.

[0120] In step S237, the coding mode violation checking unit 251 determines whether the LFNST mode is applied in the secondary conversion (lfnstIdx>0). If it is determined that the LFNST mode is applied (lfnstIdx>0), the processing proceeds to step S238.

[0121] In step S238, the mode setting unit 252 sets lfnstIdx to 0 (lfnstIdx = 0). When the processing of step S238 ends, the processing proceeds to step S239. Also, if it is determined in step S237 that the LFNST mode was not applied in the secondary conversion (lfnstIdx = 0), the processing proceeds to step S239.

[0122] In step S239, the coding mode violation check unit 251 determines whether the MTS mode was applied in the primary transform (mtsIdx>1). If it is determined that the MTS mode was not applied, the process returns to step S234, and the subsequent processes are executed. That is, in this case, the secondary transform and quantization are redone.

[0123] Also, if it is determined in step S239 that the MTS mode is applied (mtsIdx>1), the process proceeds to step S240.

[0124] In step S240, the mode setting unit 252 sets mtsIdx to 0 (mtsIdx = 0). When the processing of step S240 ends, the processing returns to step S233, and the subsequent processing is executed. That is, in this case, the primary transform, secondary transform, and quantization are redone.

[0125] Also, if it is determined in step S236 that the quantized coefficients do not violate the constraints on the coding mode, the process proceeds to FIG.

[0126] In step S251 of FIG. 17, the inverse quantization and inverse transform unit 216 performs inverse quantization and inverse transform on the quantized coefficients generated in step S235, and derives (restores) prediction residuals (residual data D').

[0127] In step S252, the second calculation unit 217 generates a locally decoded decoded image by adding the predicted image generated in step S231 to the residual data D' derived by the inverse quantization and inverse transform processing in step S251.

[0128] In step S253, the in-loop filter unit 218 performs in-loop filtering on the locally decoded image generated in step S252.

[0129] In step S254, the frame memory 219 stores the locally decoded image generated in step S252 and the locally decoded image filtered in step S253.

[0130] In step S255, the encoding unit 214 encodes the quantization coefficient LEVEL generated in step S235 to generate a bitstream. At this time, the encoding unit 214 may encode any information other than the quantization coefficient and store it in the bitstream. For example, the encoding unit 214 may encode various encoding parameters (header information HInfo, prediction mode information PInfo, transformation information TInfo, etc.) and store the encoded data in the bitstream. The encoding unit 214 may also derive residual information RInfo from the quantization coefficient LEVEL, encode the residual information RInfo, etc., and store the encoded data in the bitstream.

[0131] In step S256, the accumulation buffer 215 accumulates the bit stream derived in this manner and outputs it to the outside of the image encoding device 200 at a predetermined timing or based on a request from another device. This bit stream is transmitted to a decoding device or the like via a transmission path or a recording medium, for example. When the processing of step S256 ends, the image encoding processing ends and the processing returns to FIG. 15.

[0132] 18 is a functional block diagram of a mode decision module 301 in a VVC encoder according to an embodiment, which searches for an optimal CTU partitioning pattern. The mode decision module 301 is one of the modules constituting the control unit shown in FIG. 13. The mode decision module 301 is a partitioning unit that performs processing to partition a CTU into multiple CUs.

[0133] As shown in FIG. 18, the mode decision module 301 includes a CTU optimum division pattern search module 302 , a block division determination module 303 , an RD cost calculation module 304 , a coding mode search module 305 , and an operation cost management table 306 .

[0134] The CTU optimal division pattern search module 302 is a module that searches for the optimal division pattern of the CTU, and first calls the block division determination module 303. The CTU optimal division pattern search module 302 also recursively calls the block division determination module 303 based on the division determination result of the block division determination module 303.

[0135] The block division determination module 303 is a module that determines block division, and calls the RD cost calculation module 304. The block division determination module 303 also recursively calls the RD cost calculation module 304 based on the RD cost of the RD cost calculation module 304.

[0136] The RD cost calculation module 304 is a module that calculates the RD cost of the divided block, and calls the coding mode search module 305. The coding mode search module 305 searches for the coding mode of the divided block and returns the found coding mode to the RD cost calculation module 304.

[0137] The RD cost calculation module 304 calculates the RD cost of the divided block by referring to the coding mode output from the coding mode search module 305 and the operation cost management table 306 shown in Fig. 19, and outputs the calculated RD cost to the block division determination module 303. The block division determination module 303 makes a block division determination based on the RD cost output from the RD cost calculation module 304, and outputs the division determination result to the CTU optimal division pattern search module 302.

[0138] 19 is a diagram showing the operation cost management table 306 according to the embodiment. The operation cost management table 306 holds an upper limit value for the number of times a method is executed, and is used to limit the number of times the RD cost calculation module (method) in the mode decision module is executed.

[0139] The operation cost management table 306 can be generated before the execution of the program starts, or its values ​​can be dynamically corrected during the execution of the program.

[0140] 19 , the operation cost management table 306 stores a CU size 111, a method for calculating RD cost 112, a normalized operation execution time (TCost) 113, a number of executions (CCcost) 114, and a normalized operation cost (TCost×Ccost) 115. The operation cost management table 306 also stores a method execution count upper limit 116, a variable name 117 that holds the method execution count upper limit, and a counter variable name 118 that holds the number of method executions in the CTU.

[0141] 19, when the CU size 111 is 64×64, the method 112 for calculating the RD cost is RDC64×64(), the normalized operation execution time (TCost) 113 is 20, the number of executions (CCcost) 114 is 1, and the normalized operation cost (TCost×Ccost) 115 is 20. When the CU size 111 is 64×64 and the upper limit 116 on the number of method executions is set to 0, the operation cost reduction result is 20×0=0.

[0142] When the CU size 111 is 32×32, the method 112 for calculating the RD cost is RDC32×32(), the normalized calculation execution time (TCost) 113 is 5, the number of executions (CCcost) 114 is 4, and the normalized calculation cost (TCost×Ccost) 115 is 20. When the CU size 111 is 32×32 and the upper limit 116 on the number of method executions is set to 4, the calculation cost reduction result is 5×4=20.

[0143] When the CU size 111 is 16×16, the method 112 for calculating the RD cost is RDC16×16(), the normalized calculation execution time (TCost) 113 is 1, the number of executions (CCcost) 114 is 16, and the normalized calculation cost (TCost×Ccost) 115 is 16. When the CU size 111 is 16×16 and the upper limit 116 on the number of method executions is set to 16, the calculation cost reduction result is 1×16=16.

[0144] 20 is a functional block diagram showing a calculation cost control module 401 that controls the calculation cost of a divided CTU according to an embodiment. The calculation cost control module 401 is one of the modules that make up the control unit shown in FIG. 13.

[0145] As shown in FIG. 20, the operation cost control module 401 includes a measurement unit 411 , an operation cost calculation unit 412 , a conversion coefficient calculation unit 413 , a target operation cost calculation unit 414 , and an execution count control unit 415 .

[0146] The measurement unit 411 measures the execution time and the number of calculations performed for each of the multiple CUs into which the CTU is divided.

[0147] The operation cost calculation unit 412 calculates a normalized total operation cost for the multiple CUs based on the execution time of the method and the number of executions of the operation measured for each of the multiple CUs into which the CTU is divided. Specifically, the operation cost calculation unit 412 calculates a normalized operation cost for each of the multiple CUs based on the execution time of the method and the number of executions of the operation for each of the multiple CUs, and calculates the total operation cost by summing up the normalized operation costs calculated for each CU.

[0148] The conversion coefficient calculation unit 413 calculates a conversion coefficient for converting the total operation cost calculated by the operation cost calculation unit 412 into real time.

[0149] The target operation cost calculation unit 414 uses the transform coefficients calculated by the transform coefficient calculation unit 413 to calculate a target total operation cost for a plurality of CUs.

[0150] The execution count control unit 415 controls the number of times of execution of operations of the multiple CUs so that the total operation cost calculated by the operation cost calculation unit 412 satisfies the target total operation cost calculated by the target operation cost calculation unit 414. Then, the execution count control unit 415 outputs the controlled number of times of execution of the method. This controlled number of times of execution of the method is applied to encoding of the divided CTUs.

[0151] 5. Operation Description 5.1. Operation when searching for optimal CTU partition pattern Fig. 21 is a flowchart for explaining the process of searching for the optimal CTU partition pattern according to an embodiment. Fig. 21 illustrates a case where three types of CU sizes, 64x64, 32x32, and 16x16, are selectable, as shown in Fig. 7.

[0152] When the process of searching for the optimal division pattern of the CTU is started, first, N=32 is set as an initial value (step S301). Next, a block division determination process is executed using N as an argument (step S302), and the process of searching for the optimal division pattern of the CTU is completed.

[0153] 22 is a flowchart illustrating the block division determination process according to the embodiment. The block division determination process corresponds to step S302 in FIG.

[0154] When the block division determination process is started, the RD cost calculation process for the 2Nx2N block is executed (step S311). Next, the value of the calculated RD cost is set to a variable RDC_2N (step S312).

[0155] Next, it is determined whether N is the minimum block size (step S313). If it is determined in step S313 that N is the minimum block size (Yes in step S313), the RD cost calculation process for NxN block 1 is executed (step S314).

[0156] As shown in Figure 23, "NxN block 1" refers to the top left block of the NxN blocks obtained by dividing a 2Nx2N block. "NxN block 2" refers to the top right block of the NxN blocks obtained by dividing a 2Nx2N block. "NxN block 3" refers to the bottom left block of the NxN blocks obtained by dividing a 2Nx2N block. "NxN block 4" refers to the bottom right block of the NxN blocks obtained by dividing a 2Nx2N block.

[0157] Thereafter, the RD cost calculation process for NxN blocks 2 to 4 is executed (steps S315 to S317), the RD costs for NxN blocks 1 to 4 are summed up and set as variable RDC_N (step S318), and the process proceeds to step S319.

[0158] On the other hand, if it is determined in step S313 that N is not the minimum block size (No in step S313), the initial value is set to N=N / 2 (step S320).

[0159] Next, the division determination process for block 1 is recursively executed with N as an argument (step S321), the division determination process for block 2 is recursively executed with N as an argument (step S322), the division determination process for block 3 is recursively executed with N as an argument (step S323), and the division determination process for block 4 is recursively executed with N as an argument (step S324).Then, the RD costs of NxN blocks 1 to 4 are summed and set to variable RDC_N (step S325), and the process proceeds to step S319.

[0160] In step S319, it is determined whether the variable RDC_2N is greater than the variable RDC_N. If the variable RDC_2N is greater than the variable RDC_N in step S319 (Yes in step S319), the 2Nx2N block is determined as the block division determination result for the CTU, and RDC_2N is returned as the return value (step S326), and the block division determination process ends.

[0161] In step S319, if the variable RDC_2N is not greater than the variable RDC_N (No in step S319), the division determination result of the four NxN blocks or the recursive processing is used as the block division determination result of the CTU, and RDC_N is returned as the return value (step S327), and the block division determination process is terminated.

[0162] 22, when N is the minimum block size, the division determination result is four NxN blocks. When N is not the minimum block size, the division determination process is recursively executed with N=N / 2, and the division determination result becomes the division determination result for block size N.

[0163] Fig. 24 is a flowchart illustrating the RD cost calculation process according to the embodiment. In Fig. 24, similarly to Fig. 21, a case where three types of CU sizes, 64x64, 32x32, and 16x16, are selectable will be described.

[0164] In Fig. 24, RDC64x64(), RDC32x32(), and RDC16x16() shown in Fig. 7 etc. are processed as RDCNxN() with N as an argument. Also, Fig. 24 shows processing corresponding to the RD cost calculation method in Fig. 16.

[0165] When the RD cost calculation process is started (block size N is used as an argument), the calculation cost management table 306 is referenced and the upper limit value of the number of executions for block size N, RDC_COUNT_<N>_MAX, is set to the variable RDC_COUNT_MAX (step S331).

[0166] Next, the calculation cost management table 306 is referenced, and the execution count RDC_COUNT_<N>_COUNT for the block size N is set to the variable RDC_COUNT_N (step S332).

[0167] Next, it is determined whether the CTU division determination process is the first execution (step S333). If it is determined in step S333 that the CTU division determination process is the first execution (Yes in step S333), the calculation process execution count counter variable RDC_COUNT_N is initialized to zero (step S334), and the process proceeds to step S335.

[0168] In step S333, if it is determined that the CTU division determination process is not the first execution (No in step S333) or after the process of step S334, it is determined whether RDC_COUNT_N≧RDC_COUNT_MAX (step S335).

[0169] If RDC_COUNT_N≧RDC_COUNT_MAX (Yes in step S335), the maximum value that the variable COST_N can take is set as the cost value (step S336), and the RD cost calculation process ends (COST_N is the return value).

[0170] On the other hand, if RDC_COUNT_N≧RDC_COUNT_MAX is not satisfied (No in step S335), the coding mode of the block is searched for (step S337), the RD cost of the block in the searched coding mode is calculated, and the calculated cost value is set to the variable COST_N (step S338).

[0171] Next, the calculation process execution count counter variable RDC_COUNT_N is incremented by 1 and set in the execution count holding variable RDC_COUNT_<N>_COUNT (step S339), updating the operation cost management table 306. Then, the RD cost calculation process is terminated (COST_N is used as the return value).

[0172] <5.2. Operation for Controlling the Number of Execution Times of Calculations of Optimal Division Pattern of CTU> FIG. 25 is a flowchart for explaining the operation for controlling the number of execution times of calculations of the optimal division pattern of CTU according to the embodiment.

[0173] As shown in FIG. 25, the number of execution times of the multiple CUs into which the CTU is divided is controlled by measuring the execution time and the number of calculation execution times for each of the multiple CUs into which the CTU is divided by the measurement unit 411 (step S401).

[0174] Next, the operation cost calculation unit 412 calculates a normalized total operation cost for the multiple CUs based on the execution time and the number of operation executions of the method measured for each of the multiple CUs into which the CTU is divided (step S402).

[0175] Next, the conversion coefficient calculation unit 413 calculates a conversion coefficient for converting the total operation cost calculated by the operation cost calculation unit 412 into real time (step S403).

[0176] The target operation cost calculation unit 414 calculates a target total operation cost of a plurality of CUs using the transform coefficients calculated by the transform coefficient calculation unit 413 (step S404).

[0177] Next, the execution count control unit 415 controls the number of times that the operations are executed for the multiple CUs so that the total operation cost calculated by the operation cost calculation unit 412 satisfies the target total operation cost calculated by the target operation cost calculation unit 414 (step S405).

[0178] 6. Hardware Configuration FIG. 26 is a hardware configuration diagram showing an example of a computer 1000 that realizes an arithmetic unit of an information processing device equipped with a VVC encoder according to an embodiment.

[0179] The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0180] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0181] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, as well as programs that depend on the hardware of the computer 1000 .

[0182] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an application program according to the present disclosure, which is an example of program data 1450.

[0183] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0184] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, and semiconductor memories.

[0185] Although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, as another example, the CPU 1100 may obtain these programs from other devices via an external network 1550.

[0186] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0187] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0188] Note that the present technology may also be configured as follows. (1) An information processing device comprising: an operation cost calculation unit that calculates a normalized total operation cost for a plurality of CUs based on the execution time and number of operation executions of a method measured for each of the plurality of CUs into which a CTU is divided; a target operation cost calculation unit that calculates a target total operation cost for the plurality of CUs; ​​and an execution count control unit that controls the number of operation executions for the plurality of CUs so that the total operation cost calculated by the operation cost calculation unit satisfies the target total operation cost calculated by the target operation cost calculation unit. (2) The information processing device according to (1), further comprising: a conversion coefficient calculation unit that calculates a conversion coefficient that converts the total operation cost calculated by the operation cost calculation unit into real time, wherein the target operation cost calculation unit calculates the target total operation cost for the plurality of CUs using the conversion coefficient calculated by the conversion coefficient calculation unit. (3) The information processing device according to (1), further comprising: a measurement unit that measures the execution time and number of operation executions for each of the plurality of CUs. (4) The information processing device according to (1), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (5) The information processing device according to (4), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (6) The information processing device according to (1), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (7) The information processing device according to (1), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (8) The information processing device according to (1), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (9) The information processing device according to (2), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (10) The information processing device according to (1), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (11) The information processing device according to (1), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (12) The information processing device according to (10), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (13) The information processing device according to (12), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (14) The information processing device according to (13), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (15) The information processing device according to (14), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (16) The information processing device according to (15), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (17) The information processing device according to (16), further comprising: a division unit that performs processing to divide the CTU into the plurality of CUs. (18) The information processing device according to (18), further comprising: a division unit that (10) The information processing device according to (1), wherein the information processing device is an information processing device for measurement.(11) The information processing device according to (1), wherein the CTU is included in a frame of an image. (12) The information processing device according to (1), wherein the information processing device is an encoder. (13) The information processing device according to (1), wherein the information processing device is a software encoder. (14) The information processing device according to (1), wherein the information processing device is a hardware encoder. (15) The information processing device according to (1), wherein the information processing device is a game server that encodes and distributes game video. (16) A game system comprising: the information processing device according to (15); and a game client having a decoder that decodes the encoded video distributed from the game server and that distributes the decoded video. (17) An information processing method, comprising: calculating a normalized total operation cost for a plurality of CUs based on the execution time and number of operation executions of a method measured for each of a plurality of CUs into which a CTU is divided; calculating a target total operation cost for the plurality of CUs; ​​and controlling the number of operation executions for the plurality of CUs so that the total operation cost satisfies the target total operation cost.

[0189] 10 Game system 11 Game server 12 Game client 13 Real-time video encoder 14 Real-time video decoder 301 Mode decision module 302 CTU optimal division pattern search module 303 Block division decision module 304 RD cost calculation module 305 Coding mode search module 306 Operation cost management table 401 Operation cost control module 411 Measurement unit 412 Operation cost calculation unit 413 Transform coefficient calculation unit 414 Target operation cost calculation unit 415 Execution count control unit

Claims

1. An information processing device having: an operation cost calculation unit that calculates a normalized total operation cost for multiple CUs into which a CTU is divided based on the execution time and number of operation executions of a method measured for each of the multiple CUs; ​​a target operation cost calculation unit that calculates a target total operation cost for the multiple CUs; ​​and an execution count control unit that controls the number of operation executions of the multiple CUs so that the total operation cost calculated by the operation cost calculation unit satisfies the target total operation cost calculated by the target operation cost calculation unit.

2. An information processing device as described in claim 1, further comprising a conversion coefficient calculation unit that calculates a conversion coefficient for converting the total operation cost calculated by the operation cost calculation unit into real time, and the target operation cost calculation unit calculates the target total operation cost of the multiple CUs using the conversion coefficient calculated by the conversion coefficient calculation unit.

3. The information processing device according to claim 1, further comprising a measurement unit for measuring an execution time and a number of executions of operations for each of the plurality of CUs.

4. The information processing device according to claim 1, further comprising a division unit that performs a process of dividing the CTU into the plurality of CUs.

5. The information processing device according to claim 4, wherein an upper limit on the number of times each CU can be executed is set for each CU divided by the dividing unit.

6. The information processing device according to claim 1, wherein the operation cost calculation unit calculates a normalized operation cost for each of the multiple CUs based on the execution time of the method and the number of operation executions for each of the multiple CUs, and calculates the total operation cost by adding up the normalized operation costs calculated for each of the CUs.

7. The information processing device according to claim 1, further comprising a table storing the size of said CU, the execution time of said method associated with the size of said CU, the number of times of execution of said method, and a normalized operation cost.

8. The information processing device according to claim 1, wherein when execution of the method is omitted, an arbitrary value can be used as the cost value which is the return value of the omitted method.

9. The information processing device according to claim 2, wherein the conversion coefficients are measured for each environment in which encoding is performed.

10. The information processing device according to claim 1, wherein the information processing device is an information processing device for measurement.

11. The information processing device according to claim 1, wherein the CTU is included in a frame of an image.

12. The information processing device according to claim 1, wherein the information processing device is an encoder.

13. The information processing device according to claim 1, wherein the information processing device is a software encoder.

14. The information processing device according to claim 1, wherein the information processing device is a hardware encoder.

15. The information processing device according to claim 1, wherein the information processing device is a game server that encodes and distributes game images.

16. A game system comprising: an information processing device according to claim 15; and a game client having a decoder for decoding encoded video distributed from said game server, and distributing said decoded video.

17. An information processing method comprising: calculating a normalized total operation cost for a plurality of CUs into which a CTU is divided based on the execution time and number of operation executions of a method measured for each of the plurality of CUs; ​​calculating a target total operation cost for the plurality of CUs; ​​and controlling the number of operation executions for the plurality of CUs so that the total operation cost satisfies the target total operation cost.

Citation Information

Patent Citations

  • Filter computing unit and motion compensation device

    JP2008242594A

  • Image encoding device, image decoding device, and integrated circuit used therein

    WO2005122590A1

  • Image processing device and method

    WO2010038858A1