Information processing device and information processing method

Abstract

The present disclosure provides an information processing device comprising: a slice number determining unit that determines a number of slices on the basis of a set number of execution cores, a set encoding performance, and a set speed for an input CTU that is to be encoded; and a combination determining unit that determines a combination of the number of parallel CTUs, the number of parallel CU partitions, and the number of parallel CU modes to be performed for each slice among the number of slices determined by the slice number determining unit.

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method.

[0002] Conventional codec parallelization methods include inter-frame task pipelining, non-reference frame parallelization, and partitioned encoding. Also, slice parallelization and CTU parallelization (WPP) are known as intra-frame parallelization methods that do not require frame delay.

[0003] Here, a "slice" is a collection of CTUs. A "CTU" is a fixed-size block formed by combining several CUs. A "CU" is a screen divided into several blocks.

[0004] JP 2011-160077 A

[0005] Conventionally, even the parallelization methods of slice parallelism and CTU parallelism (WPP) have been able to meet the speed requirements of general application software programs.

[0006] However, in recent years, the emergence of video compression standards such as Versatile Video Coding (VVC) has increased the amount of codec processing required by encoders. Therefore, in order to speed up codec processing, there is a demand for codec methods with a higher degree of parallelism than conventional codec methods.

[0007] The present disclosure has been made in consideration of the above-described circumstances, and provides an information processing device and an information processing method capable of performing parallel processing of codecs with a high degree of parallelism.

[0008] The information processing device disclosed herein includes a slice number determination unit that determines the number of slices based on the set number of execution cores, set encoding performance, and set speed of the CTU to be input for encoding, and a combination determination unit that determines the combination of CTU parallel number, CU partition parallel number, and CU mode parallel number to be performed for each slice of the number of slices determined by the slice number determination unit.

[0009] FIG. 1 is a diagram for explaining an example of processing in a codec method using slice parallelism and CTU parallelism. FIG. 2 is a diagram for explaining an example of processing in a codec method using CU partition parallelism and CU mode parallelism. FIG. 3 is a diagram for explaining that the mode decision processing amount of a VVC encoder is enormous. FIG. 4 is a block diagram showing an example of the configuration of an image encoding device which is one aspect of an image processing device according to an embodiment. FIG. 5 is a block diagram showing an example of the main configuration of a transform quantization unit according to an embodiment. FIG. 6 is a flowchart for explaining an example of the flow of encoding processing executed by an image encoding device according to an embodiment. FIG. 7 is a flowchart for explaining an example of the flow of image encoding processing according to an embodiment. FIG. 8 is a functional block diagram showing functions of a VVC encoder according to an embodiment. FIG. 9 is a diagram showing a control information database according to an embodiment. FIG. 10 is a diagram showing a control information database for explaining the determination of the number of slices by a slice number determination unit according to an embodiment. FIG. 11 is a diagram for explaining the determination of encoding performance and speed by a slice number determination unit according to an embodiment. FIG. 12 is a diagram showing a speed value T(S) by a parallel method determination unit according to an embodiment. ｐ1 is a diagram illustrating a control information database for explaining a search for the number of execution cores by a parallel processing method determination unit according to an embodiment. FIG. 2 is a diagram illustrating a control information database for explaining a search for the number of execution cores by a parallel processing method determination unit according to an embodiment. FIG. 3 is a flowchart illustrating overall processing for determining a parallel processing method of an encoder according to an embodiment. FIG. 4 is a diagram illustrating a generation flowchart of a control information database of an encoder according to an embodiment. FIG. 5 is a diagram illustrating a flowchart for determining the number of slices (S) of an encoder according to an embodiment. FIG. 6 is a flowchart for determining a combination of parallel processing methods [W(n), P(n), M(n)] in each slice according to an embodiment. FIG. 7 is a flowchart for estimating a processing amount A(n) based on a past frame according to an embodiment. FIG. 8 is a flowchart for estimating a processing amount A(n) based on features of an original image or ME according to an embodiment. FIG. 9 is a flowchart for determining a parallel processing method {W(n), P(n), M(n)} for each slice based on the processing amount A(n) according to an embodiment. FIG. 10 is a flowchart for adjusting a parallel processing method pattern for an n-th slice with respect to a speed value according to an embodiment. FIG. 11 is a flowchart for adjusting a parallel processing method pattern for each slice with respect to the number of execution cores. FIG. 12 is a diagram illustrating a game server and a game client in a game system according to an embodiment. 1 is a hardware configuration diagram showing an example of a computer 1000 that realizes an arithmetic unit of an information processing device equipped with a VVC encoder according to an embodiment.

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. The description will be given in the following order.

[0011] 1. Background of the Present Disclosure 2. Problems of the Present Disclosure 3. Embodiments 3.0. Functional Block Diagram of Image Processing Device 3.1. Functional Block Diagram 3.1.1. Control Information Database Generation Unit 311 3.1.2. Number of Slices Determination Unit 312 3.1.3. Parallel Method Combination Determination Unit 313 3.1.3.1. Estimation of Processing Amount A(n) of Each Slice 3.1.3.2. Determination of Parallel Method {W(n), P(n), M(n)} of Each Slice 3.1.3.2.1. Calculation of Adjusted Weight 3.1.3.2.2. Pattern Adjustment for Speed ​​Value 3.1.3.2.3. Pattern Adjustment for Number of Execution Cores 3.2. Operation Description 3.2.1. Flowchart of Overall Processing 3.2.2. Flowchart of Generation of Control Information Database 321 3.2.3. 3.2.4. Flowchart for determining the number of slices (S) 3.2.5. Flowchart for determining the combination of parallel methods [W(n), P(n), M(n)] in each slice 3.2.6. Flowchart for estimating the processing volume A(n) based on past frames 3.2.7. Flowchart for determining the parallel method {W(n), P(n), M(n)} for each slice based on the processing volume A(n) 3.2.8. Flowchart for adjusting the speed value of the parallel method pattern in the nth slice 3.2.9. Flowchart for adjusting the parallel method pattern for each slice to the number of execution cores 4. Use cases 5. Hardware configuration

[0012] 1. Background of the Present Disclosure Fig. 1 is a diagram for explaining an example of processing of a codec method using slice parallelism and CTU parallelism. Fig. 1 shows the processing granularity of each parallelism method from the perspective of a processing flow. As shown in Fig. 1, slice parallelism is performed in block A. Then, CTU parallelism is performed within the processing in block A. CTU parallelism is wavefront parallelism that takes into account dependencies between CTUs.

[0013] In conventional codec processing, conventional parallel methods can meet the speed requirements of general applications. However, as the amount of processing required for VVC increases, a higher degree of parallelism than conventional codec processing is required to speed up codec processing. To achieve a higher degree of parallelism, it is necessary to introduce a parallel method with a finer processing granularity.

[0014] Possible fine-grained parallel methods that can be introduced to solve the parallel processing requirements of VVC include CU partition parallelism and CU mode parallelism. FIG. 2 is a diagram for explaining an example of processing in a codec method using CU partition parallelism and CU mode parallelism. FIG. 2 shows the processing granularity of each parallel method from the processing flow. In FIG. 2, block C indicates CU partition parallelism, and block D indicates CU mode parallelism.

[0015] 3 is a diagram for explaining the enormous amount of mode decision processing required by a VVC encoder. As shown in FIG. 3, when an input image I having a CTU is input, mode decision processing is performed.

[0016] The mode decision process involves block partition processing and various coding processes. The processing volume is 10^3 for Advanced Video Coding (AVC) and 10^5 for High Efficiency Video Coding (HEVC), but is a massive 10^6 for VVC.

[0017] As shown in Figure 3, the processing time for mode decision occupies more than 80% of the processing time of a VVC encoder. Therefore, it is necessary to devise parallel processing for mode decision and meet the requirement of reducing the processing time.

[0018] Since the mode decision process accounts for over 80% of the processing load of a VVC encoder, this disclosure describes a parallel method for the mode decision process.

[0019] Codec parallelization methods include inter-frame task pipelining, non-reference frame parallelization, and partitioned encoding, but the present disclosure relates to a method related to intra-frame parallelism that does not require frame delay.

[0020] <2. Problems to be Solved by the Present Disclosure> The following points can be cited as problems to be solved when combining various parallel methods in implementing a VVC encoder.

[0021] In parallel processing, the smaller the parallel granularity, the larger the processing overhead and the smaller the degree of speed improvement. This is because the number of synchronization controls (data copy, completion wait, assignment to threads, process startup, etc.) increases. Therefore, for the same number of parallel processes, the speed improvement effect is basically slice parallel > CTU parallel (WPP) > CU partition parallel > CU mode parallel.

[0022] However, when the number of CTU lines performing WPP is small, it is possible that WPP is less than CU partition parallelism or WPP is less than CU mode parallelism. The degree of WPP parallelism depends on the number of CTU lines (see Figure 1).

[0023] The impact of each parallel method on coding performance is that the larger the number of slices in parallel (= the number of slices), the worse the coding performance becomes. CTU parallelism (MD WPP), CU partition parallelism, and CU mode parallelism basically do not change the coding performance.

[0024] In addition, since MD WPP performs parallel WPP processing only on the mode decision section, the output stream does not change whether MD WPP is on or off, so this is a processing method in which the coding performance does not change.

[0025] - Issues to be addressed when combining various parallel methods There are various variations in processing resources (instance type or number of CPU cores), and it is necessary to optimize the combination of parallel methods based on the number of execution cores, required coding performance, and required speed according to requirements.

[0026] <3. Embodiment> <3.0. Functional Block Diagram of Image Processing Device> <Image Encoding Device>

[0027] Fig. 4 is a block diagram showing an example of the configuration of an image coding device, which is one aspect of an image processing device according to an embodiment. The image coding device 200 shown in Fig. 4 is a device that codes image data of a moving image. The image coding device 200 codes the image data of the moving image using a coding method such as Versatile Video Coding (VVC), Advanced Video Coding (AVC), or High Efficiency Video Coding (HEVC).

[0028] Note that Fig. 4 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 4. In other words, in the image encoding device 200, there may be processing units that are not shown as blocks in Fig. 4, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 4. This also applies to other figures that explain processing units, etc. within the image encoding device 200.

[0029] 4 , the image coding device 200 includes a control unit 201, a rearrangement buffer 211, a calculation unit 212, a transform / quantization unit 213, a coding unit 214, and an accumulation buffer 215. The image coding device 200 also includes a dequantization / inverse transform unit 216, a calculation unit 217, an in-loop filter unit 218, a frame memory 219, and a prediction unit 220.

[0030] <Control Unit> The control unit 201 divides the video data held in the rearrangement buffer 211 into blocks (CU, PU, ​​TU, etc.) based on an externally or pre-specified block size of the processing unit. The control unit 201 also determines the coding parameters (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) to be supplied to each block based on, for example, RDO (Rate-Distortion Optimization).

[0031] For example, the control unit 201 sets the quantization value of a frame, sets the quantization value of a CTU (CTU baseQp), sets the coding mode (e.g., mts_idx, lfnst_idx, etc.), and corrects the quantization value for each CU based on the subjective image quality (e.g., sets CU Qp).

[0032] Among the control units that determine the coding parameters (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) to be supplied to each block, the mode decision module searches for the optimal mode, mainly from the perspective of maximizing coding efficiency. Specifically, it searches for a CTU division pattern and searches for the optimal coding mode for the divided CTUs.

[0033] After determining the above-described coding parameters, the control unit 201 supplies them to each block. For example, the header information Hinfo is supplied to each block. The prediction mode information Pinfo is supplied to the coding unit 214 and the prediction unit 220. The transformation information Tinfo is supplied to the coding unit 214, the transform / quantization unit 213, and the inverse quantization / inverse transformation unit 216. The filter information Finfo is supplied to the coding unit 214 and the in-loop filter unit 218. Of course, the destination to which each coding parameter is supplied is arbitrary and is not limited to this example.

[0034] For example, the control unit 201 supplies the set quantization values ​​(for example, CTU baseQp and CU Qp) and control parameters for coefficient transformation (for example, mts_idx and lfnst_idx) to the transform / quantization unit 213, the encoding unit 214, and the like.

[0035] <Sorting Buffer> Each field (input image) of video data is input to the image coding device 200 in its playback order (display order). The sorting buffer 211 acquires and holds (stores) each input image in its playback order (display order). Under the control of the control unit 201, the sorting buffer 211 sorts the input images in coding order (decoding order) and divides them into blocks, which are processing units. The sorting buffer 211 supplies each processed input image to the calculation unit 212.

[0036] <Calculation Unit> The calculation unit 212 subtracts the predicted image P supplied from the prediction unit 220 from the image corresponding to the block of processing units supplied from the sorting buffer 211 to derive residual data D, and supplies it to the transformation and quantization unit 213.

[0037] <Transform / Quantization Unit> The transform / quantization unit 213 acquires residual data D supplied from the calculation unit 212. The transform / quantization unit 213 also acquires prediction mode information Pinfo and transformation information Tinfo supplied from the control unit 201. The transform / quantization unit 213 performs transform / quantization processing on the residual data D based on the prediction mode information Pinfo and the transformation information Tinfo, and derives a quantization coefficient level. The transform / quantization processing includes, for example, coefficient transformation (primary transformation or secondary transformation) and quantization. Of course, any processing may be included in the transform / quantization processing, and some of the above-described processing may be omitted, or processing other than the above-described processing may be included. The transform / quantization unit 213 supplies the derived quantization coefficient level to the encoding unit 214 and the inverse quantization / inverse transform unit 216.

[0038] <Encoding Unit> The encoding unit 214 acquires the quantization coefficient level supplied from the transform / quantization unit 213. The encoding unit 214 also acquires various encoding parameters (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) supplied from the control unit 201. The encoding unit 214 also acquires information related to filters, such as filter coefficients, supplied from the in-loop filter unit 218. The encoding unit 214 also acquires information related to the optimal prediction mode supplied from the prediction unit 220.

[0039] The encoding unit 214 performs entropy encoding (lossless encoding) on ​​the quantization coefficient level to generate a bit string (encoded data). The encoding unit 214 may apply, for example, CABAC (Context-based Adaptive Binary Arithmetic Code) as this entropy encoding. The encoding unit 214 may apply, for example, CAVLC (Context-based Adaptive Variable Length Code) as this entropy encoding. Of course, the content of this entropy encoding is arbitrary and is not limited to these examples.

[0040] Furthermore, the encoding unit 214 derives residual information Rinfo from the quantized coefficient levels, encodes the residual information Rinfo, and generates a bit string.

[0041] Furthermore, the encoding unit 214 includes information about the filter supplied from the in-loop filter unit 218 in filter information Finfo, and includes information about the optimal prediction mode supplied from the prediction unit 220 in prediction mode information Pinfo. Then, the encoding unit 214 encodes the various encoding parameters described above (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) to generate a bit string.

[0042] The encoding unit 214 multiplexes the bit strings of the various types of information generated as described above to generate encoded data, and supplies the encoded data to the accumulation buffer 215.

[0043] <Storage Buffer> The storage buffer 215 temporarily stores the encoded data obtained by the encoding unit 214. The storage buffer 215 outputs the stored encoded data, for example, as a bit stream, to the outside of the image encoding device 200 at a predetermined timing. For example, this encoded data is transmitted to the decoding side via any recording medium, any transmission medium, any information processing device, etc. In other words, the storage buffer 215 also functions as a transmission unit that transmits the encoded data (bit stream).

[0044] Information on the code amount of the encoded data stored in the storage buffer 215 is supplied to the control unit 201. Based on this information, the control unit 201 controls the rate of the quantization operation of the transform and quantization unit 213 so as to prevent overflow or underflow.

[0045] <Inverse Quantization and Inverse Transformation Unit> The inverse quantization and inverse transformation unit 216 acquires the quantization coefficient level supplied from the transform and quantization unit 213. The inverse quantization and inverse transformation unit 216 also acquires the transformation information Tinfo supplied from the control unit 201.

[0046] The inverse quantization and inverse transform unit 216 performs inverse quantization and inverse transform processing on the quantized coefficient level based on the transform information Tinfo, and derives residual data D'. This inverse quantization and inverse transform processing is the inverse processing of the transform and quantization processing performed in the transform and quantization unit 213. That is, in the inverse quantization and inverse transform processing, for example, processing such as inverse quantization and inverse coefficient transform is performed. This inverse quantization is the inverse processing of the quantization performed in the transform and quantization unit 213. Furthermore, inverse coefficient transform is the inverse processing of the coefficient transform performed in the transform and quantization unit 213.

[0047] Of course, any processing may be included in this inverse quantization and inverse transform processing, and some of the processing described above may be omitted, or processing other than that described above may be included. The inverse quantization and inverse transform unit 216 supplies the derived residual data D' to the calculation unit 217.

[0048] <Calculation Unit> The calculation unit 217 obtains the residual data D' supplied from the inverse quantization and inverse transform unit 216 and the predicted image P supplied from the prediction unit 220. The calculation unit 217 adds the residual data D' to the predicted image P corresponding to the residual data D' to derive a locally decoded image. The calculation unit 217 supplies the derived locally decoded image to the in-loop filter unit 218 and the frame memory 219.

[0049] <In-loop filter unit> The in-loop filter unit 218 acquires a locally decoded image supplied from the calculation unit 217. The in-loop filter unit 218 also acquires filter information Finfo supplied from the control unit 201. The in-loop filter unit 218 also acquires an input image (original image) supplied from the rearrangement buffer 211. Note that any information may be input to the in-loop filter unit 218, and information other than the above information may also be input. For example, as necessary, information such as a prediction mode, motion information, a code amount target value, a quantization parameter qP, a picture type, and a block (CU, CTU, etc.) may be input to the in-loop filter unit 218.

[0050] The in-loop filter unit 218 performs appropriate filtering on the locally decoded image based on the filter information Finfo. The in-loop filter unit 218 also uses the input image (original image) and other input information for the filtering, as necessary.

[0051] For example, the in-loop filter unit 218 may apply a bilateral filter as its filtering process. For example, the in-loop filter unit 218 may apply a deblocking filter (DBF (DeBlocking Filter)) as its filtering process. For example, the in-loop filter unit 218 may apply an adaptive offset filter (SAO (Sample Adaptive Offset)) as its filtering process.

[0052] For example, the in-loop filter unit 218 may apply an adaptive loop filter (ALF (Adaptive Loop Filter)) as its filtering process. Furthermore, the in-loop filter unit 218 may apply a combination of multiple of these filters as its filtering process. Note that which filters to apply and in what order to apply them are arbitrary and can be selected as appropriate. For example, the in-loop filter unit 218 applies four in-loop filters, namely, a bilateral filter, a deblocking filter, an adaptive offset filter, and an adaptive loop filter, in this order as its filtering process.

[0053] Of course, the filtering process performed by the in-loop filter unit 218 is arbitrary and is not limited to the above example. For example, the in-loop filter unit 218 may apply a Wiener filter or the like.

[0054] The in-loop filter unit 218 supplies the filtered locally decoded image to the frame memory 219. When transmitting information about the filter, such as a filter coefficient, to the decoding side, the in-loop filter unit 218 supplies the information about the filter to the encoding unit 214.

[0055] <Frame Memory> The frame memory 219 performs processing related to the storage of image data. For example, the frame memory 219 acquires and holds (stores) locally decoded images supplied from the calculation unit 217 and locally decoded images that have been filtered and supplied from the in-loop filter unit 218. The frame memory 219 also uses the locally decoded images to reconstruct and hold decoded images for each picture (storing them in a buffer within the frame memory 219). The frame memory 219 supplies the decoded images (or portions thereof) to the prediction unit 220 in response to a request from the prediction unit 220.

[0056] <Prediction Unit> The prediction unit 220 performs processing related to generation of a predicted image. For example, the prediction unit 220 obtains prediction mode information Pinfo supplied from the control unit 201. For example, the prediction unit 220 obtains an input image (original image) supplied from the reordering buffer 211. For example, the prediction unit 220 obtains a decoded image (or a part thereof) read from the frame memory 219.

[0057] The prediction unit 220 performs prediction processing such as inter prediction or intra prediction using the prediction mode information Pinfo and the input image (original image). That is, the prediction unit 220 performs prediction and motion compensation by referring to the decoded image as a reference image, and generates a predicted image P. The prediction unit 220 supplies the generated predicted image P to the calculation unit 212 and the calculation unit 217. Furthermore, the prediction unit 220 supplies information on the prediction mode selected by the above processing, i.e., the optimal prediction mode, to the encoding unit 214 as necessary.

[0058] <Transformation and quantization unit> Fig. 5 is a block diagram showing an example of the main configuration of the transformation and quantization unit 213 in Fig. 4. As shown in Fig. 5, the transformation and quantization unit 213 has a coefficient conversion unit 231, a quantization unit 232, and a violation avoidance processing unit 233.

[0059] The coefficient conversion unit 231 acquires the prediction residual (residual data D) supplied from the calculation unit 212, and performs coefficient conversion on the prediction residual to generate conversion coefficients. The coefficient conversion unit 231 has a primary conversion unit 241 and a secondary conversion unit 242.

[0060] The primary transform unit 241 acquires prediction residuals (residual data D) supplied from the calculation unit 212, and performs primary transform on the prediction residuals to generate primary transform coefficients. The primary transform unit 241 performs primary transform in a mode according to the set mts_idx. This mts_idx is set by the control unit 201 or a mode setting unit 252 (described later). For example, if mts_idx > 1, the primary transform unit 241 performs primary transform by applying the MTS mode. On the other hand, if mts_idx = 0, the primary transform unit 241 performs primary transform without applying the MTS mode. The primary transform unit 241 supplies the generated primary transform coefficients to the secondary transform unit 242.

[0061] The secondary transform unit 242 acquires the primary transform coefficients supplied from the primary transform unit 241 and performs secondary transform on the primary transform coefficients to generate transform coefficients. The secondary transform unit 242 performs the secondary transform in a mode according to the set lfnst_idx. This lfnst_idx is set by the control unit 201 or a mode setting unit 252 (described later). For example, if lfnst_idx > 0, the secondary transform unit 242 performs the secondary transform by applying the LFNST mode. On the other hand, if lfnst_idx = 0, the secondary transform unit 242 performs the secondary transform without applying the LFNST mode. The secondary transform unit 242 supplies the generated transform coefficients to the quantization unit 232.

[0062] The quantization unit 232 acquires transform coefficients supplied from the coefficient conversion unit 231 (secondary conversion unit 242) and performs quantization on the transform coefficients to generate quantized coefficients. The quantization unit 232 performs quantization using a quantization value (CTU baseQp, CU Qp, etc.) set by the control unit 201. For example, the quantization unit 232 performs quantization using CTU baseQp when the control unit 201 sets the coding mode. Furthermore, when the quantization value is corrected at the CU level, the quantization unit 232 performs quantization using CU Qp.

[0063] The quantization unit 232 supplies the generated quantization coefficients to the violation avoidance processing unit 233. The violation avoidance processing unit 233 acquires the quantization coefficients supplied from the quantization unit 232, and if the quantization coefficients violate a constraint related to the coding mode, executes processing to avoid the violation of the constraint. The violation avoidance processing unit 233 has a coding mode violation confirmation unit 251 and a mode setting unit 252.

[0064] The coding mode violation check unit 251 obtains quantized coefficients supplied from the quantization unit 232 and determines whether the quantized coefficients violate constraints related to the coding mode. For example, the coding mode violation check unit 251 determines whether the MTS mode is applied in the primary transform or the LFNST mode is applied in the secondary transform, and the only non-zero quantized coefficients are DC coefficients. For example, if it is determined that the LFNST mode is applied in the secondary transform and the only non-zero quantized coefficients are DC coefficients, the coding mode violation check unit 251 determines that a coding mode violation has occurred. Furthermore, if it is determined that the MTS mode is applied in the primary transform and the only non-zero quantized coefficients are DC coefficients, the coding mode violation check unit 251 determines that a coding mode violation has occurred.

[0065] If it is determined that the constraint is violated, the coding mode violation checking unit 251 supplies information notifying that fact to the mode setting unit 252. On the other hand, if it is determined that the constraint is not violated, the coding mode violation checking unit 251 supplies the quantized coefficients to the coding unit 214 and the inverse quantization and inverse transform unit 216.

[0066] The mode setting unit 252 acquires information indicating the result of the coding mode violation confirmation by the coding mode violation confirmation unit 251. The mode setting unit 252 sets (updates) the coding mode based on the information (i.e., based on the result of the coding mode violation confirmation by the coding mode violation confirmation unit 251). For example, the mode setting unit 252 controls whether to apply the MTS mode or the LFNST mode. That is, the mode setting unit 252 sets (updates) mts_idx and lfnst_idx. The mode setting unit 252 supplies the set (updated) control parameters (mts_idx and lfnst_idx) to the coefficient conversion unit 231 to perform coefficient conversion again. The coefficient conversion unit 231 performs coefficient conversion in accordance with the set (updated) control parameters. That is, coefficient conversion and quantization are performed again (these processes are redone).

[0067] For example, if the coding mode violation checking unit 251 determines that the LFNST mode is applied in the secondary transform and that the only non-zero quantized coefficients are DC coefficients, the mode setting unit 252 sets the control parameter (lfnst_idx) that specifies the mode of the secondary transform to a value (0) that indicates that the LFNST mode is not applied in the secondary transform. Then, the mode setting unit 252 supplies the control parameter (lfnst_idx = 0) to the secondary transform unit 242. The secondary transform unit 242 performs the secondary transform again in accordance with the control parameter (lfnst_idx = 0). That is, the secondary transform and quantization are redone.

[0068] For example, if the coding mode violation checking unit 251 determines that MTS mode is applied in the primary transform and that only DC coefficients are non-zero quantized coefficients, the mode setting unit 252 sets the control parameter (mts_idx) that specifies the mode of the primary transform to a value (0) that indicates that MTS mode is not applied in the primary transform. Then, the mode setting unit 252 supplies the control parameter (mts_idx = 0) to the primary transform unit 241. The primary transform unit 241 performs primary transform again in accordance with the control parameter (mts_idx = 0). That is, the primary transform, secondary transform, and quantization are redone.

[0069] With this configuration, the image coding device 200 can suppress violations of restrictions on coding modes.

[0070] <Flow of Encoding Process> Next, an example of the flow of the encoding process executed by the image encoding device 200 described above will be described with reference to the flowchart of FIG.

[0071] When the encoding process starts, the control unit 201 of the image encoding device 200 sets a quantization value for a frame in step S1. The control unit 201 allocates a coding amount for each frame (picture) based on coding amount prediction and feedback control so as to satisfy the constraints of the target bit rate, and determines a quantization value at the frame (picture) level.

[0072] In step S2, the control unit 201 sets a quantization value for the CTU. The control unit 201 divides the frame into CTUs and determines a quantization value CTU BaseQp for each CTU. The control unit 201 predicts the code generation amount for the entire frame from the code generation amount for each CTU, and sets the CTU BaseQp so as to satisfy the target bit rate constraint.

[0073] In step S3, the control unit 201 sets a coding mode and saves the processing result. The control unit 201 determines the CU division pattern (QT / BT / TT), the prediction mode (intra or inter), the coefficient transformation mode of the prediction residual (MTS, LFNST, etc.), etc., so as to obtain optimal coding efficiency under the constraint of the given quantization value (CTU BaseQp). The processing on a CTU-by-CTU basis can be performed sequentially or in parallel.

[0074] In step S4, the control unit 201 corrects the quantization value based on the subjective image quality. To improve the subjective image quality, the control unit 201 corrects the quantization value based on a feature amount of the original image, such as activity, for each CU block divided at the CTU level or below. That is, the control unit 201 derives a quantization value CU Qp for each CU.

[0075] In step S5, each processing unit of the image coding device 200 executes an image coding process, codes the image input to the image coding device 200, and generates a bitstream.

[0076] In step S6, the control unit 201 determines whether the bitstream obtained in step S5 satisfies the rate constraint. If it is determined that the rate constraint is not satisfied, the process returns to step S2, and the subsequent processes are repeated. If it is determined in step S6 that the bitstream satisfies the rate constraint, the process proceeds to step S7.

[0077] In step S7, the accumulation buffer 215 outputs the bitstream it holds to the outside of the image encoding device 200. The output bitstream may be transmitted to a decoding device via an arbitrary transmission path or another device, for example. The output bitstream may also be recorded on an arbitrary recording medium. When the processing of step S7 is completed, the encoding process is completed.

[0078] <Flow of Image Encoding Process> Next, an example of the flow of the image encoding process executed in step S5 of FIG. 6 will be described with reference to the flowcharts of FIGS.

[0079] When the image encoding process starts, in step S31 of FIG. 7, the prediction unit 220 generates a predicted image in the optimal prediction mode (intra prediction or inter prediction) set in step S3.

[0080] In step S32, the calculation unit 212 derives the difference (prediction residual) between the input image and the predicted image generated in step S31. That is, the calculation unit 212 derives residual data D between the input image and the predicted image. The residual data D derived in this manner has a reduced data volume compared to the original image data. Therefore, the data volume can be compressed compared to when the image is encoded as is.

[0081] In step S33, the primary transform unit 241 performs primary transform on the prediction residuals in the coefficient transform mode (mode indicated by mts_idx) set in step S3, and generates primary transform coefficients.

[0082] In step S34, the secondary transform unit 242 performs secondary transform on the primary transform coefficients in the coefficient transform mode set in step S3 (the mode indicated by lfnst_idx) to generate transform coefficients.

[0083] In step S35, the quantization unit 232 performs quantization on the transform coefficients generated in step S34 using the quantization values ​​(CU Qp) corrected in step S4, to generate quantized coefficients.

[0084] In step S36, the coding mode violation confirmation unit 251 determines whether the quantization coefficients generated in step S35 violate the constraints on the coding mode. If it is determined that the quantization coefficients violate the coding mode constraints, the process proceeds to step S37.

[0085] In step S37, the coding mode violation confirmation unit 251 determines whether the LFNST mode is applied in the secondary conversion (lfnstIdx>0). If it is determined that the LFNST mode is applied (lfnstIdx>0), the processing proceeds to step S38.

[0086] In step S38, the mode setting unit 252 sets lfnstIdx to 0 (lfnstIdx = 0). When the processing of step S38 ends, the processing proceeds to step S39. Also, if it is determined in step S37 that the LFNST mode was not applied in the secondary conversion (lfnstIdx = 0), the processing proceeds to step S39.

[0087] In step S39, the coding mode violation check unit 251 determines whether the MTS mode was applied in the primary transform (mtsIdx>1). If it is determined that the MTS mode was not applied, the process returns to step S34, and the subsequent processes are executed. That is, in this case, the secondary transform and quantization are redone.

[0088] Also, if it is determined in step S39 that the MTS mode is applied (mtsIdx>1), the process proceeds to step S40.

[0089] In step S40, the mode setting unit 252 sets mtsIdx to 0 (mtsIdx = 0). When the processing of step S40 ends, the processing returns to step S33, and the subsequent processing is executed. That is, in this case, the primary transform, secondary transform, and quantization are redone.

[0090] If it is determined in step S36 that the quantized coefficients do not violate the constraints on the coding mode, the process proceeds to FIG.

[0091] In step S51 of FIG. 8, the inverse quantization and inverse transform unit 216 performs inverse quantization and inverse transform on the quantized coefficients generated in step S35, and derives (restores) prediction residuals (residual data D').

[0092] In step S52, the calculation unit 217 generates a locally decoded decoded image by adding the predicted image generated in step S31 to the residual data D' derived by the inverse quantization and inverse transform processing in step S51.

[0093] In step S53, the in-loop filter unit 218 performs in-loop filtering on the locally decoded image generated in step S52.

[0094] In step S54, the frame memory 219 stores the locally decoded image generated in step S52 and the locally decoded image filtered in step S53.

[0095] In step S55, the encoding unit 214 encodes the quantization coefficient "level" generated in step S35 to generate a bitstream. At this time, the encoding unit 214 may encode any information other than the quantization coefficient and store it in the bitstream. For example, the encoding unit 214 may encode various encoding parameters (header information HInfo, prediction mode information PInfo, transformation information TInfo, etc.) and store the encoded data in the bitstream. The encoding unit 214 may also derive residual information RInfo from the quantization coefficient "level," encode the residual information RInfo, etc., and store the encoded data in the bitstream.

[0096] In step S56, the accumulation buffer 215 accumulates the bit stream derived in this manner and outputs it to the outside of the image encoding device 200 at a predetermined timing or based on a request from another device. This bit stream is transmitted to a decoding device or the like via a transmission path or a recording medium, for example. When the processing of step S56 is completed, the image encoding processing ends and the processing returns to FIG. 6.

[0097] <3.1. Functional Block Diagram> Fig. 9 is a functional block diagram showing the functions of a VVC encoder 310 according to an embodiment. The VVC encoder 310 in Fig. 9 is realized as the functions of the control unit 201 shown in Fig. 8. As shown in Fig. 9, the VVC encoder 310 has a control information database generation unit 311, a slice number determination unit 312, and a parallel method combination determination unit 313. In this embodiment, a software encoder that runs on an arbitrary operating system or a hardware encoder with similar functions is assumed.

[0098] <3.1.1. Control information database generating unit 311> The control information database generating unit 311 has a control information database 321. The control information database generating unit 311 generates the control information database 321 having the number of slices of a CTU (S), coding performance for the number of slices (E), speed values ​​(T) having a plurality of speed patterns for the number of slices, and the number of execution cores (parallel number) (C) having a plurality of patterns of the number of execution cores for the plurality of speed patterns.

[0099] Each of the multiple speed value patterns in the control information database 321 has a combination of a CTU parallel number W(n), a CU partition parallel number (parallel number P(n)), and a CU mode parallel number (M(n)) corresponding to the multiple speed value pattern.

[0100] The control information database generating unit 311 statistically determines the control information database 321 (based on the actual measured average value of sample data, etc.).

[0101] 10 is a diagram showing the control information database 321 according to the embodiment. Note that FIG. 10 does not show the combinations of the CTU parallel number W(n), the CU partition parallel number (parallel number P(n)), and the CU mode parallel number (M(n)) according to the patterns of a plurality of speed values.

[0102] In FIG. 10, the parameters are explained as follows: a1, a2, ..., aN: Number of parallel method combination patterns when the number of slices is 1 to N; T(N) １ , T(N)２ , ..., T(N) ａＮ : Velocity value in various parallel method combination patterns when the number of slices is N, T(N) １ <T(N) ２ <...<T(N) ａＮ Let C(N) １ , C(N) ２ , ..., C(N) ａＮ : Number of execution cores (number of parallel processes) in various parallel process combination patterns when the number of slices is N, C(N) １ <C(N) ２ <...<C(N) ａＮ Let E(1)>E(2)>...E(N).

[0103] <3.1.2. Slice number determination unit 312> The slice number determination unit 312 determines the number of slices based on the set number of execution cores, set encoding performance, and set speed of the CTU to be input for encoding. The slice number determination unit 312 determines the number of slices.

[0104] Specifically, the slice number determination unit 312 extracts from the control information database 321 a candidate pattern to be adopted that satisfies the set number of execution cores, set encoding performance, and set speed of the CTU to be subjected to the input encoding, and determines the number of slices from the extracted candidate pattern to be adopted.

[0105] 11 is a diagram showing a control information database 321 for explaining the determination of the number of slices by the slice number determination unit 312 according to an embodiment. As shown in Fig. 11 , when the number of execution cores (parallelism) (C), the set coding performance (E), and the set speed (T) are given, the slice number determination unit 312 extracts patterns that satisfy each of the conditions using the control information database 321, and designates the pattern where C(N) = C as an adoption candidate (the part surrounded by a thick line in Fig. 11 ).

[0106] Next, the slice number determination unit 312 determines the number of slices from the candidate patterns to be adopted according to whether coding performance or speed is prioritized. All candidate patterns to be adopted satisfy the set coding performance and set speed, but the coding performance and speed are different. The slice number determination unit 312 determines the number of slices (S) by determining the pattern to be adopted according to whether coding performance or speed is prioritized.

[0107] FIG. 12 is a diagram illustrating the determination of encoding performance and speed by the slice number determination unit 312 according to an embodiment. For example, when prioritizing encoding performance, the slice number determination unit 312 employs (T(a)p1, E(a)) shown in FIG. 12 and determines the number of slices (S)=a. When prioritizing speed, the slice number determination unit 312 employs (T(c)p3, E(c)) and determines the number of slices (S)=c. When prioritizing balance, the slice number determination unit 312 employs (T(b)p2, E(b)) and determines the number of slices (S)=b. In other words, the slice number determination unit 312 extracts candidate patterns to adopt according to the settings of encoding performance and speed.

[0108] 9 , the parallel method combination determination unit 313 includes a processing amount estimation unit 322 and a parallel method determination unit 323. The parallel method combination determination unit 313 determines a combination of the CTU parallel number (W(n), CU partition parallel number (P(n)), and CU mode parallel number (M(n)) to be performed for each slice of the number of slices determined by the slice number determination unit 312.

[0109] The actual processing amount A of each slice varies considerably. The method of combining CTU parallelism (WPP) (parallel number W(n)), CU partition parallelism (parallel number P(n)), and CU mode parallelism (parallel number M(n)) performed within each slice is determined according to the processing amount (A(n)) of each slice. This improves the balance of processing time.

[0110] <3.1.3.1. Estimation of the processing amount A(n) of each slice> The processing amount estimation unit 322 estimates the processing amount A(n) of each slice. The processing amount A(n) of each slice can be estimated by, for example, the following estimation methods (1) and (2).

[0111] (1) Estimation based on the processing amount of past frames This is a method of approximating the processing amount A(n) using the complexity of encoding. It is estimated as A(n) = Complexity = B × Q, where B is the slice encoding amount of the same slice at the previous time, and Q is the slice quantization value of the same slice at the previous time.

[0112] (2) Estimation based on at least one of the feature amount of the original image and the feature amount of ME (Motion Estimation) Estimation is performed using A(n)=w1×original feature amount+w2×ME feature amount.

[0113] where, original image feature amount: A(n) = sum of original image feature amounts (Var (Variance), MAD (Mean Absolute Difference), TV (Total Variation)) of blocks in a slice, ME feature amount: A(n) = sum of ME feature amounts (SAD (Sum of Absolute Difference)) of blocks in a slice, w1 is the original image feature amount value weight, and w2 is the ME feature amount value weight.

[0114] Var, MAD, TV, and SAD are generally expressed as in equations (1) to (4).

[0115] <3.1.3.2. Determination of parallelization method {W(n), P(n), M(n)} for each slice> The parallelization method determination unit 323 determines the parallelization method {W(n), P(n), M(n)} for each slice based on the processing amount A(n).

[0116] <3.1.3.2.1. Calculation of Adjusted Weight> To determine the parallelization method {W(n), P(n), M(n)} for each slice, first, the adjusted weight of the slice parallelization method is found from the ratio of A(n).

[0117] Weight(n) = A(n) / ΣA(n) × S, n = 0, 1, ..., S-1 For example, if A(n) / ΣA(n) = {0.2, 0.25, 0.25, 0.3} for 4 slices, Weight(n) = {0.8, 1.0, 1.0, 1.2}.

[0118] <3.1.3.2.2. Pattern Adjustment for Velocity Values> The pattern adjustment for velocity values ​​is performed based on Weight(n) by adjusting the velocity value pattern T(S) adopted in <3.1.2. Number of slices determination unit 312>. ｐ The search is performed using Weight(n) x T(S) ｐ A search is made for the velocity value pattern that is closest to .

[0119] FIG. 13 shows the speed value T(S) calculated by the parallelization method determination unit 323 according to the embodiment. ｐ 13 is a diagram illustrating the control information database 321 for explaining a search for a weighted average weight (n). As shown in Fig. 13, the parallel method determination unit 323 searches to the right (the side with a higher speed) when Weight(n)>1, and searches to the left (the side with a lower speed) when Weight(n)<1.

[0120] <3.1.3.2.3. Pattern Adjustment for Number of Execution Cores> By adjusting the parallel method for each slice performed in <3.1.3.2.2. Pattern Adjustment for Speed ​​Value>, the number of execution cores for each slice is adjusted to C(S). ｐ（ｎ） / S, where n=0, 1, ..., S-1.

[0121] FIG. 14 is a diagram showing the control information database 321 for explaining a search for the number of execution cores by the parallelization method determination unit 323 according to the embodiment.

[0122] The parallel method determination unit 323 determines whether Σ(C(S) ｐ（ｎ）If the number of execution cores (C) is greater than the number of execution cores (T), select the pattern adjusted in <3.1.3.2.2. Pattern adjustment for speed value> (the pattern of the number of execution cores (C) corresponding to the pattern of the speed value (T)) and shift it to the left. ｐ（ｎ） The readjustment pattern is Weight(n) × T(S). ｐ We will prioritize readjustments that are closest to the original.

[0123] The parallel method determination unit 323 determines whether Σ(C(S) ｐ（ｎ） / S) <Number of execution cores (C), select the pattern adjusted in <3.1.3.2.2. Pattern adjustment for speed value> and shift it to the right. ｐ（ｎ） The readjustment pattern is Weight(n) x T(S). ｐ Prioritize readjustments that are far more selectable (select patterns with the greatest speed-up effect).

[0124] In other words, the parallel method determination unit 323 adjusts the pattern of the number of execution cores (C) corresponding to the searched pattern of speed values ​​(T) based on the determined number of slices so that it becomes the input set number of execution cores.

[0125] <3.2. Description of Operation> <3.2.1. Flowchart of Overall Processing> FIG. 15 is a flowchart illustrating the overall processing for determining the parallel processing method of the encoder according to the embodiment.

[0126] 15, the number of execution cores (parallelism) (C) = setting C, the set coding performance (E) = setting E, and the set speed (T) = setting T are input (step S101). Next, the control information database 321 is generated (step S102). Note that the processing order of steps S101 and S102 may be reversed.

[0127] Next, the number of slices (S) is determined (step S103), and the combination of parallel methods [W(n), P(n), M(n)] in each slice is determined (step S104), and the process ends.

[0128] 16 is a flowchart showing the generation of the control information database 321 of the encoder according to the embodiment. Fig. 16 corresponds to step S102 in Fig. 15.

[0129] 16, the maximum number of slices N is set as S=1 (step S201). Next, the coding performance E(S) for the number of slices S is measured (step S202). Next, the number of parallel method combination patterns aS for the number of slices S is calculated as i=1 (step S203).

[0130] Next, the number of execution cores C(S) of the i-th parallel combination pattern in the number of slices S is ｉ (Step S204), and the velocity value T(S) of the i-th parallel combination pattern in the number of slices S is calculated. ｉ is calculated (step S205).

[0131] Set i=i+1 (step S206), and determine whether i>aS (step S207). If it is determined in step S207 that i>aS is not true (No in step S207), the process returns to step S204.

[0132] In step S207, if it is determined that i>aS (Yes in step S207), T(S) １ , T(S) ２ , ..., T(S) ａＳ are rearranged in ascending order (step S208), and C(S) １ , C(S) ２ , ..., C(S) ａＳ T(S) １ , T(S) ２ , ..., T(S) ａＳ The data are rearranged in order (step S209).

[0133] Next, E(S) and C(S) for the number of slices S １ , C(S) ２ , ..., C(S) ａＳ , T(S) １ , T(S) ２ , ..., T(S) ａＳis stored in the control information database 321 (step S210), S=S+1 is set (step S211), and it is determined whether S>N (step S212).

[0134] If S>N is satisfied in step S212 (Yes in step S212), the process ends for generating the control information database 321. If S>N is not satisfied in step S212 (No in step S212), the process returns to step S202.

[0135] 17 is a flowchart showing how the encoder according to the embodiment determines the number of slices (S). This flowchart corresponds to step S103 in FIG.

[0136] 15, the number of execution cores (parallelism) (C) = setting C, the set encoding performance (E) = setting E, and the set speed (T) = setting T are input. As shown in Fig. 17, a pattern set {E} of encoding performance patterns E(N) ≥ setting E is extracted from the control information database 321 (step S301).

[0137] Next, the speed pattern T(N) is obtained from the control information database 321. ｉ A pattern set {T} of patterns of ≧ setting T is extracted (step S302), and a pattern C(N) of the number of execution cores (parallel number) is extracted from the control information database 321. ｉ A pattern set {C} of patterns of setting C is extracted (step S303).

[0138] Next, the intersection of the pattern set {E}, the pattern set {T}, and the pattern set {C} is the candidate pattern {T(j) ｉ , E(j)} (step S304). Next, whether emphasis is placed on coding performance, speed, or balance is input (step S305).

[0139] Next, it is determined whether coding performance is important (step S306). If coding performance is important (Yes in step S306), the candidate pattern {T(S) ｉ, E(S)}, the pattern with the best coding performance E(S) is selected, and the number of slices at that time is determined as S (step S310), thereby completing the process of determining the number of slices (S).

[0140] In step S306, if coding performance is not important (No in step S306), it is determined whether speed is important (step S307). If speed is important (Yes in step S307), the candidate pattern {T(S) ｉ , E(S)}, the pattern with the best velocity value T(S) is selected, and the number of slices at that time is determined as S (step S309), and the process of determining the number of slices (S) is completed.

[0141] In step S307, if the speed is not important (No in step S307), the candidate pattern {T(S) ｉ , E(S)}, a pattern with an intermediate velocity value T(S) is selected, and the number of slices at that time is determined as S (step S308), thereby completing the process of determining the number of slices (S).

[0142] <3.2.4. Flowchart for determining the combination of parallel methods [W(n), P(n), M(n)] in each slice> FIG. 18 is a flowchart for determining the combination of parallel methods [W(n), P(n), M(n)] in each slice according to an embodiment.

[0143] As shown in FIG. 18, in determining the combination of parallel methods [W(n), P(n), M(n)] in each slice, first, the processing amount (A(n)) of each slice is estimated (step S401).

[0144] Next, based on the processing volume A(n), the parallelization method {W(n), P(n), M(n)} for each slice is determined (step S402), and the process of determining the combination of parallelization methods [W(n), P(n), M(n)] is completed (step S402).

[0145] 19 is a flowchart for estimating the amount of processing A(n) based on a past frame according to an embodiment. Fig. 19 corresponds to step S401 in Fig. 18.

[0146] As shown in Figure 19, first, n = 0 is set (step S501), and then the slice coding amount B(n) of slice (n) at the previous time and the slice quantization value Q(n) of slice (n) at the previous time are obtained (step S502).

[0147] Next, the amount of processing A(n) = B(n) x Q(n) is calculated (step S503), n = n + 1 is set (step S504), and it is determined whether n < S (step S505). If n < S in step S505 (Yes in step S505), the processing for estimating the amount of processing A(n) is terminated. If n < S in step S505 (No in step S505), the processing returns to step S502.

[0148] 20 is a flowchart for estimating the processing amount A(n) based on the feature amount of the original image or ME according to an embodiment. Fig. 20 corresponds to step S401 in Fig. 18.

[0149] 20, n=0 is set (step S601), and then an original image feature value weight W1 and an ME feature value weight W2 are input (step S602). Next, an original image feature value F(n) of slice (n) and an ME feature value M(n) of slice (n) are obtained (step S603).

[0150] Next, the processing amount A(n) = W1 x F(n) + W2 x M(n) is calculated (step S604), n = n + 1 is set (step S605), and it is determined whether n < S (step S606).

[0151] If n<S is satisfied in step S606 (Yes in step S606), the process of estimating the amount of processing A(n) is terminated. If n<S is not satisfied in step S606 (No in step S606), the process returns to step S603.

[0152] 21 is a flowchart for determining a parallelization method {W(n), P(n), M(n)} for each slice based on the throughput A(n) according to an embodiment. Fig. 21 corresponds to step S402 in Fig. 18.

[0153] As shown in Fig. 21, the processing amount (A(n)) of each slice is input (step S701), where n = 0, 1, ..., S-1. Next, the adjustment weight is calculated from the ratio of A(n) (step S702), where Weight(n) = A(n) / ΣA(n) × S, where n = 0, 1, ..., S-1.

[0154] Next, n=0 is set (step S703), and an adjustment is made to the velocity value of the parallel method pattern in the n-th slice (step S704). Next, n=n+1 is set (step S705), and it is determined whether n<S (step S706).

[0155] If n<S is satisfied in step S706 (Yes in step S706), the parallelization method pattern for each slice is adjusted with respect to the number of execution cores (step S707), and the process of determining the parallelization method {W(n), P(n), M(n)} for each slice is terminated. If n<S is not satisfied in step S706 (No in step S706), the process returns to step S704.

[0156] 22 is a flowchart of the adjustment of the velocity value of the parallel method pattern in the nth slice according to an embodiment. Fig. 22 corresponds to step S704 in Fig. 21.

[0157] As shown in FIG. 22, first, the pattern p determined by the slice number (S) determination unit, the slice adjustment weight (n), and pattern information of the slice number S from the control information database 321 are input (step S801).

[0158] Next, it is determined whether Weight(n)>1 (step S802). If it is determined in step S802 that Weight(n)>1 (Yes in step S802), i=1 is set (step S803), and T(S) ｐ＋ｉ >T(S) ｐ ×Weight(n) (step S804).

[0159] In step S804, T(S) ｐ＋ｉ >T(S) ｐ If it is determined that the value is not *Weight(n) (No in step S804), i=i+1 is set (step S805), and the process returns to step S804.

[0160] In step S804, T(S) ｐ＋ｉ >T(S) ｐ ×Weight(n) (Yes in step S804), Dis1=Abs(T(S) ｐ＋ｉ -T(S) ｐ ×Weight(n)) and Dis2 = Abs(T(S) ｐ＋ｉ－１ -T(S) ｐ ×Weight(n)) is calculated (step S806).

[0161] Next, it is determined whether Dis1<Dis2 (step S807). If it is determined in step S807 that Dis1<Dis2 is true (Yes in step S807), p=p+i is set (step S808), the combination of parallel methods {W(n), P(n), M(n)}=pth pattern in the nth slice is determined (step S809), and the adjustment process for the velocity values ​​is completed.

[0162] In step S807, if it is determined that Dis1 is not smaller than Dis2 (No in step S807), p is set to p-i (step S810), the combination of parallel methods in the n-th slice {W(n), P(n), M(n)} = p-th pattern is determined (step S809), and the adjustment process for the velocity value is completed.

[0163] On the other hand, if it is determined in step S802 that Weight(n) is not greater than 1 (No in step S802), it is determined whether Weight(n) is less than 1 (step S811). If it is determined in step S811 that Weight(n) is less than 1 (Yes in step S811), i=1 is set (step S812), and T(S) ｐ－ｉ <T(S) ｐ ×Weight(n) (step S813).

[0164] In step S813, T(S) ｐ－ｉ <T(S) ｐ If it is determined that the value is not *Weight(n) (No in step S813), i is set to i+1 (step S814), and the process returns to step S813.

[0165] In step S813, T(S) ｐ－ｉ <T(S) ｐ ×Weight(n) (Yes in step S813), Dis1=Abs(T(S) ｐ－ｉ -T(S) ｐ ×Weight(n)) and Dis2 = Abs(T(S) ｐ－ｉ＋１ -T(S) ｐ ×Weight(n)) is calculated (step S815).

[0166] Next, it is determined whether Dis1<Dis2 (step S816). If it is determined in step S816 that Dis1<Dis2 is true (Yes in step S816), p=pi is set (step S817), the pth pattern of the combination of parallel methods {W(n), P(n), M(n)} in the nth slice is determined (step S809), and the adjustment process for the velocity values ​​is completed.

[0167] In step S816, if it is determined that Dis1<Dis2 is not satisfied (No in step S816), p=p-i+1 is set (step S818), the combination of parallel methods in the n-th slice {W(n), P(n), M(n)}=p-th pattern is determined (step S809), and the adjustment process for the velocity value is completed.

[0168] <3.2.9. Flowchart for Adjusting the Parallel Method Pattern of Each Slice Depending on the Number of Execution Cores> FIG. 23 is a flowchart for adjusting the parallel method pattern of each slice depending on the number of execution cores.

[0169] First, the pattern p(n) of each slice number after adjustment for the velocity value, the adjusted weight(n) of each slice, and the pattern information of the slice number S are input from the control information database 321. Also, FirstCheck=0 is set (step S901).

[0170] Next, it is determined whether Σ(C(S)p(n) / S)>C (step S902). If it is determined in step S902 that Σ(C(S)p(n) / S)>C is true (Yes in step S902), Dis(n)=Abs(T(S) ｐ（ｎ）－１ ×Weight(n)-T(S) ｐ（ｎ） ×Weight(n)) is set (step S903), where n=0, 1, . . . , S−1.

[0171] Next, Dis(t)=Min(Dis(n)) is set (step S904), where n=0, 1, ..., S-1. Next, p(t)=p(t)-1 and FirstCheck=0 are set (step S905), and the process returns to step S902.

[0172] On the other hand, if it is determined in step S902 that Σ(C(S)p(n) / S)>C is not satisfied (No in step S902), it is determined whether FirstCheck=1 (step S906). If FirstCheck=1 in step S906 (Yes in step S906), Σ(C(S) ｐ（ｎ） It is determined whether or not the value of the variable C is less than the value of the variable S (step S907).

[0173] In step S907, Σ(C(S) ｐ（ｎ） If it is determined that Dis(n)=Abs(T(S))<C (Yes in step S907), Dis(n)=Abs(T(S)) ｐ（ｎ）＋１ ×Weight(n)-T(S) ｐ（ｎ）×Weight(n)) is set (step S908), where n=0, 1, . . . , S−1.

[0174] Next, Dis(t)=Min(Dis(n)) is set (step S909), where n=0, 1, ..., S-1. Next, p(t)=p(t)+1 and FirstCheck=0 are set (step S910), and the process returns to step S907.

[0175] On the other hand, if FirstCheck is not 1 in step S906 (No in step S906), the parallelization method combination {W(n), P(n), M(n)} in the n-th slice is set to the p-th pattern (step S911), and the process of adjusting the parallelization method pattern for each slice with respect to the number of execution cores is terminated, where n = 0, 1, ..., S-1.

[0176] 4. Use Cases The encoder according to the embodiment is used for real-time encoding in remote play of games.

[0177] Fig. 24 is a diagram showing a game server 111 and a game client 112 in a game system 110 according to an embodiment. Fig. 24 shows a block diagram focusing on a video encoder and decoder. As shown in Fig. 24, the game server 111 is connected to a game client 112 via a network. The game server 111 has a real-time video encoder 113 according to an embodiment, and the game client 112 has a real-time video decoder 114 that performs decoding corresponding to the real-time video encoder 113.

[0178] The game server 111 encodes game video in real time and transmits the encoded video and audio to the game client 112 via the network. The real-time video decoder 114 of the game client 112 decodes the video transmitted from the real-time video encoder 113 and presents the decoded video together with audio to the game player. The game client 112 also transmits player operation information to the game server 111.

[0179] The game server 111 progresses the game based on the player's operation information sent from the game client 112, encodes the game video in accordance with the progress using a real-time video encoder 113, and transmits the encoded video and audio to the game client 112.

[0180] In a game system 110 using VVC, real-time operation is required for the real-time video encoder 113 of the game server 111 and the real-time video decoder 114 of the game client 112. The real-time video encoder 113 and real-time video decoder 114 using the technology of the embodiment can meet this requirement.

[0181] The game server 111 may be an on-premise or cloud server. The parallel control method according to the embodiment can be applied to any tightly coupled HPC (High Performance Computing) system.

[0182] Furthermore, on-premise or cloud computing systems are constructed in a variety of resource configuration formats. Meanwhile, the image size, application program requirements for image quality, network bandwidth, and computing costs also vary. The parallel control method of the embodiment allows for the determination of a scalable and efficient usage (operation) configuration.

[0183] 5. Hardware Configuration FIG. 25 is a hardware configuration diagram showing an example of a computer 1000 that realizes an arithmetic unit of an information processing device equipped with a VVC encoder according to an embodiment.

[0184] The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0185] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0186] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, as well as programs that depend on the hardware of the computer 1000 .

[0187] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an application program according to the present disclosure, which is an example of program data 1450.

[0188] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0189] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, and semiconductor memories.

[0190] Although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, as another example, the CPU 1100 may obtain these programs from other devices via an external network 1550.

[0191] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0192] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0193] Note that the present technology may also be configured as follows: (1) An information processing device comprising: a slice number determination unit that determines a number of slices based on a set number of execution cores, a set coding performance, and a set speed of an input CTU to be encoded; and a combination determination unit that determines a combination of a CTU parallelism number, a CU partition parallelism number, and a CU mode parallelism number to be performed for each slice of the number of slices determined by the slice number determination unit. (2) The information processing device according to (1), further comprising: a generation unit that generates a control information database that includes the number of slices of the CTU, coding performance for the number of slices, speed values ​​having a plurality of speed patterns for the number of slices, and execution core numbers having a plurality of patterns of the number of execution cores for the plurality of speed patterns, wherein each of the plurality of speed value patterns in the control information database has a combination of the CTU parallelism number, the CU partition parallelism number, and the CU mode parallelism number corresponding to the pattern of the plurality of speed values. (3) The information processing device according to (2), wherein the slice number determination unit extracts, from the control information database, an adoptable candidate pattern that satisfies the set number of execution cores, the set encoding performance, and the set speed of the CTU to be encoded input, and determines the number of slices from the extracted adoptable candidate pattern. (4) The information processing device according to (3), wherein the extraction of the adoptable candidate pattern is performed according to the settings of the encoding performance and the speed. (5) The information processing device according to (2), wherein the combination determination unit estimates the processing amount of each slice of the determined number of slices, and determines the CTU parallel number, the CU partition parallel number, and the CU mode parallel number based on the estimated processing amount of each slice. (6) The information processing device according to (5), wherein the processing amount is estimated based on the processing amount of a past frame. (7) The information processing device according to (5), wherein the processing amount is estimated based on at least one of a feature of an original image and a feature of an ME.(8) The information processing device according to (5), wherein the combination determination unit calculates an adjustment value for searching for the speed value for each slice based on the estimated processing amount of each slice, and searches for the pattern of the speed values ​​based on the calculated adjustment value. (9) The information processing device according to (8), wherein the combination determination unit adjusts the pattern of the number of execution cores corresponding to the searched pattern of the speed values ​​based on the determined number of slices so as to match the input set number of execution cores. (10) The information processing device according to (1), wherein the information processing device is an encoder. (11) The information processing device according to (1), wherein the information processing device is a software encoder. (12) The information processing device according to (1), wherein the information processing device is a hardware encoder. (13) The information processing device according to (1), wherein the information processing device is a game server that encodes and distributes game video. (14) An information processing method that determines the number of slices based on the set number of execution cores, set encoding performance, and set speed of the CTU to be input for encoding, and determines a combination of the CTU parallel number, CU partition parallel number, and CU mode parallel number to be performed for each slice of the determined number of slices.

[0194] 310 VVC encoder 311 Control information database generation unit 312 Slice number determination unit 313 Parallel method combination determination unit 321 Control information database 322 Processing amount estimation unit 323 Parallel method determination unit

Claims

1. An information processing device having: a slice number determination unit that determines the number of slices based on the set number of execution cores, set encoding performance, and set speed of the CTU to be subjected to input encoding; and a combination determination unit that determines a combination of the CTU parallel number, CU partition parallel number, and CU mode parallel number to be performed for each slice of the number of slices determined by the slice number determination unit.

2. An information processing device as described in claim 1, comprising a generation unit that generates a control information database having the number of slices of the CTU, encoding performance for the number of slices, speed values ​​having a plurality of speed patterns for the number of slices, and numbers of execution cores having a plurality of patterns of numbers of execution cores for the plurality of speed patterns, wherein each of the plurality of speed value patterns in the control information database has a combination of the CTU parallel number, the CU partition parallel number, and the CU mode parallel number corresponding to the pattern of the plurality of speed values.

3. The information processing device described in claim 2, wherein the slice number determination unit extracts from the control information database a candidate pattern to be adopted that satisfies the set number of execution cores, the set encoding performance, and the set speed of the CTU to be subjected to the input encoding, and determines the number of slices from the extracted candidate pattern to be adopted.

4. The information processing device according to claim 3, wherein the extraction of the adoption candidate pattern is performed according to the settings of the encoding performance and the speed.

5. The information processing device according to claim 2, wherein the combination determination unit estimates the processing amount of each slice of the determined number of slices, and determines the CTU parallel number, the CU partition parallel number, and the CU mode parallel number based on the estimated processing amount of each slice.

6. The information processing device according to claim 5, wherein the processing amount is estimated based on the processing amount of a past frame.

7. The information processing device according to claim 5, wherein the amount of processing is estimated based on at least one of a feature amount of an original image and a feature amount of an ME.

8. The information processing device according to claim 5, wherein the combination determination unit calculates an adjustment value for searching the speed value for each slice based on the estimated processing amount of each slice, and searches for the pattern of the speed value based on the calculated adjustment value.

9. The information processing device according to claim 8, wherein the combination determination unit adjusts the pattern of the number of execution cores corresponding to the pattern of the speed values ​​searched for based on the determined number of slices so that it becomes the input set number of execution cores.

10. The information processing device according to claim 1, wherein the information processing device is an encoder.

11. The information processing device according to claim 1, wherein the information processing device is a software encoder.

12. The information processing device according to claim 1, wherein the information processing device is a hardware encoder.

13. The information processing device according to claim 1, wherein the information processing device is a game server that encodes and distributes game images.

14. An information processing method for determining the number of slices based on the set number of execution cores, set encoding performance, and set speed of the CTU to be encoded as input, and determining a combination of the CTU parallel number, CU partition parallel number, and CU mode parallel number to be performed for each slice of the determined number of slices.

Images