Sub-block transform method, apparatus, electronic device and computer readable medium

Abstract

Embodiments of the present disclosure disclose a sub-block transform method, device, electronic equipment and computer readable medium. A specific implementation of the method comprises: obtaining a to-be-encoded unit and a prediction unit corresponding to the to-be-encoded unit; generating an encoding residual matrix according to the to-be-encoded unit and the prediction unit; performing element replacement processing on the encoding residual matrix according to a predetermined residual threshold to obtain a replaced encoding residual matrix; performing division processing on the replaced encoding residual matrix to obtain each encoding residual sub-matrix; performing transform mode determination on each encoding residual sub-matrix to determine target sub-array division information; and performing sub-block transform processing on the encoding residual matrix according to the target sub-array division information to generate a transformed encoding residual matrix. The implementation can reduce the required computing resources and computing consumption for determining the sub-block transform mode.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to sub-block transformation methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0002] Sub-block transform is an optimization operation performed on the residual matrix in the frequency domain during video coding. There are many sub-block transform modes, each with its own advantages and disadvantages. Currently, the common approach for performing sub-block transform on the residual matrix is ​​to determine the sub-block transform mode through a rate-distortion optimization process before performing the sub-block transform.

[0003] However, when performing sub-block transformation operations using the above method, the following technical problems often arise:

[0004] First, determining the sub-block transformation mode through rate-distortion optimization often involves many processing steps and a long processing flow, leading to a waste of computing resources.

[0005] Second, the accuracy of the sub-block transformation pattern determined by a single threshold is not high, and the characteristics of each residual matrix are not taken into account, resulting in significant information distortion in the determined sub-block transformation pattern.

[0006] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this disclosure provide sub-block transformation methods, apparatuses, electronic devices, and computer-readable media to address one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this disclosure provide a sub-block transformation method, the method comprising: obtaining a unit to be encoded and a prediction unit corresponding to the unit to be encoded, wherein the unit to be encoded satisfies a pre-set encoding unit partitioning condition; generating an encoding residual matrix based on the unit to be encoded and the prediction unit; performing element replacement processing on the encoding residual matrix according to a pre-determined residual threshold to obtain a replaced encoding residual matrix; partitioning the replaced encoding residual matrix to obtain various encoding residual sub-matrices, wherein each obtained encoding residual sub-matrice corresponds to sub-matrix partitioning information; performing transformation mode determination on the various encoding residual sub-matrices to determine target sub-matrix partitioning information; and performing sub-block transformation processing on the encoding residual matrix according to the target sub-matrix partitioning information to generate a transformed encoding residual matrix.

[0010] Secondly, some embodiments of this disclosure provide a sub-block transformation apparatus, comprising: an acquisition unit configured to acquire a unit to be encoded and a prediction unit corresponding to the unit to be encoded, wherein the unit to be encoded satisfies a preset encoding unit partitioning condition; a generation unit configured to generate an encoding residual matrix based on the unit to be encoded and the prediction unit; a replacement unit configured to perform element replacement processing on the encoding residual matrix according to a preset residual threshold to obtain a replaced encoding residual matrix; a partitioning unit configured to partition the replaced encoding residual matrix to obtain various encoding residual sub-matrices, wherein each obtained encoding residual sub-matrice corresponds to sub-matrix partitioning information; a transformation mode determination unit configured to perform transformation mode determination on the various encoding residual sub-matrices to determine target sub-matrix partitioning information; and a sub-block transformation unit configured to perform sub-block transformation processing on the encoding residual matrix according to the target sub-matrix partitioning information to generate a transformed encoding residual matrix.

[0011] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0012] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0013] The above embodiments of this disclosure have the following beneficial effects: the sub-block transformation method of some embodiments of this disclosure can reduce the computing resources and computing power consumption required to determine the sub-block transformation mode. Specifically, the reason for the large computing power consumption in determining the sub-block transformation mode is that determining the sub-block transformation mode through rate-distortion optimization often involves many processing steps and a long processing flow, resulting in a waste of computing resources. Based on this, the sub-block transformation method of some embodiments of this disclosure first obtains the unit to be encoded and the prediction unit corresponding to the unit to be encoded. The unit to be encoded satisfies a pre-set encoding unit partitioning condition. Then, based on the unit to be encoded and the prediction unit, an encoding residual matrix is ​​generated. Thus, encoding residual units can be generated. Afterwards, based on a pre-determined residual threshold, the encoding residual matrix is ​​subjected to element replacement processing to obtain a replaced encoding residual matrix. Thus, the distribution of each matrix element in the entire encoding residual matrix can be determined by first replacing each matrix element with a preset value. Next, the replaced coded residual matrix is ​​partitioned to obtain various coded residual submatrices, each with corresponding subarray partitioning information. This yields the coded residual submatrices and subarray partitioning information representing different subblock transformation modes. Then, transformation mode determination is performed on each of the coded residual submatrices to determine the target subarray partitioning information. This allows for the determination of information distortion caused by each subblock transformation mode through the coded residual submatrices, identifying the subarray partitioning information corresponding to the subblock transformation mode with the smallest information distortion as the target subarray partitioning information. Finally, based on the target subarray partitioning information, the coded residual matrix undergoes subblock transformation processing to generate the transformed coded residual matrix. Thus, the coded residual matrix can be transformed using the optimal subarray transformation mode represented by the target subarray partitioning information to generate the transformed coded residual matrix. Because the subblock transformation mode determination considers the distribution of residual values ​​across the entire residual matrix, the determination process is shorter and the processing steps are simpler, thereby reducing computational power consumption. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0015] Figure 1 This is a flowchart of some embodiments of the sub-block transformation method according to this disclosure;

[0016] Figure 2 These are schematic diagrams illustrating the structure of some embodiments of the sub-block transformation device according to this disclosure;

[0017] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0023] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Figure 1 A flow 100 of some embodiments of a sub-block transformation method according to the present disclosure is shown. The sub-block transformation method includes the following steps:

[0025] Step 101: Obtain the unit to be encoded and the prediction unit corresponding to the unit to be encoded.

[0026] In some embodiments, the execution entity (e.g., a computing device) of the sub-block transformation method can obtain the unit to be encoded and the prediction unit corresponding to the unit to be encoded. The unit to be encoded satisfies a pre-defined encoding unit partitioning condition. The execution entity can be a server or a client. The unit to be encoded can be an image block obtained by partitioning the image to be encoded during the video encoding process, serving as the basic unit for subsequent processing. The prediction unit can be a prediction unit obtained by encoding prediction of the unit to be encoded using an encoding prediction algorithm. The encoding unit partitioning condition can be that the width and height of the unit to be encoded are less than or equal to a preset value. As an example, the preset value can be 64.

[0027] Step 102: Generate the coding residual matrix based on the unit to be coded and the prediction unit.

[0028] In some embodiments, the execution entity can generate a coding residual matrix based on the unit to be encoded and the prediction unit. In practice, the execution entity can subtract the unit to be encoded from the prediction unit to generate the coding residual matrix.

[0029] Step 103: Based on the predetermined residual threshold, perform element replacement processing on the encoded residual matrix to obtain the replaced encoded residual matrix.

[0030] In some embodiments, the execution entity may perform element replacement processing on the encoded residual matrix according to a predetermined residual threshold to obtain a replaced encoded residual matrix.

[0031] In some optional implementations of certain embodiments, the execution entity may perform element replacement processing on the above-mentioned encoding residual matrix according to a predetermined residual threshold through the following steps to obtain the replaced encoding residual matrix:

[0032] First, for each element in the above encoded residual submatrix, perform the following replacement steps:

[0033] The first sub-step is to determine the absolute value of the above matrix elements as the target value.

[0034] The second sub-step involves replacing the matrix elements with preset replacement values ​​in response to determining that the target value is less than the residual threshold. For example, the residual threshold could be 5. As an example, the replacement value could be 0.

[0035] Step 104: Divide the replaced coding residual matrix to obtain each coding residual submatrix.

[0036] In some embodiments, the execution entity may partition the replaced coding residual matrix to obtain various coding residual submatrices. Each obtained coding residual submatrix corresponds to submatrix partitioning information. This submatrix partitioning information may be information characterizing the partitioning method of the corresponding coding residual submatrix.

[0037] In some optional implementations of certain embodiments, the execution entity may perform the following steps to partition the replaced coding residual matrix to obtain various coding residual sub-matrices:

[0038] The first step is to determine the number of rows and columns of the above-mentioned encoded residual matrix.

[0039] The second step is to determine the median values ​​of the column and row numbers based on the aforementioned matrix column and row numbers. In practice, firstly, the executing entity can determine the median value of the matrix column number. Then, the executing entity can determine half of the matrix row number as the median value of the row number.

[0040] The third step involves partitioning the encoded residual matrix based on the aforementioned number of rows, columns, the median of the row count, and the median of the column count, resulting in various encoded residual sub-matrices. In practice, the executing entity can first determine the partitioning information for the first, second, third, fourth, fifth, sixth, seventh, and eighth sub-matrices according to the aforementioned number of rows, columns, the median of the row count, and the median of the column count. Each of these sub-matrix partitioning information represents the position range of the corresponding encoded residual sub-matrix within the encoded residual matrix. For example, the first sub-matrix partitioning information might be ((1, median row count), (1, median column count)), which indicates that the encoded residual sub-matrix corresponding to this first partitioning information is obtained by partitioning from the first row to the row corresponding to the median of the row count, and from the first column to the column corresponding to the median of the column count in the encoded residual matrix. The second subarray partitioning information mentioned above can be ((1, median row count), (median column count, number of matrix columns)). The third subarray partitioning information mentioned above can be ((median row count, number of matrix rows), (1, median column count)). The fourth subarray partitioning information mentioned above can be ((median value of row count, number of matrix rows), ((median value of column count, number of matrix columns)). The fifth subarray partitioning information mentioned above can be ((1, median value of row count), (1, number of matrix columns)). The sixth subarray partitioning information mentioned above can be ((median value of row count, number of matrix rows), (1, number of matrix columns)). The seventh subarray partitioning information mentioned above can be ((1, number of matrix rows), (1, median value of column count)). The eighth subarray partitioning information mentioned above can be ((1, number of matrix rows), (median value of column count, number of matrix columns)). Then, the execution entity can partition the above-mentioned coding residual matrix according to the determined subarray partitioning information to obtain each coding residual submatrix. Finally, the execution entity can also treat the above-mentioned coding residual matrix as a coding residual submatrix to obtain each coding residual matrix.

[0041] Step 105: Perform transformation mode determination on each encoded residual submatrix to determine the target submatrix partitioning information.

[0042] In some embodiments, the execution entity may perform transformation mode determination on each of the above-mentioned coded residual submatrices to determine the target submatrix partitioning information.

[0043] In some optional implementations of certain embodiments, the execution entity may determine the target subarray partitioning information by performing transformation mode determination on each of the above-mentioned coded residual submatrices through the following steps: determining the target subarray partitioning information by performing transformation mode determination on each of the above-mentioned coded residual submatrices according to a predetermined mode determination threshold.

[0044] Optionally, the aforementioned pattern determination threshold can be a replacement ratio coefficient threshold. Specifically, the replacement ratio coefficient can be a threshold representing the proportion of matrix elements in the encoded residual submatrix that have been replaced with the preset replacement value to the total number of matrix elements.

[0045] In some optional implementations of certain embodiments, the execution entity may determine the target submatrix partitioning information by performing transformation mode determination on each of the above-mentioned encoded residual submatrices according to a predetermined mode determination threshold through the following steps:

[0046] The first step is to perform the following steps for each of the above-mentioned encoded residual submatrices:

[0047] The first sub-step involves determining the number of elements in each encoded residual submatrix included in the aforementioned encoded residual submatrix as the total number of submatrix elements.

[0048] The second sub-step involves determining the number of elements in the target submatrix by counting the number of each element in the aforementioned encoded residual submatrix that satisfies the element value matching condition corresponding to the aforementioned preset replacement value. The aforementioned element value matching condition can be that the elements of the encoded residual submatrix are equal to the aforementioned preset replacement value.

[0049] The third sub-step is to determine the ratio of the number of elements in the target submatrix to the total number of elements in the submatrix as the replacement ratio coefficient.

[0050] The fourth sub-step is to determine the replacement ratio coefficient as the target ratio coefficient in response to determining that the replacement ratio coefficient is less than the replacement ratio coefficient threshold.

[0051] The second step involves determining the target scaling factor as less than or equal to a first preset number, and then defining the encoding residual submatrix corresponding to the target scaling factor as the target residual submatrix. The first preset number can be 1.

[0052] Third, in response to the determination that the number of the target proportional coefficients is greater than the first preset number mentioned above, the following steps are performed:

[0053] The first sub-step involves sorting the determined target proportion coefficients to obtain a target proportion coefficient sequence. In practice, the aforementioned implementing entity can sort the determined target proportion coefficients in descending order to obtain the target proportion coefficient sequence.

[0054] The second sub-step involves determining the encoded residual submatrix corresponding to the target proportional coefficients in the aforementioned target proportional coefficient sequence that satisfy a preset positional condition as the target residual submatrix. The preset positional condition can be that the target proportional coefficient corresponding to the target residual submatrix is ​​the first target proportional coefficient in the aforementioned target proportional coefficient sequence.

[0055] The third sub-step involves determining the submatrix partitioning information corresponding to the aforementioned target residual submatrix as the target submatrix partitioning information.

[0056] Optionally, the pattern determination threshold can be a residual sum of squares threshold. Specifically, the residual sum of squares threshold can be the threshold of the sum of squares of each residual value in the aforementioned encoded residual submatrix.

[0057] In some optional implementations of certain embodiments, the execution entity may determine the target submatrix partitioning information by performing transformation mode determination on each of the above-mentioned encoded residual submatrices according to a predetermined mode determination threshold through the following steps:

[0058] The first step is to perform the following steps for each of the above-mentioned encoded residual submatrices:

[0059] The first sub-step involves determining the sum of squared residuals of the elements of each encoded residual submatrix included in the aforementioned encoded residual submatrix as the submatrix residual sum of squares. In practice, the executing entity can obtain the submatrix residual sum of squares by squaring the elements of each encoded residual submatrix included in the aforementioned encoded residual submatrix and by adding the results of squaring each residual value.

[0060] The first sub-step involves determining the sum of squared residuals of the submatrix as the target sum of squared residuals in response to determining that the sum of squared residuals of the submatrix is ​​less than the sum of squared residuals threshold.

[0061] The second step involves determining that the number of the determined target residual sums of squares is less than or equal to a second preset number, and then defining the encoded residual submatrix corresponding to the determined target residual sums of squares as the target residual submatrix. The aforementioned second preset number can be 1.

[0062] Third, in response to the determination that the number of the target proportional coefficient is greater than the second preset number mentioned above, the following steps are performed:

[0063] The first sub-step involves sorting the determined target residual sums of squares to obtain a sequence of target residual sums of squares. In practice, the aforementioned execution entity can sort the determined target residual sums of squares in descending order to obtain a sequence of target residual sums of squares.

[0064] The first sub-step involves determining the encoded residual submatrix corresponding to the target residual sum of squares that satisfies a preset positional condition in the aforementioned target residual sum of squares sequence as the target residual submatrix. The preset positional condition can be that the target residual sum of squares corresponding to the target residual submatrix is ​​the first target residual sum of squares in the aforementioned target residual sum of squares sequence.

[0065] The fourth step is to determine the submatrix partitioning information corresponding to the target residual submatrix as the target submatrix partitioning information.

[0066] Optionally, the aforementioned residual threshold and adjustment threshold can be determined through the following steps:

[0067] The first step is to obtain a preset residual threshold sequence, a preset pattern determination threshold sequence, and a sample residual matrix. The preset residual threshold sequence can be any preset residual value threshold. The preset pattern determination threshold sequence can be any preset pattern determination threshold. For example, the preset residual threshold sequence could be (5, 6, 7, 8, 9, 10).

[0068] It should be noted that the aforementioned preset mode determination threshold sequence can be a preset residual sum of squares threshold sequence or a preset replacement ratio coefficient threshold sequence, depending on the implementation method of determining the target subarray partitioning information. As an example, the aforementioned preset replacement ratio coefficient threshold sequence can be (40%, 50%, 60%, 70%, 80%, 90%).

[0069] The second step is to perform the following residual threshold determination steps for each preset residual threshold in the above preset residual threshold sequence:

[0070] The first sub-step involves performing the following mode determination step for each preset mode determination threshold in the above preset mode determination threshold sequence:

[0071] Sub-step one: Based on the aforementioned preset residual threshold and the aforementioned preset mode determination threshold, perform sub-block transformation processing on the aforementioned sample residual matrix to obtain the transformed sample residual matrix. In practice, the implementation method of "performing sub-block transformation processing on the aforementioned sample residual matrix based on the aforementioned preset residual threshold and the aforementioned preset mode determination threshold to obtain the transformed sample residual matrix" can refer to the two optional implementation methods of step 105 and the implementation steps of step 106, and will not be repeated here.

[0072] Sub-step two involves determining the sample encoding distortion information based on the transformed sample residual matrix. This sample encoding distortion information can be the rate distortion corresponding to the transformed sample residual matrix. In practice, the executing entity can determine the rate distortion of the transformed sample residual matrix as the sample encoding distortion information.

[0073] The third step is to sort the determined sample coding distortion information to obtain the sample coding distortion information sequence.

[0074] The fourth step involves determining the preset residual threshold and preset mode determination threshold corresponding to the sample coding distortion information in the above sample coding distortion information sequence that satisfy the preset position condition as the residual threshold and mode determination threshold, respectively. The preset position condition can be that the sample coding distortion information corresponding to the residual threshold and mode determination threshold is the first sample coding distortion information in the sample coding distortion information sequence.

[0075] In some optional implementations of certain embodiments, the execution entity may determine the target submatrix partitioning information by performing transformation mode determination on each of the above-mentioned encoded residual submatrices through the following steps:

[0076] The first step is to perform the following steps for each of the above-mentioned encoded residual submatrices:

[0077] The first sub-step involves determining the coding residual distortion information based on the aforementioned coding residual matrix and the aforementioned coding residual submatrix. In practice, firstly, the executing entity can set each corresponding matrix element in the aforementioned coding residual matrix to 0 according to the position range represented by the submatrix corresponding to the coding residual submatrix, thus obtaining the transformed coding residual matrix. Then, the executing entity can subtract the corresponding matrix elements of the aforementioned coding residual matrix and the transformed coding residual matrix to obtain the differences between each matrix element. Finally, the executing entity can determine the coding residual distortion information as the sum of the squares of the differences between each matrix element.

[0078] The second sub-step involves generating a residual matrix to be determined based on the aforementioned coded residual sub-matrix and the aforementioned coded residual matrix. In practice, the executing entity can set each matrix element within the position range represented by the sub-matrix partitioning information corresponding to the aforementioned coded residual sub-matrix in the aforementioned coded residual matrix to 0 to generate the residual matrix to be determined.

[0079] The third sub-step involves determining the sum of squares of the residuals of the submatrix before replacement, based on the aforementioned encoded residual submatrix and the aforementioned encoded residual matrix. In practice, the executing entity can square each matrix element in the aforementioned encoded residual matrix within the position range represented by the submatrix partitioning information corresponding to the aforementioned encoded residual submatrix, and determine the sum of the squared values ​​as the sum of squares of the residuals of the submatrix before replacement.

[0080] The fourth sub-step involves determining the sum of the squared residuals of all elements in the aforementioned residual matrix to be determined as the sum of squared residuals. In practice, the executing entity can determine the sum of squared residuals by squaring each element in the aforementioned residual matrix and summing the squared results.

[0081] The fifth sub-step involves determining the variance of each element of the residual matrix to be determined, which is included in the aforementioned residual matrix to be determined, as the variance of the residual matrix to be determined. In practice, the executing entity can determine the variance of each element of the residual matrix to be determined, which is included in the aforementioned residual matrix to be determined, as the variance of the residual matrix to be determined.

[0082] The sixth sub-step involves sorting the elements of the residual matrix to be determined, resulting in a sequence of residual matrix elements. In practice, the executing entity can sort the elements of the residual matrix to be determined in descending order to obtain the sequence of residual matrix elements.

[0083] The seventh sub-step involves selecting the first residual matrix element from the sequence of elements to be determined that satisfies the first order condition. This first order condition can be that the first residual matrix element is the first element in the sequence of elements to be determined.

[0084] The eighth sub-step involves selecting, from the aforementioned sequence of residual matrix elements to be determined, an element that satisfies the second order condition as the second residual matrix element to be determined. The second order condition can be that the second residual matrix element to be determined is the last residual matrix element in the aforementioned sequence.

[0085] The ninth sub-step is to take the difference between the first residual matrix element to be determined and the second residual matrix element to be determined as the residual difference value to be determined.

[0086] The tenth sub-step is to determine the ratio of the first residual matrix element to be determined to the sum of squares of the residuals to be determined as the residual ratio to be determined.

[0087] The eleventh sub-step involves selecting from the aforementioned sequence of residual matrix elements that satisfy the positional range condition as target residual matrix elements. The positional range condition can be that the position of the target residual matrix element in the sequence of residual matrix elements is less than or equal to a preset positional position. For example, the preset positional position can be 10.

[0088] The twelfth sub-step is to determine the average value of the residual matrix elements of each of the above targets as the average value of the residual to be determined.

[0089] The thirteenth sub-step involves determining the coordinate positions of each element in the residual matrix to be determined for each target as the position information of the residual. These coordinate positions can be the coordinates of the corresponding element in the residual matrix to be determined.

[0090] The fourteenth sub-step involves determining the sum of squares of the residuals of the submatrix before replacement, the sum of squares of the residuals to be determined, the sum of variances of the residuals to be determined, the difference of the residuals to be determined, the ratio of the residuals to be determined, the average value of the residuals to be determined, and the position information of the residuals to be determined as the feature information of the residual matrix to be determined.

[0091] The fifteenth sub-step involves inputting the feature information of the residual matrix to be determined into a pre-trained transformed coding distortion prediction model to obtain the transformed bitrate prediction value. This transformed coding distortion prediction model can be used to predict the bitrate difference of the residual matrix after sub-block transformation. This transformed coding distortion prediction model can be a neural network model that takes the residual matrix feature information as input and outputs the transformed bitrate prediction value. For example, this transformed coding distortion prediction model can be a decision tree model. This transformed coding distortion prediction model can be obtained by training on each sample residual matrix, the feature information of each residual matrix corresponding to each sample residual matrix, and the bitrate differences of each sample residual matrix. The bitrate difference can be the difference between the bitrate determined by the RDO (Rate-Distortion Optimization) process after sub-block transformation of the corresponding sample residual matrix and the bitrate determined by directly performing the RDO process without sub-block transformation.

[0092] The sixteenth sub-step involves determining the transformed bitrate prediction value as the target bitrate prediction value in response to the determination that the transformed bitrate prediction value is less than the coding residual distortion information.

[0093] The second step involves determining that the number of determined target bitrate prediction values ​​is less than or equal to a third preset number, and then defining the coding residual submatrix corresponding to the determined target bitrate prediction values ​​as the target residual submatrix. The third preset number can be 1.

[0094] Third, in response to the determination that the number of the determined target bitrate prediction values ​​is greater than the third preset number mentioned above, the following steps are performed:

[0095] The first sub-step involves sorting the determined target bitrate prediction values ​​to obtain a sequence of target bitrate prediction values. In practice, the execution entity can sort the determined target bitrate prediction values ​​in descending order to obtain the target bitrate prediction value sequence.

[0096] The second sub-step involves determining the coding residual submatrix corresponding to the target bitrate prediction value that satisfies the preset bit order condition in the above target bitrate prediction value sequence as the target residual submatrix. The preset bit order condition can be that the target bitrate prediction value corresponding to the target residual submatrix is ​​the first target bitrate prediction value in the above target bitrate prediction value sequence.

[0097] The fourth step is to determine the submatrix partitioning information corresponding to the target residual submatrix as the target submatrix partitioning information.

[0098] The above embodiments, as an inventive point of this disclosure, can solve the second technical problem mentioned in the background art: "The accuracy of the sub-block transformation pattern determined by a single threshold is not high, and the various residual features of the residual matrix are not considered, resulting in significant information distortion generated by the determined sub-block transformation pattern, leading to a decrease in image quality after subsequent encoding." The factors leading to significant distortion from sub-block transformation and a decrease in image quality after encoding are often as follows: the accuracy of the sub-block transformation pattern determined by a single threshold is not high, and the various residual features of the residual matrix are not considered. If these factors are resolved, the distortion generated by sub-block transformation and the decrease in image quality after encoding can be reduced. To achieve this effect, this disclosure determines the various features of each encoded residual sub-matrix. By using the sum of squared residuals of the submatrix before replacement, the sum of squared residuals to be determined, the sum of variances of the residuals to be determined, the difference of the residuals to be determined, the ratio of the residuals to be determined, the average value of the residuals to be determined, and the location information of the residuals to be determined as feature vectors and a pre-trained post-transformation coding distortion prediction model, the coding distortion caused by the sub-block transformation mode represented by each coding residual submatrix can be determined. This allows for the identification of a better sub-block transformation mode, thereby reducing the distortion caused by sub-block transformation and minimizing the degradation of image quality after coding.

[0099] Step 106: Based on the target subarray partitioning information, perform sub-block transformation processing on the coding residual matrix to generate the transformed coding residual matrix.

[0100] In some embodiments, the execution entity can perform sub-block transformation processing on the coding residual matrix based on the target subarray partitioning information to generate a transformed coding residual matrix. In practice, the execution entity can set each matrix element in the coding residual matrix within the matrix position range represented by the target subarray partitioning information to 0, thereby performing sub-block transformation processing on the coding residual matrix to obtain the transformed coding residual matrix.

[0101] The above embodiments of this disclosure have the following beneficial effects: the sub-block transformation method of some embodiments of this disclosure can reduce the computing resources and computing power consumption required to determine the sub-block transformation mode. Specifically, the reason for the large computing power consumption in determining the sub-block transformation mode is that determining the sub-block transformation mode through rate-distortion optimization often involves many processing steps and a long processing flow, resulting in a waste of computing resources. Based on this, the sub-block transformation method of some embodiments of this disclosure first obtains the unit to be encoded and the prediction unit corresponding to the unit to be encoded. The unit to be encoded satisfies a pre-set encoding unit partitioning condition. Then, based on the unit to be encoded and the prediction unit, an encoding residual matrix is ​​generated. Thus, encoding residual units can be generated. Afterwards, based on a pre-determined residual threshold, the encoding residual matrix is ​​subjected to element replacement processing to obtain a replaced encoding residual matrix. Thus, the distribution of each matrix element in the entire encoding residual matrix can be determined by first replacing each matrix element with a preset value. Next, the replaced coded residual matrix is ​​partitioned to obtain various coded residual submatrices, each with corresponding subarray partitioning information. This yields the coded residual submatrices and subarray partitioning information representing different subblock transformation modes. Then, transformation mode determination is performed on each of the coded residual submatrices to determine the target subarray partitioning information. This allows for the determination of information distortion caused by each subblock transformation mode through the coded residual submatrices, identifying the subarray partitioning information corresponding to the subblock transformation mode with the smallest information distortion as the target subarray partitioning information. Finally, based on the target subarray partitioning information, the coded residual matrix undergoes subblock transformation processing to generate the transformed coded residual matrix. Thus, the coded residual matrix can be transformed using the optimal subarray transformation mode represented by the target subarray partitioning information to generate the transformed coded residual matrix. Because the subblock transformation mode determination considers the distribution of residual values ​​across the entire residual matrix, the determination process is shorter and the processing steps are simpler, thereby reducing computational power consumption.

[0102] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a sub-block transformation device, which are similar to... Figure 1 Corresponding to the method embodiments shown, this sub-block conversion device can be specifically applied to various electronic devices.

[0103] like Figure 2As shown, the sub-block transformation device 200 in some embodiments includes: an acquisition unit 201, a generation unit 202, a replacement unit 203, a division unit 204, a transformation mode determination unit 205, and a sub-block transformation unit 206. The acquisition unit 201 is configured to acquire the unit to be encoded and the prediction unit corresponding to the unit to be encoded, wherein the unit to be encoded satisfies a pre-set encoding unit partitioning condition; the generation unit 202 is configured to generate an encoding residual matrix based on the unit to be encoded and the prediction unit; the replacement unit 203 is configured to perform element replacement processing on the encoding residual matrix according to a pre-determined residual threshold to obtain a replaced encoding residual matrix; the partitioning unit 204 is configured to partition the replaced encoding residual matrix to obtain various encoding residual submatrices, wherein each obtained encoding residual submatric corresponds to submatrix partitioning information; the transformation mode determination unit 205 is configured to perform transformation mode determination on the various encoding residual submatrices to determine the target submatrix partitioning information; and the sub-block transformation unit 206 is configured to perform sub-block transformation processing on the encoding residual matrix according to the target submatrix partitioning information to generate a transformed encoding residual matrix.

[0104] It is understandable that the units described in the sub-block transformation device 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the sub-block transformation device 200 and the units contained therein, and will not be repeated here.

[0105] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0106] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 or a program loaded from a storage device 308 into a random access memory 303. The random access memory 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, the read-only memory 302, and the random access memory 303 are interconnected via a bus 304. An input / output interface 305 is also connected to the bus 304.

[0107] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0108] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a read-only memory 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0109] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0110] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0111] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a unit to be encoded and a prediction unit corresponding to the unit to be encoded, wherein the unit to be encoded satisfies a pre-set encoding unit partitioning condition; generate an encoding residual matrix based on the unit to be encoded and the prediction unit; perform element replacement processing on the encoding residual matrix according to a pre-determined residual threshold to obtain a replaced encoding residual matrix; partition the replaced encoding residual matrix to obtain various encoding residual sub-matrices, wherein each obtained encoding residual sub-matrice corresponds to subarray partitioning information; perform transformation mode determination on each of the encoding residual sub-matrices to determine target subarray partitioning information; and perform sub-block transformation processing on the encoding residual matrix according to the target subarray partitioning information to generate a transformed encoding residual matrix.

[0112] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0114] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including an acquisition unit, a generation unit, a replacement unit, a partitioning unit, a transformation mode determination unit, and a sub-block transformation unit. The names of these units do not necessarily limit the unit itself; for example, an acquisition unit can also be described as "a unit that acquires the unit to be encoded and the prediction unit corresponding to the unit to be encoded."

[0115] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0116] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A sub-block transformation method, comprising: Obtain the unit to be encoded and the prediction unit corresponding to the unit to be encoded, wherein the unit to be encoded satisfies a pre-set encoding unit partitioning condition; Subtract the prediction unit from the unit to be encoded to generate the encoding residual matrix; Based on a predetermined residual threshold, the encoded residual matrix is ​​subjected to element replacement processing to obtain a replaced encoded residual matrix, including: for each matrix element in the encoded residual matrix, the following replacement steps are performed: the absolute value of the matrix element is determined as a target value; in response to determining that the target value is less than the residual threshold, the matrix element is replaced with a preset replacement value, wherein the preset replacement value is 0; The replaced coding residual matrix is ​​partitioned to obtain various coding residual submatrices, wherein each obtained coding residual submatric corresponds to submatrix partitioning information; Transformation mode determination is performed on each of the encoded residual submatrices to determine the target submatrix partitioning information; Set each matrix element in the encoded residual matrix within the matrix position range represented by the target subarray partitioning information to 0, so as to perform sub-block transformation processing on the encoded residual matrix and obtain the transformed encoded residual matrix.

2. The method according to claim 1, wherein, The step of partitioning the replaced coding residual matrix to obtain various coding residual sub-matrices includes: Determine the number of rows and columns of the encoded residual matrix; Based on the number of columns and rows of the matrix, determine the median value of the number of columns and the median value of the number of rows; The coding residual matrix is ​​divided according to the number of rows, the number of columns, the median value of the number of rows, and the median value of the number of columns to obtain various coding residual submatrices.

3. The method according to claim 2, wherein, The step of determining the transformation mode of each encoded residual submatrix to determine the target submatrix partitioning information includes: Based on a predetermined mode determination threshold, the transformation mode of each encoded residual submatrix is ​​determined to identify the target submatrix partitioning information.

4. The method according to claim 3, wherein, The mode determination threshold is a replacement scaling factor threshold; and the process of performing transformation mode determination on each encoded residual submatrix according to the predetermined mode determination threshold to determine the target submatrix partitioning information includes: For each of the encoded residual submatrices, perform the following steps: The number of elements in each encoded residual submatrix included in the encoded residual submatrix is ​​determined as the total number of submatrix elements; The number of elements in the target submatrix is ​​determined by the number of each element in the encoded residual submatrix that satisfies the element value matching condition corresponding to the preset replacement value. The ratio of the number of elements in the target submatrix to the total number of elements in the submatrix is ​​determined as the replacement ratio coefficient; In response to determining that the replacement ratio coefficient is less than the replacement ratio coefficient threshold, the replacement ratio coefficient is determined as the target ratio coefficient; In response to the determination that the number of the determined target scaling coefficients is less than or equal to the first preset number, the coding residual submatrix corresponding to the determined target scaling coefficients is determined as the target residual submatrix; In response to the determination that the number of the determined target proportional coefficients is greater than the first preset number, the following steps are performed: The determined target proportion coefficients are sorted to obtain a target proportion coefficient sequence; The encoded residual submatrix corresponding to the target proportional coefficients that satisfy the preset position order conditions in the target proportional coefficient sequence is determined as the target residual submatrix; The submatrix partitioning information corresponding to the target residual submatrix is ​​determined as the target submatrix partitioning information.

5. The method according to claim 3, wherein, The pattern determination threshold is the residual sum of squares threshold; The process of performing transformation mode determination on each encoded residual submatrix according to a predetermined mode determination threshold to determine the target submatrix partitioning information includes: For each of the encoded residual submatrices, perform the following steps: The sum of squared residuals of each element of the encoded residual submatrix is ​​determined as the sum of squared residuals of the submatrix. In response to determining that the sum of squared residuals of the submatrix is ​​less than the residual sum of squared threshold, the sum of squared residuals of the submatrix is ​​determined as the target residual sum of squared; In response to determining that the number of the determined target residual sums of squares is less than or equal to the second preset number, the encoded residual submatrix corresponding to the determined target residual sums of squares is determined as the target residual submatrix; In response to the determination that the number of the determined target scaling factor is greater than the second preset number, the following steps are performed: The determined target residual sums of squares are sorted to obtain a sequence of target residual sums of squares; The encoded residual submatrix corresponding to the target residual sum of squares that satisfies the preset position condition in the target residual sum of squares sequence is determined as the target residual submatrix; The submatrix partitioning information corresponding to the target residual submatrix is ​​determined as the target submatrix partitioning information.

6. The method according to any one of claims 3-5, wherein, The residual threshold and the mode determination threshold are determined through the following steps: Obtain the preset residual threshold sequence, the preset pattern determination threshold sequence, and the sample residual matrix; For each preset residual threshold in the preset residual threshold sequence, the following residual threshold determination steps are performed: For each preset mode determination threshold in the preset mode determination threshold sequence, the following mode determination threshold determination steps are performed: Based on the preset residual threshold and the preset mode determination threshold, the sample residual matrix is ​​subjected to sub-block transformation processing to obtain the transformed sample residual matrix; Based on the transformed sample residual matrix, the sample encoding distortion information is determined; The determined sample coding distortion information is sorted to obtain a sample coding distortion information sequence; The preset residual threshold and preset mode determination threshold corresponding to the sample encoding distortion information that meets the preset position condition in the sample encoding distortion information sequence are determined as the residual threshold and mode determination threshold.

7. A sub-block conversion device, comprising: The acquisition unit is configured to acquire a unit to be encoded and a prediction unit corresponding to the unit to be encoded, wherein the unit to be encoded satisfies a pre-set encoding unit partitioning condition; The generating unit is configured to subtract the predicting unit from the unit to be encoded to generate an encoding residual matrix; The replacement unit is configured to perform element replacement processing on the encoded residual matrix according to a predetermined residual threshold to obtain a replaced encoded residual matrix, including: for each matrix element in the encoded residual matrix, performing the following replacement steps: determining the absolute value of the matrix element as a target value; in response to determining that the target value is less than the residual threshold, replacing the matrix element with a preset replacement value, wherein the preset replacement value is 0; The partitioning unit is configured to partition the replaced coding residual matrix to obtain various coding residual submatrices, wherein each obtained coding residual submatric corresponds to submatrix partitioning information; The transformation mode determination unit is configured to perform transformation mode determination on each of the encoded residual submatrices to determine the target submatrix partitioning information. The sub-block transformation unit is configured to set each matrix element in the encoded residual matrix within the matrix position range represented by the target subarray partitioning information to 0, so as to perform sub-block transformation processing on the encoded residual matrix and obtain the transformed encoded residual matrix.

8. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images