VVC coding unit fast division method and system based on image frequency domain characteristics

Abstract

The present application discloses a method and system for fast partitioning of VVC coding units based on image frequency domain features. The method uses a first image frequency domain feature, a first classification label and a first weight to train a group of binary classifiers based on a lightweight gradient boosting model to obtain a group of initial binary classifiers; performs feature selection through a group of initial binary classifiers, and uses the selected features to train a group of initial binary classifiers to obtain a group of reduced dimension binary classifiers; adjusts parameters of a group of reduced dimension binary classifiers to reduce the size of a group of reduced dimension binary classifiers to obtain a group of partition decision classifiers; initializes online sample data, and makes decisions on the partition mode in each partition decision classifier based on the online sample data to obtain a group of target partition modes; and quickly partitions the VVC coding unit based on a group of target partition modes. The present application improves the partitioning efficiency and decision accuracy of the coding unit, ensures video quality and bit rate, and reduces encoding time.

Description

Technical Field

[0001] The present application relates to the field of video coding technology, and in particular to a method and system for fast division of VVC coding units based on image frequency domain features. Background Art

[0002] The optimization directions of the encoder include compression efficiency and encoding speed. Specifically, compression efficiency refers to the balance between image quality and bit rate, which is manifested as a smaller rate-distortion cost (RD Cost) during the encoding process and a BDBR (Bjontegaard Delta Bit Rate) less than 0 in the encoding result. The encoding speed is measured by the encoding time.

[0003] The original rate-distortion optimization (RDO) method performs the complete encoding process for all allowed partitioning modes to obtain the RD Cost, and obtains the optimal mode by comparing the cost. Although the optimal partitioning mode can always be found, it leads to great time complexity and poor cost performance. The fast algorithm seeks a balance between time and compression performance, that is, to reduce the encoding time while ensuring high image quality and low bit rate (that is, the RD Cost is as small as possible), but it often cannot fully remove the partitioning modes with low probability, resulting in little time saving, or the decision accuracy is poor and the better partitioning modes are skipped, resulting in a decrease in video quality and an increase in bit rate.

[0004] Therefore, the above-mentioned existing coding unit division method is not efficient in dividing the coding units, and the decision accuracy is poor, resulting in a decrease in video quality and an increase in bit rate. Summary of the invention

[0005] The present application aims to propose a method and system for quickly dividing VVC coding units based on image frequency domain features, which can improve the efficiency of coding unit division, improve decision accuracy, and significantly reduce encoding time while ensuring video quality and bit rate.

[0006] In a first aspect, an embodiment of the present application provides a method for quickly dividing a VVC coding unit based on image frequency domain features, the method comprising:

[0007] Using an encoder to run on each video in a video training set, obtaining a first image frequency domain feature, a first classification label, and a first weight corresponding to the video training set;

[0008] Using the first image frequency domain feature, the first classification label and the first weight to train a set of binary classifiers based on a lightweight gradient boosting model to obtain a set of initial binary classifiers, wherein the binary classifiers based on the lightweight gradient boosting model include two division modes;

[0009] Performing feature selection through the set of initial binary classifiers, and using the selected features to train the set of initial binary classifiers to obtain a set of reduced-dimensional binary classifiers;

[0010] Adjusting parameters of the set of dimension reduction binary classifiers to reduce the size of the set of dimension reduction binary classifiers to obtain a set of partition decision classifiers;

[0011] Initializing online sample data, and making decisions on the partitioning modes in each of the partitioning decision classifiers according to the online sample data to obtain a set of target partitioning modes;

[0012] The VVC coding unit is quickly divided according to the set of target division modes.

[0013] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0014] The method uses an encoder to run on each video in a video training set to obtain the first image frequency domain features, the first classification label and the first weight corresponding to the video training set; uses the first image frequency domain features, the first classification label and the first weight to train a set of binary classifiers based on a lightweight gradient boosting model to obtain a set of initial binary classifiers, wherein the binary classifiers based on the lightweight gradient boosting model include two partitioning modes; performs feature selection through a set of initial binary classifiers, and uses the selected features to train a set of initial binary classifiers to obtain a set of dimension reduction binary classifiers; adjusts parameters of a set of dimension reduction binary classifiers to reduce the size of a set of dimension reduction binary classifiers to obtain a set of partition decision classifiers; initializes online sample data, and makes decisions on the partition mode in each partition decision classifier according to the online sample data to obtain a set of target partition modes; and quickly partitions the VVC coding unit according to a set of target partition modes. In this way, the frequency domain feature decision partitioning is introduced to improve the multi-scale expression ability, information density and robustness of the features, and improve the performance of the partition decision model. Then, by performing feature selection on the initial binary classifier and adjusting the parameters of a set of dimension reduction binary classifiers, the size of a set of dimension reduction binary classifiers can be reduced, which can effectively select features and control the model size, save decision time and have good decision performance. In addition, online sample data is used to further make decisions on the partitioning mode in each of the partitioning decision classifiers, which can improve the partitioning efficiency of the coding unit, improve the decision accuracy, and significantly reduce the coding time while ensuring the video quality and bit rate.

[0015] In some implementations, the initializing online sample data and making decisions on the partitioning modes in each of the partitioning decision classifiers according to the online sample data to obtain a set of target partitioning modes include:

[0016] Obtain the second image frequency domain features of the unit to be encoded at the current moment in the target video, and initialize the second classification labels and second weights corresponding to the encoded units before the current moment as online sample data;

[0017] Inputting the second image frequency domain feature into the group of partition decision classifiers to obtain a first probability output by each partition decision classifier;

[0018] Using the online sample data to modify the first probability output by each partition decision classifier to obtain a second probability;

[0019] A decision is made on the partition mode in each of the partition decision classifiers according to the second probability to obtain a set of target partition modes.

[0020] In some implementations, the using the online sample data to modify the first probability to obtain a second probability includes:

[0021] ;

[0022] in, represents the second probability, represents the number of online samples, represents the first probability, represents a weight factor of an offline sample relative to an online sample, wherein the offline sample includes the first image frequency domain feature, the first classification label and the first weight, represents the number of offline samples, represents the weighted mean of offline samples, Indicates Classification labels of online samples, represents the geometric progression attenuation factor, Indicates The weight of the online samples.

[0023] In some implementations, making a decision on a partitioning mode in each of the partitioning decision classifiers according to the second probability to obtain a set of target partitioning modes includes:

[0024] Presetting an upper threshold and a lower threshold for each of the partition decision classifiers, wherein the upper threshold is greater than the lower threshold;

[0025] Compare the second probability with the upper threshold and the lower threshold to obtain a comparison result;

[0026] A decision is made on the partitioning mode in each of the partitioning decision classifiers according to the comparison result to obtain a set of target partitioning modes.

[0027] In some implementations, before obtaining the second image frequency domain feature of the to-be-encoded unit at the current moment in the target video, the method further includes:

[0028] Determine the size of the unit to be encoded at the current moment in the target video, and if the size of the unit to be encoded meets the size condition, downsample the unit to be encoded at the current moment in the target video to obtain a downsampled feature; and calculate the second image frequency domain feature of the unit to be encoded at the current moment in the target video according to the downsampled feature;

[0029] If the size of the unit to be encoded does not meet the size condition, the second image frequency domain features of the unit to be encoded at the current moment in the target video are not calculated, and multiple division modes are directly decided by rate-distortion optimization, and the VVC coding unit is quickly divided according to the decided division mode.

[0030] In some implementations, after initializing the online sample data and making a decision on the partition mode in each of the partition decision classifiers according to the online sample data to obtain a set of target partition modes, the method further includes:

[0031] If the group of target partition modes includes at least two partition modes, and two of the at least two partition modes belong to the same partition decision classifier, then updating the online sample data belonging to the same partition decision classifier according to the rate-distortion costs of the two partition modes belonging to the same partition decision classifier;

[0032] If the group of target partition modes includes at least two partition modes, and the remaining partition modes except the two partition modes belonging to the same partition decision classifier do not have two partition modes belonging to the same partition decision classifier, then the online sample data of the partition decision classifiers corresponding to the remaining partition modes are not updated;

[0033] If the group of target partition modes includes at least two partition modes, and there are no two partition modes belonging to the same partition decision classifier among all the partition modes, then the online sample data of the partition decision classifiers corresponding to all the partition modes are not updated;

[0034] If the set of target partition modes includes only one partition mode, the online sample data of all partition decision classifiers are not updated.

[0035] In some implementations, the fast partitioning of the VVC coding unit according to the set of target partitioning modes includes:

[0036] If the set of target partition modes includes only one target partition mode, the one target partition mode is used to quickly partition the VVC coding unit;

[0037] If the group of target partition modes includes at least two target partition modes, rate-distortion optimization is used to decide the target partition mode among the at least two target partition modes to obtain the decided target partition mode; and the VVC coding unit is quickly divided using the decided target partition mode.

[0038] In a second aspect, an embodiment of the present application further provides a VVC coding unit fast division system based on image frequency domain features, the system comprising:

[0039] A data acquisition unit, configured to use an encoder to run on each video in a video training set to acquire a first image frequency domain feature, a first classification label, and a first weight corresponding to the video training set;

[0040] A first training unit is used to train a set of binary classifiers based on a lightweight gradient boosting model by using the first image frequency domain feature, the first classification label and the first weight to obtain a set of initial binary classifiers, wherein the binary classifiers based on the lightweight gradient boosting model include two division modes;

[0041] A second training unit is used to perform feature selection through the set of initial binary classifiers, and train the set of initial binary classifiers using the selected features to obtain a set of reduced-dimensional binary classifiers;

[0042] A scale reduction unit, used for adjusting parameters of the set of dimension reduction binary classifiers to reduce the scale of the set of dimension reduction binary classifiers to obtain a set of partition decision classifiers;

[0043] A partitioning mode decision unit, used for initializing online sample data, and making a decision on the partitioning mode in each of the partitioning decision classifiers according to the online sample data, to obtain a set of target partitioning modes;

[0044] A fast partitioning unit is used to quickly partition the VVC coding unit according to the set of target partitioning modes.

[0045] In the third aspect, an embodiment of the present application also provides an electronic device, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned method for quickly dividing VVC coding units based on image frequency domain features.

[0046] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned method for quickly dividing VVC coding units based on image frequency domain features.

[0047] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0049] Figure 1 It is a flowchart of an embodiment of a method for fast division of VVC coding units based on image frequency domain features provided by the present application;

[0050] Figure 2 It is a schematic diagram of the overall process of the best embodiment of the method for fast division of VVC coding units based on image frequency domain features provided by the present application;

[0051] Figure 3 It is a structural schematic diagram of an embodiment of a VVC coding unit fast division system based on image frequency domain features provided by the present application. DETAILED DESCRIPTION

[0052] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0053] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0054] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0055] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0056] The optimization directions of the encoder include compression efficiency and encoding speed. Specifically, compression efficiency refers to the balance between image quality and bit rate, which is manifested as a smaller rate-distortion cost (RD Cost) during the encoding process and a BDBR (Bjontegaard Delta Bit Rate) less than 0 in the encoding result. The encoding speed is measured by the encoding time.

[0057] The original rate-distortion optimization (RDO) method performs the complete encoding process for all allowed partitioning modes to obtain the RD Cost, and obtains the optimal mode by comparing the cost. Although the optimal partitioning mode can always be found, it leads to great time complexity and poor cost performance. The fast algorithm seeks a balance between time and compression performance, that is, to reduce the encoding time while ensuring high image quality and low bit rate (that is, the RD Cost is as small as possible), but it often cannot fully remove the partitioning modes with low probability, resulting in little time saving, or the decision accuracy is poor and the better partitioning modes are skipped, resulting in a decrease in video quality and an increase in bit rate.

[0058] Therefore, the above-mentioned existing coding unit division method is not efficient in dividing the coding units, and the decision accuracy is poor, resulting in a decrease in video quality and an increase in bit rate.

[0059] In order to solve the problem that the above-mentioned existing coding unit division method has low efficiency in coding unit division and poor decision accuracy resulting in reduced video quality and increased bit rate, the present application proposes a VVC coding unit fast division method and system based on image frequency domain features.

[0060] Reference Figure 1 The embodiment of the present application provides a method for quickly dividing VVC coding units based on image frequency domain features, and the method comprises the following steps:

[0061] Step S100: using an encoder to run on each video in a video training set to obtain a first image frequency domain feature, a first classification label, and a first weight corresponding to the video training set;

[0062] Step S200: training a set of binary classifiers based on a lightweight gradient boosting model using the first image frequency domain feature, the first classification label and the first weight to obtain a set of initial binary classifiers, wherein the binary classifiers based on the lightweight gradient boosting model include two division modes;

[0063] Step S300, performing feature selection through a set of initial binary classifiers, and using the selected features to train the set of initial binary classifiers to obtain a set of reduced-dimensional binary classifiers;

[0064] Step S400, adjusting parameters of a set of dimension reduction binary classifiers to reduce the size of the set of dimension reduction binary classifiers to obtain a set of partition decision classifiers;

[0065] Step S500, initializing online sample data, and making decisions on the partitioning patterns in each partitioning decision classifier according to the online sample data to obtain a set of target partitioning patterns;

[0066] Step S600: quickly divide the VVC coding unit according to a set of target division modes.

[0067] In this embodiment, the encoder is used to run on each video in the video training set to obtain the first image frequency domain features, the first classification label and the first weight corresponding to the video training set; the first image frequency domain features, the first classification label and the first weight are used to train a group of binary classifiers based on the lightweight gradient boosting model to obtain a group of initial binary classifiers, wherein the binary classifiers based on the lightweight gradient boosting model include two division modes; feature selection is performed through a group of initial binary classifiers, and the selected features are used to train a group of initial binary classifiers to obtain a group of dimension reduction binary classifiers; parameter adjustment is performed on a group of dimension reduction binary classifiers to reduce the size of a group of dimension reduction binary classifiers to obtain a group of division decision classifiers; online sample data is initialized, and the division mode in each division decision classifier is decided according to the online sample data to obtain a group of target division modes; the VVC coding unit is quickly divided according to a group of target division modes. In this way, the frequency domain feature decision division is introduced to improve the multi-scale expression ability, information density and robustness of the feature, and improve the performance of the division decision model. Then, by performing feature selection on the initial binary classifier and adjusting the parameters of a set of dimensionality reduction binary classifiers to reduce the size of a set of dimensionality reduction binary classifiers, the features can be effectively selected and the model size can be controlled, which saves decision time and has good decision performance. In addition, online sample data is used to further make decisions on the partitioning mode in each partitioning decision classifier, which can improve the partitioning efficiency of the coding unit, improve the decision accuracy, and significantly reduce the encoding time while ensuring the video quality and bit rate.

[0068] The above encoder may be an encoder known to those skilled in the art, and this embodiment does not impose any specific limitation thereto.

[0069] The first image frequency domain feature may be obtained by running an encoder on each video in a video training set to extract the frequency domain feature of each coding unit in each video image.

[0070] The first classification label may be expressed by the magnitude relationship of rate-distortion costs (RD Cost) of two division modes.

[0071] The first weight may be calculated by using the absolute difference between the rate-distortion costs of the two partitioning modes.

[0072] The binary classifier based on the lightweight gradient boosting model may be a binary classifier built based on LightGBM.

[0073] The above feature selection through a set of initial binary classifiers can adopt the concept of the initial binary classifier itself. Since the binary classifier can be built based on the LightGBM model, the model can measure the importance of the feature by calculating the number of times each feature is used in the tree model or the importance of being used for splitting.

[0074] The above-mentioned parameter adjustment of a set of dimensionality reduction binary classifiers to reduce the size of a set of dimensionality reduction binary classifiers can be achieved by adjusting the parameters of the binary classifier model itself to reduce the model size and improve the decision speed of CU division. For LightGBM, the classification performance and running speed of the classifier mainly depend on the number of leaf nodes and the number of iterations. The size of a set of dimensionality reduction binary classifiers can be adjusted by controlling the leaf nodes and the number of iterations.

[0075] In the above, the partitioning mode in each partitioning decision classifier is decided based on the online sample data. Since each partitioning decision classifier contains two partitioning modes, the online sample data can be further used to make a decision on the partitioning mode in each partitioning decision classifier to terminate the worse partitioning mode in advance.

[0076] The VVC coding unit is quickly divided according to a set of target division modes, and the VVC coding unit is quickly divided from the last selected set of target division modes. If there is only one division mode in a set of target division modes, the VVC coding unit is directly divided by the division mode; if a set of target division modes contains multiple division modes, a decision algorithm based on rate-distortion optimization (RDO) is used to select a division mode from the last multiple division modes to quickly divide the VVC coding unit.

[0077] In some implementations, online sample data is initialized, and a partitioning mode in each partitioning decision classifier is decided based on the online sample data to obtain a set of target partitioning modes, including:

[0078] Obtain the second image frequency domain features of the unit to be encoded at the current moment in the target video, and initialize the second classification labels and second weights corresponding to the encoded units before the current moment as online sample data;

[0079] Inputting the second image frequency domain feature into a group of partition decision classifiers to obtain a first probability output by each partition decision classifier;

[0080] Using online sample data to correct the first probability output by each partition decision classifier to obtain a second probability;

[0081] A decision is made on the partitioning mode in each partitioning decision classifier according to the second probability to obtain a set of target partitioning modes.

[0082] In this embodiment, the second image frequency domain feature is input into a group of partition decision classifiers to obtain the first probability output by each partition decision classifier; then the first probability output by each partition decision classifier is corrected using online sample data to obtain the second probability; then the partition mode in each partition decision classifier is decided based on the second probability to obtain a group of target partition modes. Since each partition decision classifier is trained based on offline samples, the offline samples include samples of the first image frequency domain feature, the first classification label and the first weight, and then weighted prediction is performed through online samples, thus designing a weighted prediction scheme with scene adaptation capability, without the need for scene conversion or saving each online sample, making full use of the partition mode actually run by the CU, and performing weighted prediction through the generated online samples and the offline trained model, the adaptability in different coding scenarios is improved, and the coding efficiency is increased.

[0083] The target video may be a video to be encoded.

[0084] The to-be-encoded unit may be a coding unit (CU) that needs to be divided at the current moment.

[0085] The above-mentioned initialization of the second classification label and the second weight corresponding to the encoded unit before the current moment as online sample data can be to obtain the second classification label and the second weight of the encoded unit corresponding to each partition decision classifier, and initialize the second classification label and the second weight as the online sample data of the partition decision classifier.

[0086] The above-mentioned use of online sample data to correct the first probability output by each partition decision classifier may be to use the online sample data corresponding to each partition decision classifier to correct the first probability output by each partition decision classifier.

[0087] The above-mentioned making a decision on the partitioning mode in each partitioning decision classifier according to the second probability may be to use the second probability corresponding to each partitioning decision classifier to make a decision on the partitioning mode in each partitioning decision classifier to determine which partitioning modes will be retained.

[0088] In some implementations, the first probability is modified using online sample data to obtain a second probability, including:

[0089] ;

[0090] in, represents the second probability, represents the number of online samples, represents the first probability, represents a weight factor of an offline sample relative to an online sample, wherein the offline sample includes a first image frequency domain feature, a first classification label, and a first weight, represents the number of offline samples, represents the weighted mean of offline samples, Indicates Classification labels of online samples, represents the geometric progression attenuation factor, Indicates The weight of the online samples.

[0091] In some implementations, a decision is made on the partitioning mode in each partitioning decision classifier according to the second probability to obtain a set of target partitioning modes, including:

[0092] Presetting an upper threshold and a lower threshold for each partition decision classifier, wherein the upper threshold is greater than the lower threshold;

[0093] Compare the second probability with the upper threshold and the lower threshold to obtain a comparison result;

[0094] According to the comparison results, a decision is made on the partitioning mode in each partitioning decision classifier to obtain a set of target partitioning modes.

[0095] In this embodiment, an upper threshold and a lower threshold are preset for each partition decision classifier; then the second probability is compared with the upper threshold and the lower threshold to obtain a comparison result; and then a decision is made on the partition mode in each partition decision classifier according to the comparison result to obtain a set of target partition modes. In this way, by further making a decision on the partition mode in each partition decision classifier, a set of target partition modes is obtained, which lays a good data foundation for improving the partition efficiency of the coding unit and improving the decision accuracy in the later stage.

[0096] The above-mentioned decision is made on the partitioning mode in each partitioning decision classifier according to the comparison result to obtain a set of target partitioning modes, which may include the following situations:

[0097] 1. For the QT division mode, the second probability is given by the QT-BH classifier ,like Then the QT partition mode is terminated in advance and jumps to the BH and TH partition modes in 3;

[0098] 2. For the QT division mode, the probability is given by the QT-BV classifier ,like Then the QT division mode is terminated in advance;

[0099] 3. For BH and TH partitioning modes, the probability is given by the BH-BV classifier ,like Then the BH partition mode and the TH partition mode are terminated in advance;

[0100] 4. For the BV and TV partition modes, the probability is given by the BH-BV classifier ,like The BV partition mode and the TV partition mode are terminated in advance;

[0101] 5. For the TH partition mode, if the BH partition mode has been run but is not the best partition mode that has been run, or the BH partition mode is terminated early but the BV partition mode has been run, the TH partition mode is terminated early, otherwise the BH-TH classifier gives the probability ,like Then the TH partitioning mode is terminated in advance;

[0102] 6. For the TV partition mode, if the BV partition mode has been run but is not the best partition mode that has been run, or the BV partition mode is terminated early but the BH partition mode has been run, then the TV partition mode is terminated early, otherwise the BV-TV classifier gives the probability ,like The TV division mode is terminated early.

[0103] The above analysis of the statistical distribution of different partitions focuses on retaining BT partitions when making partition decisions, and more strictly screens QT and TT partitions, reducing the risk of missing BT, reducing the encoding time of redundant QT and TT, and improving encoding efficiency and encoding time cost-effectiveness.

[0104] The division and decision of the above-mentioned QT-BH classifier, QT-BV classifier, BH-BV classifier, BH-TH classifier and BV-TV classifier can be labeled and participate in training as long as the CU sample has a valid RD Cost in both modes of the mode pair, without the need to include the optimal division mode, which greatly increases the sample utilization and avoids sample bias. In addition, even if the optimal division mode is mistakenly skipped, it is not easy to select a division mode with a very high cost.

[0105] In some implementations, before obtaining the second image frequency domain feature of the to-be-encoded unit at the current moment in the target video, the method further includes:

[0106] Determine the size of the unit to be encoded at the current moment in the target video, and if the size of the unit to be encoded meets the size condition, downsample the unit to be encoded at the current moment in the target video to obtain the downsampled features; calculate the second image frequency domain features of the unit to be encoded at the current moment in the target video according to the downsampled features;

[0107] If the size of the unit to be encoded does not meet the size condition, the second image frequency domain features of the unit to be encoded at the current moment in the target video are not calculated, and multiple division modes are directly decided by rate-distortion optimization, and the VVC coding unit is quickly divided according to the decided division mode.

[0108] In this embodiment, by judging the size of the unit to be encoded at the current moment in the target video, if the size of the unit to be encoded meets the size condition, the unit to be encoded at the current moment in the target video is downsampled to obtain the downsampled features; according to the downsampled features, the second image frequency domain features of the unit to be encoded at the current moment in the target video are calculated; if the size of the unit to be encoded does not meet the size condition, the second image frequency domain features of the unit to be encoded at the current moment in the target video are not calculated, and multiple division modes are directly decided by rate-distortion optimization, and the VVC coding unit is quickly divided according to the division mode after the decision. In this way, by selecting different decision methods for coding units of different sizes to obtain a suitable division mode, it can better adapt to different coding scenarios, thereby improving the adaptability in different coding scenarios and increasing coding efficiency.

[0109] The size of the unit to be encoded meets the size condition, which may be that the size condition is set to be no less than 8×8, that is, the size of the unit to be encoded needs to be greater than or equal to 8×8.

[0110] The above-mentioned calculation of the second image frequency domain features of the unit to be encoded at the current moment in the target video based on the down-sampled features can be that the brightness blocks of the unit to be encoded at the current moment in the target video are first down-sampled on average (8*8 CU does not need to be down-sampled) to obtain an 8*8 brightness matrix M, and then, based on the brightness matrix M, an 8*8 DCT transform kernel C is applied to calculate the DCT coefficient matrix F of the CU, and the coefficient matrix F is the second image frequency domain features of the unit to be encoded at the current moment.

[0111] The above-mentioned rate-distortion optimization (RDO) can be a method of improving video quality in video compression. The name refers to optimizing the amount of distortion (loss of video quality) for the amount of data (rate) required to encode the video. While it is primarily used by video encoders, rate-distortion optimization can be used to improve quality in any encoding situation (e.g., images, video, audio, or other) where decisions must be made that affect both file size and quality.

[0112] The multiple partitioning modes may include a quadtree partitioning (QT) mode, a horizontal binary partitioning (BH) mode, a vertical binary partitioning (BV) mode, a horizontal ternary partitioning (TH) mode, and a vertical ternary partitioning (TV) mode.

[0113] In some implementations, after initializing the online sample data and making a decision on the partitioning mode in each partitioning decision classifier according to the online sample data to obtain a set of target partitioning modes, the method further includes:

[0114] If a set of target partition modes includes at least two partition modes, and two of the at least two partition modes belong to the same partition decision classifier, then the online sample data belonging to the same partition decision classifier is updated according to the rate distortion cost of the two partition modes belonging to the same partition decision classifier;

[0115] If a group of target partitioning modes contains at least two partitioning modes, and the remaining partitioning modes except the two partitioning modes belonging to the same partitioning decision classifier do not have two partitioning modes belonging to the same partitioning decision classifier, then the online sample data of the partitioning decision classifiers corresponding to the remaining partitioning modes are not updated;

[0116] If a group of target partitioning modes contains at least two partitioning modes, and there are no two partitioning modes belonging to the same partitioning decision classifier among all the partitioning modes, then the online sample data of the partitioning decision classifiers corresponding to all the partitioning modes will not be updated;

[0117] If a set of target partition modes contains only one partition mode, the online sample data of all partition decision classifiers will not be updated.

[0118] In this embodiment, after making a decision on the partitioning mode in each partitioning decision classifier based on the online sample data and obtaining a set of target partitioning modes, the online sample data is updated through the obtained set of target partitioning modes, thereby laying a good data foundation for better new coding unit division in the later stage.

[0119] In some implementations, the VVC coding unit is quickly partitioned according to a set of target partitioning modes, including:

[0120] If a set of target partitioning modes contains only one target partitioning mode, then the one target partitioning mode is used to quickly partition the VVC coding unit;

[0121] If a group of target partition modes includes at least two target partition modes, rate-distortion optimization is used to decide the target partition mode among the at least two target partition modes to obtain the decided target partition mode; the decided target partition mode is used to quickly divide the VVC coding unit.

[0122] In this embodiment, the VVC coding unit is quickly divided through a set of target division modes that are finally decided, which can better improve the division efficiency of the coding unit and improve the decision accuracy.

[0123] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0124] First, the common fast decision-making methods use image features such as original images (original pixel values ​​or predicted residuals), texture features (such as gradient direction, mainly some edge information obtained by filtering), and image complexity (such as variance and entropy). The use of these features has the following defects:

[0125] (1) Weak information extraction capability.

[0126] (2) Poor information density and high redundancy.

[0127] (3) The scale is single and does not fit the decision-making needs of the specific structure of the division model.

[0128] (4) Poor noise resistance and easily affected by local interference.

[0129] Second, existing learning-based fast algorithms usually only examine the best partitioning mode after encoding and compare the best partitioning mode with all non-optimal partitioning modes. This is not a reasonable way to set classification labels, and has the following main defects:

[0130] (1) The correlation between partitioning patterns is ignored.

[0131] (2) The information contained in the labels is complex, which increases the difficulty of learning.

[0132] (3) Common classification models used in existing learning-based fast algorithms include support vector machines, decision trees, or neural networks. Neural network design is usually complex, has high embedding costs, and requires a large number of matrix operations. Encoding a frame of an image often requires the calculation of tens of thousands of CUs. Therefore, neural networks are not ideal as decision models at the CU level. Other machine learning methods other than decision trees often require manual feature screening. Although general decision tree models can use weights to screen features, they are prone to falling into local optimality, have low computational efficiency, and have difficulty processing large-scale data.

[0133] (4) In addition, the decision-making mechanism of existing fast algorithms is usually based on experience or offline training. However, the CU division results are different under different encoder configurations (such as quantization parameter settings and tool switches, etc.) and when encoding video sequences with different characteristics, and the adaptability to different scenarios is poor.

[0134] Due to the above four aspects, namely image features, classification labels, classification models and offline training, it is difficult for the fast algorithm to decide the best coding division mode, and redundant coding decision processes are not fully eliminated. Finally, the compression efficiency of the encoder is not high and the encoding speed is not fully improved.

[0135] The main reasons for the above defects include the following:

[0136] The general process of the fast algorithm for coding unit partitioning is to extract image features and coding information before the partitioning mode is executed, and then judge whether the partitioning mode needs to be fully run according to empirical formulas or trained classifiers. Therefore, the core determining the performance of the fast algorithm lies in whether the defined empirical formulas and the speed of the trained classifiers are fast enough and accurate enough.

[0137] From the above analysis, it can be seen that the currently selected internal image features of the fast algorithm for CUs are mainly general complexity estimates or texture and edge information. Taking variance analysis as an example, variance can only represent the magnitude of the numerical fluctuations of the pixels inside the CU, losing the image details of the CU and being difficult to describe the distribution characteristics of the brightness inside the CU. Therefore, it is not specific enough as a basis for CU partitioning; while the gradient examines the numerical changes between adjacent pixels. Since it is a local feature, it is not convenient to be directly used as a basis for CU partitioning and still requires further processing. There is also a problem of scale control. The size of the CU to be partitioned can range from 8×8 to 64×64, but the result of CU partitioning is always not to partition or to partition into 2 to 4 sub-blocks. Then, using the gradient as an image feature will encounter problems of size adaptation and feature dimension adaptation. In addition, features such as variance and gradient are very sensitive to noise and fine high-frequency textures, so they are prone to misleading the CU partitioning and affecting the compression efficiency of the final coding.

[0138] From the perspective of the classification label setting, there are usually two common ways in the existing fast algorithms. The first is to decide the best mode in the order of "whether to use quadtree partitioning (QT)", "whether to use horizontal partitioning", and "whether to use binary tree (BT) partitioning". There is ambiguity in the labeling process of this kind of classification label. Suppose the RD Cost of a sample CU satisfies horizontal TT (i.e., horizontal ternary tree) < QT < vertical TT (i.e., vertical ternary tree) < vertical BT (i.e., vertical binary tree) < horizontal BT (i.e., horizontal binary tree). Then, "whether to use horizontal partitioning" is true only when considering TT, and false when considering BT. In short, this classification method attempts to merge situations that may be contradictory. The usual way to eliminate ambiguity is to set the above three labels only according to the best partitioning mode. This will increase the risk of classification errors and the sample utilization rate is low, ignoring the comparison between sub-optimal partitioning modes.

[0139] The second method is to use "whether it is the best encoding partition mode" as the basis for classification. This criterion can avoid the ambiguity of the first method, but the number of models to be trained increases from 3 to 5, and each classification problem can be specifically divided into multiple problems. Taking horizontal BT as an example, "horizontal BT is the best mode" also means "horizontal BT is better than QT", "horizontal BT is better than vertical BT", "horizontal BT is better than horizontal TT", and "horizontal BT is better than vertical TT". Setting labels in this way can also train the corresponding classifier, but this concept is more difficult to fully learn than the comparison between two single modes, especially the lack of similarity between BT in a certain direction and TT in different directions, so it is difficult to make a targeted comparison. From the perspective of machine learning, it is a more confusing concept than the strong contrast between horizontal and vertical. Therefore, mixing these classification problems of different difficulty together will lead to poor overall classification performance due to the "barrel effect". In addition, setting labels in this way will lead to a greater risk of classification errors because the correlation between the partition structures is not utilized. Once the optimal partition mode is skipped by the fast algorithm, the suboptimal partition mode cannot be guided by probability to win among the remaining partition modes, which ultimately leads to poor decision performance and reduced compression rate.

[0140] The choice of decision model is also very important. Since CU-level decisions need to be run thousands of times within one frame of an image, a lightweight and efficient model must be used. Complex models will increase the computational burden of fast algorithms and reduce time benefits, while models with poor generalization capabilities will lead to loss of compression efficiency and poor time-compression ratio cost-effectiveness. In addition, models that do not have the ability to automatically screen features will lead to feature redundancy, which will also increase computing time.

[0141] Finally, since the encoder will be used in different scenarios, it will adopt various parameter settings, enable different tools, and encode video sequences with different characteristics. If the fast algorithm cannot adapt to the changes in the scene, it is likely to have the problem of too deep or too shallow division. For example, in the low-quality and high-compression scenario, the quantization parameter (QP) is generally large, and the encoder will tend to save bit rate and reduce quality requirements. The best CU division will tend to be shallow division, or if the video image tends to be flat and stable, it will also tend to be shallow division. The activation of each tool will also affect the RD Cost of each mode. Therefore, unchanging decisions will lead to poor adaptability and loss of compression efficiency.

[0142] The technical solution of this embodiment aims at the problem that the CU division fast algorithm under the current H.266 / VVC standard is difficult to fully eliminate redundant modes and the decision accuracy is not high, and a scene-adaptive coding unit (CU) fast division method with better coding efficiency and coding time balance based on frequency domain features is proposed. To achieve this goal, the technical solution of this embodiment adopts the following four measures:

[0143] Measure 1: Use image frequency domain features.

[0144] In this embodiment, the original image brightness value of the CU is subjected to a two-dimensional discrete cosine transform (Discrete Cosine Transform, DCT), and the obtained DCT coefficients are used as the input of the decision algorithm. This needs to be distinguished from the transform module in the encoder. The encoder's transform transforms the residual obtained by intra-frame or inter-frame prediction, and the obtained transform coefficients are the input of the quantization module. The technical solution of this embodiment is to transform the original CU to obtain the frequency domain features of the image, which are used for division decisions, and occur before the steps of prediction, transformation, quantization, etc. The image frequency domain features have the following advantages:

[0145] (1) DCT coefficients are a frequency domain feature. Compared with spatial domain features, they can naturally decompose images into components from low frequency to high frequency, thereby directly capturing the directionality, periodicity and complexity of the image. They have strong expressive power and well avoid the problem that spatial domain features (such as variance and gradient) can only reflect general complexity or local edge characteristics.

[0146] (2) It retains most of the original CU information, with high information density and less redundant information, especially the regional distribution characteristics of brightness, which has a strong targeted prompting effect on CU division;

[0147] (3) DCT transform can perform frequency domain transform on the downsampled image with a fixed size to obtain a series of eigenvalues ​​at fixed frequencies, and each transform coefficient has a reliable physical meaning. The low-frequency component represents the overall brightness fluctuation of the CU, while the high-frequency component represents the richness of texture details. Therefore, CU information can be captured from multiple scales, taking into account the overall complexity of the CU and local details.

[0148] (4) Compared with spatial domain features, frequency domain features are naturally robust. The DCT transform discards high-frequency noise and complex details in the image that have little effect on CU partitioning, and is insensitive to geometric transformations such as rotation and translation, and can guide partitioning decisions more stably.

[0149] (5) Feature extraction is done in one step, requiring only one matrix operation, without the need for pixel-by-pixel calculation and statistics.

[0150] In summary, using frequency domain features as the input of the fast algorithm for partitioning patterns is conducive to extracting CU information in a multi-scale and consistent manner, suppressing the interference of noise and complex details, and efficiently and comprehensively guiding CU partitioning decisions, thereby speeding up decision-making while ensuring the performance of the decision-making model.

[0151] Measure 2: Set classification tasks according to the criteria of partitioning structure relevance and minimizing information loss.

[0152] The correlation of partition structure refers to the connection between different partition modes, the main connection being the direction of the partition and the number of sub-CUs divided; the criterion of minimizing information loss means that when labeling data, the classification basis only involves two partition modes, avoiding the ambiguity caused by the merging of modes. This embodiment associates 3 types of 5 pairs of partition modes to construct a classifier. When labeling, only CU samples with valid RD Cost in both partition modes of the partition mode pair can be labeled and participate in training, without the need to include the optimal partition mode. This greatly increases sample utilization and avoids sample bias. In addition, even if the optimal partition mode is mistakenly skipped, it is not easy to select a partition mode with a very high cost. The label design of the 5 classifiers is as follows:

[0153] (1) QT-BH classifier: QT partitioning and BT partitioning have partially overlapping partitioning boundaries. The only difference is whether there is a boundary in one direction. Therefore, there is an obvious competitive relationship in the partitioning of larger CUs. Generally, CUs with high texture complexity but weak directionality tend to be QT partitioned, while CUs with strong directionality tend to be BT partitioned.

[0154] (2) QT-BV classifier: Similar to (1), the classification of horizontal BT and QT and the classification of vertical BT and QT are separated here to avoid information loss such as “BT is better than QT” relative to “horizontal BT is better than QT but QT is better than vertical BT” and to simplify the learning difficulty of the classification task.

[0155] (3) BH-BV classifier: The two BT divisions have the same sub-CU area but different directions. Therefore, when comparing the two divisions, we can focus more on the texture directionality inside the CU. Here, the comparison of horizontal TT and vertical TT is separated to avoid information loss such as "horizontal division is better than vertical division" relative to "horizontal BT is better than vertical BT but vertical TT is better than horizontal TT". This simplifies the learning difficulty of the classification task and does not require the TT mode of the sample to be available, thus weakening the sample bias.

[0156] (4) BH-TH classifier: For BT and TT in the same direction, the key to comparison lies in the regional distribution of brightness, while the directional factors can be ignored.

[0157] (5) BV-TV Classifier: Similar to (4), the comparison between BT and TT in horizontal and vertical cases is split here to avoid information loss.

[0158] By obtaining the RD Cost of two partitioning modes in the coded intermediate results respectively according to the above three partitioning modes, the classification labels can be obtained. Compared with using "whether it is the best coding partitioning mode" as the classification label, this classification label setting makes the partitioning modes participating in classification and the factors affecting classification as few as possible (because "being the best coding partitioning mode" implies the superiority and inferiority relationship between the current mode and the other four partitioning modes). Each model needs to consider fewer features, and the partitioning structure has strong relevance, strong pertinence and strong flexibility, which not only improves the decision-making speed but also the decision-making quality. From the perspective of risk, even if a certain classifier makes a mistake, the impact on the overall decision-making is controllable. Consider the following example:

[0159] Assume that the size relationship of the RD Cost of each partitioning structure is horizontal BT < QT < vertical BT < horizontal TT < vertical TT. Taking "whether it is the best coding partitioning mode" as the label, once the horizontal BT is skipped by mistake, any one of the other four partitionings may or may not be adopted; while if the above mode pair method is used to set the label, even if a certain classification judgment is wrong, the sub-optimal mode can still be used, and the overall risk is smaller.

[0160] Measure 3: Adopt a lightweight gradient boosting model for feature selection and classification.

[0161] The Light Gradient Boosting Machine (LightGBM) is an ensemble learning method based on decision trees. Compared with other machine learning classification methods for CU partitioning decision-making, it has the following advantages:

[0162] (1) It can give the importance of each feature after training, as the basis for feature selection, and avoid the additional computational burden brought by redundant features.

[0163] (2) Integrate multiple decision trees in the gradient boosting framework, and each tree fits the residual of the previous tree. It has stronger fitting ability than a single learner such as a decision tree or a support vector machine.

[0164] (3) The training process is highly parallel and is specially optimized for industrial-scale massive data, suitable for the analysis and processing of massive CU data.

[0165] (4) The inference process of the model is a series of if-else logical judgments, and the time complexity is extremely low.

[0166] (5) The number of decision trees can be set according to the loss change rate of multiple rounds of iterations, and the number of leaves of a single decision tree can be determined according to the best fitting loss, so as to flexibly control the scale of the model and balance speed and accuracy.

[0167] Measure 4: Online samples are combined with offline models to adapt to different encoding scenarios.

[0168] For each partition mode pair in the above measure 2, the output Pred of the corresponding classifier is a floating point number representing the probability that the result of the binary classifier model is true, and the value range is between 0 and 1. This embodiment sets a default upper threshold ThrHi and a lower threshold ThrLo for each group of partition mode pairs, wherein, taking the "BH-BV" classifier as an example, the label of the classifier is "BH is better than BV", when Pred is greater than ThrHi, the BV partition mode is disabled, when Pred is less than ThrLo, the BH partition mode is disabled, and when Pred is between ThrHi and ThrLo, both BH and BV partition modes are retained. In this case, after both partition modes are run to obtain the RD Cost, such CUs can be used as online samples and jointly decided with the offline trained model, thereby achieving the purpose of scene adaptation and reducing the compression efficiency loss caused by different scenes.

[0169] The specific technical solution of this embodiment is:

[0170] The standard CU partitioning mode decision is implemented through the RDO process, that is, all available partitioning modes are run on the CU in turn, the RD Cost of each partitioning mode is obtained, and the mode with the lowest cost is selected. Brute force solution can theoretically obtain the partition with the best compression performance, but the time complexity is extremely high. In order to reduce the encoding time without affecting the encoding quality as much as possible, this embodiment proposes a CU fast partitioning method based on frequency domain features, which is suitable for H.266 / VVC.

[0171] The basic idea of ​​the method of this embodiment is to extract the frequency domain features of the CU (i.e., the frequency domain features of the image) and the RD Cost of each partition mode during the encoding process for training the classifier; then reduce the scale of the classifier model through a variety of methods to reduce the time required for decision-making; then reasonably use the correlation between the partition modes and prior knowledge to plan the decision-making process, try to retain the possible better partition mode, skip the bad partition mode, and achieve a balance between time and compression efficiency; at the end of the CU decision, use the RD Cost of the fully run partition mode to update the threshold of the offline trained classifier online, so that the decision algorithm can adapt to encoding scenarios with different characteristics. Specifically, it includes the following contents:

[0172] 1. About frequency domain features: The frequency domain features used are not the transform coefficients obtained by transforming the prediction residual by the transform module in the encoder, but directly performing DCT transform on the original image (luminance block of CU), which is a method for extracting features from the image. The specific process is to first average downsample the luminance block of the square CU (8*8 CU does not need downsampling) to obtain an 8*8 luminance matrix M, and then apply the 8*8 DCT transform kernel C based on the luminance matrix M to calculate the DCT coefficient matrix F of the CU. The specific calculation formula is as follows:

[0173] ;

[0174] ;

[0175] By extracting the RD Cost of each partitioning mode of the CU sample, we can judge the quality of the mode and set the classification label and sample weight. There are 5 classifiers in total, and the tasks, labels and weight settings of the classifiers are shown in Table 1.

[0176] Table 1 Classifier parameter configuration

[0177]

[0178] 2. Regarding the setting of classification tasks and labeling: There are two common practices. The first is to set a classification task for each partition mode separately to distinguish it from all other partition modes. The key to classification is whether this partition mode is the optimal mode, that is, it is classified as true only when this mode is better than other modes. This method ignores the correlation between partition modes and implies a comparative relationship with multiple partition modes. In this way, more features need to be considered, and the partition task becomes more difficult, which affects both the decision-making speed and accuracy. The other is to select the best mode according to whether the best mode is QT, whether it is horizontal division, and whether it is binary division. In addition to the problems in the first approach, "the best division mode is horizontal division" is equivalent to "BH and TH are better than BV and TV" in practical application, but in fact there is information bias between the two concepts. For example, even if BH is the best mode, TH may not be as good as BV or TV. This situation may lead to greater costs when classification errors occur because there is a lack of support for classifications other than the best mode. For the second case mentioned, even if we give up starting from the best mode and use "whether horizontal division is better than vertical division" and "whether binary division is better than trifurcated division" as the basis for labeling, there will still be ambiguity problems because there are situations in the training samples where BH is better than BV and TH is worse than TV.

[0179] In view of the above problems, this embodiment proposes that each classifier is only associated with two specific partitioning modes. In this case, there are many combinations of partitioning modes, and it is obviously not necessary to compare all possible combinations. Therefore, it is considered to use the structural correlation and statistical prior knowledge between the partitioning modes. MTT partitions with the same direction only need to compare the number and distribution of sub-blocks, and BT or TT with different directions only need to compare the directionality. QT is mainly used for large-size CU partitioning in flat areas. It has a certain degree of competitive relationship with BT partitioning, and has a weaker competitive relationship with TT applicable to complex problem areas. Therefore, only common partitioning modes can be selected to compare their differences to establish a competitive relationship, thereby reducing the number of classifiers required and the features that each classifier needs to refer to. In addition, from the statistical results, BT has more advantages than QT and TT at most depths, which means that skipping BT is more risky than skipping QT and TT. Therefore, differentiated processing of BT can be considered. Even if all the five designed classifiers involve the BT mode, only the BH-BV classifier is used to terminate BH and BV in advance, and the other classifiers are used to terminate QT and TT with lower probability in advance. The advantage of this is that when BT is available, there is always a direction of BT division mode that will be retained, which significantly reduces the risk of erroneously skipping BT. Stricter early termination of QT and TT can save more time at a lower cost, thereby achieving a better balance between coding efficiency and coding time.

[0180] 3. Sample weight setting: It is the absolute difference between the RD Cost of the two modes, which represents the rate-distortion benefit or cost brought by the correct or incorrect selection of the CU in the two partitioning modes. Since the RD Cost is the optimization target used in the encoder to evaluate the quality of the partitioning mode and the RD Cost of multiple CUs is additive (usually the RD Cost of the parent CU is obtained by adding the RDCost of the child CU and the overhead of the partitioning operation), it is very suitable as a weight. The loss calculated in this way also faithfully reflects the additional loss introduced by the decision model.

[0181] 4. About feature selection: After the first training of the binary classifier based on the LightGBM model, feature importance indicators can be obtained from the model. These indicators can be used for feature selection, which can improve the model speed, reduce the number of iterations required for model convergence, and reduce the scale of the model. The specific method of selecting features according to feature importance is to include features in order of feature importance from large to small. When the feature importance reaches a certain ratio of the total importance of all features, a reduced-dimensional feature set can be obtained. It should be noted that this certain ratio in this embodiment can be set to 90%, and can also be changed according to actual conditions. This embodiment does not impose specific restrictions.

[0182] 5. Regarding model scale control: In addition to feature selection, the model scale can also be reduced and the speed can be improved by adjusting the parameters of the model itself. For the LightGBM model, the classification performance and running speed of the classifier mainly depend on the number of leaf nodes and the number of iterations. The number of iterations is also the number of gradient boosting decision trees included in LightGBM. The number of leaf nodes affects the fitting ability of a single gradient boosting decision tree and determines the upper limit of the model's fitting ability. The more the number of iterations, the smaller the prediction residuals of the model until it reaches the upper limit of the fitting ability.

[0183] Therefore, the strategy for adjusting the model scale is to first train a model with a sufficient number of leaf nodes and iterations. The judgment method is that the prediction loss remains stable after the number of leaf nodes and iterations of the model increase. Then determine the minimum number of leaf nodes, gradually reduce the number of leaf nodes and retrain the model until the prediction loss increases significantly. For example, record the sum of the absolute values of the changes in the prediction loss when the number of leaf nodes is reduced for the first 3 times (which is a very small value compared to the change in the prediction loss when the fitting ability deteriorates). When the increase in the prediction loss caused by reducing the number of leaf nodes in a certain time is greater than 3 times this sum of absolute values, it is considered that the prediction loss has increased significantly. It should be noted that the numerical range of the significant increase in the prediction loss can be changed according to the actual situation, and this embodiment does not make specific limitations. Select the number of leaf nodes and the classifier before the last adjustment, and then plot the change in the loss during the iteration of the classifier to find the number of iterations when the slope of the loss curve begins to slow down significantly. Use the classifier of this iteration as the final decision model. For example, obtain the sum of the absolute values of the changes in the prediction loss in the last 3 iteration rounds, and traverse forward. If the change in the prediction loss in a certain round is greater than 3 times this sum of absolute values, then adopt this iteration round. It should be noted that the numerical range of the significant slowdown in the slope of the loss curve can be changed according to the actual situation, and this embodiment does not make specific limitations.

[0184] 6. Regarding online update: For each classifier, two partitioning modes A and B are associated with it. The probability p (i.e., the first probability) output by the classifier needs to be corrected through online samples to obtain the probability (i.e., the second probability), which is the probability that mode A is better than B, and it is a floating-point number between 0 and 1. To make decisions on modes A and B respectively, a lower threshold ThrLo and an upper threshold ThrHi are set, where ThrLo < ThrHi. Generally, when the partitioning mode A will be terminated in advance. When the partitioning mode B will be terminated in advance (BT is an exception and may only be terminated in advance when comparing BH and BV). In particular, if Between ThrLo and ThrHi, both modes A and B can run. If both modes A and B are run after the CU partition decision is completed and the RD Cost is obtained, an online sample belonging to the AB classifier can be obtained.

[0185] Specifically, assuming that the offline sample The sample weight mean is , the label of the i-th online sample is , is a label of 0 or 1, and the weight is , according to the previous The weighted prediction results after the online samples are updated Satisfies the following formula:

[0186] ;

[0187] in, is the weight factor of offline samples relative to online samples, satisfying 0< <1. Directly using the above formula requires saving the label and weight of each online sample. Therefore, the above formula is optimized into the following formula:

[0188] ;

[0189] in, is the historical weight decay factor, satisfying 0< <1Due to the scene switching in the video encoding process, the online samples of the earlier scenes have no advantage over the offline samples for the encoding of the new scene CU, so the weight should be attenuated according to the time of appearance. Here, the online samples use the geometric progression attenuation factor, let , , , ,but:

[0190] and ;

[0191] ;

[0192] in, Indicates The probability of an online sample.

[0193] In this way, the weights of the offline model (i.e., the binary classifier based on the lightweight gradient boosting model) only need to be initialized once, and the online samples do not need to be saved, only the intermediate results need to be updated. and In addition, there is no need to explicitly handle scene conversion. After several online samples of the new scene are added, the online samples of the old scene will automatically eliminate the impact.

[0194] According to the descriptions of 1 to 6 above, see Figure 2 , Figure 2 : is a flow chart of the technical solution of this embodiment, and the specific steps of the technical solution of this embodiment are as follows:

[0195] Step 1: Run the encoder on the video in the video training set to obtain frequency domain features (i.e., first image frequency domain features), classification labels (i.e., first classification labels), and weights (i.e., first weights);

[0196] Step 2: Use the frequency domain features, classification labels and weights in step 1 to train a set of binary classifiers based on LightGBM to obtain a set of initial binary classifiers;

[0197] Step 3: According to the feature importance of a set of initial binary classifiers in step 2, perform feature selection on a set of initial binary classifiers, and retrain a set of initial binary classifiers based on the reselected features to obtain a set of reduced-dimensional binary classifiers, and save the iterative loss;

[0198] Step 4: Adjust the parameters of the set of dimension reduction binary classifiers in step 3 to reduce the size of a set of initial binary classifiers to obtain a final set of partition decision classifiers;

[0199] Step 5: Integrate a set of partition decision classifiers obtained in step 4 into the encoder, initialize the online sample data of each partition decision classifier, and automatically make partition mode decisions;

[0200] Among them, the description of automated decision-making:

[0201] (1) For a square CU with a side length greater than or equal to 8, the 64-dimensional frequency domain features (i.e., the frequency domain features of the second image) are calculated as in step 1. The five classifiers only need to be calculated once, and the reduced-dimensional features can be obtained as needed during prediction and classification.

[0202] (2) By default, the encoding is carried out in the order of no division, QT division mode, BH division mode, BV division mode, TH division mode and TV division mode;

[0203] (3) For the QT division mode, the second probability is given by the QT-BH classifier ,like Then the QT partition mode is terminated early and jumps to the BH and TH partition modes in (5);

[0204] (4) For the QT partitioning pattern, the probability is given by the QT-BV classifier ,like Then the QT division mode is terminated in advance;

[0205] (5) For the BH and TH partitioning modes, the probability is given by the BH-BV classifier ,like Then the BH partition mode and the TH partition mode are terminated in advance;

[0206] (6) For the BV and TV partition modes, the probability is given by the BH-BV classifier ,like The BV partition mode and the TV partition mode are terminated in advance;

[0207] (7) For the TH partition mode, if the BH partition mode has been run but is not the best partition mode that has been run, or the BH partition mode is terminated early but the BV partition mode has been run, the TH partition mode is terminated early, otherwise the BH-TH classifier gives the probability ,like Then the TH partitioning mode is terminated in advance;

[0208] (8) For the TV partition mode, if the BV partition mode has been run but is not the best partition mode that has been run, or the BV partition mode is terminated early but the BH partition mode has been run, then the TV partition mode is terminated early, otherwise the BV-TV classifier gives the probability ,like The TV division mode is terminated in advance;

[0209] Step 6: After the current CU completes the partitioning mode decision of step 5, if there is any partitioning mode corresponding to the partitioning decision classifier in step 5 that has not been terminated in advance, the online sample data of this partitioning decision classifier is updated according to the RD Cost of these two partitions. For example, after executing step 5, assuming that the QT partitioning mode, TH partitioning mode, and TV partitioning mode are terminated, and the BH partitioning mode and BV partitioning mode are retained, for the QT-BH classifier, only the BH partitioning mode and the BV partitioning mode of the BH-BV classifier have available RD Cost at the same time, so only the online sample information of the BH-BV classifier is updated; and for the QT-BH classifier, only the BH partitioning mode has RD Cost, so the online sample information of the QT-BH classifier is not updated. If the two partitioning modes in a classifier do not have RD Cost, the online sample information of the corresponding classifier is not updated.

[0210] For better explanation, the following experiments were performed in this embodiment:

[0211] In order to verify the feasibility and performance of the method of this embodiment, the method of this embodiment is implemented on the VVC reference software VTM18.2 encoder and runs in All Intra mode. The encoding time change and BDBR change compared with the original VTM18.2 encoder are tested when QP is 22, 27, 32 and 37.

[0212] BDBR (Bjontegaard Delta Bit Rate) is used to compare the bit rate difference between two video encoders at the same quality. If it is less than 0, it means that the second encoder requires a lower bit rate than the first encoder to compress the video of the same quality, and the encoding efficiency is higher. If it is greater than 0, it means that the second encoder loses encoding efficiency.

[0213] The method of this embodiment is a fast algorithm, and its goal is to lose less coding efficiency in exchange for coding speed and save coding time. Table 2 shows the performance of the optimized encoder on the test sequence. Cactus, RaceHorsesC, RaceHorses and FourPeople in Table 2 are the file names of common test files, and Cactus, RaceHorsesC, RaceHorses and FourPeople indicate that there are four video sequences.

[0214] Table 2 Performance of optimization methods on test sequences

[0215]

[0216] Experiments show that the method of this embodiment can significantly save encoding time and has little impact on encoding quality.

[0217] Compared with the prior art, the technical solution of this embodiment has the following advantages:

[0218] 1. The problems existing in the spatial domain features of images were analyzed, and the image frequency domain feature decision-making partitioning mode was introduced to improve the multi-scale expression ability, information density and robustness of the features, and improve the performance of the partitioning decision model;

[0219] 2. The correlation between the partition modes was analyzed, and the partition modes with strong correlation and competitiveness were selected for pairwise comparison and appropriate sample weights were set, thus avoiding one-to-many and many-to-many comparisons, reducing the difficulty of model training and eliminating the ambiguity of the classification task;

[0220] 3. The LightGBM model is introduced to analyze the frequency domain features of the image to make decisions on CU division, which can effectively select features and control the model scale, saving decision-making time and having good decision-making performance;

[0221] 4. The statistical distribution of different partitioning modes was analyzed, and the BT partitioning was retained with emphasis when making partitioning mode decisions. The QT and TT partitioning modes were screened more strictly, which reduced the risk of missing BT, reduced the encoding time of redundant QT and TT, and improved the encoding efficiency and encoding time cost performance;

[0222] 5. A weighted prediction scheme with scene adaptability is designed. It does not require scene conversion or saving each online sample. It makes full use of the partitioning mode of CU actual operation, and generates weighted predictions of online samples and offline training models, which improves the adaptability in different coding scenarios and increases coding efficiency.

[0223] Reference Figure 3 The embodiment of the present application further provides a VVC coding unit fast division system based on image frequency domain features, the system comprising a data acquisition unit 100, a first training unit 200, a second training unit 300, a scale reduction unit 400, a division mode decision unit 500 and a fast division unit 600, wherein:

[0224] The data acquisition unit 100 is used to use an encoder to run on each video in the video training set to acquire a first image frequency domain feature, a first classification label and a first weight corresponding to the video training set;

[0225] The first training unit 200 is used to train a set of binary classifiers based on the lightweight gradient boosting model by using the first image frequency domain feature, the first classification label and the first weight to obtain a set of initial binary classifiers, wherein the binary classifiers based on the lightweight gradient boosting model include two division modes;

[0226] The second training unit 300 is used to perform feature selection through a set of initial binary classifiers, and use the selected features to train the set of initial binary classifiers to obtain a set of reduced-dimensional binary classifiers;

[0227] A scale reduction unit 400 is used to adjust parameters of a set of dimension reduction binary classifiers to reduce the scale of the set of dimension reduction binary classifiers to obtain a set of partition decision classifiers;

[0228] A partitioning mode decision unit 500 is used to initialize online sample data and make a decision on the partitioning mode in each partitioning decision classifier according to the online sample data to obtain a set of target partitioning modes;

[0229] The fast partitioning unit 600 is used to quickly partition the VVC coding unit according to a set of target partitioning modes.

[0230] It should be noted that, since the VVC coding unit fast division system based on image frequency domain features in this embodiment and the above-mentioned VVC coding unit fast division method based on image frequency domain features are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0231] An embodiment of the present application also provides an electronic device, comprising: at least one control processor and a memory for communicating with the at least one control processor.

[0232] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0233] The non-transient software program and instructions required to implement the VVC coding unit fast division method based on image frequency domain features of the above embodiment are stored in the memory. When executed by the processor, the VVC coding unit fast division method based on image frequency domain features in the above embodiment is executed, for example, the above described method is executed. Figure 1 The method comprises steps S100 to S600.

[0234] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0235] The embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by one or more control processors, so that the one or more control processors can execute a VVC coding unit fast division method based on image frequency domain features in the above method embodiment, for example, executing the above described Figure 1 The functions of method steps S100 to S600 in the embodiment.

[0236] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0237] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

[0238] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A method for fast division of VVC coding units based on image frequency domain features, characterized in that: The method comprises: Using an encoder to run on each video in a video training set, obtaining a first image frequency domain feature, a first classification label, and a first weight corresponding to the video training set; Using the first image frequency domain feature, the first classification label and the first weight to train a set of binary classifiers based on a lightweight gradient boosting model to obtain a set of initial binary classifiers, wherein the binary classifiers based on the lightweight gradient boosting model include two division modes; Performing feature selection through the set of initial binary classifiers, and using the selected features to train the set of initial binary classifiers to obtain a set of reduced-dimensional binary classifiers; Adjusting parameters of the set of dimension reduction binary classifiers to reduce the size of the set of dimension reduction binary classifiers to obtain a set of partition decision classifiers; Initialize online sample data, and make decisions on the partitioning modes in each of the partitioning decision classifiers according to the online sample data to obtain a set of target partitioning modes, specifically including: Obtain the second image frequency domain features of the unit to be encoded at the current moment in the target video, and initialize the second classification labels and second weights corresponding to the encoded units before the current moment as online sample data; Inputting the second image frequency domain feature into the group of partition decision classifiers to obtain a first probability output by each partition decision classifier; Using the online sample data to modify the first probability output by each partition decision classifier to obtain a second probability; Making a decision on a partitioning mode in each of the partitioning decision classifiers according to the second probability to obtain a set of target partitioning modes; The VVC coding unit is quickly divided according to the set of target division modes.

2. The method for fast dividing VVC coding units based on image frequency domain features according to claim 1, characterized in that: The using the online sample data to modify the first probability to obtain a second probability includes: ； in, represents the second probability, represents the number of online samples, represents the first probability, represents a weight factor of an offline sample relative to an online sample, wherein the offline sample includes the first image frequency domain feature, the first classification label and the first weight, represents the number of offline samples, represents the weighted mean of offline samples, Indicates Classification labels of online samples, represents the geometric progression attenuation factor, Indicates The weight of the online samples.

3. The method for fast dividing VVC coding units based on image frequency domain features according to claim 1, characterized in that: The step of making a decision on the partition mode in each of the partition decision classifiers according to the second probability to obtain a set of target partition modes includes: Presetting an upper threshold and a lower threshold for each of the partition decision classifiers, wherein the upper threshold is greater than the lower threshold; Compare the second probability with the upper threshold and the lower threshold to obtain a comparison result; A decision is made on the partitioning mode in each of the partitioning decision classifiers according to the comparison result to obtain a set of target partitioning modes.

4. The method for fast dividing VVC coding units based on image frequency domain features according to claim 1, characterized in that: Before obtaining the second image frequency domain feature of the to-be-encoded unit at the current moment in the target video, the method further includes: Determine the size of the unit to be encoded at the current moment in the target video, and if the size of the unit to be encoded meets the size condition, downsample the unit to be encoded at the current moment in the target video to obtain a downsampled feature; and calculate the second image frequency domain feature of the unit to be encoded at the current moment in the target video according to the downsampled feature; If the size of the unit to be encoded does not meet the size condition, the second image frequency domain features of the unit to be encoded at the current moment in the target video are not calculated, and multiple division modes are directly decided by rate-distortion optimization, and the VVC coding unit is quickly divided according to the decided division mode.

5. The method for fast dividing VVC coding units based on image frequency domain features according to claim 1, characterized in that: After initializing the online sample data and making a decision on the partition mode in each of the partition decision classifiers according to the online sample data to obtain a set of target partition modes, the method further includes: If the group of target partition modes includes at least two partition modes, and two of the at least two partition modes belong to the same partition decision classifier, then updating the online sample data belonging to the same partition decision classifier according to the rate-distortion costs of the two partition modes belonging to the same partition decision classifier; If the group of target partition modes includes at least two partition modes, and the remaining partition modes except the two partition modes belonging to the same partition decision classifier do not have two partition modes belonging to the same partition decision classifier, then the online sample data of the partition decision classifiers corresponding to the remaining partition modes are not updated; If the group of target partition modes includes at least two partition modes, and there are no two partition modes belonging to the same partition decision classifier among all the partition modes, then the online sample data of the partition decision classifiers corresponding to all the partition modes are not updated; If the set of target partition modes includes only one partition mode, the online sample data of all partition decision classifiers are not updated.

6. The method for fast dividing VVC coding units based on image frequency domain features according to claim 1, characterized in that: The fast partitioning of the VVC coding unit according to the set of target partitioning modes includes: If the set of target partition modes includes only one target partition mode, the one target partition mode is used to quickly partition the VVC coding unit; If the group of target partition modes includes at least two target partition modes, rate-distortion optimization is used to decide the target partition mode among the at least two target partition modes to obtain the decided target partition mode; and the VVC coding unit is quickly divided using the decided target partition mode.

7. A VVC coding unit fast division system based on image frequency domain features, characterized in that: The system comprises: A data acquisition unit, configured to use an encoder to run on each video in a video training set to acquire a first image frequency domain feature, a first classification label, and a first weight corresponding to the video training set; A first training unit is used to train a set of binary classifiers based on a lightweight gradient boosting model by using the first image frequency domain feature, the first classification label and the first weight to obtain a set of initial binary classifiers, wherein the binary classifiers based on the lightweight gradient boosting model include two division modes; A second training unit is used to perform feature selection through the set of initial binary classifiers, and train the set of initial binary classifiers using the selected features to obtain a set of reduced-dimensional binary classifiers; A scale reduction unit, used for adjusting parameters of the set of dimension reduction binary classifiers to reduce the scale of the set of dimension reduction binary classifiers to obtain a set of partition decision classifiers; The partition mode decision unit is used to initialize the online sample data and make a decision on the partition mode in each of the partition decision classifiers according to the online sample data to obtain a set of target partition modes, specifically including: Obtain the second image frequency domain features of the unit to be encoded at the current moment in the target video, and initialize the second classification labels and second weights corresponding to the encoded units before the current moment as online sample data; Inputting the second image frequency domain feature into the group of partition decision classifiers to obtain a first probability output by each partition decision classifier; Using the online sample data to modify the first probability output by each partition decision classifier to obtain a second probability; Making a decision on a partitioning mode in each of the partitioning decision classifiers according to the second probability to obtain a set of target partitioning modes; A fast partitioning unit is used to quickly partition the VVC coding unit according to the set of target partitioning modes.

8. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the VVC coding unit fast division method based on image frequency domain features as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the VVC coding unit fast division method based on image frequency domain features according to any one of claims 1 to 6.

Images