Coding partitioning method and apparatus, device, and storage medium

Abstract

The present application discloses a coding partitioning method and apparatus, a device, and a storage medium. The coding partitioning method comprises: acquiring a target coding frame type corresponding to a target image frame; determining target coding information on the basis of the target coding frame type; performing partitioning depth prediction on the basis of a target partitioning prediction model corresponding to the target coding frame type, and the target coding information, to obtain a target partitioning depth value corresponding to a reference coding block in the target image frame, wherein the target partitioning prediction model is obtained by performing training in advance on the basis of sample coding information corresponding to a sample image frame and a label partitioning depth value corresponding to a reference coding block in the sample image frame, and the label partitioning depth value is determined on the basis of an actual coding block partitioning result of the sample image frame; and on the basis of the target partitioning depth value, determining a target coding block partitioning result corresponding to the target image frame.

Description

Coding division method, device, apparatus and storage medium

[0001] The present application claims priority to the Chinese patent application No. 202411311840.X, filed on September 19, 2024, with the Chinese Patent Office, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the computer technology field, and particularly relates to a coding division method, device, apparatus and storage medium. BACKGROUND

[0003] With the rapid development of computer technology, it is often necessary to encode image frames based on a coding mode such as High Efficiency Video Coding (HEVC) to reduce storage space and transmission costs.

[0004] In the encoding process, the image frames need to be divided into multiple coding blocks, and each coding block needs to be subjected to pixel compression processing. In the coding division, the division mode of the image frames is usually enumerated based on multiple coding block sizes to obtain the optimal coding block division result.

[0005] However, at least the following problem exists in the related art: the above coding division mode needs to consume a large amount of time, greatly reducing the encoding speed. SUMMARY

[0006] The present application provides a coding division method, device, apparatus and storage medium to realize fast and efficient coding division, improve the encoding speed while ensuring the encoding quality.

[0007] The present application provides a coding division method, comprising: obtaining a target coding frame type corresponding to a target image frame to be encoded; determining target coding information corresponding to the target image frame based on the target coding frame type; performing division depth prediction on the target image frame based on a target division prediction model corresponding to the target coding frame type and the target coding information, to obtain a target division depth value corresponding to a reference coding block in the target image frame, wherein the target division prediction model is obtained by pre-training based on sample coding information corresponding to a sample image frame and a label division depth value corresponding to a reference coding block in the sample image frame, and the label division depth value is determined based on an actual coding block division result of the sample image frame; and determining a target coding block division result corresponding to the target image frame based on the target division depth value.

[0008] The application further provides an encoding partitioning device, comprising: a target encoding frame type acquisition module configured to acquire a target encoding frame type corresponding to a target image frame to be encoded; a target encoding information determination module configured to determine target encoding information corresponding to the target image frame based on the target encoding frame type; a partitioning depth prediction module configured to perform partitioning depth prediction on the target image frame based on a target partitioning prediction model corresponding to the target encoding frame type and the target encoding information, to obtain a target partitioning depth value corresponding to a reference encoding block in the target image frame, wherein the target partitioning prediction model is obtained by training based on sample encoding information corresponding to a sample image frame and label partitioning depth values corresponding to reference encoding blocks in the sample image frame, and the label partitioning depth values are determined based on actual encoding block partitioning results of the sample image frame; and a partitioning result determination module configured to determine a target encoding block partitioning result corresponding to the target image frame based on the target partitioning depth value.

[0009] The application further provides an electronic device, comprising: one or more processors; a memory configured to store one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the encoding partitioning method provided by any of the embodiments of the application.

[0010] The application further provides a computer-readable storage medium, having a computer program stored thereon, which, when executed by a processor, implements the encoding partitioning method described above. BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is a flowchart of an encoding partitioning method provided by an embodiment of the application;

[0012] FIG. 2 is an example diagram of a luminance component of an image frame provided by an embodiment of the application;

[0013] FIG. 3 is an example diagram of two determination manners of a label partitioning depth value provided by an embodiment of the application;

[0014] FIG. 4 is an example diagram of a network architecture of a target partitioning prediction model provided by an embodiment of the application;

[0015] FIG. 5 is an example diagram of a network architecture of another target partitioning prediction model provided by an embodiment of the application;

[0016] FIG. 6 is an example diagram of a network architecture of a target partitioning prediction model during training provided by an embodiment of the application;

[0017] FIG. 7 is a flowchart of another encoding partitioning method provided by an embodiment of the application;

[0018] FIG. 8 is an example diagram of a spatial consistency adjustment process according to an embodiment of the present application;

[0019] FIG. 9 is a structural diagram of an encoding division apparatus according to an embodiment of the present application;

[0020] FIG. 10 is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] FIG. 1 is a flowchart of an encoding division method according to an embodiment of the present application. The embodiment can be applied to the case of encoding block division of an image frame in an encoding process of the image frame. The method can be executed by an encoding division apparatus, which can be implemented by software and / or hardware and integrated in an electronic device. As shown in FIG. 1, the method includes the following steps:

[0022] S110, obtaining a target encoding frame type corresponding to a target image frame to be encoded.

[0023] The target image frame can refer to an image frame currently to be encoded. The image frame can refer to a frame in a video or a single image. In a real-time communication scenario, the target image frame can refer to an image frame currently to be encoded in a real-time communication video. The real-time communication scenario can be live streaming, remote video conference or cloud gaming. The target encoding frame type refers to an encoding frame type matched with the target image frame, that is, an encoding frame to which the target image frame needs to be encoded. The target encoding frame type can include an intra-frame encoding frame or an inter-frame encoding frame. The intra-frame encoding frame only needs to refer to blocks within the frame for encoding and does not need to refer to other frames. The inter-frame encoding frame needs to refer to other frames for encoding. For example, the intra-frame encoding frame can refer to a key frame (i.e., an I frame) or an instant decoding refresh frame (i.e., an IDR frame). The inter-frame encoding frame can refer to a forward prediction encoding frame (i.e., a P frame) or a bidirectional prediction encoding frame (i.e., a B frame). The forward prediction encoding frame needs to refer to a front frame of the current frame for encoding generation. The bidirectional prediction encoding frame needs to refer to both the front frame and the rear frame of the current frame for encoding generation.

[0024] When the target image frame is encoded by using an encoder, the target encoding frame type to which the target image frame needs to be encoded can be determined based on any manner, so as to accurately perform encoding division on the target image frame based on the obtained target encoding frame type.

[0025] S120, determining target encoding information corresponding to the target image frame based on the target encoding frame type.

[0026] The target coding information can refer to information required for coding the target image frame. For example, the target coding information can include a luminance component of the target image frame, or can include a luminance component of a target reference frame in addition to the luminance component of the target image frame. The luminance component refers to luminance information in a color image frame, which can reflect luminance variation of the image frame, as shown in FIG. 2. The target reference frame refers to another image frame referenced when coding the target image frame. For example, when the target coding frame type is a P frame, the target reference frame can refer to a previous frame of the target image frame. When the target coding frame type is a B frame, the target reference frame can refer to a previous frame and a next frame of the target image frame. Different target coding frame types correspond to different target coding information.

[0027] For example, S120 can include: if the target coding frame type is an intra coding frame, determining the luminance component of the target image frame as the target coding information corresponding to the target image frame; and if the target coding frame type is an inter coding frame, determining the luminance component of the target image frame and the luminance component of the target reference frame as the target coding information corresponding to the target image frame.

[0028] When the target coding frame type is an intra coding frame, only the current frame information is required when coding the target image frame, and the target coding information includes the luminance component of the target image frame at this time. When the target coding frame type is an inter coding frame, in addition to the current frame information, other frames are required when coding the target image frame, and the target coding information includes the luminance component of the target image frame and the luminance component of the reference image frame at this time.

[0029] For example, when the target coding frame type is an inter coding frame, the target coding information further includes quantization parameter information of the target image frame. The quantization parameter information (QP) is set in advance for the target image frame, and is used to adjust the details of the target image frame, and ultimately to adjust the picture quality. For example, the larger the QP, the more a pixel value is used to assign a value in a larger area of the coded image. The more the number of different pixel values in the image, the more storage space is required. However, the larger the QP, the worse the quality of the coded image and the less clear the image, so the quantization parameter information also affects the coding effect of the inter coding frame. Therefore, when inter coding, the target coding information needs to include the quantization parameter information in addition to the luminance component of the target image frame and the luminance component of the reference image frame, so as to further improve the coding quality.

[0030] S130, performing target partition depth prediction on the target image frame based on the target partition prediction model corresponding to the target coding frame type and the target coding information, to obtain a target partition depth value corresponding to a reference coding block in the target image frame, wherein the target partition prediction model is obtained by pre-training based on sample coding information corresponding to a sample image frame and label partition depth values corresponding to reference coding blocks in the sample image frame, and the label partition depth values are determined based on actual coding block partition results of the sample image frame.

[0031] The target partition prediction model can be an intra-frame partition prediction model corresponding to an intra-frame coding frame or an inter-frame partition prediction model corresponding to an inter-frame coding frame. The intra-frame partition prediction model is a network model for predicting partition depth values of reference coding blocks in a target image frame based on target coding information corresponding to an intra-frame coding frame. The inter-frame partition prediction model is a network model for predicting partition depth values of reference coding blocks in a target image frame based on target coding information corresponding to an inter-frame coding frame. A coding block refers to a partial region divided in an image frame. A reference coding block refers to a coding block with a preset size after initial partition. A reference coding block refers to an initial partition unit. A reference coding block corresponds to a partition depth value one-to-one. The partition depth value can refer to the depth value of the reference coding block in the coding tree. For example, starting from a 64x64 coding block, the partition result can have four sizes of coding blocks, i.e., 64x64, 32x32, 16x16, and 8x8, and their partition depth values are 0, 1, 2, and 3, respectively. The preset size of the reference coding block can be used to represent the size of the coding block represented by a partition depth value, which can be set based on business requirements in advance. For example, the preset size of the reference coding block can be 16x16 or 8x8, etc. The target partition prediction model can perform initial partition on the target image frame based on the preset size to obtain each reference coding block with the preset size, and predict the target partition depth value of each reference coding block based on the target coding information.

[0032] The sample image frame can be an existing image frame used for training the model. The sample coding information is determined based on a target coding frame type corresponding to the sample image frame, and the determination process is the same as that of the target coding information, which will not be described herein. The label division depth value can refer to an actual division depth value corresponding to a reference coding block in the sample image frame. The actual coding block division result is a coding block division result when the coding quality of the sample image frame is optimal, so as to ensure the coding division quality of the target division prediction model. The label division depth value corresponding to each reference coding block in the sample image frame can be determined based on a preset size of the reference coding block and the actual coding block division result of the sample image frame. For example, two determination methods are given in FIG. 3. The first determination method is that when the preset size of the reference coding block is 16x16, the 32x32 coding block in the actual coding block division result is divided into four 16x16 reference coding blocks, and the label division depth value corresponding to each reference coding block is determined as 1; the 16x16 coding block in the actual coding block division result is directly taken as a reference coding block, and the label division depth value is determined as 2; and the four 8x8 coding blocks in the actual coding block division result are combined into one 16x16 reference coding block, and the label division depth value is determined as 3, thereby obtaining the label division depth value corresponding to each 16x16 reference coding block in the sample image frame. The second determination method is that when the preset size of the reference coding block is 8x8, the 32x32 coding block in the actual coding block division result is divided into 16 8x8 reference coding blocks, and the label division depth value corresponding to each reference coding block is determined as 1; the 16x16 coding block is divided into four 8x8 reference coding blocks, and the label division depth value corresponding to each reference coding block is determined as 3; and the 8x8 coding block in the actual coding block division result is directly taken as a reference coding block, and the label division depth value is determined as 3, thereby obtaining the label division depth value corresponding to each 8x8 reference coding block in the sample image frame.

[0033] Since a 64x64 coding block does not appear during actual coding division, only three sizes of division depth values, i.e., 32x32, 16x16 and 8x8, are labeled in the sample image frame. The number of label division depth values obtained by the first determination method is less than that obtained by the second determination method, so that model training by the first determination method can further simplify the network complexity of the model and further improve the division prediction speed of the model. For example, when the preset size of the reference coding block is 16x16, for an image frame with a resolution of 720x1280, the label size of the whole image division depth is (720 / 16)x(1280 / 16), and each 16x16 reference coding block is represented by one label division depth value.

[0034] The supervised model training of the depth value can be performed in advance based on the sample coding information corresponding to the sample image frame under each coding frame type and the label corresponding to the reference coding block in the sample image frame. The training prediction model corresponding to each coding frame type after the training is completed, such as the intra-frame partition prediction model and the inter-frame partition prediction model. In actual application, the target partition prediction model corresponding to the target coding frame type is determined based on the training prediction model corresponding to each coding frame type obtained in advance, and the target coding information can be input into the target partition prediction model to predict the partition depth value of each reference coding block in the target image frame. The predicted target partition depth value is output, so as to obtain the target partition depth value corresponding to each reference coding block in the target image frame.

[0035] Exemplarily, before the target coding information is input into the target partition prediction model, the luminance component in the target coding information can be pixel filled based on the preset size of the reference coding block, to obtain a luminance component with a preset image size, so as to input the filled luminance component into the target partition prediction model for prediction of the partition depth.

[0036] In the traditional coding, the partition result of each coding block is affected by the partition result of the surrounding coding block and whether there is a coding block around. In order to simulate the partition logic in the traditional coding process as much as possible, the luminance component of the image frame is pixel filled in the up, down, left and right four directions before the partition prediction of each image frame, such as filling the pixel value of 114, so that the filling width of each direction is 16, such as the image resolution of the 720*1280 image after filling is changed to 752*1312, and the 16*16 small block with all values of 114 represents the non-existent coding block. By pixel filling the luminance component, the accuracy of coding partition can be further guaranteed.

[0037] S140, based on the target partition depth value, determining a target coding block partition result corresponding to the target image frame.

[0038] The target partition depth value can be used to represent the partition relationship between the reference coding block and the finally partitioned target coding block. For example, the target partition depth value can be used to represent whether the target coding block is obtained by merging the reference coding block, or the target coding block is obtained by partitioning the reference coding block, or the reference coding block is directly taken as the target coding block. The target coding block partition result corresponding to the target image frame includes a plurality of finally partitioned target coding blocks in the target image frame. For example, the target partition depth value can include a first depth value, a second depth value or a third depth value. The first depth value (such as 1) is used to represent that the reference coding blocks in the partition region (such as a region of 2x2 reference coding blocks) need to be merged into one target coding block. The second depth value (such as 2) is used to represent that the reference coding block is the finally partitioned target coding block. The third depth value (such as 3) is used to represent that the reference coding block needs to be further partitioned into a plurality of target coding blocks.

[0039] Based on the target partition depth value corresponding to each reference coding block in the target image frame, all reference coding blocks in the target image frame are processed to obtain the target coding block partition result corresponding to the target image frame at one time. The process of determining the target coding block partition result based on the target partition depth value is a reciprocal process of the process of determining the label partition depth value based on the actual coding block partition result, which can be referred to the above description.

[0040] For example, when the preset size of the reference coding block is 16x16, if the target partition depth value corresponding to a reference coding block in the target image frame is the first depth value 1, the reference coding block and the surrounding 2x2 reference coding blocks are merged into one target coding block which is finally partitioned as 32x32. If the target partition depth value corresponding to the reference coding block is the second depth value 2, the reference coding block is taken as the finally partitioned target coding block without further partitioning. If the target partition depth value corresponding to the reference coding block is the third depth value 3, the reference coding block is further partitioned into 2x2 target coding blocks of 8x8.

[0041] The technical scheme of the embodiment determines the target coding information matched with the target image frame based on the target coding frame type corresponding to the target image frame to be coded, and performs division depth prediction on the target image frame based on the target division prediction model corresponding to the target coding frame type and the target coding information, so that the target division depth value corresponding to the reference coding block in the target image frame can be quickly obtained, and the target coding block division result corresponding to the target image frame can be determined at one time based on the target division depth value, realizing fast and efficient coding division. The target division prediction model is obtained by pre-training based on the sample coding information corresponding to the sample image frame and the label division depth value corresponding to the reference coding block in the sample image frame, and the label division depth value is determined based on the actual coding block division result of the sample image frame, so that the trained target division prediction model can be used to perform fast and efficient coding division while ensuring coding quality, improving coding speed.

[0042] On the basis of the above technical scheme, when the target coding information corresponding to each coding frame type only includes information of the same type, the division prediction model corresponding to each coding frame type has the same network architecture but different network weight values, which need to be determined based on model training. For example, when the target coding information corresponding to the intra-frame coding frame and the target coding information corresponding to the inter-frame coding frame both only include the luminance component, the intra-frame division prediction model and the inter-frame division prediction model have the same network architecture but different network weight values. Illustratively, the division prediction model corresponding to each coding frame type can include a feature extraction sub-model, a semantic segmentation sub-model and a division prediction sub-model. The information flow processing process in the division prediction model corresponding to each coding frame type is the same. The target division prediction model determined from the division prediction model corresponding to each coding frame type can also include a feature extraction sub-model, a semantic segmentation sub-model and a division prediction sub-model.

[0043] The feature extraction sub-model is configured to extract low-level features of the image from the luminance component, such as contour features, edge features, texture features, shape features, and the like. The semantic segmentation sub-model can be configured to extract target features in the image. For example, the semantic segmentation sub-model can include multiple Atrous spatial pyramid pooling (ASPP) modules in parallel, so as to extract features of targets at different scales, and further improve the accuracy of the segmentation prediction. For example, the semantic segmentation sub-model is composed of 4 parallel ASPP modules, each of which corresponds to a different convolution rate, so as to obtain a larger receptive field and extract more context information. The segmentation prediction sub-model can be a classification network model configured to predict the segmentation depth value. For example, the segmentation prediction sub-model can be obtained by concatenating convolution layers and pooling layers in a neural network. The segmentation prediction sub-model uses depth separable convolution instead of a regular convolution layer, so as to improve the calculation speed.

[0044] Exemplarily, referring to FIG. 4, S130 can include: inputting the target coding information into the feature extraction sub-model to extract features, to obtain first feature information; inputting the first feature information into the semantic segmentation sub-model to extract semantic features, to obtain second feature information; splicing the target coding information, the first feature information, and the second feature information, and inputting the spliced target feature information into the segmentation prediction sub-model to predict the segmentation depth of the target image frame; and based on the output of the segmentation prediction sub-model, obtaining the target segmentation depth value corresponding to the reference coding block in the target image frame.

[0045] When the target partition prediction model is an intra-frame partition prediction model, the luminance component of the target image frame is input into the feature extraction sub-model for feature extraction to obtain first feature information, and the first feature information is input into the semantic segmentation sub-model for feature extraction at different scales to obtain second feature information. The luminance component of the target image frame, the first feature information, and the second feature information are spliced (i.e., feature concatenation), and the spliced target feature information is input into the partition prediction sub-model for partition depth prediction to obtain and output the target partition depth value corresponding to each reference coding block in the target image frame. Alternatively, when the target partition prediction model is an inter-frame partition prediction model, the luminance component of the target image frame and the luminance component of the target reference frame are spliced and input into the feature extraction sub-model for feature extraction to obtain first feature information, and the first feature information is input into the semantic segmentation sub-model for feature extraction at different scales to obtain second feature information. The luminance component of the target image frame, the luminance component of the target reference frame, the first feature information, and the second feature information are spliced (i.e., feature concatenation), and the spliced target feature information is input into the partition prediction sub-model for target image frame partition depth prediction to obtain and output the target partition depth value corresponding to each reference coding block in the target image frame.

[0046] On the basis of the above technical solution, when the target coding frame type is an inter-frame coding frame, and the target coding information includes not only the luminance component of the target image frame and the luminance component of the target reference frame, but also the quantization parameter information of the target image frame, the target partition prediction model is an inter-frame partition prediction model. In this case, the target partition prediction model includes not only the feature extraction sub-model, the semantic segmentation sub-model, and the partition prediction sub-model, but also a quantization coding sub-model, as shown in FIG. 5.

[0047] The quantization parameter information is a constant matrix, and each value on the matrix represents the quantization value of each reference coding block in the target image frame. The quantization value can take 52 values, with the maximum being 51 and the minimum being 0. The quantization coding sub-model can be used to one-hot encode or normalize the input quantization parameter information. Compared with the one-hot encoding method, the normalization processing method used by the quantization coding sub-model can obtain higher prediction speed and accuracy. For the same target image frame and reference image frame, if the given quantization parameter information is different, the corresponding inter-frame partition result will also be different.

[0048] Exemplarily, inputting the spliced target feature information into the division prediction sub-model for division depth prediction of the target image frame can include: inputting the quantization parameter information of the target image frame into the quantization coding sub-model for information coding to obtain third feature information; performing information superposition on the spliced target feature information and the third feature information, and inputting the superposed target feature information into the division prediction sub-model for division depth prediction of the target image frame.

[0049] Referring to FIG. 5, when the target division prediction model is an inter-frame division prediction model, the quantization parameter information of the target image frame is input into the quantization coding sub-model for information coding to obtain third feature information. The luminance component of the target image frame, the luminance component of the target reference frame, the first feature information output by the feature extraction sub-model, and the second feature information output by the semantic segmentation sub-model are spliced (i.e., feature concatenation) to obtain spliced target feature information. The target feature information and the third feature information are superimposed (i.e., feature addition), and the superimposed target feature information is input into the division prediction sub-model for division depth prediction to obtain and output the target division depth value corresponding to each reference coding block in the target image frame. By using the quantization coding sub-model to code the input quantization parameter information, the accuracy of inter-frame division prediction can be further improved.

[0050] On the basis of the above technical solution, the target division prediction model is obtained by pre-training based on sample coding information corresponding to a sample image frame and label division depth values and label coding modes corresponding to reference coding blocks in the sample image frame. In training the target division prediction model, the target division prediction model further includes a mode prediction sub-model, which is used to predict the coding mode based on the spliced target feature information or the superimposed target feature information.

[0051] The label coding mode can refer to the coding mode used by the reference coding block when the coding quality is optimal. The coding mode refers to the way of compression processing of the pixel values in the reference coding block. For example, the coding mode can be a vertical mode or a horizontal mode, etc. The horizontal mode is to represent each row of pixel values in the reference coding block by using one pixel value. The vertical mode is to represent each column of pixel values in the reference coding block by using one pixel value, such as performing average processing on all pixel values in the same column to obtain a mean value representing the entire pixel value of the reference coding block, thereby saving storage space and transmission cost.

[0052] In training the partition prediction model corresponding to each encoding frame type, a mode prediction sub-model can be added in the partition prediction model, so as to predict the encoding mode while predicting the depth, and the predicted encoding mode is used as an auxiliary loss for model training, thereby improving the prediction effect of the partition depth. For example, referring to FIG. 6, in training the target partition prediction model, the input information of the mode prediction sub-model and the partition prediction sub-model is the same, which is the spliced target feature information. The training process of the target partition prediction model is as follows: the sample encoding information corresponding to the sample image frame is input into the target partition prediction model to be trained to predict the partition depth and the encoding mode, and the sample partition depth value and the sample encoding mode corresponding to each reference encoding block in the sample image frame are obtained and output. Based on the loss function, the training error is determined according to the sample partition depth value, the sample encoding mode, the label partition depth value and the label encoding mode corresponding to each reference encoding block, and the training error is back propagated to the target partition prediction model to be trained, the network weight in the target partition prediction model to be trained is adjusted, until a preset convergence condition is reached, such as the iteration number being equal to a preset number, or the training error tends to be stable, the training is determined to be ended, and the target partition prediction model is obtained. Similarly, in training the target partition prediction model shown in FIG. 5, the input information of the mode prediction sub-model and the partition prediction sub-model is also the same, which is the superimposed target feature information, and the training process can be referred to the above description. By introducing the prediction of the encoding mode in training the model, the prediction accuracy of the partition depth can be further improved, and the encoding effect is further improved.

[0053] For example, after the training of the target partition prediction model is completed, the mode prediction sub-model in the target partition prediction model can be deleted, so as to only use the partition prediction sub-model to predict the partition depth, or the mode prediction sub-model can be retained, so as to simultaneously predict the partition depth value and the encoding mode by using the target partition prediction model, and the encoding speed is further improved.

[0054] FIG. 7 is a flowchart of another encoding partition method provided by an embodiment of the present application, which illustrates the step of “determining the target encoding block partition result corresponding to the target image frame based on the target partition depth value” based on the above embodiment. The explanation of the same or corresponding terms in the above embodiment is not repeated here.

[0055] Referring to FIG. 7, the another encoding partition method provided by the embodiment includes the following steps:

[0056] S210, obtaining a target encoding frame type corresponding to a target image frame to be encoded.

[0057] S220, determining target encoding information corresponding to the target image frame based on the target encoding frame type.

[0058] S230, performing partition depth prediction on the target image frame based on the target partition prediction model corresponding to the target coding frame type and the target coding information, to obtain a target partition depth value corresponding to a reference coding block in the target image frame.

[0059] S240, adjusting the target partition depth value based on a spatial consistency adjustment manner of the partition depth value, so that the adjusted target partition depth value meets the spatial consistency.

[0060] The spatial consistency adjustment manner is determined in advance based on position restriction information of each partition depth value in the image frame, so that the partition result can meet the purpose of spatial consistency.

[0061] When predicting the partition depth value corresponding to each reference coding block in the target image frame by using the target partition prediction model, the restriction of spatial consistency is not considered, such as that all partition depth values in a partition region are the same, or a depth value cannot exist alone in a partition region. After obtaining the target partition depth value corresponding to each reference coding block in the target image frame, it is necessary to detect whether the target partition depth value corresponding to each reference coding block meets the spatial consistency. If there is a target partition depth value that does not meet the spatial consistency, the spatial consistency adjustment manner needs to be used to adjust the target partition depth value, so that all adjusted target partition depth values meet the spatial consistency, thereby further improving the accuracy of coding partition.

[0062] Exemplarily, S240 can include: determining the number of target reference coding blocks in each partition region in the target image frame; and adjusting the target partition depth value corresponding to the reference coding block in each partition region based on the number of target reference coding blocks corresponding to each partition region, so that the adjusted target partition depth value meets the spatial consistency.

[0063] Each partition region includes a first number of reference coding blocks, such as a neighborhood region composed of 2x2 reference coding blocks. The target partition depth value corresponding to the target reference coding block is a first depth value. The first depth value is used to represent that the reference coding blocks in the partition region need to be merged into one coding block. For example, the first depth value is 1, which represents that 2x2 reference coding blocks need to be merged into one final partition target coding block.

[0064] For each divided region in the target image frame, based on the target divided depth value corresponding to each reference coding block in the divided region, the number of target reference coding blocks with the target divided depth value being a first depth value is counted in the divided region. Whether adjustment of the target divided depth value in the divided region is needed and the adjustment manner can be determined based on the number of target reference coding blocks, so that the adjusted divided region has spatial consistency. Through the unit of the divided region, spatial consistency adjustment can be more quickly and accurately performed.

[0065] The target divided depth value can be a second depth value and a third depth value in addition to the first depth value. The second depth value (such as 2) is used to represent that the reference coding block is a finally divided target coding block. The third depth value (such as 3) is used to represent that the reference coding block needs to be divided into multiple target coding blocks again. When the target divided depth value is the second depth value and the third depth value, spatial consistency is satisfied and adjustment is not needed. When the target divided depth value is the first depth value, adjustment is needed based on other target divided depth values in the divided region in which the target divided depth value is located, so that the divided region satisfies spatial consistency, and thus the number of target reference coding blocks with the target divided depth value being the first depth value in each divided region needs to be counted. For example, the limitation of spatial consistency is that when the preset size of the reference coding block is 16x16, if the target divided depth value corresponding to a reference coding block is the first depth value 1, it indicates that the reference coding block should be located in the divided result of 32x32 coding blocks, and thus the target divided depth values corresponding to the other three reference coding blocks in the 2x2 divided region in which the reference coding block is located need to be adjusted to 1, so that the final divided result of 2x2 16x16 reference coding blocks forming 1 32x32 coding block can be satisfied.

[0066] Exemplarily, based on the number of target reference coding blocks corresponding to each divided region, the target divided depth value corresponding to each reference coding block in the divided region is adjusted, so that the adjusted target divided depth value satisfies spatial consistency, which can include: if the number of target reference coding blocks corresponding to the current divided region is one, the target divided depth value corresponding to the target reference coding block is adjusted to the second depth value; if the number of target reference coding blocks corresponding to the current divided region is less than a first number and greater than or equal to a second number, the target divided depth value corresponding to the remaining reference coding blocks other than the target reference coding block in the current divided region is adjusted to the first depth value.

[0067] The second depth value is used to represent that the reference coding block is a finally divided target coding block. The second number can be half of the first number. For example, the first number is 4, and the second number can be 2.

[0068] Each segmented region in the target image frame can be used as the current segmented region for adjustment. If only one target reference coding block exists in the current segmented region, since the first depth value cannot exist independently (meaning the prediction result of the target reference coding block is incorrect), the target segmentation depth value corresponding to the target reference coding block can be adjusted from the first depth value 1 to the second depth value 2, as shown in the upper left segmented region of Figure 8. If at least a second number of target reference coding blocks exist in the current segmented region, it indicates that all reference coding blocks in the current segmented region need to be merged. In this case, the target segmentation depth values ​​corresponding to the remaining reference coding blocks in the current segmented region (excluding the target reference coding block) can also be adjusted to the first depth value 1, so that the target segmentation depth value of all reference coding blocks in the current segmented region is the first depth value 1, as shown in the upper right segmented region of Figure 8. If the number of target reference coding blocks corresponding to the current segmented region is zero (meaning there are no target reference coding blocks in the current segmented region), no adjustment is needed.

[0069] S250. Based on the adjusted target segmentation depth value, determine the target coding block segmentation result corresponding to the target image frame.

[0070] Based on the target partitioning depth value corresponding to each reference coding block that satisfies spatial consistency, processing all reference coding blocks in the target image frame can more accurately obtain the target coding block partitioning result corresponding to the target image frame. For example, referring to Figure 8, after spatial consistency adjustment, if the target partitioning depth value corresponding to the reference coding block is the first depth value 1, then the reference coding block is merged with the surrounding 2×2 reference coding blocks into a final target coding block of 32×32. If the target partitioning depth value corresponding to the reference coding block is the second depth value 2, then the reference coding block is taken as the final target coding block and no further partitioning is needed. If the target partitioning depth value corresponding to the reference coding block is the third depth value 3, then the reference coding block is further partitioned into 2×2 8×8 target coding blocks.

[0071] The technical solution of this embodiment adjusts the target segmentation depth value by means of spatial consistency adjustment based on the segmentation depth value, so that the adjusted target segmentation depth value satisfies spatial consistency. Based on the adjusted target segmentation depth value, the target coding block segmentation result corresponding to the target image frame can be determined more accurately, thereby further improving the accuracy of coding segmentation.

[0072] The following are embodiments of the encoding partitioning device provided in this application. This device and the encoding partitioning method described above belong to the same concept. For details not described in detail in the embodiments of the encoding partitioning device, please refer to the embodiments of the encoding partitioning method described above.

[0073] Fig. 9 is a structural schematic diagram of an encoding division apparatus provided by an embodiment of the present application. The embodiment can be applied to the case of encoding block division of an image frame in the process of encoding the image frame. As shown in Fig. 9, the apparatus comprises a target encoding frame type obtaining module 310, a target encoding information determining module 320, a division depth prediction module 330 and a division result determining module 340.

[0074] The target encoding frame type obtaining module 310 is configured to obtain a target encoding frame type corresponding to a target image frame to be encoded. The target encoding information determining module 320 is configured to determine target encoding information corresponding to the target image frame based on the target encoding frame type. The division depth prediction module 330 is configured to perform division depth prediction on the target image frame based on a target division prediction model corresponding to the target encoding frame type and the target encoding information, to obtain a target division depth value corresponding to a reference encoding block in the target image frame. The target division prediction model is obtained by training based on sample encoding information corresponding to a sample image frame and label division depth values corresponding to reference encoding blocks in the sample image frame. The label division depth values are determined based on actual encoding block division results of the sample image frame. The division result determining module 340 is configured to determine a target encoding block division result corresponding to the target image frame based on the target division depth value.

[0075] The technical scheme of the embodiment determines target encoding information matched with a target image frame based on a target encoding frame type corresponding to the target image frame to be encoded, and performs division depth prediction on the target image frame based on a target division prediction model corresponding to the target encoding frame type and the target encoding information, so that a target division depth value corresponding to a reference encoding block in the target image frame can be quickly obtained, and a target encoding block division result corresponding to the target image frame can be determined at one time based on the target division depth value, thereby realizing fast and efficient encoding division. The target division prediction model is obtained by training based on sample encoding information corresponding to a sample image frame and label division depth values corresponding to reference encoding blocks in the sample image frame. The label division depth values are determined based on actual encoding block division results of the sample image frame. Thus, the trained target division prediction model can be used to perform fast and efficient encoding division while ensuring encoding quality, thereby improving encoding speed.

[0076] The target encoding information determining module 320 is configured to:

[0077] If the target encoding frame type is an intra-frame encoding frame, the luminance component of the target image frame is determined as the target encoding information corresponding to the target image frame. If the target encoding frame type is an inter-frame encoding frame, the luminance component of the target image frame and the luminance component of a target reference frame are determined as the target encoding information corresponding to the target image frame.

[0078] When the target coding frame type is an inter-coding frame, the target coding information further comprises quantization parameter information of the target image frame.

[0079] The target partition prediction model comprises a feature extraction sub-model, a semantic segmentation sub-model and a partition prediction sub-model; the partition depth prediction module 330 comprises: a first feature information acquisition unit configured to input the target coding information into the feature extraction sub-model to perform feature extraction and obtain first feature information; a second feature information acquisition unit configured to input the first feature information into the semantic segmentation sub-model to perform semantic feature extraction and obtain second feature information; a partition depth prediction unit configured to perform information splicing on the target coding information, the first feature information and the second feature information, and input the spliced target feature information into the partition prediction sub-model to perform partition depth prediction of the target image frame; and a target partition depth value acquisition unit configured to obtain a target partition depth value corresponding to a reference coding block in the target image frame based on an output of the partition prediction sub-model.

[0080] When the target coding frame type is an inter-coding frame and the target coding information further comprises quantization parameter information of the target image frame, the target partition prediction model further comprises a quantization coding sub-model; and the partition depth prediction unit is configured to: input the quantization parameter information of the target image frame into the quantization coding sub-model to perform information coding and obtain third feature information; perform information superposition on the spliced target feature information and the third feature information, and input the superposed target feature information into the partition prediction sub-model to perform partition depth prediction of the target image frame.

[0081] The target partition prediction model is obtained by training based on sample coding information corresponding to a sample image frame and label partition depth values and label coding modes corresponding to reference coding blocks in the sample image frame; when training the target partition prediction model, the target partition prediction model further comprises a mode prediction sub-model, which is used to perform coding mode prediction based on spliced target feature information or superposed target feature information.

[0082] The partition result determination module 340 comprises: a spatial consistency adjustment unit configured to adjust the target partition depth value based on a spatial consistency adjustment manner of the partition depth value, so that the adjusted target partition depth value satisfies spatial consistency; and a partition result determination unit configured to determine a target coding block partition result corresponding to the target image frame based on the adjusted target partition depth value.

[0083] The space consistency adjustment unit comprises: a target reference coding block quantity determination subunit, configured to determine a target reference coding block quantity in each partition region in the target image frame, wherein each partition region comprises a first quantity of reference coding blocks, and a target reference coding block corresponds to a target partition depth value of a first depth value, the first depth value being used to represent that the reference coding blocks in the partition region need to be merged into one coding block; and a space consistency adjustment subunit, configured to adjust the target partition depth value corresponding to the reference coding blocks in each partition region based on the target reference coding block quantity corresponding to each partition region, so that the adjusted target partition depth value satisfies the space consistency.

[0084] The space consistency adjustment subunit is configured to: if the target reference coding block quantity corresponding to the current partition region is one, adjust the target partition depth value corresponding to the target reference coding block to a second depth value, the second depth value being used to represent that the reference coding block is a finally partitioned target coding block; and if the target reference coding block quantity corresponding to the current partition region is less than the first quantity and greater than or equal to a second quantity, adjust the target partition depth value corresponding to the remaining reference coding blocks in the current partition region except the target reference coding block to the first depth value.

[0085] The encoding partitioning apparatus provided by the embodiments of the present application can execute the encoding partitioning method provided by any of the embodiments of the present application, and has the corresponding function modules and effects of executing the encoding partitioning method.

[0086] In the embodiments of the encoding partitioning apparatus, the plurality of units and modules are only divided according to the function logic, but are not limited to the above division, as long as the corresponding functions can be implemented; in addition, the names of the plurality of function units are only for the convenience of mutual differentiation, and do not limit the protection scope of the present application.

[0087] FIG. 10 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. FIG. 10 shows a block diagram of an exemplary electronic device 12 suitable for implementing the embodiments of the present application. The electronic device 12 shown in FIG. 10 is only one example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0088] As shown in FIG. 10, the electronic device 12 is in the form of a general computing device. The components of the electronic device 12 can include one or more processors or processing units 16, a system memory 28, and a bus 18 connecting the different system components, including the system memory 28 and the processing unit 16.

[0089] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0090] Electronic device 12 includes a variety of computer system readable media. These media can be any available media that is accessible by electronic device 12 and includes both volatile and nonvolatile media, removable and non-removable media.

[0091] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory 30 (RAM) and / or cache memory 32. Electronic device 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (typically called a "hard drive"). Computer system 12 can also provide a floppy disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk (e.g., a CD-ROM, DVD-ROM or other optical media). In these instances, each drive can be connected to bus 18 by one or more data media interfaces. System memory 28 can include a number of program products for use by electronic device 12, such as at least one program product for implementing embodiments of the present application.

[0092] Program / utility 40 having a set of program modules 42 can be stored in system memory 28 by way of example, such program modules 42 include an operating system, one or more application programs, other program modules, and program data, each or a combination of which

[0093] Electronic device 12 can also communicate with one or more external devices 14 such as a keyboard or pointing device, a display 24, etc. ; other devices or systems that enable a user to interact with electronic device 12 and / or one or more devices or systems that enable electronic device 12 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface(s) 22. Still yet, electronic device 12 can communicate with one or more networks such as a local area network (LAN), a wide area network (WAN), and / or the Internet through network adapter 20. As an example, network adapter 20 can include a modem, a network card (wireless or wired), or other well-known interface devices. As depicted, network adapter 20 communicates with the other

[0094] Processing unit(s) 16 can execute instructions and manipulate data to perform a variety of functions, such as implementing an encoding partitioning method according to any of the embodiments described herein.

[0095] As will be appreciated by persons skilled in the art, a processor can also be implemented as an embodiment of the encoding partitioning method according to any of the embodiments described herein.

[0096] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement steps of an encoding division method provided by any embodiment of the application. The method comprises the following steps: obtaining a target encoding frame type corresponding to a target image frame to be encoded; determining target encoding information corresponding to the target image frame based on the target encoding frame type; performing division depth prediction on the target image frame based on a target division prediction model corresponding to the target encoding frame type and the target encoding information, to obtain a target division depth value corresponding to a reference encoding block in the target image frame, wherein the target division prediction model is obtained by training based on sample encoding information corresponding to a sample image frame and label division depth values corresponding to reference encoding blocks in the sample image frame, and the label division depth values are determined based on actual encoding block division results of the sample image frame; and determining a target encoding block division result corresponding to the target image frame based on the target division depth value.

[0097] The computer storage medium of the embodiment of the application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. Examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, device or component.

[0098] The computer readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which a computer readable program code is borne. Such a propagated data signal can take multiple forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can send, propagate or transmit a program for use by or in connection with an instruction execution system, apparatus or device.

[0099] The program code embodied on the computer readable media can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0100] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a LAN or a WAN, or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0101] Those skilled in the art should appreciate that the modules or steps of the present application described above can be implemented with general computing devices, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and they can be implemented with program codes executable by the computing devices, so that they can be stored in storage devices and executed by the computing devices, or they can be respectively made into multiple integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

Claims

1. A coding partitioning method, comprising: obtaining a target coding frame type corresponding to a target image frame to be coded; determining target coding information corresponding to the target image frame based on the target coding frame type; performing partition depth prediction on the target image frame based on a target partition prediction model corresponding to the target coding frame type and the target coding information to obtain a target partition depth value corresponding to a reference coding block in the target image frame, wherein the target partition prediction model is obtained by pre-training based on sample coding information corresponding to a sample image frame and label partition depth values corresponding to reference coding blocks in the sample image frame, and the label partition depth values are determined based on actual coding block partitioning results of the sample image frame; determining a target coding block partitioning result corresponding to the target image frame based on the target partition depth value.

2. The method of claim 1, wherein, The determining of the target coding information corresponding to the target image frame based on the target coding frame type comprises: in response to the target coding frame type being an intra coding frame, determining a luminance component of the target image frame as the target coding information corresponding to the target image frame; in response to the target coding frame type being an inter coding frame, determining a luminance component of the target image frame and a luminance component of a target reference frame as the target coding information corresponding to the target image frame.

3. The method of claim 2, wherein, When the target coding frame type is the inter coding frame, the target coding information further comprises quantization parameter information of the target image frame.

4. The method of claim 1, wherein, The target partition prediction model comprises a feature extraction sub-model, a semantic segmentation sub-model, and a partition prediction sub-model. The performing of the partition depth prediction on the target image frame based on the target partition prediction model corresponding to the target coding frame type and the target coding information to obtain the target partition depth value corresponding to the reference coding block in the target image frame comprises: inputting the target coding information into the feature extraction sub-model to perform feature extraction to obtain first feature information; inputting the first feature information into the semantic segmentation sub-model to perform semantic feature extraction to obtain second feature information; concatenating the target coding information, the first feature information, and the second feature information, and inputting the concatenated target feature information into the partition prediction sub-model to perform partition depth prediction on the target image frame; obtaining the target partition depth value corresponding to the reference coding block in the target image frame based on an output of the partition prediction sub-model.

5. The method of claim 4, wherein, When the target coding frame type is the inter coding frame and the target coding information further comprises the quantization parameter information of the target image frame, the target partition prediction model further comprises a quantization coding sub-model. The inputting of the concatenated target feature information into the partition prediction sub-model to perform the partition depth prediction on the target image frame comprises: inputting the quantization parameter information of the target image frame into the quantization coding sub-model to perform information coding to obtain third feature information; The target feature information after splicing and the third feature information are superimposed, and the superimposed target feature information is input into the division prediction sub-model to perform division depth prediction on the target image frame.

6. The method of claim 4 or 5, wherein, The target division prediction model is obtained by training based on sample coding information corresponding to a sample image frame and label division depth values and label coding modes corresponding to reference coding blocks in the sample image frame; During training of the target division prediction model, the target division prediction model further includes a mode prediction sub-model configured to predict a coding mode based on the target feature information after splicing or the target feature information after superimposing.

7. The method of claim 1, wherein, The target coding block division result corresponding to the target image frame is determined based on the target division depth values, including: adjusting the target division depth values based on a spatial consistency adjustment mode to make the adjusted target division depth values meet spatial consistency; determining the target coding block division result corresponding to the target image frame based on the adjusted target division depth values.

8. The method of claim 7, wherein, The target division depth values are adjusted based on a spatial consistency adjustment mode to make the adjusted target division depth values meet spatial consistency, including: determining the number of target reference coding blocks in each division region in the target image frame, wherein each division region includes a first number of reference coding blocks, the target reference coding blocks correspond to a first depth value, and the first depth value is used to represent that the reference coding blocks in the division region need to be merged into one coding block; adjusting the target division depth values of the reference coding blocks in each division region based on the number of target reference coding blocks corresponding to each division region to make the adjusted target division depth values meet spatial consistency.

9. The method of claim 8, wherein, The target division depth values of the reference coding blocks in each division region are adjusted based on the number of target reference coding blocks corresponding to each division region to make the adjusted target division depth values meet spatial consistency, including: in response to the number of target reference coding blocks corresponding to a current division region being one, adjusting the target division depth value corresponding to the target reference coding block to a second depth value, and the second depth value is used to represent that the reference coding block is a finally divided target coding block; in response to the number of target reference coding blocks corresponding to a current division region being less than the first number and greater than or equal to a second number, adjusting the target division depth values corresponding to the remaining reference coding blocks in the current division region except the target reference coding block to the first depth value.

10. An encoding division apparatus, comprising: a target coding frame type acquisition module configured to acquire a target coding frame type corresponding to a target image frame to be encoded; a target coding information determination module configured to determine target coding information corresponding to the target image frame based on the target coding frame type; The division depth prediction module is configured to perform division depth prediction on the target image frame based on a target division prediction model corresponding to the target coding frame type and the target coding information, to obtain a target division depth value corresponding to a reference coding block in the target image frame, wherein the target division prediction model is obtained by training based on sample coding information of a sample image frame and label division depth values corresponding to reference coding blocks in the sample image frame, and the label division depth values are determined based on actual coding block division results of the sample image frame. The division result determination module is configured to determine a target coding block division result corresponding to the target image frame based on the target division depth value. 11.An electronic device, comprising: at least one processor; a memory configured to store at least one program; when the at least one program is executed by the at least one processor, the at least one processor implements the coding division method according to any one of claims 1-9. 12.A computer readable storage medium storing a computer program, which when executed by a processor, implements the coding division method according to any one of claims 1-9.

Images