Image division method, intelligent encoding method, intelligent decoding method and device

Abstract

The application discloses an image division method, an intelligent encoding method, an intelligent decoding method and device. The image division method comprises the following steps: determining an image block; inputting the image block into a division strategy confirmation network to obtain a target division strategy matched with the image block; the target division strategy is a division strategy in a division strategy set preset for a preset division depth, and the division strategy set at least comprises a division strategy with a difference value greater than or equal to 2 from the division depth of the image block; and dividing the image block according to a division mode combination corresponding to the target division strategy. The application can improve the efficiency of image division.

Description

Technical Field

[0001] This application relates to the field of block partitioning technology, and in particular to an image partitioning method, an intelligent encoding method, an intelligent decoding method, and an apparatus. Background Technology

[0002] In video encoding, the input is a series of image frames. However, encoding a single frame requires dividing it into several LCUs (Maximum Coding Units), and then further dividing each LCU into different CUs (Coding Units). Video encoding is performed using CUs as the unit. Within each CU, prediction blocks (PUs) and transform blocks (TUs) are further divided.

[0003] The current image block partitioning process is a recursive process that checks all partitioning patterns to determine the partitioning pattern of the image block. This results in a large number of partitioning pattern RDO processes being required during the image block partitioning process, leading to high complexity and low efficiency in image block partitioning. Summary of the Invention

[0004] This application provides an image segmentation method, an intelligent encoding method, an intelligent decoding method, and an apparatus, which can improve the efficiency of image segmentation.

[0005] To achieve the above objectives, this application provides an image segmentation method, which includes:

[0006] Identify image blocks;

[0007] The image patch is input into the partitioning strategy confirmation network to obtain the target partitioning strategy that matches the image patch; the target partitioning strategy is a partitioning strategy in a set of partitioning strategies preset for a preset partitioning depth, and the set of partitioning strategies includes at least partitioning strategies whose difference from the partitioning depth of the image patch is greater than or equal to 2.

[0008] The image block is divided according to the combination of division modes corresponding to the target division strategy.

[0009] Wherein, the segmentation depth of the image block is equal to 0, and the preset segmentation depth is the maximum segmentation depth in the target encoding and decoding technology standard.

[0010] The partitioning strategy confirmation network is trained using sample image blocks and their label vectors as training data, with the partitioning strategy that has the highest usage probability of the output sample image block being the partitioning strategy corresponding to the largest element in the label vector as the training target. The label vector is determined based on the encoding cost of the sample image block under various partitioning strategies.

[0011] in,

[0012] The values ​​of each element in the label vector of the sample image block are negatively correlated with the cost difference of the partitioning strategy corresponding to each element. The cost difference of the partitioning strategy is equal to the difference between the encoding cost of the sample image block under the partitioning strategy and the minimum encoding cost of the sample image block; or,

[0013] In the label vector of the sample image block, the element corresponding to the partitioning strategy with the minimum encoding cost has a value of 1, and the values ​​of the remaining elements are 0.

[0014] The step of inputting image patches into a segmentation strategy verification network to obtain a target segmentation strategy that matches the image patches includes:

[0015] The network is validated by the partitioning strategy, and the probability of partitioning the image patch using various partitioning strategies in the set of partitioning strategies is predicted.

[0016] Based on the obtained probabilities, the target partitioning strategy is selected from the set of partitioning strategies.

[0017] The step of selecting the target partitioning strategy from the partitioning strategy set based on the obtained probabilities includes:

[0018] The partitioning strategy with the highest probability in the set of partitioning strategies is taken as the target partitioning strategy; or...

[0019] A preset number of partitioning strategies are selected from the set of partitioning strategies, wherein the probability of selecting the preset number of partitioning strategies is greater than the probability of all unselected partitioning strategies; the image block is partitioned according to the partitioning mode combination corresponding to each partitioning strategy in the preset number of partitioning strategies; the encoding cost corresponding to each partitioning strategy is calculated based on the partitioning result corresponding to each partitioning strategy; the partitioning strategy with the minimum encoding cost is selected as the target partitioning strategy.

[0020] Before inputting image patches into the partitioning strategy confirmation network, the method further includes:

[0021] Confirm whether the segmentation depth of the image block is within a preset range;

[0022] If it is confirmed that the segmentation depth of the image block is within a preset range, then the steps of inputting the image block into the segmentation strategy confirmation network to obtain a target segmentation strategy that matches the image block, and segmenting the image block according to the segmentation mode combination corresponding to the target segmentation strategy are executed, and the preset segmentation depth is equal to the maximum value contained in the preset range.

[0023] If it is confirmed that the segmentation depth of the image block is not within the preset range, the image block is segmented using various segmentation modes specified in the target encoding and decoding technology standard, and the encoding cost corresponding to each segmentation mode is determined; the segmentation mode corresponding to the minimum value among the determined encoding costs is taken as the segmentation module of the image block.

[0024] To achieve the above objectives, this application also provides an intelligent decoding method, which includes:

[0025] Obtain the bitstream after encoding the image blocks;

[0026] The image block partitioning strategy is parsed from the bitstream, and the image block partitioning strategy is obtained based on the image partitioning method described above;

[0027] The image block is divided according to the division strategy, and the division result of the image block is decoded.

[0028] To achieve the above objectives, this application also provides an image segmentation method, which includes:

[0029] The first network determines whether to divide the image into blocks;

[0030] In response to the first network determining to divide the image block, the second network obtains the probability of dividing the image block through each of the various division modes among all division modes;

[0031] Based on the obtained probabilities, a partitioning mode is selected from all partitioning modes, and the result of partitioning the image block by the selected partitioning mode is determined.

[0032] Wherein, the size of the first network and the size of the image patch correspond; and / or

[0033] The second network corresponds to the size of the image block.

[0034] The step of selecting a partitioning mode from all partitioning modes based on the obtained probabilities, and determining the result of partitioning the image patch using the selected partitioning mode, includes:

[0035] Select a preset number of partitioning patterns from all partitioning patterns, wherein the probability of selecting the preset number of partitioning patterns is greater than the probability of all unselected partitioning patterns;

[0036] When the preset number is greater than or equal to 2, the image block is divided using each of the preset number of division patterns; the encoding cost corresponding to each division pattern is calculated based on the division result corresponding to each division pattern; and the division pattern corresponding to the minimum value among the determined encoding costs is taken as the final division pattern of the image block.

[0037] When the preset number is less than 2, the image block is divided using the selected division mode.

[0038] The step of selecting a preset number of partitioning patterns from all partitioning patterns includes:

[0039] A preset number of partitioning patterns are selected from all feasible partitioning patterns of the image block as specified in the video coding standard to which the method is applied; wherein the probability of selecting the preset number of partitioning patterns is greater than the probability of all unselected partitioning patterns among all feasible partitioning patterns.

[0040] The step of determining whether to divide the image patch using the first network includes the following:

[0041] Confirm whether the segmentation depth of the image block is within a preset range;

[0042] If it is confirmed that the segmentation depth of the image block is within the preset range, then the following steps are executed: determining whether to segment the image block through the first network; responding to the determination by the first network to segment the image block, obtaining the probability of segmenting the image block through various segmentation modes among all segmentation modes through the second network; and selecting a segmentation mode from all segmentation modes based on the obtained probabilities, and determining the result of segmenting the image block through the selected segmentation mode.

[0043] If it is confirmed that the segmentation depth of the image block is not within the preset range, the image block is segmented using various segmentation modes specified in the video coding technology standard applied by the method, and the coding cost corresponding to each segmentation mode is determined; the segmentation mode corresponding to the minimum value among the determined coding costs is taken as the segmentation module of the image block.

[0044] To achieve the above objectives, this application also provides an image decoding method, the method comprising:

[0045] Obtain the bitstream after encoding the image blocks;

[0046] The image block partitioning pattern is parsed from the bitstream, and the image block partitioning pattern is obtained based on the image partitioning method described above;

[0047] The image block is divided according to the division pattern, and the division result of the image block is decoded.

[0048] To achieve the above objectives, this application also provides an intelligent coding method, which includes:

[0049] The image block segmentation results are determined based on the above image segmentation method;

[0050] The index information corresponding to the partitioning result is encoded to obtain the encoded bitstream.

[0051] To achieve the above objectives, this application also provides a decoder, which includes a processor; the processor is used to execute instructions to implement the steps of the above method.

[0052] To achieve the above objectives, this application also provides an encoder that includes a processor; the processor is configured to execute instructions to implement the steps of the above method.

[0053] To achieve the above objectives, this application also provides a computer-readable storage medium for storing instruction / program data that can be executed to implement the above methods.

[0054] This application inputs image blocks into a partitioning strategy confirmation network to obtain target partitioning strategies that match the image blocks. Then, the image blocks are partitioned according to the partitioning pattern combination corresponding to the target partitioning strategy. Moreover, this application uses a set of partitioning strategies that includes partitioning strategies with a partitioning depth difference greater than or equal to 2 to determine the target partitioning strategy of the image blocks. This allows at least two partitioning patterns corresponding to the partitioning depths to be confirmed at once. This eliminates the need to confirm the partitioning patterns of the image blocks step by step, saving time in selecting image block partitioning patterns, significantly reducing the complexity of image block partitioning pattern selection, improving the efficiency of image partitioning, and increasing the richness of image block partitioning. Attached Figure Description

[0055] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0056] Figure 1 This is a flowchart illustrating one embodiment of the image segmentation method of this application;

[0057] Figure 2 This is a schematic diagram of the structure of the partitioning strategy confirmation network in one embodiment of the image partitioning method of this application;

[0058] Figure 3 This is a schematic diagram of the combination of segmentation modes corresponding to the segmentation strategy in one embodiment of the image segmentation method of this application;

[0059] Figure 4 This is a schematic diagram of the partitioning mode in the AVS3 standard;

[0060] Figure 5 This is a flowchart illustrating another embodiment of the image segmentation method of this application;

[0061] Figure 6 This is a schematic diagram of the structure of the first network in another embodiment of the image segmentation method of this application;

[0062] Figure 7 This is a schematic diagram of the structure of the second network in another embodiment of the image segmentation method of this application;

[0063] Figure 8 This is a flowchart illustrating one embodiment of the intelligent coding method of this application;

[0064] Figure 9 This is a flowchart illustrating one embodiment of the intelligent decoding method of this application;

[0065] Figure 10 This is a schematic diagram of one embodiment of the encoder of this application;

[0066] Figure 11 This is a schematic diagram of the structure of one embodiment of the decoder of this application;

[0067] Figure 12 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. In addition, unless otherwise specified (e.g., "or additionally" or "or in alternatives"), the term "or" as used herein refers to a non-exclusive "or" (i.e., "and / or"). Furthermore, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0069] Currently, the partitioning of CUs is done recursively. The main process of recursively partitioning CUs from LCUs is as follows:

[0070] (1) Traverse all partitioning patterns of the current partitioning node and perform the following (2) operation for each partitioning pattern;

[0071] (2) Determine whether the partitioning pattern currently being traversed is usable based on the current node information;

[0072] (2.1) If the current partitioning mode can be used on the current node, there are two cases:

[0073] (2.1.1) The current mode is no partitioning, the leaf nodes of the tree structure are obtained, the partitioning stops, and the next partitioning mode in (1) continues;

[0074] (2.1.2) The current mode is other partitioning modes. According to the partitioning strategy, several partitioning child nodes can be obtained. Traverse all child nodes, and each child node is used as the new current node to perform all operations (1) and (2).

[0075] (2.2) If the current partitioning mode cannot be used on the current node, the current node skips the current partitioning mode and continues to the next partitioning mode in (1);

[0076] (3) The partitioning process ends when all partitioning strategies of the root node have been traversed.

[0077] The image block partitioning process described above is actually a process of recursively determining the partitioning pattern of an image block by considering all partitioning patterns. This results in a large number of partitioning pattern RDO processes being required during the image block partitioning process, leading to high complexity in image block partitioning.

[0078] Based on this, such as Figure 1 As shown, this application provides an image segmentation method according to a first embodiment, which includes the following steps. It should be noted that the step numbers are for simplification only and are not intended to limit the execution order of the steps. The execution order of each step in this embodiment can be arbitrarily changed without departing from the technical concept of this application.

[0079] S101: Determine the image block.

[0080] The current image block to be divided in the image can be determined so that the current image block to be divided can be divided using steps S102 and S103.

[0081] In one feasible implementation, the image partitioning method of this embodiment can be applied to the H.265 video coding standard. In the H.265 coding standard, an image can be divided into CTU, CU, PU, ​​and TU, etc. In this implementation, the image block determined in step S101 can refer to CTU or CU, etc., in the H.265 coding standard.

[0082] In another possible implementation, the image partitioning method of this embodiment can be applied to the AVS3 audio and video coding standard. In the AVS3 coding standard, an image can be divided into CTU, CU, PU, ​​and TU, etc. In this implementation, the image block determined in step S101 can refer to CTU or CU, etc., in the AVS3 coding standard.

[0083] S102: Input the image patch into the partitioning strategy confirmation network to obtain the target partitioning strategy that matches the image patch.

[0084] After confirming the image patch, the image patch can be input into the partitioning strategy confirmation network to confirm the target partitioning strategy that matches the image patch, so as to obtain the result of partitioning the image patch according to the partitioning pattern combination corresponding to the target partitioning strategy.

[0085] Among them, the target partitioning strategy is a partitioning strategy from the set of partitioning strategies preset for a preset partitioning depth.

[0086] The preset segmentation depth can be set according to actual conditions and is not limited here. For example, it can be the maximum segmentation depth in the target codec standard. The target codec standard can be understood as the video codec standard applied to the image segmentation method of this embodiment.

[0087] The aforementioned set of partitioning strategies for a preset partitioning depth may include multiple partitioning strategies within the preset partitioning depth.

[0088] In one feasible approach, the set of partitioning strategies preset for a preset partitioning depth can include all partitioning strategies within the preset partitioning depth. That is, the set of partitioning strategies preset for a preset partitioning depth can be the union of all partitioning strategies at all partitioning depths less than or equal to the preset partitioning depth. For example, if the preset partitioning depth is 6, then the set of partitioning strategies preset for the preset partitioning depth can include all partitioning strategies at partitioning depth 6, all partitioning strategies at partitioning depth 5, all partitioning strategies at partitioning depth 4, all partitioning strategies at partitioning depth 3, all partitioning strategies at partitioning depth 2, all partitioning strategies at partitioning depth 1, and all partitioning strategies at partitioning depth 0.

[0089] Furthermore, all partitioning strategies within the preset partitioning depth are also related to the partitioning depth of the image block itself. Specifically, all partitioning strategies within the preset partitioning depth can be understood as the union of all partitioning strategies at all partitioning depths greater than or equal to the partitioning depth of the image block itself and less than or equal to the preset partitioning depth. For example, if the partitioning depth of the image block itself is 0 and the preset partitioning depth is 2, then the set of partitioning strategies preset for the preset partitioning depth may include all partitioning strategies at a partitioning depth of 2, all partitioning strategies at a partitioning depth of 1, and all partitioning strategies at a partitioning depth of 0. All partitioning strategies at a partitioning depth of 0 refer to strategies that do not partition the image block. All partitioning strategies at a partitioning depth of 1 include strategies that use the partitioning pattern in the standard applied by the image partitioning method of this application to partition the image block once to obtain at least two image sub-blocks. Partitioning strategies at a partitioning depth of 2 refer to strategies that use the partitioning pattern in the standard applied by the image partitioning method of this application to partition the image block once to obtain at least two image sub-blocks, and then use the partitioning pattern in the standard applied by the image partitioning method of this application to partition at least one of the at least two image sub-blocks.

[0090] In this implementation, before step S102, all feasible partitioning strategies conforming to the target encoding / decoding technology standard within the preset partitioning depth can be listed to obtain a partitioning strategy set.

[0091] In another possible implementation, the preset set of partitioning strategies for a preset partitioning depth may include a subset of all partitioning strategies within the preset partitioning depth. These subset of partitioning strategies within the preset partitioning depth may be user-preferred partitioning strategies, thus allowing image partitioning based on user preferences.

[0092] In this implementation, before step S102, all feasible partitioning strategies conforming to the target codec technology standard within the preset partitioning depth can be listed; some strategies are selected from the listed partitioning strategies, and the selected strategies are used as a partitioning strategy set.

[0093] Furthermore, the partitioning strategy set also includes at least partitioning strategies with a partitioning depth difference greater than or equal to 2. The partitioning depth difference refers to the difference between the partitioning depth of the partitioning strategy and the partitioning depth of the image patch itself. In other words, at least some partitioning strategies in the partitioning strategy set have a partitioning depth difference greater than or equal to 2 compared to the partitioning depth of the image patch itself. Conversely, the partitioning depth of all sub-blocks obtained by partitioning the image patch according to a partitioning strategy with a partitioning depth difference equal to X has a partitioning depth difference less than or equal to X compared to the partitioning depth of the image patch itself, and at least some sub-blocks obtained by partitioning the image patch according to a partitioning strategy with a partitioning depth difference equal to X have a partitioning depth difference equal to X compared to the partitioning depth of the image patch itself. By using the target partitioning strategy for the image patch determined using a partitioning strategy set including those with a partitioning depth difference greater than or equal to 2, the partitioning pattern corresponding to at least two partitioning depths can be confirmed at once. This eliminates the need to confirm the partitioning pattern of the image patch step by step, saving time in selecting the image patch partitioning pattern, significantly reducing the complexity of image patch partitioning pattern selection, and increasing the richness of image patch partitioning. Of course, the partitioning strategy set can also include partitioning strategies with a partitioning depth difference of less than 2. That is, the partitioning strategy set can include partitioning strategies with a partitioning depth difference of 1 (i.e., strategies that use the partitioning mode in the target encoding and decoding technology standard to partition the image block once to obtain at least two image sub-blocks) and / or partitioning strategies with a partitioning depth difference of 0 (i.e., strategies that do not partition the image block).

[0094] Preferably, the image block is divided at a depth of 0, i.e., the image block is a CTU. The preset division depth is the maximum division depth of the standard used by the image division method, and the division strategy set includes at least the division strategy of the preset division depth. In this way, the division mode combination of the final CU obtained by dividing from the CTU can be directly selected in one step by the division strategy set that includes the division strategy of the preset division depth. This avoids the step-by-step determination of the division mode for each division depth and avoids a large number of division mode RDO processes, thus greatly reducing the complexity of block division mode selection.

[0095] In step S102, the image block can be directly input into the partitioning strategy confirmation network to obtain the probability of partitioning the image block using various partitioning strategies in the partitioning strategy set output by the partitioning strategy confirmation network; then, based on the obtained probabilities, the target partitioning strategy is selected from the partitioning strategy set.

[0096] The partitioning strategy confirms that the network can be a classification network, and its specific network structure is not restricted. For example, it can be a convolutional network (CNN) for image classification, a recurrent network (RNN) for image classification, or a Transformer network structure for image classification.

[0097] For example, the partitioning strategy confirms the network structure as follows: Figure 2As shown, it includes convolutional layers, residual blocks, spatial attention models, and fully connected layers.

[0098] Before step S102, the partitioning strategy confirmation network can be trained so that it can predict the probability of partitioning an image patch using various partitioning strategies in the partitioning strategy set.

[0099] The training data for the partitioning strategy confirmation network can be sample image patches and their label vectors. This ensures that the probability distribution predicted by the trained network closely approximates the label vectors. Furthermore, the category of the largest element in the label vector of the sample image patch corresponds to the partitioning strategy with the lowest cost. This ensures that the partitioning strategy with the highest probability predicted by the trained network closely approximates the partitioning strategy with the lowest cost. Thus, processing image patches with the partitioning strategy confirmation network can also identify the partitioning strategy with the lowest cost, without needing to traverse the encoding costs of all partitioning strategies. This significantly reduces the complexity of block partitioning mode selection while maintaining effective block partitioning.

[0100] Specifically, the process of training the partitioning strategy to confirm the network may include:

[0101] 1. List all the partitioning strategies in the partitioning strategy set and their corresponding numbers, and design a partitioning strategy confirmation network such that the number of classification categories in the partitioning strategy confirmation network is equal to the total number of categories of the partitioning strategy, and the classification categories correspond one-to-one with the partitioning strategy.

[0102] 2. Encode the training dataset using an encoder. Each sample image block in the training dataset and its partitioning strategy are treated as a sample. Record the encoding cost of each sample image block under each partitioning strategy.

[0103] When encoding the training dataset, you can disable methods such as fast algorithms that skip certain pattern selections.

[0104] 3. Based on the encoding cost of each sample image block under each partitioning strategy, determine the value of the element corresponding to each partitioning strategy in the label vector of each sample image block, thus obtaining the classification label vector of each sample image block.

[0105] In one feasible approach, the values ​​of the elements corresponding to each partitioning strategy in the label vector of each sample image block are negatively correlated with the cost difference of each sample image block under each partitioning strategy. Specifically, the cost difference of each sample image block under each partitioning strategy is equal to the difference between the encoding cost of each sample image block under each partitioning strategy and the minimum encoding cost of each sample image block. The encoding cost can be rate-distortion cost, etc.

[0106] Specifically, the encoding cost of each sample image block under all partitioning strategies can be normalized to obtain the normalized value of the encoding cost of each sample image block under each partitioning strategy. The value of the element corresponding to each partitioning strategy in the label vector of each sample image block is equal to 1 minus the value of the normalized value of the encoding cost of each sample image block under each partitioning strategy. Furthermore, the output of the partitioning strategy confirmation network is also normalized. The specific method of normalization is not limited; for example, it can be softmax.

[0107] In another possible implementation, the element in the label vector of the sample image patch corresponding to the partitioning strategy with the minimum encoding cost is set to 1, and the remaining elements are set to 0. That is, the label vector for each sample image patch is determined using the OneHot encoding method.

[0108] 4. Using the partitioning strategy to confirm the vectors obtained by the network in processing the sample image patches and the classification label vectors of the sample image patches, calculate the loss of the partitioning strategy confirmation network.

[0109] There are no restrictions on the type of loss function used to calculate the loss of the partitioning strategy confirmation network. For example, any regression function can be used as the loss function for the partitioning strategy confirmation network.

[0110] 5. The network is trained based on the loss decision to determine the partitioning strategy.

[0111] Furthermore, the method for training the network based on the partitioning strategy is not limited; for example, it can be a deep learning method such as gradient backpropagation.

[0112] The step of selecting the target partitioning strategy from the set of partitioning strategies based on the obtained probabilities may include: directly selecting the partitioning strategy with the highest probability as the target partitioning strategy that matches the image patch.

[0113] In other embodiments, the step of selecting the target partitioning strategy from the set of partitioning strategies based on the obtained probabilities may include: selecting a preset number of partitioning strategies from all partitioning strategies, wherein the probability of the selected preset number of partitioning strategies is greater than the probability of the remaining unselected partitioning strategies; traversing each of the selected preset number of partitioning strategies, and determining the partitioning strategy with the minimum encoding cost among the preset number of partitioning strategies as the target partitioning strategy. This combines fast partitioning strategy selection with the traditional RDO method. The RDO method allows for determining the actual cost of each partitioning strategy at the encoding end, resulting in the optimal partitioning strategy. Simultaneously, the fast mode selection method reduces complexity. The preset number can be set according to actual conditions and is not limited here; for example, it can be 2, 3, 4, or 6.

[0114] S103: Divide the image block according to the combination of partitioning modes corresponding to the target partitioning strategy.

[0115] After determining the target segmentation strategy that matches the image patch based on the above steps, the result of segmenting the image patch using the combination of segmentation patterns corresponding to the target segmentation strategy can be obtained.

[0116] In step S103, the image block can be divided using the combination of division modes corresponding to the target division strategy.

[0117] Of course, if a preset number of partitioning strategies are selected in step S102, the image blocks can be partitioned by using the partitioning mode combination corresponding to the target partitioning strategy during the process of traversing the preset number of partitioning strategies; in this way, after the target partitioning strategy is determined, the partitioning result corresponding to the target partitioning strategy determined during the traversal can be directly exported.

[0118] Among them, the combination of partitioning modes corresponding to the target partitioning strategy refers to the combination of all partitioning modes in the target partitioning strategy.

[0119] For example, the target segmentation strategy is Figure 3 The partitioning strategy shown includes the following partitioning pattern combinations: quadtree partitioning of image blocks, quadtree partitioning of the top left sub-block of image blocks, BT horizontal partitioning of the top right sub-block of image blocks, EQT vertical partitioning of the bottom left sub-block of image blocks, BT vertical partitioning of the bottom right sub-block of image blocks, BT vertical partitioning of the top left block of the top left sub-block of image blocks, BT vertical partitioning of the top block of the bottom right sub-block of image blocks, and BT vertical partitioning of the bottom block of the bottom right sub-block of image blocks.

[0120] In this embodiment, an image block can be input into a partitioning strategy confirmation network to obtain a target partitioning strategy that matches the image block. Then, the image block is partitioned according to the partitioning pattern combination corresponding to the target partitioning strategy. Moreover, this application uses a set of partitioning strategies that includes partitioning strategies with a partitioning depth difference greater than or equal to 2 to determine the target partitioning strategy of the image block. This allows at least two partitioning patterns corresponding to the partitioning depth to be confirmed at one time. This eliminates the need to confirm the partitioning pattern of the image block step by step, saving the time of image block partitioning pattern selection, greatly reducing the complexity of image block partitioning pattern selection, and increasing the richness of image block partitioning.

[0121] Furthermore, before step S102, it can be confirmed whether the image block's partitioning depth is within a preset range. If the partitioning depth of the image block is confirmed to be within the preset range, steps S102 and S103 are executed to select a partitioning strategy from the preset partitioning strategy set for the preset partitioning depth and partition the image block. The preset partitioning depth is equal to the maximum value contained in the preset range, thus determining the partitioning strategy of the image block from multiple partitioning strategies within the preset partitioning depth. If the partitioning depth of the image block is confirmed to be outside the preset range, the image block is partitioned using various partitioning modes specified in the target codec technology standard, and the encoding cost corresponding to each partitioning mode is determined. The partitioning mode corresponding to the minimum value among the determined encoding costs is used as the partitioning module of the image block. That is, when the partitioning depth of the image block itself is outside the preset range, the RDO decision is normally applied to determine the partitioning mode of the image block.

[0122] The available image block partitioning modes are related to the target codec standard. For example, if the target codec standard is AVS3, then the available image block partitioning modes could include... Figure 4 At least one of the six partitioning patterns shown, wherein, Figure 4 The mode 0 shown is the no-division mode. Figure 4 The mode1 shown is the QT partitioning mode (SPLIT_QUAD). Figure 4 The mode2 shown is a BT vertical partition (SPLIT_BI_VER). Figure 4 The mode 3 shown is the BT horizontal partition (SPLIT_BI_HOR). Figure 4 The mode4 shown is a vertical split of EQT (SPLIT_EQT_VER). Figure 4 The mode5 shown is the EQT horizontal division (SPLIT_EQT_HOR).

[0123] The preset range can be set according to the actual situation and is not limited here. For example, it can be [2,6], or [3,4].

[0124] Furthermore, after confirming the target segmentation strategy of the image block through the above method, the segmentation mode combination corresponding to the target segmentation strategy of the image block can be injected into the bitstream so that the decoding end can decode the target segmentation strategy of the image block from the bitstream and segment the image block according to the segmentation mode combination corresponding to the decoded target segmentation strategy.

[0125] Of course, in other embodiments, each partitioning strategy in the partitioning strategy combination can be assigned a number. After confirming the target partitioning strategy of the image block through the above method, the number of the target partitioning strategy of the image block can be directly written into the bitstream. In this way, after the decoding end decodes the number of the target partitioning strategy of the image block from the bitstream, the decoding end can directly use the partitioning mode combination corresponding to the number to partition the image block.

[0126] This application also provides a second embodiment of the image segmentation method, such as... Figure 5 As shown, the image segmentation method of this embodiment includes the following steps. It should be noted that the step numbers are for simplification only and are not intended to limit the execution order of the steps. The execution order of each step in this embodiment can be arbitrarily changed without departing from the technical concept of this application.

[0127] S201: Determine whether to divide the image block using the first network.

[0128] In this embodiment, the first network first determines whether to divide the image block; if it is confirmed that the image block should be divided, the process proceeds to step S202 to confirm the specific division mode of the image block; if it is confirmed that the image block should not be divided, the image block is not divided.

[0129] In step S201, the image patch can be directly input into the first network to obtain a binary classification result output by the first network indicating whether the image patch should be classified. Specifically, the first network can obtain the probability of classifying the image patch and the probability of not classifying it; then, based on the probability of classifying the image patch and the probability of not classifying it obtained by the first network, it can be determined whether to classify the image patch. For example, if the probability of classifying the image patch is greater than a first threshold, it is determined that the image patch should be classified; otherwise, the image patch should not be classified. The first threshold can be set according to the actual situation and is not limited here.

[0130] The first network can be a binary classification network, and its specific network structure is not limited. For example, it can be a convolutional network (CNN) for image classification, a recurrent network (RNN) for image classification, or a Transformer network structure for image classification.

[0131] For example, the first network may be as follows Figure 6 As shown, it includes several sub-networks. Specifically, the first network may include convolutional layers, fully connected layers, LSTM layers, concat layers, and fully connected layers, etc.

[0132] Before step S201, the first network can be trained so that it has the ability to determine whether an image patch should be divided.

[0133] The training data for the first network includes sample image patches and their binary classification labels indicating whether or not the sample image patches were partitioned. The goal is to train the network so that its predictions approximate the binary classification labels indicating whether or not the sample image patches were partitioned. When the encoding cost of partitioning the sample image patches is less than the encoding cost of not partitioning them, the element in the label vector corresponding to partitioning the sample image patches has a value of 1, and the element corresponding to not partitioning the sample image patches has a value of 0. Conversely, when the encoding cost of partitioning the sample image patches is greater than the encoding cost of not partitioning them, the element in the label vector corresponding to partitioning the sample image patches has a value of 0, and the element corresponding to not partitioning the sample image patches has a value of 1.

[0134] In one feasible approach, the first network corresponds to the size of the image patch. Thus, in step S201, the first network corresponding to the size of the image patch can be determined, and the first network corresponding to the size of the image patch can be used to determine whether the image patch should be divided. That is, a corresponding first network is set for each possible image patch size, so that by using a first network specifically trained for each size, it is possible to determine more accurately whether the image patch should be divided.

[0135] For example, in the AVS3 standard, according to the partitioning restrictions and encoder configuration specified in the standard, a CTU can be recursively partitioned into CU blocks of different sizes. According to the standard, all possible sizes are listed, such as 128x128, 128x64, 64x128, 64x64, 64x32, 32x64, 64x16, 16x64, 32x32, 32x16, 16x32, 32x8, 8x32, 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, 4x4, and a first network of each of the above sizes is trained.

[0136] In another possible implementation, in step S201, if the size of the image patch is not a predetermined size, the size of the image patch is converted to a predetermined size, and then the converted image patch is input into the first network to determine whether the image patch needs to be divided. Thus, although there may be multiple sizes of image patches, the size of the image patches input to the first network is unified through the size conversion operation, so only one first network needs to be trained. Furthermore, the size conversion method is not limited; for example, it can be upsampling, downsampling, or convolution.

[0137] S202: In response to the first network's determination to divide the image block, the second network obtains the probability of dividing the image block through each of the various division modes among all division modes.

[0138] After the first network determines the segmentation of the image block, the second network can obtain the probability of segmenting the image block using each of the various segmentation modes. Based on the obtained probabilities, the specific segmentation mode of the image block is determined. Here, "all segmentation modes" corresponding to the second network refers to all modes used for segmentation. For example, the "all segmentation modes" corresponding to the second network could be mode1, mode2, mode3, mode4, and mode5 in the AVS3 standard.

[0139] In step S202, the image block can be directly input into the second network to obtain the probability of dividing the image block by various division modes among all division modes; so that the division mode of the image block can be selected from all division modes based on the obtained probabilities.

[0140] The second network can be a classification network, and its specific network structure is not limited. For example, it can be a convolutional network (CNN) for image classification, a recurrent network (RNN) for image classification, or a Transformer network structure for image classification.

[0141] For example, the network structure of the second network can be as follows: Figure 7 The Transformer network structure shown.

[0142] Before step S202, the second network can be trained to enable it to predict the probability of dividing an image patch by various partitioning patterns among all partitioning modes.

[0143] The second network can be trained using sample image blocks and their segmentation pattern label vectors as training data, so that the prediction results of the trained second network are close to the segmentation pattern label vectors.

[0144] The method for determining the segmentation pattern label vector can be the same as the method for determining the label vector of the sample image block shown in step S102, and will not be elaborated here.

[0145] In one feasible approach, the second network corresponds to the size of the image patch. Thus, in step S202, the second network corresponding to the size of the image patch can be determined, and the probability of dividing the image patch using various partitioning modes among all partitioning patterns can be determined using the second network corresponding to the size of the image patch. That is, a corresponding second network is set up for each possible image patch size, so that the partitioning mode of the image patch can be determined more accurately by using a second network specifically trained for each size.

[0146] For example, in the AVS3 standard, according to the partitioning restrictions and encoder configuration specified in the standard, a CTU can be recursively partitioned into CU blocks of different sizes. According to the standard, all possible sizes are listed, such as 128x128, 128x64, 64x128, 64x64, 64x32, 32x64, 64x16, 16x64, 32x32, 32x16, 16x32, 32x8, 8x32, 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, 4x4, and a second network of each of the above sizes is trained.

[0147] In another possible implementation, in step S202, if the size of the image patch is not a predetermined size, the size of the image patch is converted to a predetermined size, and then the converted image patch is input into a second network to determine the probability of dividing the image patch using various partitioning modes among all partitioning patterns. Thus, although the image patch size may vary, the size of the image patch input to the second network is unified through the size conversion operation, so only one second network needs to be trained. Furthermore, the size conversion method is not limited; for example, it can be upsampling, downsampling, or convolution.

[0148] S203: Based on the obtained probabilities, select a partitioning mode from all partitioning modes, and determine the result of partitioning the image block by the selected partitioning mode.

[0149] After obtaining the probability of dividing the image block using various division modes among all division modes through the second network, a division mode can be selected from all division modes based on the obtained probabilities, and the result of dividing the image block using the selected division mode can be determined.

[0150] Optionally, a preset number of partitioning patterns can be selected from all partitioning patterns, wherein the probability of the selected preset number of partitioning patterns is greater than the probability of all unselected partitioning patterns.

[0151] Specifically, when the preset number is greater than or equal to 2, the image block is divided using each of the preset number of partitioning modes; the encoding cost corresponding to each partitioning mode is calculated based on the partitioning results corresponding to each partitioning mode; the partitioning mode corresponding to the minimum value among the calculated encoding costs is taken as the partitioning mode of the image block, and in this process, the result of partitioning the image block by the image block partitioning mode can be obtained. Thus, by combining fast partitioning mode selection with the traditional RDO method, the actual cost of each partitioning mode can be determined at the encoding end to obtain the optimal partitioning mode, while the combination with the fast mode selection method reduces the complexity.

[0152] When the preset number is less than 2, the image block can be directly divided using the selected division mode in step S203, thus obtaining the result of dividing the image block using the selected division mode.

[0153] In addition, the preset quantity can be set according to the actual situation, and there is no restriction here. For example, it can be 2, 3 or 6.

[0154] Furthermore, since the image block partitioning mode selected by the second network for the current block may not be a partitioning allowed by the standard's constraints, in this case, it can be set that the current block should either not be partitioned or can select the partitioning mode with the highest probability value corresponding to the neural network output among the partitioning modes within the allowed partitioning range. Specifically, selecting a preset number of partitioning modes from all partitioning modes can be: selecting a preset number of partitioning modes from all feasible partitioning modes of the image block specified in the video coding standard to which the method is applied; wherein the probability of the selected preset number of partitioning modes is greater than the probability of all unselected partitioning modes among all feasible partitioning modes.

[0155] After dividing the image block into at least two blocks using the above method, the division method shown in steps S201-S203 can be used to divide each of the at least two blocks.

[0156] Of course, in another embodiment, after dividing the image block into at least two blocks using the above method, the division method shown in steps S101-S103 can also be used to divide each of the at least two blocks.

[0157] Alternatively, in another embodiment, after dividing the image block into at least two blocks using the above method, each of the at least two blocks can be divided using various partitioning modes specified in the target codec technology standard, and the coding cost corresponding to each partitioning mode can be determined. The partitioning mode corresponding to the minimum value among the determined coding costs is used as the partitioning module for each of the at least two blocks, and the RDO decision can be applied normally to partition each of the at least two blocks. Alternatively, the image block is obtained by normally applying the RDO decision. In a specific example, before step S201, it can also be confirmed whether the partitioning depth of the image block is within a preset range. If it is confirmed that the partitioning depth of the image block is within the preset range, steps S201, S202, and S203 are executed to determine the partitioning mode of the image block through the first network and the second network. If it is confirmed that the segmentation depth of the image block is not within the preset range, the image block is segmented using various segmentation modes specified in the target encoding and decoding technology standard, and the encoding cost corresponding to each segmentation mode is determined. The segmentation mode corresponding to the minimum value among the determined encoding costs is taken as the segmentation module of the image block. That is, when the segmentation depth of the image block itself is not within the preset range, the RDO decision is normally applied to determine the segmentation mode of the image block. The preset range can be set according to the actual situation and is not limited here. For example, it can be [2,6], or [3,4].

[0158] In this embodiment, a first network determines whether to divide the image block; in response to the first network determining that the image block should be divided, a second network obtains the probability of dividing the image block using various division modes among all division modes; based on the obtained probabilities, a division mode is selected from all division modes, and the result of dividing the image block using the selected division mode is determined. Thus, by combining the division determination and the fast division mode selection method, the division result can be obtained directly from top to bottom without the encoding cost of traversing all division cases, which greatly reduces the complexity of block division mode selection. Furthermore, since it is also divided downwards according to a tree structure, it can be divided more appropriately for image textures compared to one-step division.

[0159] Please see Figure 8 , Figure 8 This is a flowchart illustrating one embodiment of the intelligent encoding method of this application. This intelligent encoding method can be applied to image encoding and video encoding. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily use it. Figure 8 The illustrated process sequence is limited. In this embodiment, the intelligent coding method includes the following steps:

[0160] S301: Determine the image block segmentation result based on any of the above image segmentation methods.

[0161] S302: Encode the index information corresponding to the partitioning result to obtain the encoded bitstream.

[0162] In the image segmentation method of the first embodiment, step S302 may refer to: encoding the index information of the segmentation pattern combination corresponding to the segmentation result; or encoding the number of the segmentation strategy corresponding to the segmentation result.

[0163] In the image segmentation method of the second embodiment, step S302 may refer to: encoding the index information of the segmentation pattern corresponding to the segmentation result.

[0164] Please see Figure 9 , Figure 9 This is a flowchart illustrating one embodiment of the intelligent decoding method of this application. This intelligent decoding method can be applied to video decoding. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily use it in the same way. Figure 9 The illustrated process sequence is limited. In this embodiment, the intelligent decoding method includes the following steps:

[0165] S401: Obtain the bitstream after encoding the image block.

[0166] S402: Parse the image block partitioning pattern from the bitstream.

[0167] The image block division pattern is obtained based on the image division method described in the above embodiments.

[0168] Corresponding to the image segmentation method of the first embodiment, in step S402, the segmentation strategy of the image block can be parsed from the bitstream first, and then the segmentation mode combination corresponding to the segmentation strategy of the image block can be determined. In this way, the segmentation mode of the image block can be parsed from the bitstream.

[0169] S403: Divide the image blocks according to the partitioning pattern and decode the partitioning results of the image blocks.

[0170] Please see Figure 10 , Figure 10 This is a schematic diagram of one embodiment of the encoder of this application. The encoder 10 includes a processor 12, which executes instructions to implement the above-described image segmentation method and intelligent encoding method. For detailed implementation processes, please refer to the description of the above embodiments, which will not be repeated here.

[0171] Processor 12 can also be referred to as a CPU (Central Processing Unit). Processor 12 may be an integrated circuit chip with signal processing capabilities. Processor 12 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor, or processor 12 can be any conventional processor.

[0172] The encoder 10 may further include a memory 11 for storing instructions and data required for the processor 12 to run.

[0173] The processor 12 is used to execute instructions to implement the methods provided by any embodiment of the image segmentation method and intelligent coding method of this application and any non-conflicting combination thereof.

[0174] Please see Figure 11 , Figure 11 This is a schematic diagram of one embodiment of the decoder of this application. The decoder 20 includes a processor 22, which executes instructions to implement the above-described image segmentation method and intelligent decoding method. For detailed implementation processes, please refer to the description of the above embodiments; they will not be repeated here.

[0175] Processor 22 can also be referred to as CPU (Central Processing Unit). Processor 22 may be an integrated circuit chip with signal processing capabilities. Processor 22 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor, or processor 22 can be any conventional processor.

[0176] The decoder 20 may further include a memory 21 for storing instructions and data required for the processor 22 to run.

[0177] The processor 22 is used to execute instructions to implement the methods provided by any embodiment and any non-conflicting combination of the image segmentation method and intelligent decoding method of this application.

[0178] Please see Figure 12 , Figure 12This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of this application. The computer-readable storage medium 30 in this embodiment stores instruction / program data 31. When executed, this instruction / program data 31 implements the methods provided by any embodiment of the image segmentation method, intelligent decoding method, and intelligent encoding method, as well as any non-conflicting combination thereof. The instruction / program data 31 can be formed into a program file and stored in the storage medium 30 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium 30 includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0180] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0181] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0182] The above are merely embodiments of this application and do not limit the scope of this patent application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. An image segmentation method, characterized in that, The method includes: Identify image blocks; The image patch is input into the partitioning strategy confirmation network to obtain the target partitioning strategy that matches the image patch; the target partitioning strategy is a partitioning strategy in a set of partitioning strategies preset for a preset partitioning depth, and the set of partitioning strategies includes at least partitioning strategies whose difference from the partitioning depth of the image patch is greater than or equal to 2. The image block is divided according to the combination of division modes corresponding to the target division strategy.

2. The image segmentation method according to claim 1, characterized in that, The image block is divided into segments with a depth of 0, and the preset segmentation depth is the maximum segmentation depth in the target encoding / decoding technology standard.

3. The image segmentation method according to claim 1, characterized in that, The partitioning strategy confirmation network is trained using sample image blocks and their label vectors as training data, with the partitioning strategy that has the highest usage probability of the output sample image block being close to the partitioning strategy corresponding to the largest element in the label vector as the training target. The label vector is determined based on the encoding cost of the sample image block in various partitioning strategies.

4. The image segmentation method according to claim 3, characterized in that, The value of each element in the label vector of the sample image block is negatively correlated with the cost difference of the partitioning strategy corresponding to each element. The cost difference of the partitioning strategy is equal to the difference between the encoding cost of the sample image block under the partitioning strategy and the minimum encoding cost of the sample image block. or, In the label vector of the sample image block, the element corresponding to the partitioning strategy with the minimum encoding cost has a value of 1, and the values ​​of the remaining elements are 0.

5. The image segmentation method according to claim 1, characterized in that, The step of inputting image patches into a segmentation strategy verification network to obtain a target segmentation strategy that matches the image patches includes: The network is validated by the partitioning strategy, and the probability of partitioning the image patch using various partitioning strategies in the set of partitioning strategies is predicted. Based on the obtained probabilities, the target partitioning strategy is selected from the set of partitioning strategies.

6. The image segmentation method according to claim 5, characterized in that, The step of selecting the target partitioning strategy from the set of partitioning strategies based on the obtained probabilities includes: The partitioning strategy with the highest probability in the set of partitioning strategies is taken as the target partitioning strategy; or... A preset number of partitioning strategies are selected from the set of partitioning strategies, wherein the probability of selecting the preset number of partitioning strategies is greater than the probability of all unselected partitioning strategies; the image block is partitioned according to the partitioning mode combination corresponding to each partitioning strategy in the preset number of partitioning strategies; the encoding cost corresponding to each partitioning strategy is calculated based on the partitioning result corresponding to each partitioning strategy; the partitioning strategy with the minimum encoding cost is selected as the target partitioning strategy.

7. The image segmentation method according to claim 1, characterized in that, Before inputting image patches into the partitioning strategy confirmation network, the process also includes: Confirm whether the segmentation depth of the image block is within a preset range; If it is confirmed that the segmentation depth of the image block is within a preset range, then the steps of inputting the image block into the segmentation strategy confirmation network to obtain a target segmentation strategy that matches the image block, and segmenting the image block according to the segmentation mode combination corresponding to the target segmentation strategy are executed, and the preset segmentation depth is equal to the maximum value contained in the preset range. If it is confirmed that the segmentation depth of the image block is not within the preset range, the image block is segmented using various segmentation modes specified in the target encoding and decoding technology standard, and the encoding cost corresponding to each segmentation mode is determined; the segmentation mode corresponding to the minimum value among the determined encoding costs is taken as the segmentation module of the image block.

8. An intelligent decoding method, characterized in that, The method includes: Obtain the bitstream after encoding the image blocks; The image block partitioning strategy is parsed from the bitstream, and the image block partitioning strategy is obtained based on the image partitioning method according to any one of claims 1-7; The image block is divided according to the division strategy, and the division result of the image block is decoded.

9. An intelligent coding method, characterized in that, The method includes: The image block segmentation result is determined based on the image segmentation method according to any one of claims 1-7; The index information corresponding to the partitioning result is encoded to obtain an encoded bitstream.

10. A decoder, characterized in that, The decoder includes a processor; the processor is configured to execute instructions to implement the steps of the method as described in claim 8.

11. An encoder, characterized in that, The encoder includes a processor; the processor is configured to execute instructions to implement the steps of the method as described in any one of claims 1-7 or 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instruction / program data for execution to implement the method of any one of claims 1-7, 8 or 9.

Images