Image block division method, device, system, encoding and decoding method, and electronic device

Abstract

The embodiments of the present application provide a method, device, system, encoding and decoding method, and electronic device for dividing image blocks. The method for dividing image blocks includes: obtaining motion features corresponding to the image frames to be divided through motion feature extraction, where the motion features are used to characterize the object motion information between two adjacent image frames; using a pre-trained prediction model to perform inter-frame division depth prediction on a plurality of image blocks of the image frames to be divided based on the motion features, and obtaining an inter-frame division depth prediction result; determining the inter-frame coding units divided from the plurality of image blocks based on the inter-frame division depth prediction results corresponding to each image block, so as to perform inter-frame coding based on the inter-frame coding units. The solution provided by this embodiment realizes one-way decoupling of the prediction process and the coding process, can execute the prediction process and the coding process in parallel, saves the time spent on dividing the coding units during the coding process, and improves the efficiency and performance of video coding.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, device, system, encoding and decoding method, and electronic device for dividing image blocks. Background Art

[0002] As people's demand for high-definition / ultra-high-definition videos, various 360-degree panoramic videos, and ultra-high-definition videos continues to increase, video coding standards are also constantly evolving.

[0003] Generally, when video encoding is performed, the image frame of the video is divided into multiple image blocks of standard sizes, and the image blocks are further divided into multiple coding units (CUs), and subsequent encoding processing is performed based on the CUs.

[0004] However, currently, when dividing an image block into coding units, it is necessary to traverse the image block to determine multiple ways to divide the coding units, compare the coding costs of each method, and use the method with the lowest cost. This method leads to a high complexity in dividing the image block into coding units, which greatly affects the overall encoding speed. Summary of the Invention

[0005] In view of this, an embodiment of the present application provides a solution for dividing image blocks to at least partially solve the above-mentioned problem.

[0006] According to a first aspect of an embodiment of the present application, a method for dividing an image block is provided, comprising: obtaining motion features corresponding to an image frame to be divided by motion feature extraction, the motion features being used to characterize object motion information between two adjacent image frames; using a pre-trained prediction model, performing inter-frame division depth prediction on several image blocks of the image frame to be divided according to the motion features, and obtaining inter-frame division depth prediction results, so as to determine inter-frame coding units divided from several image blocks according to the inter-frame division depth prediction results corresponding to each image block, and performing inter-frame coding according to the inter-frame coding units.

[0007] According to the second aspect of an embodiment of the present application, a device for dividing an image block is provided, comprising: a motion feature extraction model, configured to obtain motion features corresponding to an image frame to be divided through motion feature extraction, wherein the motion features are used to characterize object motion information between two adjacent image frames; and a prediction model, configured to perform inter-frame division depth prediction on a plurality of image blocks of the image frame to be divided according to the motion features, to obtain an inter-frame division depth prediction result, and to determine inter-frame coding units divided from the plurality of image blocks according to the inter-frame division depth prediction results corresponding to each of the image blocks, so that the encoder performs inter-frame coding according to the inter-frame coding units.

[0008] According to a third aspect of an embodiment of the present application, a coding method is provided, which includes: obtaining an inter-frame division depth prediction result corresponding to an image frame to be divided, wherein the inter-frame division depth prediction result adopts a pre-trained prediction model, and performs inter-frame division depth prediction on several image blocks of the image frame to be divided according to the motion features corresponding to the image frame to be divided; according to the inter-frame division depth prediction results corresponding to each of the image blocks, determining the inter-frame coding units divided by several image blocks, so as to perform inter-frame coding according to the inter-frame coding units.

[0009] According to a fourth aspect of an embodiment of the present application, a system for dividing an image block is provided, comprising a GPU and a CPU, wherein the GPU is configured to obtain motion features corresponding to an image frame to be divided by extracting motion features; and, using a pre-trained prediction model, performs inter-frame division depth prediction on several image blocks of the image frame to be divided according to the motion features to obtain an inter-frame division depth prediction result; and the CPU is configured to determine, based on the inter-frame division depth prediction results corresponding to each of the image blocks, inter-frame coding units divided from the several image blocks, so as to perform inter-frame coding according to the inter-frame coding units.

[0010] According to a fifth aspect of an embodiment of the present application, a decoding method is provided, comprising: obtaining a code stream to be decoded, and parsing the code stream to obtain inter-frame coding units divided into image blocks, wherein the inter-frame coding units are determined by the method described above; and decoding the inter-frame coding units of the image blocks.

[0011] According to the sixth aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method described above.

[0012] According to the image block partitioning scheme provided in the embodiments of the present application, motion features corresponding to the image frame to be partitioned, which are used to characterize object motion information between the image frame to be partitioned and the reference image frame, can be obtained through motion feature extraction. A pre-trained prediction model is then used to perform inter-frame partitioning depth prediction on a plurality of image blocks of the image frame to be partitioned based on the motion features, obtaining inter-frame partitioning depth prediction results. Furthermore, based on the inter-frame partitioning depth prediction results corresponding to each of the image blocks, a plurality of inter-frame coding units divided from the image blocks are determined, so that inter-frame coding is performed based on the inter-frame coding units. Compared to partitioning image blocks by an encoder, the process of performing motion feature extraction and prediction using a pre-trained prediction model in the scheme provided in this embodiment can be performed based on the video to be encoded, rather than the encoding process. However, the encoding process can be performed based on the determined inter-frame partitioning depth prediction results. Therefore, the scheme provided in this embodiment achieves one-way decoupling of the prediction process from the encoding process, enabling the prediction process and the encoding process to be performed in parallel, while saving the time spent on partitioning coding units during the encoding process and improving the efficiency and performance of video encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0014] Figure 1 A structural block diagram of a coding system applicable to an embodiment of the present application;

[0015] Figure 2A 1 is a flow chart of a method for dividing an image block according to an embodiment of the present application;

[0016] Figure 2B for Figure 2A A schematic diagram of dividing an image block in the illustrated embodiment;

[0017] Figure 2C for Figure 2A A schematic diagram of an example scenario in the illustrated embodiment;

[0018] Figure 3A is a flowchart of another method for dividing image blocks according to an embodiment of the present application;

[0019] Figure 3B for Figure 3A A schematic structural diagram of an optical flow map extraction model in the illustrated embodiment;

[0020] Figure 3C for Figure 3A A schematic diagram of a convolution module in the illustrated embodiment;

[0021] Figure 4 is a structural block diagram of a device for dividing image blocks according to an embodiment of the present application;

[0022] Figure 5 is a structural block diagram of a system for dividing image blocks according to an embodiment of the present application;

[0023] Figure 6 1 is a flowchart of a decoding method according to an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a usage scenario according to an embodiment of the present application;

[0025] Figure 8 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0027] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0028] Video codec system

[0029] Figure 1 A structural block diagram of an encoding system according to an embodiment of the present application is shown as follows: Figure 1 As shown, the encoding system includes an encoding side and a decoding side. The encoding side can encode the captured image source and transmit it to the decoding side through the network. The decoding side can decode the encoded video and display it.

[0030] The encoding side and the decoding side can be the same or different electronic devices. Specifically, the electronic devices can be, for example, computers, servers, mobile terminals, etc. The network can be a cellular network, a local area network, or the Internet, etc., which is not limited in this embodiment.

[0031] The acquisition device for acquiring the image source may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image or comment (for screen content encoding, some text on the screen is also considered to be part of the image or image to be encoded) generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images and / or any combination thereof (e.g., augmented reality (AR) images)).

[0032] The encoder encodes the image source and generates encoded image data, which is then sent to the decoding device via a communication interface. The decoding device includes a decoder (e.g., a video decoder), and may also include a communication interface or communication unit, a post-processing unit, and a display device. The decoder receives the encoded image data and provides decoded image data, which can be displayed on a display device.

[0033] Video encoding and decoding methods

[0034] The new generation of international video coding standard VVC (Versatile Video Coding) is based on image blocks. When using VVC as the standard for video encoding, the image frame is generally divided into multiple image blocks, and the image blocks can be specifically, for example, coding tree units (CTUs). The image blocks can then be divided into a quadtree to obtain the first layer of leaf nodes, and the first layer of leaf nodes can be further divided to obtain multiple layers of leaf nodes, each layer of leaf nodes can correspond to a layer of depth. After multiple divisions, the bottom layer of leaf nodes, namely the coding units (CUs), can be obtained.

[0035] After the CU is divided, intra-frame prediction or inter-frame prediction can be performed on the CU to obtain a residual, and encoding is performed based on the residual.

[0036] However, in general, when performing division, it is necessary to use the above method to divide the image block, obtain multiple ways of dividing the coding units at different division depths, and compare the coding costs of various methods to use the division method with the lowest cost. This method leads to a high complexity in dividing the image block into coding units, which greatly affects the overall encoding speed.

[0037] To this end, in this embodiment, a scheme for dividing image blocks is provided, which is mainly used for performing depth division between frames to solve or alleviate the above-mentioned problem.

[0038] See also Figure 2A, shows a flow chart of a method for dividing an image block, as shown in the figure, which includes:

[0039] S201 : obtaining motion features corresponding to the image frame to be divided by motion feature extraction.

[0040] The motion feature is used to represent the object motion information between two adjacent image frames.

[0041] In this embodiment, since there is a relative displacement between the acquisition device and the captured picture during the video acquisition process, and the objects in the picture may also be displaced, the position of the object between two adjacent image frames of the acquired video will change, that is, there is object motion information between the two adjacent image frames.

[0042] By extracting motion features, motion features corresponding to the image frame to be divided can be obtained. For example, the motion features may include at least one of the following: an optical flow map, motion vectors divided according to the image blocks, and motion tracking information of objects in the image frame to be divided.

[0043] The optical flow map includes the optical flow corresponding to the image. The optical flow expresses the changes between two adjacent image frames. Since it contains information about the movement of the target, it can be used by the observer to determine the movement of the target. The optical flow map in this embodiment can specifically be an optical flow field map, which refers to a two-dimensional (2D) instantaneous velocity field composed of all pixels in the image, wherein the two-dimensional velocity vector is the projection of the three-dimensional velocity vector of the visible point in the scene on the imaging surface. Therefore, the optical flow field map not only contains the motion information of the observed object, but also contains rich information about the three-dimensional structure of the scene. The optical flow field map can make the inter-frame depth prediction result in step S202 more accurate.

[0044] The motion vectors divided according to the image blocks may specifically include: a vector between the most similar image blocks between two adjacent image frames. For example, in two adjacent image frames, the preceding image frame may include image block A, and the succeeding image frame may include image block B, and image block B is the image block in the succeeding image frame that is most similar to image block A. In this way, for any image block in the preceding image frame, the image block that is most similar to it can be determined in the succeeding image frame, thereby determining the vector between the two most similar image blocks between the two adjacent image frames.

[0045] The motion tracking information of the object in the image frame to be divided may specifically include tracking information of the motion process of the object in the image frame, and specifically, may include a motion field map.

[0046] Optionally, in this embodiment, motion features can be extracted from the image frame to be divided and its reference image frame using a preset feature extraction model to obtain motion features corresponding to the image frame to be divided. Different feature extraction models can be used for different motion features, and this embodiment does not limit this.

[0047] S202 : Using a pre-trained prediction model, perform inter-frame partitioning depth prediction on a plurality of image blocks of the image frame to be divided according to the motion features to obtain an inter-frame partitioning depth prediction result.

[0048] In this embodiment, the pre-trained prediction model can be any model, such as a classifier, etc., and this embodiment does not limit this.

[0049] Through the pre-trained prediction model, inter-frame partitioning depth prediction can be performed on several image blocks of the image frame to be divided according to the motion information of the image frame to be divided, and the inter-frame partitioning depth prediction result can be obtained.

[0050] For example, if the image frame to be divided can be divided into 10 image blocks, the inter-frame division depth prediction result in this embodiment may include the inter-frame division depths corresponding to the 10 image blocks. The inter-frame division depth of each image block may specifically include the division depth of each region of the image block.

[0051] Taking the quadtree as an example, see Figure 2B The size of the image block y1(U) can be 64*64, and the corresponding inter-frame partition depth prediction result can be a 4*4 matrix. Each element in the matrix can identify the layer number of the 16*16 block at the corresponding position in the quadtree. For example, the number of partition depth levels in the quadtree can include four layers, namely 0, 1, 2, and 3. If it is in layer 0, the 16*16 block is in the y1 layer where the 64*64 is located, and the depth division is not continued, that is, the entire image block is not divided into depths; if it is in layer 1, the 16*16 block is in the y2 layer where the 32*32 is located, and the depth division is not continued. At this time, the blocks on the same quadtree branch as the block are all in the y2 layer where the 32*32 is located; if it is in layer 2, the 16*16 block is in the y3 layer where the 16*16 is located, and the depth division is not continued. At this time, the blocks on the same quadtree branch as the block are all in the y3 layer where the 16*16 is located; if it is in layer 3, the 16*16 block is in the 8*8 layer, and the depth division is not continued, that is, the block is divided into 4 8*8 blocks.

[0052] S203 : Determine, according to the inter-frame division depth prediction results corresponding to each of the image blocks, a number of inter-frame coding units divided by the image blocks, and perform inter-frame coding according to the inter-frame coding units.

[0053] In this embodiment, according to the above content, it can be seen that the inter-frame division depth prediction result can represent the division depth of the image block, and the inter-frame coding unit divided by the image block can be directly determined according to the inter-frame division depth prediction result, so that inter-frame coding is performed according to the inter-frame coding unit, and the corresponding code stream can be obtained.

[0054] It should be noted that step S203 in this embodiment is executed by the encoder, and steps S201-S202 can be executed by the encoder or by a device or program other than the encoder, which is not limited in this embodiment.

[0055] See also Figure 2C , shows a scenario schematic diagram provided by an embodiment of the present application. As shown in the figure, the video may include multiple video frames to be encoded.

[0056] The feature extraction unit can extract motion features for multiple video frames to be encoded respectively, and each time the motion features of a video frame to be encoded are extracted, they are transmitted to the pre-trained prediction model.

[0057] The pre-trained prediction model can perform inter-frame partitioning depth prediction based on motion features, obtain inter-frame partitioning depth prediction results, and send the inter-frame partitioning depth prediction results to the encoder.

[0058] Based on the inter-frame depth prediction results, the encoder can determine the coding units divided into several image blocks of the video frame to be encoded, perform inter-frame encoding, and obtain the encoded bitstream. At this time, the feature extraction unit can simultaneously extract motion features of the next video frame to be encoded, and the pre-trained prediction model can also perform inter-frame depth prediction for the next video frame to be encoded, thus achieving parallelization.

[0059] The solution provided by this embodiment obtains motion features corresponding to the image frame to be divided, which are used to characterize object motion information between the image frame to be divided and the reference image frame, through motion feature extraction. A pre-trained prediction model is then used to perform inter-frame depth prediction on a plurality of image blocks of the image frame to be divided based on the motion features, obtaining inter-frame depth prediction results. Furthermore, based on the inter-frame depth prediction results corresponding to each image block, a plurality of inter-frame coding units are determined, into which the image blocks are divided, so as to perform inter-frame coding based on the inter-frame coding units. Compared to dividing image blocks by an encoder, the process of extracting motion features and performing prediction using a pre-trained prediction model in the solution provided by this embodiment can be performed based on the video to be encoded, rather than the encoding process. However, the encoding process can be performed based on the determined inter-frame depth prediction results. Therefore, the solution provided by this embodiment achieves one-way decoupling of the prediction process from the encoding process, enabling the prediction and encoding processes to be performed in parallel, while saving the time spent on dividing the coding units during the encoding process, thereby improving the efficiency and performance of video encoding.

[0060] See also Figure 3A , shows a flow chart of a method for dividing an image block provided by the present application, as shown in the figure, which includes:

[0061] S301 : obtaining motion features corresponding to the image frame to be divided by motion feature extraction.

[0062] The motion feature is used to represent the object motion information between two adjacent image frames.

[0063] Taking the motion feature being an optical flow map as an example, in this embodiment, a preset optical flow map extraction model can be used to extract motion features of the image frame to be divided and its reference image frame to obtain an optical flow map corresponding to the image frame to be divided.

[0064] For example, see Figure 3B The optical flow map extraction model can be a learnable feature extractor (LFE) based on a deep learning optical flow algorithm. The optical flow map extraction model can extract optical flow information from the image frame to be divided (frame image2 shown in the figure) and its reference image frame (frame image1 shown in the figure), thereby forming an optical flow map that can be used as input for subsequent steps.

[0065] like Figure 3BAs shown in the figure, it includes: a head enhanced pooling pyramid feature extractor (HEPP), which is used to extract features through multi-layer convolution and pooling layers; a center dense dilation correlation (CDDC) layer, which is used to construct a compact cost volume cost colume based on the features extracted from two image frames, thereby maintaining a large search radius and reducing the computational burden; a switching decoder (SBD) is used to output the predicted optical flow information predicted flow; the predicted optical flow information after upsample is adjusted (wrapped) and then input into the center dense dilation correlation (CDDC) layer again to construct a compact cost volume cost colume, and is again used to output the predicted optical flow information predicted flow as the final output optical flow map through the switching decoder (SBD).

[0066] For example, the specific extraction process can be expressed as F:x->o, where F is the optical flow algorithm used in the optical flow map extraction model, x is the spatiotemporal information input tensor formed by fusing the Y channel information of the image frame to be segmented and the Y channel information of the reference image frame, and o is the optical flow field map extracted from the spatiotemporal information. If x is a tensor of shape (2, H, W), then o is also a tensor of shape (2, H, W), where the values ​​of the two channels H and W of a pixel at a certain coordinate represent the horizontal and vertical components of the motion vector of that pixel, respectively.

[0067] In addition, the optical flow field extraction model can extract features at multiple scales and output a multi-scale optical flow field map matrix to increase the number of extracted optical flow features, thereby improving the accuracy of the inter-frame depth prediction results output by the subsequent prediction model.

[0068] S302. Using a pre-trained prediction model, perform inter-frame partitioning depth prediction on the plurality of image blocks of the image frame to be divided according to the motion features to obtain an inter-frame partitioning depth prediction result. The inter-frame partitioning depth prediction result may be a vector composed of a plurality of partitioning depth flags.

[0069] In this embodiment, the pre-trained prediction model is a convolutional neural network model. For example, the inter-frame partition depth prediction result may be a vector consisting of a plurality of partition depth flags; determining the inter-frame coding units divided from the image blocks based on the inter-frame partition depth prediction results corresponding to the image blocks includes: dividing the image blocks according to the partition depths indicated by the values ​​of the partition depth flags included in the inter-frame partition depth prediction results to obtain the plurality of inter-frame coding units.

[0070] In the embodiment of the present application, during the division process, an image block can be divided into several sub-image blocks, and the sub-image blocks can be further divided into smaller sub-image blocks. The division depth increases by one each time, and the number of sub-image blocks increases exponentially. For example, the number of image blocks and the sub-image blocks divided therefrom can be Wherein, A is the number of blocks that the sub-image block can be further divided into, for example, if a quadtree is used, A=4, if a ternary tree is used, A=3; when i=0, it represents the image block that has not been divided, and when i≥1, it represents the division depth of the divided sub-image block, for example Figure 2B As shown, the division depth of the divided sub-image blocks is 1, which corresponds to y1 in the figure; the division depth is 2, which corresponds to y2 in the figure; and the division depth is 3, which corresponds to y3 in the figure.

[0071] Specifically, the depth flag can be set in the following two ways:

[0072] First, each sub-image block with the highest partition depth corresponds to a partition depth flag, that is, each minimum sub-image block that can be divided corresponds to a partition depth flag, and the value of the partition depth flag can be the value of the predicted partition depth corresponding to the sub-image block.

[0073] For example, taking the quadtree as an example, when the maximum partition depth is 2, a 64*64 image block can be divided into 16 sub-image blocks, corresponding to Figure 2B The value of the split depth flag bit can be the split depth of the corresponding image block. For example, if y3(U 2，2 ) corresponds to a depth of 1, then the value obtained after the division is completed is y3(U2). 2，2 ) The value of the division depth flag corresponding to other sub-image blocks belonging to the same upper sub-image block is also 1, that is, y3(U 2，1 )、y3(U 2，3 )、y3(U 2，4 )The value of the corresponding division depth flag is also 1.

[0074] Second, each sub-image block corresponds to a division depth flag, that is, from a division depth of 0 to a maximum division depth, each division depth corresponds to a group of sub-image blocks that can be divided, and each sub-image block corresponds to a division depth flag. The division depth flag is used to indicate whether the sub-image block is divided according to the division depth corresponding to the sub-image block. The value of the division depth flag can be 0 or 1. The subsequent steps S304-S306 of this application are executed when the second division depth flag is used.

[0075] At this time, for an image block of size k*k, according to the n-ary tree division method, there may be multiple groups of flag bits, wherein the i-th group of flag bits is used to indicate that the size is k / (n i )*k / (n i ) whether to continue dividing the block, i = 0, 1, 2...

[0076] When n=4, that is, taking the quadtree as an example, assuming that the highest partition depth is 2, that is, the maximum value of i is 2, when the partition depth is 1, the corresponding Figure 2B The y1 image block in the image is 2, which corresponds to Figure 2B The four y2 sub-image blocks in the image are divided into two groups when the depth is 2. Figure 2B 16 y3 sub-image blocks in the y3 sub-image blocks. The corresponding number of split depth flags is 1+4+16,=21. Correspondingly, the vector composed of several split depth flags can be a 21-dimensional vector. Each split depth flag can specifically include: 1 flag indicating whether the 64*64 CTU is further divided, corresponding to Figure 2B The y1 in , 4 flags indicating whether the 32*32 size sub-CU continues to be divided correspond to Figure 2B The 4 y2 and 16 16*16 size sub-CUs are marked as whether to continue to be divided, that is, corresponding to Figure 2B The i in y2(Ui) can be used to represent the position of the block, and y3(U m，n ) can be used to represent the coordinates of the block to represent the position of the block.

[0077] Of course, the above is only an example. In other implementations of the present application, vectors composed of flag bits in other forms are also within the scope of protection of the present application.

[0078] In addition, the value of each flag can be 0 or 1, 0 is used to indicate that the division will not continue, and 1 is used to indicate that the division will continue. Since each flag has two possible values, the prediction model can be specifically a number of binary classifiers, and the number of binary classifiers is the same as the number of flags. In addition, the process of training the prediction model can be regarded as multiple binary classification tasks, and the sum of the binary cross entropy loss function can be selected as the total loss function, which can be expressed as:

[0079] l n =-w[y n ·logx n +(1-y n )·log(1-x n )].

[0080] Wherein, xn represents the predicted value of the 21-dimensional partition vector, yn represents the true value, and w represents the weight of each dimension. In this embodiment, w is 1.

[0081] In this embodiment, the pre-trained prediction model is a convolutional neural network model.

[0082] In order to obtain better prediction results, the convolutional neural network model includes several convolution modules, which perform inter-frame division depth prediction on several image blocks of the image frame to be divided according to the motion features to obtain inter-frame division depth prediction results, including: inputting the motion features into several connected convolution modules, and extracting features of the motion features in turn through the convolution modules, wherein the input of the convolution module currently performing feature extraction is the fusion of the output of the previous convolution module and the motion features; and determining the inter-frame division depth prediction results of the image to be divided according to the outputs of the several convolution modules.

[0083] See also Figure 3C , shows a schematic diagram of a convolution module, specifically the T-th layer convolution module, whose input is the concatenation and fusion of the C*H*W feature data output by the T-1th layer convolution module and the optical flow features of size 2*H*W. The T-th layer convolution module outputs feature data of size (2*C)*(H / 2)*(W / 2), which is used as the input of the T+1th layer convolution module. If the T+1th layer convolution module does not exist, the feature data output by the T-th layer convolution module can be output directly.

[0084] In this embodiment, each convolution module may include a convolution layer, a batch normalization layer, an activation function Relu layer, a convolution layer, a batch normalization layer, an activation function Relu layer and an average pooling layer connected in sequence, and the step size of the average pooling layer can be 2.

[0085] For example, the input and output sizes of each module in the model can be shown in Table 1 below.

[0086]

[0087] Table 1

[0088] Conv1, Conv2, Conv3, Conv4, Conv5, and Conv6 in Table 1 correspond to the convolution modules mentioned above; Linear1 and Linear2 are linear layers (also known as fully connected layers) in the neural network; ReLU and Sigmoid are parameter-free activation functions.

[0089] In addition, the prediction model in this embodiment can be trained by the following method:

[0090] Obtain sample data, the sample data including a sample image frame, a reference image frame of the sample image frame, and labels of inter-frame coding units divided from each image block of the sample image frame; input the sample image frame and the reference image frame into a feature extraction network, and extract motion features of the sample image frame through the feature extraction network; input the motion features into the prediction model, and perform inter-frame division depth prediction on several image blocks of the image frame to be divided according to the motion features through the prediction model to obtain an inter-frame division depth prediction result; adjust the prediction model according to the difference between the inter-frame coding unit represented by the inter-frame division depth prediction result and the inter-frame coding unit represented by the inter-frame coding unit.

[0091] The sample data may include a CTU group of a sample image frame, where the CTU group may include multiple image blocks, each image block corresponding to an inter-frame coding unit label, which is used to characterize the division result of the CTU into inter-frame CUs; ​​the sample data also includes a CTU group of a reference image frame, which is used to extract motion features of the CTU of the sample image.

[0092] The specific training process can refer to the relevant supervised training or semi-supervised training schemes, which will not be repeated here.

[0093] S303: Compare the value of the split depth flag with a preset first split threshold th_up and a preset second split threshold th_down, where the first split threshold is greater than the second split threshold.

[0094] S304: If it is greater than or equal to the first division threshold th_up, determine to perform division according to the division depth indicated by the division depth flag, and execute step S323 for the division depth flag corresponding to the divided sub-image block to continue division.

[0095] S305: If the value is less than or equal to the second division threshold th_down, determine to perform division according to the division depth indicated by the division depth flag and stop the division.

[0096] S306 : If the value is smaller than the first division threshold th_up and larger than the second division threshold th_down, perform division using the depth division method of the video encoder.

[0097] It should be noted that steps S303-S306 in this embodiment are executed by the encoder, and steps S301-S302 can be executed by the encoder or by a device or program other than the encoder, which is not limited in this embodiment.

[0098] In this embodiment, when the inter-frame split depth prediction result includes a vector consisting of several split depth flags, the value of each split depth flag may be a floating point number rather than an integer. For example, if during the training process, the value of each flag can be 0 or 1, 0 is used to indicate that the split will not continue, and 1 is used to indicate that the split will continue, then when executing step S303, the value of each split depth flag may be 0 to 1, and the closer to 1, the higher the probability of continuing the split, and the closer to 0, the higher the probability of not continuing the split.

[0099] Therefore, in this embodiment, the sub-image blocks to be further divided and the sub-image blocks not to be further divided are determined by the first division threshold th_up and the second division threshold th_down. The sub-image blocks that cannot be determined can be divided by the video encoder according to its own built-in depth division method.

[0100] The specific values ​​of the first division threshold th_up and the second division threshold th_down can be obtained through experiments or set by relevant personnel, which are not limited in this embodiment.

[0101] Specifically, when performing splitting, the current frame to be split, the reference frame of the current frame, the CU variable, the first splitting threshold th_up and the second splitting threshold th_down can be pre-set. Two flags, is_skip_Current_flag and is_skip_split_flag, can also be set. The is_skip_Current_flag indicates the end of splitting, while the is_skip_Current_flag indicates the continuation of splitting.

[0102] When performing the division, you can first set the two flags to false, and assign the CU variable to the CU that currently needs to be divided, and then for the current CU:

[0103] Determine whether the position information of the current CU exceeds the image boundary to be divided; if it exceeds the boundary, the flag bit is_skip_Current_flag corresponding to the current CU can be set to true, indicating that the current CU is skipped. For example, an image frame may include multiple image blocks, that is, multiple y1. If the image frame to be divided is Figure 2BThe four sub-image blocks CU corresponding to y2 in the image block y1 are obtained. After obtaining the corresponding CU, it can be determined whether the CU exceeds the image boundary of the current image block y1. If it exceeds, it indicates that the image block y1 cannot be divided according to the inter-frame depth division prediction result, and the flag is_skip_Current_flag can be set to true. If it does not exceed the boundary, the depth value depth_value of the current CU and the position number of the current CU in the image block CTU are obtained. The position number can be, for example, Figure 2B The corresponding serial numbers of the four sub-image blocks corresponding to y2 in are obtained respectively, and the value of the split depth flag corresponding to the current CU is obtained according to the depth value and the position information. The value of the split depth flag is compared with the first split threshold th_up and the second split threshold th_down. If it is greater than or equal to the first split threshold th_up, it is determined that the split depth indicated by the split depth flag is to be split, and the flag is_skip_Current_flag corresponding to the current CU can be set to true, indicating that the current CU is skipped; if it is less than or equal to the second split threshold th_down, it is determined that the split depth indicated by the split depth flag is to be split and no further splitting is to be performed, and the flag is_skip_split_flag corresponding to the current CU can be set to true, indicating that no further splitting is to be performed.

[0104] Afterwards, the values ​​of the two flags is_skip_Current_flag and is_skip_split_flag corresponding to the current CU may be returned to end the operation on the current CU; and the two flags may be set to false again, and the CU variable may be assigned to the next CU to be split to continue execution.

[0105] Subsequently, for the CU whose flag bit is_skip_Current_flag is true, the CU after the division can be determined, and the above steps are continued for the divided CUs until there is no CU whose flag bit is_skip_Current_flag is true.

[0106] Afterwards, for the CU whose flag is_skip_split_flag is true, the split depth value corresponding to each CU can be used as the final split depth value. For the CU whose flag is_skip_current_flag is false or the flag is_skip_split_flag is false, it can be split according to the depth splitting method of the video encoder.

[0107] For example, referring to the following Table 2, a first partitioning threshold th_up and a second partitioning threshold th_down used for image blocks of different sizes under different quantization parameters (QP) are shown.

[0108]

[0109] Table 2

[0110] Through the solution provided by this embodiment, motion features such as optical flow maps can be used to extract motion information corresponding to the video frames to be divided, and a convolutional neural network model CNN can be used to perform inter-frame division depth prediction based on the optical flow map to obtain an inter-frame division depth prediction result, and the process of extracting the optical flow map and performing inter-frame division depth prediction through CNN can be accelerated by GPU, thereby accelerating the video encoding process through GPU parallel computing. In addition, the solution provided by this embodiment is not limited to obtaining motion features through pixel-level information, that is, it is not limited to obtaining motion features by comparing pixels one by one, but uses a preset feature extraction model to extract the optical flow map to represent the motion features, and can approximate the motion vector of each image block through the optical flow map, and finally uses the generated tensor as the input of CNN, and outputs the inter-frame division depth prediction result through CNN, and uses the inter-frame division depth prediction result as the basis for skipping division during the encoding process, thereby accelerating the video encoding process.

[0111] See also Figure 4 , shows a schematic structural diagram of an image block division device provided in an embodiment of the present application, as shown in the figure, which includes:

[0112] A motion feature extraction model 401 is used to obtain motion features corresponding to the image frames to be divided by motion feature extraction, wherein the motion features are used to represent the object motion information between two adjacent image frames;

[0113] A prediction model 402 is configured to perform inter-frame partitioning depth prediction on a plurality of image blocks of the image frame to be partitioned according to the motion features, to obtain an inter-frame partitioning depth prediction result;

[0114] The division module 403 is embedded in the encoder and is used to determine the inter-frame coding units divided by the image blocks according to the inter-frame division depth prediction results corresponding to each image block, so that the encoder performs inter-frame coding according to the inter-frame coding units.

[0115] The process of motion feature extraction and prediction through a pre-trained prediction model in the solution provided by this embodiment can be executed based on the video to be encoded, rather than the encoding process. However, the encoding process can be executed based on the determined inter-frame division depth prediction result. Therefore, the solution provided by this embodiment realizes the one-way decoupling of the prediction process and the encoding process, can execute the prediction process and the encoding process in parallel, and saves the time spent on dividing the coding units in the encoding process, thereby improving the efficiency and performance of video encoding.

[0116] See also Figure 5 , shows a structural diagram of an image block division system provided by this embodiment, as shown in the figure, which includes a GPU and a CPU.

[0117] In this embodiment, the GPU is used to obtain motion features corresponding to the image frame to be divided by motion feature extraction; and, using a pre-trained prediction model, perform inter-frame division depth prediction on a plurality of image blocks of the image frame to be divided according to the motion features to obtain an inter-frame division depth prediction result;

[0118] The CPU is configured to determine, based on the inter-frame division depth prediction results corresponding to each of the image blocks, a plurality of inter-frame coding units divided by the image blocks, and perform inter-frame coding according to the inter-frame coding units. The CPU may also be configured to perform inter-frame coding and output a code stream.

[0119] In the scheme provided by this embodiment, the process of extracting motion features and performing predictions through a pre-trained prediction model can be executed based on the video to be encoded, rather than the encoding process. However, the encoding process can be executed based on the determined inter-frame division depth prediction results. Therefore, the scheme provided by this embodiment realizes a one-way decoupling of the prediction process and the encoding process, can execute the prediction process and the encoding process in parallel, can also accelerate the execution of the prediction process through a GPU, and saves the time spent on dividing the encoding units during the encoding process, thereby improving the efficiency and performance of video encoding.

[0120] See also Figure 6 , shows a schematic flow chart of a decoding method provided by this embodiment, as shown in the figure, which includes:

[0121] S601: Acquire a code stream to be decoded, and parse the code stream to obtain inter-frame coding units divided by image blocks.

[0122] The inter-frame coding unit is determined by the method described in the above embodiment.

[0123] S602: Decode according to the inter-frame coding unit of the image block.

[0124] Reference Figure 7, shows a schematic diagram of a usage scenario. As shown in the figure, the usage scenario may include user equipment 1, a communication network, and user equipment 2.

[0125] In some embodiments, the communication network can be any appropriate combination of one or more wired and / or wireless networks, and the communication network can include a transmission intermediary, for example, user device 1 can transmit data to a service node via the communication network, and the service node then forwards the data to user device 2 via the communication network. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The electronic device can be connected to the communication network via one or more communication links (e.g., a communication link), and the communication network can be linked to the cloud service end via one or more communication links (e.g., a communication link). The communication link can be any communication link suitable for transmitting data between the electronic device and the cloud service end, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0126] User device 1 can be any electronic device capable of running programs, capturing images, encoding images, and transmitting code streams. In some embodiments, user device 1 can include any suitable type of device. For example, in some embodiments, user device 1 can include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a cloud computing terminal, a game console, a media player, a vehicle entertainment system, and / or any other suitable type of electronic device.

[0127] User device 2 can be any electronic device capable of video decoding and video playback. Similar to user device 1, user device 2 can include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a cloud computer terminal, a game console, a media player, a vehicle entertainment system and / or any other suitable type of electronic device.

[0128] In a conference scenario, user device 1 can be an electronic device of a user participating in the conference. User device 1 can capture the video corresponding to the shared desktop or the video captured by the camera, encode the video using the above solution, and transmit the encoded code stream to user device 2 via the communication network. User device 2 can decode and play the decoded video using the above solution.

[0129] In a live broadcast scenario, user device 1 can be the electronic device of a host. User device 1 can capture the host's video, encode the video using the above solution, and transmit the encoded code stream to user device 2 via a communication network. User device 2 can decode and play the decoded video using the above solution, thereby watching the live broadcast.

[0130] Reference Figure 8 , shows a structural diagram of an electronic device according to the fifth embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0131] like Figure 8 As shown, the electronic device may include: a processor (processor) 802 , a communication interface (Communications Interface) 804 , a memory (memory) 806 , and a communication bus 808 .

[0132] in:

[0133] The processor 802 , the communication interface 804 , and the memory 806 communicate with each other via a communication bus 808 .

[0134] The communication interface 804 is used to communicate with other electronic devices or servers.

[0135] The processor 802 is configured to execute the program 810 , and specifically to execute the relevant steps in the above-mentioned embodiment of the method for dividing image blocks.

[0136] Specifically, the program 810 may include program codes, which include computer operation instructions.

[0137] The processor 802 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0138] The memory 806 is used to store the program 810. The memory 806 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0139] The program 810 can be specifically used to enable the processor 802 to execute operations corresponding to the method described in any one of the aforementioned method embodiments.

[0140] The specific implementation of each step in program 810 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0141] An embodiment of the present application further provides a computer storage medium including a computer program, wherein the computer program is used to implement any image block division method in the above-mentioned multiple method embodiments.

[0142] An embodiment of the present application further provides a computer program product, comprising computer instructions, which instruct a computing device to execute operations corresponding to any image block division method in the above-mentioned multiple method embodiments.

[0143] Codec Experiment

[0144] After applying the solution provided in the above embodiment to the encoder and performing a comparative test, the results are shown in Table 3. In the experiment, the performance of the encoder was evaluated using two indicators: BDrate (Bjotegaard Delta rate) and TS.

[0145] BDBR represents the bitrate difference between two encoding methods at the same objective quality. It is obtained by encoding the same video at four QP values ​​(22, 27, 32, and 37) and calculating the bitrate and PSNR. BDBR comprehensively reflects the bitrate and quality of a video. A larger BDBR value indicates that the proposed fast algorithm has a higher bitrate than the original encoder, but also indicates poorer compression performance.

[0146] TS is used to measure the degree to which the fast algorithm reduces the encoding time based on the original encoder. It is calculated as follows:

[0147]

[0148] Among them, T p is the total encoding time after the fast algorithm is embedded in the encoder, T o is the total encoding time of the original encoder.

[0149]

[0150] Table 3 Experimental results

[0151] Experiments show that the BDBR of the encoder using the above scheme increased by 0.77%, indicating that the bit rate did not increase significantly, which effectively guaranteed the compression performance of the encoder. The encoding time reduction degree TS indicates that the encoding time of the encoder after applying the above scheme was reduced by 9.9% compared with the original encoder, which effectively shortened the encoding time and improved the encoding efficiency.

[0152] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0153] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or can be implemented as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0154] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0155] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A method for dividing an image block, comprising: Obtaining motion features corresponding to the image frames to be divided by motion feature extraction, wherein the motion features are used to characterize the object motion information between two adjacent image frames; Using a pre-trained prediction model, performing inter-frame division depth prediction on the plurality of image blocks of the image frame to be divided according to the motion features to obtain an inter-frame division depth prediction result, determining inter-frame coding units divided by the plurality of image blocks according to the inter-frame division depth prediction results corresponding to each of the image blocks, and performing inter-frame coding according to the inter-frame coding units; The pre-trained prediction model is a convolutional neural network model, and the convolutional neural network model includes a plurality of convolution modules. The inter-frame division depth prediction is performed on the plurality of image blocks of the image frame to be divided according to the motion features to obtain the inter-frame division depth prediction result, including: Inputting the motion features into a plurality of connected convolution modules, and sequentially extracting features from the motion features through the convolution modules, wherein the input of the convolution module currently performing feature extraction is a fusion of the output of the previous convolution module and the motion features; According to the output of the last convolution module, the inter-frame division depth prediction result of the image to be divided is determined.

2. The method according to claim 1, wherein The step of obtaining motion features corresponding to the image frames to be divided by motion feature extraction includes: Using a preset feature extraction model, motion features are extracted from the image frame to be divided and its reference image frame to obtain motion features corresponding to the image frame to be divided; wherein the motion features include at least one of the following: Optical flow map, motion vectors divided according to the image blocks, and motion tracking information of objects in the image frame to be divided.

3. The method according to claim 2, wherein: If the motion feature includes the optical flow map, the step of extracting the motion feature to obtain the motion feature corresponding to the image frame to be divided includes: The motion features of the image frame to be divided and its reference image frame are extracted by using a preset optical flow map extraction model to obtain an optical flow map corresponding to the image frame to be divided.

4. A device for dividing an image block, comprising: A motion feature extraction model is used to obtain motion features corresponding to the image frames to be divided by motion feature extraction, wherein the motion features are used to characterize the object motion information between two adjacent image frames; A prediction model is configured to perform inter-frame division depth prediction on the plurality of image blocks of the image frame to be divided according to the motion features to obtain an inter-frame division depth prediction result, and determine inter-frame coding units divided by the plurality of image blocks according to the inter-frame division depth prediction results corresponding to the respective image blocks, so that an encoder performs inter-frame coding according to the inter-frame coding units; The prediction model is a convolutional neural network model, which includes several convolution modules. The prediction model is specifically used to input the motion features into several connected convolution modules, and extract the motion features in turn through the convolution modules, wherein the input of the convolution module currently performing feature extraction is the fusion of the output of the previous convolution module and the motion features; based on the output of the last convolution module, the inter-frame division depth prediction result of the image to be divided is determined.

5. A coding method, wherein: include: Obtaining an inter-frame division depth prediction result corresponding to the image frame to be divided, wherein the inter-frame division depth prediction result adopts a pre-trained prediction model, and performs inter-frame division depth prediction on a plurality of image blocks of the image frame to be divided according to the motion features corresponding to the image frame to be divided, wherein the pre-trained prediction model is a convolutional neural network model, and the convolutional neural network model includes a plurality of convolution modules, and the inter-frame division depth prediction result of the image to be divided is inputting the motion features into a plurality of connected convolution modules, and sequentially extracting the motion features through the convolution modules, wherein the input of the convolution module currently performing feature extraction is a fusion of the output of the previous convolution module and the motion features; and is determined according to the output of the last convolution module; According to the inter-frame division depth prediction results corresponding to each of the image blocks, a number of inter-frame coding units divided by the image blocks are determined, so as to perform inter-frame coding according to the inter-frame coding units.

6. The method according to claim 5, wherein: The inter-frame division depth prediction result includes: a vector composed of a plurality of division depth flag bits; and determining, according to the inter-frame division depth prediction result corresponding to each of the image blocks, a plurality of inter-frame coding units divided by the image blocks, including: The image block is divided according to the division depth indicated by the value of the division depth flag included in the inter-frame division depth prediction result to obtain a plurality of inter-frame coding units.

7. The method according to claim 6, wherein: The image block is divided into multiple layers according to the division depth, and each layer corresponds to multiple sub-image blocks; Each sub-image block with the highest division depth corresponds to a division depth flag, and the value of the division depth flag is the predicted division depth corresponding to the sub-image block; or, each sub-image block corresponds to a division depth flag, and the value of the division depth flag is used to indicate whether to divide according to the division depth of the corresponding sub-image block.

8. The method according to claim 7, wherein: If each of the sub-image blocks corresponds to a split depth flag, the image block is divided according to the split depth indicated by the value of the split depth flag included in the inter-frame split depth prediction result to obtain a plurality of inter-frame coding units, including: Comparing the value of the split depth flag with a preset first split threshold and a second split threshold, the first split threshold being greater than the second split threshold; If the value of the depth flag is greater than or equal to the first division threshold, determining to divide the image block according to the division depth indicated by the division depth flag, and comparing the value of the division depth flag with the preset first division threshold and the second division threshold for the division sub-image block, and if the first division threshold is greater than the second division threshold, then continuing the division; If it is less than or equal to the second division threshold, determine to divide according to the division depth indicated by the division depth flag bit and stop dividing; If it is less than the first division threshold and greater than the second division threshold, division is performed using the depth division method of the video encoder.

9. A system for dividing image blocks, comprising a GPU and a CPU, The GPU is used to obtain motion features corresponding to the image frame to be divided by motion feature extraction; and, using a pre-trained prediction model, perform inter-frame division depth prediction on several image blocks of the image frame to be divided according to the motion features to obtain an inter-frame division depth prediction result, wherein the pre-trained prediction model is a convolutional neural network model, and the convolutional neural network model includes several convolution modules. The inter-frame division depth prediction result of the image to be divided is inputting the motion features into several connected convolution modules, and extracting the motion features in sequence through the convolution modules, wherein, The input of the convolution module currently performing feature extraction is a fusion of the output of the previous convolution module and the motion feature; determined according to the output of the last convolution module; The CPU is configured to determine, based on the inter-frame division depth prediction results corresponding to each of the image blocks, a number of inter-frame coding units divided by the image blocks, so as to perform inter-frame coding according to the inter-frame coding units.

10. A decoding method comprising: Obtaining a code stream to be decoded, and parsing the code stream to obtain inter-frame coding units divided by image blocks, wherein the inter-frame coding units are determined by the method according to any one of claims 1 to 3 and 5 to 8; The inter-frame coding unit of the image block is decoded.

11. An electronic device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1-3, 5-8, and 10.

Images