Video processing method and device, video coding method and device, video decoding method and device, and electronic equipment

Abstract

The embodiment of the invention provides a video processing method and device, a video coding method and device, a video decoding method and device and electronic equipment. The video processing method comprises the following steps: acquiring a current frame image in a video to be processed; performing prediction processing on each pixel of the current frame image to obtain prediction frame data; the prediction frame data comprises prediction data of each pixel of the current frame image, and the prediction data comprises offset information and reference frame information corresponding to the offset information; and performing motion compensation on the current frame image based on the predicted frame data to obtain a predicted frame image.

Description

Technical Field

[0001] The present disclosure relates to the field of video technology, and in particular to a video processing method and device, a video encoding method and device, a video decoding method and device, and an electronic device. Background Art

[0002] Block matching is a motion estimation technique widely used in digital video processing and image analysis. The basic idea behind block matching is to divide the current frame into several pixel blocks and then identify the best matching block in the reference frame to obtain a predicted image for the current frame. However, block matching-based motion estimation techniques often suffer from issues such as large bitstream size and low coding efficiency.

[0003] Deformable convolution is an emerging image processing technology. The main difference between it and ordinary convolution is that the pixel position of the input convolution is learnable, and it can be adaptively adjusted according to the pixel changes in the input image to achieve accurate modeling of the image. Summary of the Invention

[0004] The embodiments of the present disclosure provide a video processing method and apparatus, a video encoding method and apparatus, a video decoding method and apparatus, and an electronic device.

[0005] In a first aspect, an embodiment of the present disclosure provides a video processing method, the method comprising:

[0006] Get the current frame image in the video to be processed;

[0007] Performing prediction processing on each pixel of the current frame image to obtain predicted frame data; the predicted frame data includes predicted data of each pixel of the current frame image, and the predicted data includes offset information and reference frame information corresponding to the offset information;

[0008] Motion compensation is performed on the current frame image based on the predicted frame data to obtain a predicted frame image.

[0009] In a second aspect, an embodiment of the present disclosure provides a video encoding method, applied to a video encoding end, the method comprising:

[0010] The current frame image is divided into multiple coding blocks. For any of the coding blocks:

[0011] Encoding using a first encoding method to obtain a target pixel value of a target matching block corresponding to the encoding block; the first encoding method is based on a second algorithm for encoding;

[0012] Using a second encoding method to perform prediction processing and motion compensation to obtain predicted pixel values of the coding block; the second encoding method is encoded based on a target prediction model;

[0013] Calculating a second difference between a target pixel value of the target matching block and an original pixel value of the coding block; and calculating a third difference between a predicted pixel value of the coding block and an original pixel value of the coding block;

[0014] If the second difference is less than the third difference, the coding block is encoded according to the first coding method to obtain first coding information; if the second difference is greater than the third difference, the coding block is predicted and motion compensated according to the second coding method to obtain second coding information; the second coding information includes at least identification information, and the identification information is used to indicate that the coding block corresponds to the second coding method.

[0015] In a third aspect, an embodiment of the present disclosure provides a video decoding method, applied to a video decoding end, the method comprising:

[0016] Determining a decoding method corresponding to the video coding information to be decoded, and decoding a plurality of coding blocks to be decoded, according to the video coding information to be decoded, wherein the video coding information includes first coding information and second coding information;

[0017] If the encoding information is the first encoding information, decoding the encoding block to be decoded using a first decoding method to obtain a first decoded block; the first decoding method is based on a second algorithm for decoding;

[0018] If the encoding information is the second encoding information, each pixel of the encoding block to be decoded is predicted to obtain decoded block data; the decoded block data includes offset information of each pixel of the decoding block and reference frame information corresponding to the offset information; and motion compensation is performed on the encoding block to be decoded based on the decoded block data to obtain a second decoded block; the second decoding method is based on the target prediction model.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a video processing device, the video processing device comprising an acquisition unit, a prediction unit, and a compensation unit;

[0020] The acquisition unit is configured to acquire a current frame image in the video to be processed;

[0021] The prediction unit is configured to perform prediction processing on each pixel of the current frame image to obtain predicted frame data; the predicted frame data includes prediction data of each pixel of the current frame image, and the prediction data includes offset information and reference frame information corresponding to the offset information;

[0022] The compensation unit is configured to perform motion compensation on the current frame image based on the predicted frame data to obtain a predicted frame image.

[0023] In a fifth aspect, an embodiment of the present disclosure provides a video encoding device, applied to a video encoding end, the video encoding device including an encoding unit, a calculation unit, and a judgment unit;

[0024] The encoding unit is configured to divide the current frame image into a plurality of encoding blocks, and for any of the encoding blocks: encode using a first encoding method to obtain a target pixel value of a target matching block corresponding to the encoding block; encode using the first encoding method based on a second algorithm; and perform prediction processing and motion compensation using a second encoding method to obtain a predicted pixel value of the encoding block; and encode using the second encoding method based on a target prediction model;

[0025] The calculation unit is configured to calculate a second difference between the target pixel value of the target matching block and the original pixel value of the coding block; and calculate a third difference between the predicted pixel value of the coding block and the original pixel value of the coding block;

[0026] The judgment unit is configured to, if the second difference is less than the third difference, encode the coding block according to the first coding method to obtain first coding information; if the second difference is greater than the third difference, perform prediction processing and motion compensation on the coding block according to the second coding method to obtain second coding information; the second coding information includes at least identification information, and the identification information is used to indicate that the coding block corresponds to the second coding method.

[0027] In a sixth aspect, an embodiment of the present disclosure provides a video decoding device, applied to a video decoding end, the video decoding device including a determination unit and a decoding unit;

[0028] The determining unit is configured to determine a decoding method corresponding to the video coding information to be decoded, and decode the plurality of coding blocks to be decoded; the video coding information includes first coding information and second coding information;

[0029] The decoding unit is configured to, if the encoding information is the first encoding information, decode the encoding block to be decoded using a first decoding method to obtain a first decoding block; the first decoding method performs decoding based on a second algorithm; and if the encoding information is the second encoding information, perform prediction processing on each pixel of the encoding block to be decoded to obtain decoding block data; the decoding block data includes bias information of each pixel of the decoding block and reference frame information corresponding to the bias information; and perform motion compensation on the encoding block to be decoded based on the decoding block data to obtain a second decoding block; the second decoding method performs decoding based on a target prediction model.

[0030] In a seventh aspect, an embodiment of the present disclosure provides an electronic device, the electronic device including a memory and a processor;

[0031] The memory is used to store a computer program that can be run on the processor;

[0032] The processor is configured to execute the method according to any one of the first aspect, the method according to any one of the second aspect, or the method according to any one of the third aspect when running the computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of motion estimation and motion compensation provided by an embodiment of the present disclosure;

[0034] Figure 2 A schematic diagram of a common convolution and a deformable convolution provided in an embodiment of the present disclosure;

[0035] Figure 3 A schematic diagram of a deformable convolution process provided by an embodiment of the present disclosure;

[0036] Figure 4 A flowchart of a video processing method provided by an embodiment of the present disclosure;

[0037] Figure 5 A schematic diagram of a prediction layer of a target prediction model provided in an embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram of a motion compensation layer of a target prediction model provided by an embodiment of the present disclosure;

[0039] Figure 7 A schematic diagram of a video encoding method according to an embodiment of the present disclosure;

[0040] Figure 8 A schematic diagram of a video decoding method according to an embodiment of the present disclosure;

[0041] Figure 9 A schematic diagram of the structure of a video processing device provided in an embodiment of the present disclosure;

[0042] Figure 10 A schematic diagram of the structure of a video encoding device provided in an embodiment of the present disclosure;

[0043] Figure 11 A schematic diagram of the structure of a video decoding device provided by an embodiment of the present disclosure;

[0044] Figure 12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure Figure 1 ;

[0045] Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure Figure 2 . DETAILED DESCRIPTION

[0046] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. It should be understood that the specific embodiments described herein are only used to explain the relevant applications and are not intended to limit the relevant applications. It should also be noted that for ease of description, only the portions relevant to the relevant applications are shown in the drawings.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art of the present disclosure. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0048] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present disclosure are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described here can be implemented in an order other than that illustrated or described here.

[0050] Traditional video motion estimation algorithms are primarily based on the concept of block matching, and include two parts or processes: motion estimation and motion compensation. Specifically, the current frame image can be divided into several fixed-size pixel blocks, and it is assumed that all pixels within the same pixel block have consistent motion characteristics. By comparing the similarity between the current pixel block and the candidate pixel blocks in the reference frame image, the position of the best matching block is determined, thereby obtaining data such as residuals and motion vectors. After obtaining this data, the position of the pixel blocks in the reference frame image is adjusted based on the residuals, motion vectors, and the reference frame number (i.e., the number of the reference frame image) to generate a predicted frame image of the current frame image.

[0051] See also Figure 1 , which shows a schematic diagram of motion estimation and motion compensation provided by an embodiment of the present disclosure, wherein, Figure 1 (a) is the motion estimation process. Figure 1 (b) is the motion compensation process. Figure 1 As shown in (a) in the figure, multiple candidate pixel blocks in the adjacent area (also called the search area) in the reference frame image are compared with the current pixel block in the current frame image one by one to obtain the best matching block corresponding to the current pixel block.

[0052] It should be noted that the difference between the pixel values of the best matching block and the pixel values of the current pixel block is called the residual. The difference between the pixel values of the best matching block and the pixel values of the current pixel block is the smallest. The relative position between the current pixel block and the best matching block is called the motion vector. After the residual, motion vector, and reference frame number are determined, they are encoded into the bitstream and parsed and reconstructed at the decoder. The bitstream, also known as the bitrate or video bitrate, refers to the amount of data transmitted or processed per unit time.

[0053] like Figure 1 As shown in (b) in the figure, at the video decoding end, the motion vector and reference frame number are used to find the best matching block (as indicated by the asterisk in the reference frame image), and then the best matching block is added to the residual to obtain the final display pixel, that is, the predicted pixel block of the current pixel block (as indicated by the asterisk in the current frame image).

[0054] However, block matching-based motion estimation technology has at least the following problems: for example, in cases of complex motion, texture loss, and noise, large errors will be generated, resulting in larger residual values, which in turn increases the code stream volume and reduces coding efficiency; and in the inter-frame coding stage, the residuals, motion vectors, and reference frame numbers that need to be transmitted in the code stream occupy a significant portion of the code stream volume.

[0055] Deformable convolution adjusts the sampling points at each position in the convolution kernel by sampling and offsetting the input feature map, thereby adapting to different target deformations. Specifically, deformable convolution treats each position on the input feature map as a control point and adjusts the sampling points at each position in the convolution kernel through a learnable offset, thereby forming a deformable convolution kernel. It can be understood that deformable convolution actually means adding an offset to the sampling position in ordinary convolution, so that the convolution kernel can be expanded to a wide range during training.

[0056] See also Figure 2 , which shows a schematic diagram of a common convolution and a deformable convolution provided by an embodiment of the present disclosure, wherein, Figure 2 (a) in the figure is a normal convolution. Figure 2 (b) in is a deformable convolution. Figure 2 As shown in the figure, the top layer of images is the activation unit on objects of different sizes; the middle layer of images is the sampling process performed to obtain the top layer of activation units (which can be regarded as a convolution operation, the 3×3 convolution kernel corresponds to nine points, and the nine points finally get the upper point, that is, a value); the bottom layer of images is the sampling area performed to obtain the middle layer of images.

[0057] It should be noted that Figure 2 (a) in the figure is a standard 3×3 square matrix sampling. Figure 2 (b) in the figure is a non-standard shape sampling, but the sampling points are still 3×3, that is, 3×3 convolution kernel. Figure 2 As shown in Figure 2, deformable convolution can be closer to the shape and size of the object during sampling and is more robust.

[0058] Deformable convolution introduces an independent control point prediction layer, which enables the convolution kernel to adaptively follow local changes in the image, thereby better capturing the details of the image. Figure 3 , which shows a schematic diagram of a deformable convolution process provided by an embodiment of the present disclosure. Figure 3 As shown in the figure, the input feature map passes through a convolution layer (conv) to obtain an offset field, and then generates specific offsets through the offset field. These offsets determine the sampling position of the convolution kernel on the input feature map; the input feature map is then deformed and convolved with the offset to obtain the output feature map.

[0059] It should be noted that the offset field is used to store the dynamic offset of each sampling point of the convolution kernel. The number of channels of the offset field is 2N, which means that the convolution kernel learns the offsets in the X and Y directions respectively. N can be obtained according to the convolution kernel, which is specifically the number of convolution kernel sampling points. For example, N of a 3×3 convolution kernel is 9, corresponding to 18 channels.

[0060] Based on this, an embodiment of the present disclosure provides a video processing method, which includes: obtaining a current frame image in a video to be processed; performing prediction processing on each pixel of the current frame image to obtain predicted frame data; the predicted frame data includes prediction data for each pixel of the current frame image, and the prediction data includes bias information and reference frame information corresponding to the bias information; and performing motion compensation on the current frame image based on the predicted frame data to obtain a predicted frame image. In this way, first, each pixel of the current frame image is predicted to obtain predicted frame data, and then motion compensation is performed on the current frame image based on the predicted frame data, thereby achieving pixel-level motion estimation and motion compensation for the current frame image. In addition, the prediction data for each pixel includes not only bias information but also reference frame information corresponding to the bias information, thereby improving the prediction precision and accuracy of the predicted frame image based on the bias information and the reference frame information corresponding to the bias information. Furthermore, when the video processing method is used for video encoding and video decoding, the code stream volume can be significantly reduced, and the encoding efficiency and decoding efficiency can be improved.

[0061] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0062] In one embodiment of the present disclosure, see Figure 4 , which shows a flow chart of a video processing method provided by an embodiment of the present disclosure. Figure 4 As shown, the video processing method may include:

[0063] S101: Obtain a current frame image in a video to be processed.

[0064] It should be noted that the embodiments of the present disclosure provide a video processing method, specifically a method for end-to-end video motion estimation and compensation based on deformable convolution. Specifically, the video processing method is implemented based on a deformable convolution model, and implements pixel prediction and motion compensation at the video encoding and decoding ends.

[0065] It should also be noted that the video processing method can be applied to a video processing device or an electronic device integrated with the video processing device. Here, the electronic device can be a computer, a smart phone, a tablet computer, a laptop computer, a server, etc., and the present disclosure does not specifically limit this.

[0066] Here, the video to be processed refers to a video in which the predicted pixel values of all pixels have not yet been completely obtained. The video to be processed may include multiple frames, such as a reference frame image in which the predicted pixel values of each pixel have been obtained and a frame image in which the predicted pixel values of each pixel have not yet been obtained. The current frame image refers to a frame image in which the predicted pixel values of each pixel are to be obtained.

[0067] S102: performing prediction processing on each pixel of the current frame image to obtain predicted frame data; the predicted frame data includes the predicted data of each pixel of the current frame image, and the predicted data includes offset information and reference frame information corresponding to the offset information.

[0068] It should be noted that bias information can also be called an offset value or offset; reference frame information specifically refers to the reference frame ID (ref ID), i.e., the reference frame image ID. Each bias information has a corresponding reference frame information. A pixel that has already received a predicted pixel value can be determined based on the bias information and the reference frame information corresponding to the bias information.

[0069] In one solution, prediction processing is performed on a block-by-block basis; in the disclosed embodiment, prediction processing is performed on a pixel-by-pixel basis, which can improve prediction precision and accuracy. Furthermore, in one solution, only bias information is obtained for each pixel after prediction processing; in the disclosed embodiment, the prediction data for each pixel includes not only the bias information but also the reference frame information corresponding to the bias information. This allows for more precise prediction of each pixel's predicted value based on this prediction data, further improving prediction precision and accuracy.

[0070] In some embodiments, the video processing method may further include:

[0071] Perform prediction processing on the pixels of the current frame image one by one to obtain predicted frame data;

[0072] The prediction process for the pixels of the current frame image includes:

[0073] Determine a prediction block where a target prediction pixel to be predicted is located, and corresponding pixel blocks in a plurality of related frame images related to the current frame image;

[0074] The target prediction model performs prediction processing on the prediction block and the corresponding pixel blocks in multiple related frame images to obtain bias information corresponding to the target prediction pixel and reference frame information corresponding to the bias information; the reference frame image corresponding to the reference frame information is one of the multiple related frame images.

[0075] In the disclosed embodiment, prediction processing is performed on a pixel of the current frame image to obtain prediction data for the pixel. Prediction processing is then performed on all pixels of the current frame image one by one according to the method, and then predicted frame data is assembled based on the obtained prediction data for each pixel. Here, when prediction processing is required for a pixel of the current frame image, the pixel is determined as a target prediction pixel.

[0076] It should be noted that the related frame image refers to a frame image that is close to and associated with the current frame image, and is a frame image in which the predicted pixel value of each pixel has been determined. Specifically, the related frame image can be a frame image adjacent to the current frame image, or it can be a frame image separated by 2, 3, or other numbers of frames from the current frame image, and there is no specific limitation on this. In addition, the related frame image can be ahead of or behind the current frame image in time, and multiple related frame images can include both frame images that are ahead of and lag behind the current frame image, or can only include frame images that are ahead of or lag behind the current frame image, and there is no specific limitation on this. Furthermore, the number of related frame images is related to the size of the first convolution kernel in the target prediction model, and there is no specific limitation on the number of related frame images.

[0077] It should also be noted that after determining multiple related frame images based on the current frame image, a pixel block related to the prediction block can be determined in each related frame image. The prediction block and the corresponding related pixel blocks in the multiple related frame images are then input into the target prediction model to obtain bias information and reference frame information corresponding to the bias information. In other words, after determining the target prediction pixel, the reference frame image corresponding to the target prediction pixel can be determined in the multiple related frame images.

[0078] Here, the target prediction model is implemented based on the original deformable convolution model. Specifically, an input channel is added so that the prediction block and the corresponding multiple pixel blocks in multiple related frame images can be input into the target prediction model. In addition to the pixel blocks of the current frame image, the input of the prediction layer of the target prediction model also includes the corresponding multiple pixel blocks in multiple related frame images, and the output is also adjusted to a dual-channel output, so that each pixel will obtain the corresponding bias information and the reference frame information corresponding to the bias information; while the bias prediction layer of the original deformable convolution model only uses one layer of feature input, and each pixel will only obtain a corresponding bias value.

[0079] In some embodiments, the video processing method may further include:

[0080] Determining, according to a position of a target prediction pixel, the target prediction pixel and a plurality of first adjacent pixels within a first preset range as a prediction block; the first preset range matches a first convolution kernel size of a target prediction model, and the first preset range is located in a first direction and a second direction of the target prediction pixel;

[0081] In the multiple related frame images, corresponding pixel blocks in the multiple related frame images are determined according to the position of the prediction block and the first preset range.

[0082] It should be noted that when the prediction block is input into the target prediction model, the pixel value of the target prediction pixel can be replaced with 0, or with the pixel value of the corresponding position in the previous frame image for which the prediction pixel value has been determined, or with the pixel value of the corresponding position in any one of multiple related frame images, without specific limitation. In this way, by filling the position of the target prediction pixel with the pixel value of the related frame image, prediction can be performed based on the pixel value of the related frame image, which can improve prediction efficiency.

[0083] In addition, the disclosed embodiments are applied to inter-frame prediction. All pixel blocks in the initial frame image are intra-frame prediction blocks, and do not include inter-frame prediction blocks. For example, when the current frame image is the second frame image, the previous frame image has data, but the previous two frames do not have data. The portions without data can be filled with 0.

[0084] It should also be noted that the prediction block needs to have a range restriction, namely a first preset range. It can be understood that the prediction block consists of the target prediction pixel and multiple first adjacent pixels within the first preset range of the target prediction pixel. The first preset range matches the first convolution kernel size of the target prediction model. The first convolution kernel size refers to the size of the first convolution kernel; the first convolution kernel is the convolution kernel within the convolution layer used for prediction processing to obtain bias information and reference frame information corresponding to the bias information.

[0085] In addition, the first preset range may be located in a first direction and a second direction of the target prediction pixel, the first direction may refer to the left, and the second direction may refer to the top, but this is not specifically limited. When the target prediction pixel is located in the lower right corner of the prediction block, the first direction is the left, and the second direction is the top. Exemplarily, if the first convolution kernel is a 3×3 convolution kernel, then the target prediction pixel and the first adjacent pixels of the two columns to the left, the two rows above, the current row, and the current column of the target prediction pixel constitute the prediction block; if the first convolution kernel is a 5×5 convolution kernel, then the target prediction pixel and the first adjacent pixels of the four columns to the left, the four rows above, the current row, and the current column of the target prediction pixel constitute the prediction block.

[0086] It should also be noted that, according to the position of the prediction block in the current frame image and the size of the first preset range, a pixel block corresponding to the prediction block is determined in a plurality of related frame images.

[0087] See also Figure 5 , which shows a schematic diagram of the prediction layer of a target prediction model provided by an embodiment of the present disclosure. Figure 5As shown, exemplarily, the input of the prediction layer is the pixel blocks of three frames of images, namely the prediction block in the current frame image, the pixel block in the previous frame image, and the pixel block in the historical frame image. After prediction processing, the bias information of the target predicted pixel and the reference frame information corresponding to the bias information can be obtained.

[0088] It should be noted that the previous frame image and the historical frame image are related frame images. Figure 5 Except for the pixel in the lower right corner of the predicted block in the previous frame, the remaining pixels in the predicted block, the pixels in the previous frame of the previous image, and the pixels in the previous frame of the previous image have all been predicted and have determined predicted pixel values. For example, the pixel value of the target predicted pixel is replaced with the pixel value at the corresponding position in the previous frame of the image.

[0089] In addition, exemplarily, the 8 first adjacent pixels to the left and above the position of the target prediction pixel constitute a first preset range, and the prediction block includes 9 pixels; and according to the position of the prediction block and the first preset range, the pixel block in the previous frame image and the pixel block in the historical frame image also include 9 pixels at the corresponding positions respectively.

[0090] S103: Perform motion compensation on the current frame image based on the predicted frame data to obtain a predicted frame image.

[0091] After each pixel of the current frame image is predicted one by one to obtain predicted frame data, motion compensation is performed on the current frame image according to the predicted frame data to obtain a predicted frame image.

[0092] In some embodiments, the video processing method may further include:

[0093] Performing motion compensation on pixels of the current frame image one by one based on the predicted frame data to obtain a predicted frame image;

[0094] The motion compensation is performed on the pixels of the current frame image based on the predicted frame data, including:

[0095] Determine the compensation block where the target compensation pixel to be compensated is located;

[0096] The target prediction model performs motion compensation on the target compensation pixel based on the prediction data of each pixel of the compensation block to obtain a predicted pixel value of the target compensation pixel.

[0097] In the disclosed embodiment, after motion compensation is performed on a pixel in the current frame image and a predicted pixel value is obtained for that pixel, motion compensation is then performed on all pixels in the current frame image one by one according to the method. A predicted frame image is then formed based on the obtained predicted pixel value for each pixel. When motion compensation is required for a pixel in the current frame image, that pixel is determined as a target compensation pixel.

[0098] It should be noted that, except for the target compensation pixel, the remaining pixels in the compensation block are pixels whose predicted pixel values have been determined according to the corresponding prediction data.

[0099] In some embodiments, the video processing method may further include:

[0100] According to the position of the target compensation pixel, determining the target compensation pixel and a plurality of second adjacent pixels within a second preset range as a compensation block;

[0101] Among them, multiple second adjacent pixels are pixels obtained in the corresponding reference frame image according to the corresponding bias information, the second preset range matches the second convolution kernel size of the target prediction model, and the second preset range is located in the first direction and the second direction of the target compensation pixel.

[0102] It should be noted that the compensation block also needs to have a range restriction, namely a second preset range. It can be understood that the compensation block consists of the target compensation pixel and multiple second adjacent pixels within the second preset range of the target compensation pixel. The second preset range matches the second convolution kernel size of the target prediction model. The second convolution kernel size refers to the size of the second convolution kernel; the second convolution kernel is the convolution kernel within the convolution layer used for motion compensation to obtain the predicted pixel value.

[0103] In addition, the second preset range can be located in the first direction and the second direction of the target compensation pixel, the first direction can refer to the left, and the second direction can refer to the top, but this is not specifically limited. When the target prediction pixel is located in the lower right corner of the compensation block, the first direction is the left and the second direction is the top. Exemplarily, if the second convolution kernel is a 3×3 convolution kernel, then the target compensation pixel and the target compensation pixel's two columns to the left, two rows above, current row, and the second adjacent pixels of the current column constitute the compensation block; if the second convolution kernel is a 5×5 convolution kernel, then the target compensation pixel and the target compensation pixel's four columns to the left, four rows above, current row, and the second adjacent pixels of the current column constitute the compensation block.

[0104] In the disclosed embodiment, based on the position of the target compensation pixel, the positions of multiple second-neighboring pixels within a second preset range are determined. Then, based on the positions of the multiple second-neighboring pixels, the corresponding offset information and reference frame information corresponding to the offset information are obtained. The pixels corresponding to the offset information are then transferred from the corresponding reference frame image to form a compensation block. The second preset range matches the size of the second convolution kernel of the target prediction model; that is, the size of the compensation block matches the size of the second convolution kernel of the target prediction model.

[0105] Furthermore, after the compensation block is determined, a convolution operation is performed on the compensation block and the second convolution kernel to obtain a predicted pixel value of the target compensation pixel.

[0106] In one solution, the convolution implementation layer of the original deformable convolution model can move pixels at positions corresponding to the bias information based on the predicted bias information, then stitch the moved pixels into a compensation block that matches the size of the second convolution kernel, and then perform a convolution operation on the compensation block and the second convolution kernel. In the disclosed embodiment, the motion compensation layer of the target prediction model moves pixels at positions corresponding to the bias information in the corresponding reference frame image based on the bias information output by the prediction layer and the reference frame information corresponding to the bias information; then stitch the moved pixels into a compensation block whose size is the same as the size of the second convolution kernel of the target prediction model; and then perform a convolution operation on the compensation block and the second convolution kernel.

[0107] See also Figure 6 , which shows a schematic diagram of a motion compensation layer of a target prediction model provided by an embodiment of the present disclosure. Figure 6 As shown in the figure, for example, pixels at positions corresponding to the offset information are moved from different reference frame images to form a compensation block. The compensation block includes 9 pixels. After performing a convolution operation on the compensation block and a 3×3 convolution kernel, the predicted pixel value of the target compensation pixel is obtained.

[0108] The disclosed embodiment provides a video processing method, which first performs prediction processing on each pixel of a current frame image to obtain predicted frame data, and then performs motion compensation on the current frame image based on the predicted frame data, thereby realizing pixel-level motion estimation and motion compensation of the current frame image; in addition, the prediction data of each pixel includes not only bias information, but also reference frame information corresponding to the bias information, thereby improving the prediction precision and accuracy of the predicted frame image based on the bias information and the reference frame information corresponding to the bias information; further, when the video processing method is used for video encoding and decoding, compared with the traditional block matching algorithm, there is no need to transmit data such as residuals, motion vectors and reference frame numbers, thereby significantly reducing the code stream volume and improving encoding efficiency and decoding efficiency.

[0109] In some embodiments, the target prediction model is obtained by training the initial prediction model using a model training method, and the model training method includes:

[0110] Obtain a sample frame image from a sample video;

[0111] Determine, according to a first algorithm, a sample original block in a sample frame image and a plurality of sample reference blocks corresponding to the sample original block;

[0112] Inputting the sample original block and multiple sample reference blocks into the initial prediction model, performing prediction processing and motion compensation on the pixels of the sample original block one by one, and obtaining predicted pixel values of the sample original block;

[0113] Calculating a first difference between a predicted pixel value of the sample original block and an original pixel value of the sample original block;

[0114] If the first difference does not meet the preset error range, the model training step is iterated until the first difference meets the preset error range, and the initial prediction model is determined as the target prediction model.

[0115] Here, sample videos can be obtained using various public video datasets; sample frame images refer to frame images used to train the initial prediction model and which have not yet been subjected to prediction processing and motion compensation.

[0116] It should be noted that the first algorithm can be a rate-distortion optimization algorithm of a traditional encoder. This algorithm can determine the sample original block suitable for inter-frame prediction and the sample reference blocks surrounding the sample original block. Furthermore, the best matching block corresponding to the sample original block can be determined based on a traditional block matching algorithm. The rate-distortion optimization algorithm is a technology that balances bit rate (data volume) and distortion (quality loss) in video coding. It uses a mathematical model to find the optimal balance between the two, achieving the best trade-off between efficient compression and visual quality.

[0117] It should also be noted that by inputting the sample original block and multiple sample reference blocks into the initial prediction model, a predicted pixel value of one of the pixels can be obtained, and then by determining the predicted pixel value of each pixel of the sample original block one by one, the predicted pixel value of the sample original block can be obtained. Particularly, when determining the predicted pixel value of one of the pixels, it can be determined according to the aforementioned video processing method. Specifically, the initial prediction model performs prediction processing on the sample original block and multiple sample reference blocks to obtain bias information corresponding to the target predicted pixel and reference frame information corresponding to the bias information, thereby obtaining prediction data for each pixel; then, the compensation block where the target compensation pixel to be compensated is located is determined, and the initial prediction model performs motion compensation on the target compensation pixel based on the prediction data of each pixel of the compensation block to obtain the predicted pixel value of the target compensation pixel.

[0118] Details not disclosed in the embodiments of the present disclosure may be understood by referring to the description of the aforementioned embodiments.

[0119] It should also be noted that the original pixel value of the sample original block can be obtained according to the initial pixel value of each pixel of the sample original block.

[0120] The preset error range is generally an empirical value and is not specifically limited. Specifically, in lossless mode, the predicted pixels undergo quantization and entropy coding in traditional encoding, which also introduce errors. The preset error range depends on whether it is greater than the errors introduced by the encoder backend and other processes.

[0121] During the training process of the target prediction model, the goal of model learning is to minimize the difference between the predicted pixel value and the expected output (i.e., the original pixel value). This difference is usually called loss. Therefore, during the training process, it is hoped to minimize the loss function.

[0122] Specifically, during the training process, the model is fed with reference block pixels (i.e., sample reference blocks) and original block pixels (i.e., sample original blocks). The model's predicted output (i.e., predicted pixel values) is then calculated through forward propagation. The predicted output is then compared with the expected output, and the difference between them is calculated. Finally, the model parameters are adjusted through backpropagation and optimization algorithms to minimize the loss, thereby ensuring that the predicted pixel values output by the model are consistent with the original pixel values. Forward propagation refers to starting with the input data, calculating and passing data layer by layer through each layer of the neural network, and ultimately obtaining the model's output results. Backpropagation refers to calculating the gradients inversely using the chain rule based on the difference between the model's predicted results and the true labels, and propagating the gradients from the output layer back to each layer of the network to update the model's parameters.

[0123] It should also be noted that the sample reference block is used as input data for the training model, while the best matching block and the sample original block can be used to compare with the predicted pixel values output by the model. Specifically, if the difference between the predicted pixel value output by the model and the original pixel value is less than the difference between the pixel value of the best matching block and the original pixel value, it proves that the coding efficiency of the model is higher than that of the traditional block matching algorithm. In this way, the use of this model can improve the coding efficiency and decoding efficiency during the encoding or decoding process; if the difference between the predicted pixel value output by the model and the original pixel value is less than the allowable error range (i.e., the preset error range), it can be achieved that there is no need to transmit data such as residuals, motion vectors, and reference frame numbers during the code stream transmission.

[0124] It should also be noted that the trained target prediction model can be used for encoding at the encoding end and for decoding at the decoding end, and the model structure and model parameters deployed at both the encoding end and the decoding end are consistent.

[0125] In another embodiment of the present disclosure, see Figure 7 , which shows a flow chart of a video encoding method provided by an embodiment of the present disclosure. The video encoding method is applied to a video encoding end, such as Figure 7 As shown, the video encoding method may include:

[0126] S201: Divide the current frame image into multiple coding blocks. For any coding block: use the first coding method to encode and obtain the target pixel value of the target matching block corresponding to the coding block; use the first coding method to encode based on the second algorithm; and use the second coding method to perform prediction processing and motion compensation to obtain the predicted pixel value of the coding block; use the second coding method to encode based on the target prediction model.

[0127] S202: Calculate a second difference between a target pixel value of the target matching block and an original pixel value of the coding block; and calculate a third difference between a predicted pixel value of the coding block and an original pixel value of the coding block.

[0128] S203: If the second difference is less than the third difference, encode the coding block according to the first coding method to obtain first coding information; if the second difference is greater than the third difference, perform prediction processing and motion compensation on the coding block according to the second coding method to obtain second coding information; the second coding information at least includes identification information, and the identification information is used to indicate that the coding block corresponds to the second coding method.

[0129] It should be noted that the video encoding method can be applied to a video encoding device or an electronic device incorporating the video encoding device. Here, the electronic device can be a computer, a smartphone, a tablet computer, a laptop computer, a server, etc., and the present disclosure does not specifically limit this.

[0130] It should also be noted that a coding block is equivalent to the aforementioned prediction block or compensation block. Specifically, when performing prediction processing, a coding block is equivalent to a prediction block; when performing motion compensation, a coding block is equivalent to a compensation block.

[0131] Here, the second algorithm may refer to a traditional block matching algorithm. The first encoding method is based on the second algorithm and may include two parts: motion estimation and motion compensation. The first encoding information obtained includes at least residuals, motion vectors and reference frame numbers. The embodiments of the present disclosure will not be elaborated here.

[0132] It should also be noted that when encoding any coding block based on a target prediction model, the coding block and multiple related pixel blocks corresponding to the coding block can be input into the target prediction model to obtain a predicted pixel value for one of the pixels. The predicted pixel value of each pixel in the coding block is then determined one by one, thereby obtaining the predicted pixel value for the coding block. The predicted pixel value for one of the pixels can be determined according to the aforementioned video processing method, which will not be further described here.

[0133] Details not disclosed in the embodiments of the present disclosure may be understood by referring to the description of the aforementioned embodiments.

[0134] In the embodiment of the present disclosure, the encoding method corresponding to each coding block can be determined based on the comparison result, so that different coding blocks in the current frame image are encoded using different encoding methods, and then the next frame image is encoded based on the same method, which can improve the encoding efficiency. When the second encoding method is used for encoding at the video encoding end, that is, when deformable convolution is used to implement motion estimation, pixel-level motion estimation can be directly based on pixel prediction. In addition, when the prediction is accurate enough, all coding blocks can be encoded using the second encoding method, so that the residual, motion vector, and reference frame number do not need to be encoded into the video code stream, which can greatly reduce the code stream volume and improve the encoding efficiency.

[0135] In another embodiment of the present disclosure, see Figure 8 , which shows a flow chart of a video decoding method provided by an embodiment of the present disclosure. The video decoding method is applied to a video decoding end, such as Figure 8 As shown, the video decoding method may include:

[0136] S301: Determine a decoding method corresponding to video coding information to be decoded, and decode multiple coding blocks to be decoded; the video coding information includes first coding information and second coding information.

[0137] S302: If the coded information is first coded information, decode the coded block to be decoded using a first decoding method to obtain a first decoded block; the first decoding method performs decoding based on a second algorithm.

[0138] S303: If the coding information is the second coding information, each pixel of the coding block to be decoded is predicted to obtain decoded block data; the decoded block data includes offset information of each pixel of the decoding block and reference frame information corresponding to the offset information; and motion compensation is performed on the coding block to be decoded based on the decoded block data to obtain a second decoded block; the second decoding method is based on the target prediction model.

[0139] It should be noted that the video decoding method can be applied to a video decoding device or an electronic device integrated with the video decoding device. Here, the electronic device can be a computer, a smart phone, a tablet computer, a laptop computer, a server, etc., and the embodiments of the present disclosure do not specifically limit this.

[0140] Here, the decoding method corresponds to the encoding method. When the identification information in the second encoded information is recognized, the target prediction model is used for direct decoding. The remaining pixel blocks are decoded using a block matching algorithm. The decoded first decoded block and the second encoded block are then concatenated to obtain the predicted frame image corresponding to the current frame image.

[0141] It should also be noted that when decoding a coding block to be decoded based on a target prediction model, the coding block to be decoded and multiple related pixel blocks corresponding to the coding block to be decoded can be input into the target prediction model to obtain a predicted pixel value for one of the pixels. Then, the predicted pixel value for each pixel of the coding block to be decoded is determined one by one, thereby determining a second decoded block. Determining the predicted pixel value for one of the pixels can be performed according to the aforementioned video processing method, which will not be further described here.

[0142] Details not disclosed in the embodiments of the present disclosure may be understood by referring to the description of the aforementioned embodiments.

[0143] In the embodiment of the present disclosure, the bias information and the reference frame information corresponding to the bias information are directly calculated at the video decoding end using a target prediction model with the same weight as that at the video encoding end; and by placing two identical models in the encoder and decoder respectively (i.e., the model structure and model parameters are consistent), the parameters of the model are static (i.e., after the model training is completed, the parameters of the model are no longer updated with the input data), so that the subsequent improvement of the prediction performance only needs to update the weights of the model without changing the hardware. In addition, in the encoding and decoding stages, the parameters of the model do not need to be updated, so they do not need to be transmitted in the bitstream, which improves the encoding efficiency and decoding efficiency; if the subsequent encoder version is updated, the update can also be achieved by only updating the parameters. That is to say, in the embodiment of the present disclosure, not only can video motion estimation and compensation from the encoding end to the decoding end be achieved, but the flexibility and upgradeability of the system can also be maintained at the same time.

[0144] In another embodiment of the present disclosure, see Figure 9 , which shows a schematic diagram of the composition structure of a video processing device provided by an embodiment of the present disclosure. Figure 9 As shown, the video processing device 40 may include an acquisition unit 401, a prediction unit 402 and a compensation unit 403;

[0145] An acquisition unit 401 is configured to acquire a current frame image in a video to be processed;

[0146] The prediction unit 402 is configured to perform prediction processing on each pixel of the current frame image to obtain predicted frame data; the predicted frame data includes predicted data of each pixel of the current frame image, and the predicted data includes offset information and reference frame information corresponding to the offset information;

[0147] The compensation unit 403 is configured to perform motion compensation on the current frame image based on the predicted frame data to obtain a predicted frame image.

[0148] In some embodiments, the prediction unit 402 is further configured to perform prediction processing on the pixels of the current frame image one by one to obtain predicted frame data; determine the prediction block where the target predicted pixel to be predicted is located, and the corresponding pixel blocks in multiple related frame images related to the current frame image; use the target prediction model to perform prediction processing on the prediction block and the corresponding pixel blocks in the multiple related frame images to obtain bias information corresponding to the target predicted pixel and reference frame information corresponding to the bias information; the reference frame image corresponding to the reference frame information is one of the multiple related frame images.

[0149] In some embodiments, the compensation unit 403 is further configured to perform motion compensation on pixels of the current frame image one by one based on the predicted frame data to obtain a predicted frame image; determine the compensation block where the target compensation pixel to be compensated is located; and perform motion compensation on the target compensation pixel based on the predicted data of each pixel of the compensation block by the target prediction model to obtain a predicted pixel value of the target compensation pixel.

[0150] In some embodiments, the prediction unit 402 is further configured to determine the target prediction pixel and multiple first adjacent pixels within a first preset range as a prediction block based on the position of the target prediction pixel; the first preset range matches the first convolution kernel size of the target prediction model, and the first preset range is located in the first direction and the second direction of the target prediction pixel; in multiple related frame images, the corresponding pixel blocks in the multiple related frame images are determined based on the position of the prediction block and the first preset range.

[0151] In some embodiments, the compensation unit 403 is further configured to determine the target compensation pixel and multiple second adjacent pixels within a second preset range as a compensation block based on the position of the target compensation pixel; wherein the multiple second adjacent pixels are pixels obtained according to corresponding bias information in the corresponding reference frame image, the second preset range matches the second convolution kernel size of the target prediction model, and the second preset range is located in the first direction and the second direction of the target compensation pixel.

[0152] In some embodiments, the target prediction model is obtained by training the initial prediction model using a model training method. Figure 9 As shown, the video processing device 40 may further include a training unit 404;

[0153] The training unit 404 is configured to obtain a sample frame image in a sample video; determine a sample original block and multiple sample reference blocks corresponding to the sample original block in the sample frame image according to a first algorithm; input the sample original block and the multiple sample reference blocks into an initial prediction model, perform prediction processing and motion compensation on the pixels of the sample original block one by one, and obtain a predicted pixel value of the sample original block; calculate a first difference between the predicted pixel value of the sample original block and the original pixel value of the sample original block; if the first difference does not meet a preset error range, iterate the model training steps until the first difference meets the preset error range, and determine the initial prediction model as the target prediction model.

[0154] In another embodiment of the present disclosure, see Figure 10 , which shows a schematic diagram of the composition structure of a video encoding device provided by an embodiment of the present disclosure. Figure 10 As shown, the video encoding device 50 may include an encoding unit 501, a calculation unit 502 and a judgment unit 503;

[0155] The encoding unit 501 is configured to divide the current frame image into a plurality of encoding blocks, and for each encoding block: encoding using a first encoding method to obtain a target pixel value of a target matching block corresponding to the encoding block; encoding using the first encoding method based on a second algorithm; and performing prediction processing and motion compensation using a second encoding method to obtain a predicted pixel value of the encoding block; encoding using the second encoding method based on a target prediction model;

[0156] A calculation unit 502 is configured to calculate a second difference between a target pixel value of the target matching block and an original pixel value of the coding block; and calculate a third difference between a predicted pixel value of the coding block and an original pixel value of the coding block;

[0157] The judgment unit 503 is configured to encode the coding block according to the first coding method to obtain first coding information if the second difference is less than the third difference; if the second difference is greater than the third difference, perform prediction processing and motion compensation on the coding block according to the second coding method to obtain second coding information; the second coding information includes at least identification information, and the identification information is used to indicate that the coding block corresponds to the second coding method.

[0158] In another embodiment of the present disclosure, see Figure 11 , which shows a schematic diagram of the composition structure of a video decoding device provided by an embodiment of the present disclosure. Figure 11 As shown, the video decoding device 60 may include a determining unit 601 and a decoding unit 602;

[0159] The determining unit 601 is configured to determine a decoding method corresponding to the video coding information to be decoded, and decode the plurality of coding blocks to be decoded; the video coding information includes first coding information and second coding information;

[0160] The decoding unit 602 is configured to, if the encoding information is the first encoding information, decode the encoding block to be decoded using the first decoding method to obtain a first decoded block; the first decoding method is based on the second algorithm for decoding; and if the encoding information is the second encoding information, perform prediction processing on each pixel of the encoding block to be decoded to obtain decoded block data; the decoded block data includes bias information of each pixel of the decoding block and reference frame information corresponding to the bias information; and perform motion compensation on the encoding block to be decoded based on the decoded block data to obtain a second decoded block; the second decoding method is based on the target prediction model for decoding.

[0161] It is understood that in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular system. Furthermore, the various components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The aforementioned integrated units can be implemented in the form of hardware or software functional modules.

[0162] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0163] Therefore, this embodiment provides a computer storage medium, which stores a computer program. When the computer program is executed by at least one processor, it implements the steps of the video processing method, video encoding method, or video decoding method described in any one of the aforementioned embodiments.

[0164] Based on the above-mentioned composition of a video processing device 40 or a video encoding device 50 or a video decoding device 60 and a computer storage medium, see Figure 12 , which shows a schematic diagram of the composition structure of an electronic device provided by an embodiment of the present disclosure Figure 1 .like Figure 12 As shown, the electronic device 70 may include: a communication interface 701, a memory 702 and a processor 703; each component is coupled together via a bus system 704. It is understood that the bus system 704 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 12 Various buses are labeled as bus system 704. Among them, the communication interface 701 is used to receive and send signals in the process of sending and receiving information between other external network elements;

[0165] Memory 702, used to store computer programs that can be run on processor 703;

[0166] The processor 703 is configured to execute the aforementioned video processing method, video encoding method, or video decoding method when running the computer program.

[0167] It is understood that the memory 702 in the embodiment of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 702 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0168] The processor 703 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 703 or by software instructions. The processor 703 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this disclosure can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 702, and the processor 703 reads the information in the memory 702 and, in conjunction with its hardware, completes the steps of the above method.

[0169] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present disclosure, or a combination thereof.

[0170] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0171] Optionally, as another embodiment, the processor 703 is further configured to execute the method described in any one of the aforementioned embodiments when running the computer program.

[0172] In yet another embodiment of the present disclosure, see Figure 13 , which shows a schematic diagram of the composition structure of an electronic device provided by an embodiment of the present disclosure Figure 2 .like Figure 13 As shown, the electronic device 70 at least includes the video processing device 40 and / or the video encoding device 50 and / or the video decoding device 60 according to any one of the aforementioned embodiments.

[0173] It should be noted that the electronic device 70 may include only the video processing device 40 or the video encoding device 50 or the video decoding device 60, or may include the video processing device 40 and the video encoding device 50, or may include the video processing device 40 and the video decoding device 60, or may include the video encoding device 50 and the video decoding device 60, or may include the video processing device 40, the video encoding device 50 and the video decoding device 60, and there is no specific limitation on this.

[0174] For example, the electronic device 70 includes a device, for example, Figure 13 As shown in (a) of FIG. , the electronic device 70 includes a video processing device 40; Figure 13 As shown in (b) of FIG. 1 , the electronic device 70 includes a video encoding device 50; Figure 13 As shown in (c) in FIG. 8 , the electronic device 70 includes a video decoding device 60 .

[0175] In the embodiment of the present disclosure, for the electronic device 70, the code stream volume can be reduced and the encoding efficiency and decoding efficiency can be improved.

[0176] The above description is merely a preferred embodiment of the present disclosure and is not intended to limit the scope of protection of the present disclosure.

[0177] It should be noted that, in this disclosure, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0178] The serial numbers of the above-mentioned embodiments of the present disclosure are for description only and do not represent the advantages or disadvantages of the embodiments.

[0179] The methods disclosed in the several method embodiments provided in this disclosure can be arbitrarily combined without conflict to obtain new method embodiments.

[0180] The features disclosed in the several product embodiments provided in this disclosure can be arbitrarily combined without conflict to obtain new product embodiments.

[0181] The features disclosed in several method or device embodiments provided in this disclosure may be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0182] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A video processing method, comprising: Get the current frame image in the video to be processed; After performing prediction processing on each pixel of the current frame image, predicted frame data is obtained; the predicted frame data includes predicted data of each pixel of the current frame image, and the predicted data includes offset information and reference frame information corresponding to the offset information; Motion compensation is performed on the current frame image based on the predicted frame data to obtain a predicted frame image.

2. The method according to claim 1, wherein The method further comprises: Performing prediction processing on pixels of the current frame image one by one to obtain the predicted frame data; The step of performing prediction processing on the pixels of the current frame image includes: Determining a prediction block where a target prediction pixel to be predicted is located, and corresponding pixel blocks in a plurality of related frame images related to the current frame image; The target prediction model performs prediction processing on the prediction block and the corresponding pixel blocks in the multiple related frame images to obtain the bias information corresponding to the target prediction pixel and the reference frame information corresponding to the bias information; the reference frame image corresponding to the reference frame information is one of the multiple related frame images.

3. The method according to claim 1, wherein The method further comprises: Performing motion compensation on pixels of the current frame image one by one based on the predicted frame data to obtain the predicted frame image; The step of performing motion compensation on pixels of the current frame image based on the predicted frame data includes: Determine the compensation block where the target compensation pixel to be compensated is located; The target prediction model performs motion compensation on the target compensation pixel based on the prediction data of each pixel of the compensation block to obtain a predicted pixel value of the target compensation pixel.

4. The method according to claim 2, wherein: The method further comprises: Determining, according to a position of the target predicted pixel, the target predicted pixel and a plurality of first adjacent pixels within a first preset range as the prediction block; the first preset range matches a first convolution kernel size of the target prediction model, and the first preset range is located in a first direction and a second direction of the target predicted pixel; In the multiple related frame images, corresponding pixel blocks in the multiple related frame images are determined according to the position of the prediction block and the first preset range.

5. The method according to claim 3, wherein: The method further comprises: According to the position of the target compensation pixel, determining the target compensation pixel and a plurality of second adjacent pixels within a second preset range as the compensation block; Among them, the multiple second adjacent pixels are pixels obtained in the corresponding reference frame image according to the corresponding bias information, the second preset range matches the second convolution kernel size of the target prediction model, and the second preset range is located in the first direction and the second direction of the target compensation pixel.

6. The method according to claim 2, wherein: The target prediction model is obtained by training the initial prediction model using a model training method, wherein the model training method includes: Obtain a sample frame image from a sample video; Determine, according to a first algorithm, a sample original block in the sample frame image and a plurality of sample reference blocks corresponding to the sample original block; Inputting the sample original block and the plurality of sample reference blocks into the initial prediction model, performing prediction processing and motion compensation on pixels of the sample original block one by one, and obtaining predicted pixel values of the sample original block; Calculating a first difference between a predicted pixel value of the sample original block and an original pixel value of the sample original block; If the first difference does not meet the preset error range, the model training step is iterated until the first difference meets the preset error range, and the initial prediction model is determined as the target prediction model.

7. A video encoding method, applied to a video encoding end, the method comprising: The current frame image is divided into multiple coding blocks. For any of the coding blocks: Encoding using a first encoding method to obtain a target pixel value of a target matching block corresponding to the encoding block; The first encoding method is based on a second algorithm for encoding; Performing prediction processing and motion compensation using a second encoding method to obtain predicted pixel values of the encoding block; The second encoding method is based on the target prediction model for encoding; Calculating a second difference between a target pixel value of the target matching block and an original pixel value of the coding block; and calculating a third difference between the predicted pixel value of the coding block and the original pixel value of the coding block; If the second difference is less than the third difference, encoding the coding block according to the first coding mode to obtain first coding information; If the second difference is greater than the third difference, performing prediction processing and motion compensation on the coding block according to the second coding mode to obtain second coding information; The second coding information includes at least identification information, where the identification information is used to indicate that the coding block corresponds to the second coding mode.

8. A video decoding method, applied to a video decoding end, the method comprising: Determining a decoding method corresponding to the video encoding information according to the video encoding information to be decoded, and decoding the multiple encoding blocks to be decoded; The video encoding information includes first encoding information and second encoding information; If the encoding information is the first encoding information, decoding the encoding block to be decoded using a first decoding method to obtain a first decoded block; The first decoding method performs decoding based on a second algorithm; If the encoding information is the second encoding information, performing prediction processing on each pixel of the encoding block to be decoded to obtain decoded block data; the decoded block data includes offset information of each pixel of the decoding block and reference frame information corresponding to the offset information; and performing motion compensation on the coding block to be decoded based on the decoding block data to obtain a second decoding block; The second decoding method performs decoding based on a target prediction model.

9. A video processing device, comprising an acquisition unit, a prediction unit, and a compensation unit; The acquisition unit is configured to acquire a current frame image in the video to be processed; The prediction unit is configured to perform prediction processing on each pixel of the current frame image to obtain predicted frame data; the predicted frame data includes prediction data of each pixel of the current frame image, and the prediction data includes offset information and reference frame information corresponding to the offset information; The compensation unit is configured to perform motion compensation on the current frame image based on the predicted frame data to obtain a predicted frame image.

10. A video encoding device, applied to a video encoding end, comprising an encoding unit, a calculation unit, and a judgment unit; The encoding unit is configured to divide the current frame image into a plurality of encoding blocks, and for any of the encoding blocks: encode the block using a first encoding method to obtain a target pixel value of a target matching block corresponding to the encoding block; The first encoding method is based on a second algorithm for encoding; and performing prediction processing and motion compensation using a second encoding method to obtain predicted pixel values of the encoding block; The second encoding method is based on the target prediction model for encoding; The calculation unit is configured to calculate a second difference between the target pixel value of the target matching block and the original pixel value of the coding block; and calculating a third difference between the predicted pixel value of the coding block and the original pixel value of the coding block; The judging unit is configured to encode the coding block according to the first coding mode to obtain first coding information if the second difference is smaller than the third difference; If the second difference is greater than the third difference, performing prediction processing and motion compensation on the coding block according to the second coding mode to obtain second coding information; The second coding information includes at least identification information, where the identification information is used to indicate that the coding block corresponds to the second coding mode.

11. A video decoding device, applied to a video decoding end, the video decoding device comprising a determination unit and a decoding unit; The determining unit is configured to determine a decoding method corresponding to the video coding information to be decoded, and decode the plurality of coding blocks to be decoded; the video coding information includes first coding information and second coding information; The decoding unit is configured to, if the encoding information is the first encoding information, decode the encoding block to be decoded using a first decoding method to obtain a first decoded block; The first decoding method is based on a second algorithm for decoding; and if the encoding information is the second encoding information, prediction processing is performed on each pixel of the encoding block to be decoded to obtain decoded block data; the decoded block data includes offset information of each pixel of the decoding block and reference frame information corresponding to the offset information; and performing motion compensation on the coding block to be decoded based on the decoding block data to obtain a second decoding block; The second decoding method performs decoding based on a target prediction model.

12. An electronic device comprising a memory and a processor; The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 6, the method according to claim 7, or the method according to claim 8 when running the computer program.

Images