Video coding method, video decoding method and device

Abstract

The embodiment of the invention provides a video coding method and device and a video decoding method and device, and relates to the technical field of video coding and decoding. Comprising the following steps: acquiring a target image block which is a rectangular image block obtained by performing block division on a video frame; obtaining a boundary of the target image block according to the texture edge in the target image block, wherein the boundary is used for dividing the target image block into two image areas; obtaining two end segments of a segmentation curve according to the coded video content so as to obtain a segmentation starting point and a segmentation ending point; obtaining a segmentation curve and control information of the segmentation curve according to the segmentation starting point, the segmentation ending point and the boundary; segmenting the target image block into two sub-image blocks based on the segmentation curve of the target image block; and according to the control information of the segmentation curve and the two sub-image blocks, obtaining code stream data corresponding to the target image block. Some embodiments of the present application are used for reducing code rate overhead brought by representing a segmentation curve.

Description

Technical Field

[0001] Some embodiments of the present application relate to the technical field of video encoding and decoding, and more specifically, to a video encoding method, a video decoding method and a device. Background Art

[0002] A complete image in a video is usually called a video frame. The content within a video frame and between video frames is often highly correlated, so there is a lot of redundant information in the video. The purpose of video encoding is to remove as much redundant information as possible in the video to reduce the amount of data representing the video content.

[0003] At present, when encoding video frames, mainstream video coding standards first divide video frames into coding tree units (CTUs) of different sizes, and use the resulting CTUs as the minimum unit for encoding and decoding. The encoding and decoding tasks for CTUs can be further refined to divide the CTUs into multiple coding units (CUs), a process called block partitioning. The coding performance of a video is closely related to the block partitioning. In order to improve the video coding performance, the related art proposes a block partitioning scheme that obtains the segmentation curve of an image block based on the internal texture edge of the image and uses the segmentation curve to partition the image block. The block partitioning scheme supports the partitioning of blocks into asymmetric and irregular coding blocks, and can make the block partitioning as close to the real image edge as possible, thereby improving the video coding performance. In addition, in order to ensure the consistency of video encoding and decoding, the bitstream needs to contain the complete information of the segmentation curve. However, the segmentation curve is generally composed of many pixels. If the position information of all pixels is directly encoded, it will undoubtedly cause a very large bitrate overhead. Therefore, how to reduce the bitrate overhead caused by representing the segmentation curve is an urgent problem to be solved. Summary of the invention

[0004] The exemplary embodiments of the present application provide a video encoding method, a video decoding method and an apparatus for reducing the bit rate overhead caused by representing a segmentation curve.

[0005] Some embodiments of the present application provide the following technical solutions:

[0006] In a first aspect, some embodiments of the present application provide a video encoding method, including:

[0007] Obtaining a target image block, wherein the target image block is a rectangular image block obtained by dividing the video frame into blocks;

[0008] Acquire a boundary line of the target image block according to a texture edge in the target image block, wherein the boundary line is used to divide the target image block into two image areas;

[0009] Obtain two endpoints of the segmentation curve according to the encoded video content to obtain the segmentation start point and segmentation end point;

[0010] Acquire the segmentation curve and control information of the segmentation curve according to the segmentation starting point, the segmentation end point and the dividing line;

[0011] Splitting the target image block into two sub-image blocks based on a segmentation curve of the target image block;

[0012] According to the control information of the segmentation curve and the two sub-image blocks, the code stream data corresponding to the target image block is obtained.

[0013] In a second aspect, some embodiments of the present application provide a video decoding method, including:

[0014] Acquire bitstream data corresponding to a target image block, the bitstream data comprising: first bitstream data, second bitstream data and third bitstream data; the first bitstream data is bitstream data obtained by encoding control information of a segmentation curve, the segmentation curve is a curve for dividing the target image block into two sub-image blocks, and the second bitstream data and the third bitstream data are bitstream data obtained by encoding the two sub-image blocks respectively;

[0015] Acquire control information of the segmentation curve according to the first bitstream data;

[0016] Obtain two endpoints of the segmentation curve according to the decoded video content to obtain a segmentation start point and a segmentation end point;

[0017] Acquire the segmentation curve according to the control information of the segmentation curve, the segmentation starting point, and the segmentation end point;

[0018] Segmenting the target image block based on the segmentation curve to determine the two sub-image blocks;

[0019] Reconstructing the two sub-image blocks according to the second code stream data and the third code stream data;

[0020] The reconstructed target image block is obtained according to the two reconstructed sub-image blocks.

[0021] In a third aspect, some embodiments of the present application provide a video encoding device, including:

[0022] a memory configured to store a computer program;

[0023] The processor is configured to, when calling a computer program, enable the video encoding device to implement the video encoding method described in the first aspect.

[0024] In a fourth aspect, some embodiments of the present application provide a video decoding device, including:

[0025] a memory configured to store a computer program;

[0026] The processor is configured to, when calling a computer program, enable the video decoding device to implement the video decoding method described in the second aspect.

[0027] In a fifth aspect, some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the video encoding method described in the first aspect or the video decoding method described in the second aspect.

[0028] In a sixth aspect, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the video encoding method described in the first aspect or the video decoding method described in the second aspect.

[0029] It can be seen from the above technical scheme that the video encoding method provided in the embodiment of the present application, when encoding the target image block obtained by block division of the video frame, first obtains the dividing line for dividing the target image block into two image areas according to the texture edge in the target image block, and then obtains the two end segments of the dividing curve according to the video content that has been encoded to obtain the dividing starting point and the dividing end point, and then obtains the dividing curve and the control information of the dividing curve according to the dividing starting point, the dividing end point and the dividing line, and divides the target image block into two sub-image blocks based on the dividing curve of the target image block, and finally obtains the code stream data corresponding to the target image block according to the control information of the dividing curve and the two sub-image blocks. Since the segmentation starting point and segmentation end point of the segmentation curve for segmenting the target image block into two sub-image blocks in the embodiment of the present application are obtained according to the video content that has been encoded, the decoding end can also obtain the segmentation starting point and the segmentation end point according to the video content that has been encoded. Therefore, the encoding end only needs to add the control information of the segmentation curve to the bitstream data corresponding to the target image block, and the decoding end can obtain the complete segmentation curve information. Therefore, the embodiment of the present application can avoid adding the segmentation starting point and the segmentation end point of the segmentation curve to the bitstream data corresponding to the target image block, thereby reducing the bit rate overhead brought by representing the segmentation curve. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the implementation methods of some embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0031] Figure 1 A block diagram of a video decoding system in some embodiments of the present application is shown;

[0032] Figure 2 A schematic diagram showing the structure of a video encoder in some embodiments of the present application is shown;

[0033] Figure 3 A schematic diagram showing the structure of a video decoder in some embodiments of the present application is shown;

[0034] Figure 4 One of the flow charts of the steps of the video encoding method in some embodiments of the present application is shown;

[0035] Figure 5 The second flowchart of the video encoding method in some embodiments of the present application is shown;

[0036] Figure 6 A schematic diagram showing non-maximum suppression in some embodiments of the present application is shown;

[0037] Figure 7 A schematic diagram showing a set of image regions in some embodiments of the present application;

[0038] Figure 8 A schematic diagram showing a merging scheme in some embodiments of the present application;

[0039] Fig. 9 A schematic diagram showing spatially adjacent pixel points at four edges of a target image area in some embodiments of the present application is shown;

[0040] Fig.10 A schematic diagram showing pixel value step points in some embodiments of the present application is shown;

[0041] Fig.11 A schematic diagram showing a third-order Bezier curve in some embodiments of the present application is shown;

[0042] Fig.12 A schematic diagram showing a first pixel point and a second pixel point in some other embodiments of the present application is shown;

[0043] Fig.13 A schematic diagram showing a second-order Bezier curve in some embodiments of the present application is shown;

[0044] Fig.14Schematic diagrams showing target pixel points in other embodiments of the present application;

[0045] Fig.15 A schematic diagram showing sampling results of segmentation curves in other embodiments of the present application is shown;

[0046] Fig.16 A schematic diagram showing two sub-image blocks in some embodiments of the present application;

[0047] Fig.17 A flowchart showing the steps of a video decoding method in some embodiments of the present application is shown. DETAILED DESCRIPTION

[0048] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0049] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0050] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0051] References to "some implementations", "some embodiments", etc. in the specification indicate that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may include the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same implementation. In addition, when a specific feature, structure, or characteristic is described in connection with an embodiment, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in connection with other implementations (whether or not explicitly described herein).

[0052] The embodiments of the present application relate to the field of video coding and decoding technology. The following first describes a video coding and decoding framework for executing the video encoding method and video decoding method provided by the embodiments of the present application.

[0053] Video can be regarded as a sequence of multiple video frames (images). Video playback can be regarded as the display of video frames at a preset speed (for example, 24 frames per second, 30 frames per second, and 60 frames per second) in the order of the sequence. In theory, the amount of video data is positively correlated with the resolution of the video frame. The higher the resolution of the video frame, the larger the amount of video data. If the pixel data of each pixel of all video frames is directly saved in the video file, the amount of video data will be very large, which will make the video difficult to store and transmit. Video encoding and decoding is proposed to solve this problem to a certain extent. Video decoding mainly includes: video encoding and video decoding. Among them, video encoding can be understood as the process of compressing the original video frame, and video decoding can be understood as the process of reconstructing the video frame based on the compressed video data.

[0054] Reference Figure 1 The block diagram of the video decoding system in some embodiments of the present application is shown in FIG. Figure 1 As shown, the video decoding system 100 includes: a source device 10 and a destination device 20. The source device 10 can obtain original video data through a video source 101, and encode the original video frame through a video encoder 102 to obtain video encoding data, and provide the video encoder 102 output video encoding data to the destination device 20 through an output interface 103. The destination device 20 can obtain the video encoding data provided by the source device 10 through an input interface 201, and decode the video encoding data through a video decoder 202 to obtain video decoding data, and input the video decoding data into a player 203 to play the video. The source device 10 and the destination device 20 may include any of a wide range of devices, such as: a personal computer (Program Counter), a notebook computer, a tablet computer, a set-top box, a mobile phone, a television, a camera, a display, a digital media player, a video game console, a video streaming device, etc.

[0055] In some embodiments, the video source 101 of the source device 10 may be a video shooting device, such as a camera. In other embodiments, the video source 101 may be a component capable of generating video based on computer graphics, such as a screen recording component, an animation generation component, etc.

[0056] In some embodiments, the destination device 20 may receive the video encoding data provided by the source device 10 via a computer-readable medium. The computer-readable medium may include any type of medium or device capable of moving the video encoding data from the source device 10 to the destination device 20. In one example, the computer-readable medium may include a communication medium. The communication medium may modulate the video encoding data according to a communication standard (e.g., a wireless communication protocol) and transmit it to the destination device 20. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet network (e.g., a local area network, a wide area network, or a global network, such as the Internet). The communication medium may include a router, a switch, a base station, or any other device that can be used to facilitate communication from the source device 10 to the destination device 20.

[0057] In some examples, the video encoding data may be output from the output interface 103 of the source device 10 to a storage device. Accordingly, the video encoding data may be accessed from the storage device by the input interface 203 of the destination device 20. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing video encoding data. In another example, the storage device may be a server or an intermediate storage device for storing the video encoding data generated by the source device 10. The destination device 20 may obtain the stored video encoding data from the storage device via streaming or downloading. The file server may be any type of server capable of storing encoded video data and transmitting the encoded video data to the destination device 20. In some embodiments. The file server includes a network server (e.g., for a website), an FTP server, a network attached storage device, or a local disk drive. The destination device 20 may access the encoded video data through any standard data connection (including an Internet connection). This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both suitable for accessing encoded video data stored on a file server. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0058] At present, the video coding standard has gradually evolved from ISO / IECMPEG-1 (International StandardizationOrganization / International Electrotechnical CommissionMoving Picture ExpertsGroup-1), through ISO / IECMPEG-2, ISO / IECMPEG-4, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), etc. to Versatile Video Coding (VVC). The video encoding method and video decoding method provided in the embodiments of the present application can be applied to any suitable video decoding standard. For example: HEVC standard, VVC standard, etc.

[0059] Reference Figure 2 As shown, Figure 2The schematic diagram of the structure of the video encoder in some embodiments of the present application. The video encoder 200 provided in the embodiment of the present application includes: a segmentation module 21, a motion estimation module 22, a motion compensation module 23, a residual calculation module 24, a quantization transformation module 25 and an entropy coding module 26. The process of video encoding by the video encoder 200 includes: first, the video frame to be encoded is divided into a plurality of rectangular image blocks by the segmentation module 21, and the segmentation module 21 can decide whether to further segment the rectangular image block according to the actual encoding situation. Specifically: Since selecting a larger encoding block is more conducive to improving the encoding efficiency, for rectangular image blocks with relatively flat pixels, fewer segmentation times can be selected, and the segmentation depth is shallow. On the contrary, for rectangular image blocks with more complex textures or located at the edge of the image, more segmentation times can be selected, and the segmentation depth is deeper. The segmentation module 21 will also segment at least one rectangular image block into two heterogeneous sub-image blocks based on a segmentation curve that fits the real texture edge. Then, the motion estimation module 22 performs motion estimation (ME) on each coding block (including: rectangular image blocks and irregular sub-image blocks that are not further divided) to obtain a reference block (RB) corresponding to each coding block, and then the motion compensation module 23 performs motion compensation (MC) on the reference block to obtain a prediction block and a motion vector (MV) corresponding to each coding block, and the residual calculation module 24 calculates the residual between the prediction block and the corresponding coding block to obtain a prediction residual, and finally the quantization transformation module 25 transforms and quantizes the prediction residual to obtain a transformation coefficient, and the entropy coding module 26 performs entropy coding on the segmentation curve, motion vector, transformation coefficient and other information to obtain the bit stream data of the video frame to be encoded.

[0060] It should be noted that the above encoding process only describes some steps in the video encoding process. In addition to the above steps, the video encoding process also includes other steps. For example, inputting information such as the prediction mode and the transformation mode into the entropy encoding module 26, and adding the entropy encoding result of the prediction mode, the transformation mode and the like information to the encoded data of the video frame to be encoded.

[0061] Reference Figure 3 As shown, Figure 3The schematic diagram of the structure of the video decoder in some embodiments of the present application. The video encoder 300 provided in the embodiment of the present application includes: an entropy decoding module 31, a motion estimation module 32, a motion compensation module 33, an inverse quantization transformation module 34, a fusion module 35 and a reconstruction module 36. The process of video decoding by the video encoder 300 includes: firstly, entropy decoding is performed on the bit stream data of the video frame to be encoded by the entropy decoding module 31, and at least one of the information such as a segmentation curve for segmenting the rectangular image block of the video frame to be reconstructed into two irregular sub-image blocks, motion vectors of the two sub-image blocks, transformation coefficients of the two sub-image blocks, motion vectors of the rectangular image block, and transformation coefficients of the rectangular image block is obtained according to the entropy decoding result. Then, on the one hand, the motion estimation module 32 determines the reference block corresponding to each block to be reconstructed (including: rectangular image blocks and irregular sub-image blocks that are not further divided), and the motion compensation module 33 performs motion compensation on the reference block according to the motion vector to obtain the prediction block corresponding to the block to be reconstructed; on the other hand, the inverse quantization transformation module 34 performs inverse quantization and inverse transformation operations on the transformation coefficient to obtain the prediction residual, and the fusion module 35 adds and fuses the prediction residual with the prediction block to obtain the reconstructed data of the block to be reconstructed, and finally the reconstruction module 36 reconstructs each block to be reconstructed according to the reconstruction data of the block to be reconstructed to obtain the reconstructed video frame.

[0062] The present application embodiment provides a video encoding method, referring to Figure 4 As shown, the video encoding method includes the following steps S41 to S46:

[0063] S41, obtaining a target image block.

[0064] The target image block is a rectangular image block obtained by dividing the video frame into blocks.

[0065] Under video coding standards such as HEVC and VVC, after receiving the video frame to be encoded, the encoder will first divide the video frame to be encoded into multiple image blocks, and encode the image blocks obtained by dividing the video frame to be encoded as the minimum coding unit. In addition, the encoding task for the image block can be further refined. For example: In the HEVC standard, the image block obtained by dividing the video frame to be encoded is called a coding tree unit (Coding Tree Unit, CTU), and the coding tree unit can be further divided into smaller square coding units (Coding unit, CU) by quadtree division. The maximum size of the coding tree unit can be supported to 64×64 and the minimum size can be supported to 16×16. In order to meet the needs of ultra-high-definition video encoding such as 4K and 8K, the VVC standard expands the maximum size of CTU to 128×128, and further division methods are no longer limited to quadtree division, but also support binary tree and ternary tree division. The target image block in the embodiment of the present application can be a coding tree unit, or it can be a coding unit obtained by further dividing the coding tree unit, and the embodiment of the present application does not limit this.

[0066] S42: Acquire a boundary line of the target image block according to a texture edge in the target image block.

[0067] The dividing line is used to divide the target image block into two image areas.

[0068] In some embodiments, the above step S42 (obtaining the boundary line of the target image block according to the texture edge in the target image block) includes: dividing the target image block into two image areas based on the texture edge in the target image block, and obtaining the boundary line of the two image areas as the boundary line of the target image block.

[0069] It should be noted that, in some cases, the brightness of the pixels in the image block is relatively flat, and the texture edge in the image block cannot divide the target image block into two image areas. For such image blocks, the embodiment of the present application may not further divide the sub-image blocks, but directly encode such image blocks as a coding unit. In other cases, the texture edge in the image block will divide the image block into image areas greater than 2. For such image blocks, multiple image areas can be first merged into two image areas, and then the boundary line of the two image areas can be obtained as the boundary line of the target image block.

[0070] S43: Obtain two end segments of the segmentation curve according to the encoded video content to obtain a segmentation start point and a segmentation end point.

[0071] The video content that has been encoded in the embodiment of the present application refers to the video content that has been encoded before encoding the target image block. The video content that has completed video encoding can be the content in the video frame that is close to the current video frame in time domain, or it can be the content in the current video frame that is close to the target image block in spatial domain.

[0072] S44: Acquire the segmentation curve and control information of the segmentation curve according to the segmentation starting point, the segmentation end point, and the dividing line.

[0073] In the embodiment of the present application, there is no restriction on the type of segmentation curve. When the segmentation curve is a curve of different types, the control information of the segmentation curve is control information of different types. Therefore, the embodiment of the present application does not restrict the control information of the segmentation curve, and the segmentation curve can be reconstructed according to the segmentation starting point, the segmentation end point and the control information of the segmentation curve. For example, when the segmentation curve is a second-order Bezier curve, the control information of the segmentation curve is the position information of the control point of the second-order Bezier curve. For another example, when the segmentation curve is a third-order Bezier curve, the control information of the segmentation curve is the position information of the first control point and the position information of the second control point of the third-order Bezier curve.

[0074] S45 . Divide the target image block into two sub-image blocks based on the segmentation curve of the target image block.

[0075] It should be noted that the segmentation curve is a continuous curve. The continuous segmentation curve may pass through the smallest unit (pixel point) of the digital image, but the smallest unit of the digital image cannot be segmented. Therefore, the segmentation curve cannot be directly applied to the segmentation of the digital image. Therefore, when the target image block is segmented into two sub-image blocks based on the segmentation curve of the target image block, it is necessary to sample the segmentation curve into a discrete curve, and then segment the target image block into two sub-image blocks through the sampled curve.

[0076] S46: Acquire code stream data corresponding to the target image block according to the control information of the segmentation curve and the two sub-image blocks.

[0077] The video encoding method provided by the embodiment of the present application first obtains a dividing line for dividing the target image block into two image areas according to a texture edge in the target image block when encoding a target image block obtained by block division of a video frame, and then obtains two end segments of the dividing curve according to the encoded video content to obtain a dividing starting point and a dividing end point, and then obtains the dividing curve and control information of the dividing curve according to the dividing starting point, the dividing end point and the dividing line, and divides the target image block into two sub-image blocks based on the dividing curve of the target image block, and finally obtains code stream data corresponding to the target image block according to the control information of the dividing curve and the two sub-image blocks. Since the segmentation starting point and segmentation end point of the segmentation curve for segmenting the target image block into two sub-image blocks in the embodiment of the present application are obtained according to the video content that has been encoded, the decoding end can also obtain the segmentation starting point and the segmentation end point according to the video content that has been encoded. Therefore, the encoding end only needs to add the control information of the segmentation curve to the bitstream data corresponding to the target image block, and the decoding end can obtain the complete segmentation curve information. Therefore, the embodiment of the present application can avoid adding the segmentation starting point and the segmentation end point of the segmentation curve to the bitstream data corresponding to the target image block, thereby reducing the bit rate overhead brought by representing the segmentation curve.

[0078] As an extension and refinement of the above embodiment, the present application embodiment provides another video encoding method, referring to Figure 5 As shown, the video encoding method includes the following steps S501 to S515:

[0079] S501: Obtain a target image block.

[0080] The target image block is a rectangular image block obtained by dividing the video frame into blocks.

[0081] S502: Segment the target image block into two image areas based on a texture edge in the target image block, and obtain a boundary line of the two image areas as a boundary line of the target image block.

[0082] In some embodiments, segmenting the target image block into two image regions based on a texture edge in the target image block comprises the following steps a to g:

[0083] Step a: converting the target image block into a grayscale image to obtain a grayscale image block corresponding to the target image block.

[0084] In some embodiments, the color space of the target image block is a YUV color space, and the pixel value of each pixel of the target image block is composed of a brightness signal Y representing a brightness value and two chrominance signals U and V representing color values. The implementation method of converting the target image block into a grayscale image may include: deleting the chrominance signals U and V in the pixel value of each pixel and retaining only the brightness signal Y, thereby converting the target image block into a grayscale image.

[0085] In some embodiments, the color space of the target image block is an RGB color space, and the implementation method of converting the target image block into a grayscale image may include: converting the color space of the target image block from the RGB color space to the YUV color space, and then deleting the chrominance signals U and V in the pixel value of each pixel point, and only retaining the brightness signal Y, thereby converting the target image block into a grayscale image.

[0086] Since the human eye is much more sensitive to brightness than chromaticity, brightness is often used as important information in image processing. Therefore, the embodiment of the present application first converts the target image block into a grayscale image and filters out the chromaticity signal in the target image block, thereby reducing the amount of data processing in the subsequent edge detection process.

[0087] Step b: filtering the grayscale image block based on a Gaussian kernel of a preset size.

[0088] In some embodiments, the Gaussian kernel of the preset size may be a 5*5 Gaussian kernel.

[0089] In some embodiments, the Gaussian kernel G σ The definition of can be shown as follows:

[0090]

[0091] Where σ is the standard deviation, x and y are the distance values ​​from the center of the Gaussian kernel in the X-axis direction and the Y-axis direction respectively.

[0092] The grayscale image block is represented as I, and the grayscale image block after filtering is represented as I σ , the Gaussian kernel of the preset size is represented by G σ , then the above step b can be expressed as the following formula (2):

[0093]

[0094] in, Represents a convolution operation.

[0095] It should be noted that in the embodiment of the present application, two-dimensional Gaussian kernels of other sizes may be selected to filter the grayscale image block, for example: a 7*7 Gaussian kernel is selected to filter the grayscale image block, and another example: a 3*3 Gaussian kernel is selected to filter the grayscale image block. The embodiment of the present application does not limit the size of the Gaussian kernel, and in actual use, the Gaussian kernel can be selected according to the resolution of the grayscale image block. That is, when the resolution of the grayscale image block is large, a larger Gaussian kernel can be selected, and when the resolution of the grayscale image block is small, a smaller Gaussian kernel can be selected.

[0096] Since the embodiment of the present application also performs filtering processing on the target image block through the above step b, the embodiment of the present application can filter out the noise in the target image block, thereby improving the accuracy of subsequent texture edge detection.

[0097] Step c: performing texture edge detection on the target image block to obtain at least one texture edge of the target image block.

[0098] In the embodiment of the present application, the texture edge of the target image block refers to an image region in the target image block where the brightness changes significantly.

[0099] In the embodiment of the present application, the edge detection algorithm used for texture edge detection of the target image block is not limited, and the edge detection algorithm used is subject to the ability to obtain the texture edge of the target image block. Exemplarily, in the embodiment of the present application, the texture edge detection of the target image block can be performed using edge detection algorithms such as Sobel, Prewitt, Roberts, Canny, and Marr-Hildreth.

[0100] The above step c is described below by using the Canny edge detection algorithm to perform texture edge detection on the grayscale image block. Using the Canny edge detection algorithm to perform texture edge detection on the grayscale image block includes the following steps c1 to c3:

[0101] Step c1, calculating the gradient value and gradient direction of the grayscale image block after filtering.

[0102] In the embodiment of the present application, the gradient value and gradient direction of the grayscale image block refer to the global gradient value and global gradient direction of the grayscale image block. The global gradient value includes the gradient value of each pixel in the grayscale image block, and the global gradient direction includes the gradient direction of each pixel in the grayscale image block.

[0103] Exemplarily, calculating the gradient value and gradient direction of the grayscale image block after filtering may include: calculating the gradient value and gradient direction of the grayscale image block after filtering based on a Sobel operator, a Roberts operator, a Prewitt operator, or a Laplacian operator.

[0104] Exemplarily, the above step c1 is described below by taking the calculation of the gradient value and gradient direction of the grayscale image block after filtering based on the Sobel operator as an example.

[0105] The templates of the Sobel operator in the X-axis direction and the Y-axis direction are shown in the following formulas (3) and (4) respectively:

[0106]

[0107]

[0108] If the gradient values ​​of the grayscale image block after filtering in the X-axis direction and the Y-axis direction are represented as G x and G y , then the calculation formulas for the gradient values ​​of the grayscale image block in the X-axis direction and the Y-axis direction after filtering can be respectively as follows:

[0109] G x =I σ *S x (5)

[0110] G y =I σ *S y (6)

[0111] The gradient value and gradient direction of the grayscale image block after filtering are calculated according to the gradient values ​​of the grayscale image block after filtering in the X-axis direction and the Y-axis direction and the following formulas (7) and (8):

[0112]

[0113]

[0114] Wherein, G is the gradient value of the grayscale image block after filtering, and θ is the gradient direction of the grayscale image block after filtering.

[0115] Step c2: performing non-maximum suppression on the gradient value of the grayscale image block after filtering according to the gradient value and gradient direction of the grayscale image block after filtering.

[0116] Although the global gradient value of the grayscale image block after filtering is obtained through the above step c2, the texture edge directly obtained based on the gradient value is too rough, so it is necessary to perform non-maximum suppression on the gradient value to refine the edge.

[0117] In some embodiments, the above-mentioned step c2 (performing non-maximum suppression on the gradient value of the grayscale image block after filtering according to the gradient value and gradient direction of the grayscale image block after filtering) includes: traversing the pixel points of the grayscale image block after filtering, and determining whether the gradient value of the current pixel point is the maximum value among the surrounding pixel points with similar gradient direction; if so, retaining the current pixel point as a candidate edge point, if not, deleting the current pixel point from the candidate edge points.

[0118] In some embodiments, the implementation method for determining whether the gradient value of the current pixel is the maximum value of the surrounding pixels with similar gradient directions may be: discretizing the gradient direction θ into eight directions of 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°, traversing each pixel belonging to the texture edge, obtaining the gradient value of the pixel closest to the corresponding first direction and the gradient value of the pixel closest to the corresponding second direction, if the gradient value of the current pixel is greater than the gradient value of the pixel in the first direction, and the gradient value of the current pixel is greater than the gradient value of the pixel in the second direction, then retain the current pixel as a pixel belonging to the texture edge, otherwise suppress the current pixel and remove it from the texture edge. Among them, the first direction corresponding to the pixel is the direction with the smallest angle with the gradient direction of the pixel among the eight directions, and the second direction corresponding to the pixel is the opposite direction of the first direction corresponding to the pixel.

[0119] For example, refer to Figure 6 As shown in the figure, the gradient direction of pixel a is θ, and the direction with the smallest angle with the gradient direction θ of pixel a among the eight directions obtained by discrete gradient direction is the 45° direction. The pixel closest to pixel a in the 45° direction is pixel b. The opposite direction of the 45° direction is the 225° direction. The pixel closest to pixel a in the 225° direction is pixel c. Therefore, the gradient value G of pixel b is obtained. b and the gradient value G of pixel c c , if the gradient value G of pixel a a Greater than the gradient value G of pixel b b , and the gradient value G of pixel a a Greater than the gradient value G of pixel c c , then pixel a is retained as a pixel belonging to the texture edge, and if the gradient value G of pixel a a Less than the gradient value G of pixel b b and / or the gradient value G of pixel a aLess than the gradient value G of pixel c c , then the pixel a is suppressed and removed from the texture edge.

[0120] Step c3: determining the texture edge of the target image block based on the first threshold and the second threshold.

[0121] The first threshold is smaller than the second threshold.

[0122] The above step c3 (determining the texture edge of the target image block based on the first threshold and the second threshold) includes: determining that the pixel point with a gradient value greater than the second threshold is a strong edge point, and the strong edge point belongs to the texture edge, determining that the pixel point with a gradient value less than the first threshold does not belong to the texture edge, determining that the pixel point with a gradient value between the second threshold and the first threshold belongs to a weak edge point, and further determining whether the weak edge point belongs to the texture edge based on whether the weak edge point is connected to the strong edge. Exemplarily, along the contour line of the strong edge until the end point s(x,y) of the strong edge, it can be determined whether there is a weak edge point in the 8-neighborhood of the end point s(x,y), if so, the weak edge point is classified as a texture edge, recorded as r(x,y), and the above operation is repeated with r(x,y) as the starting point, so as to obtain the texture edge of the current image block.

[0123] Step d: performing image region segmentation on the target image block based on the at least one texture edge to obtain an image region set corresponding to the target image block.

[0124] For example, refer to Figure 7 As shown, the texture edge of the target image block includes: texture edge 71 and texture edge 72. Therefore, the image area set corresponding to the target image block obtained based on the texture edge of the target image block includes a total of 3 image areas, and the 3 image areas are image area A, image area B and image area C.

[0125] Step e: obtaining the number of image regions in the image region set.

[0126] That is, the number of image regions obtained by dividing the image region of the target image block by the texture edge of the target image block (the at least one texture edge) is determined.

[0127] As above Figure 7In the example shown, the target image block is likely to have more than one texture edge, which means that the target image block will be divided into more than 2 image areas. Taking into account the bit rate cost, too many sub-image areas cannot bring matching performance improvements. Therefore, when the number of image areas is greater than 2, the embodiment of the present application will first merge the image areas into two image areas. If the number of image areas is equal to 2, no merging process is required. That is, if the number of image areas in the image area set is less than 2, the target image block is directly encoded as a whole. If the number of image areas in the image area set is equal to 2, the boundary line of the two image areas is directly obtained as the boundary line of the target image block. If the number of image areas in the image area set is greater than 2, the following step f is performed, and then the boundary line of the two image areas is obtained as the boundary line of the target image block.

[0128] Step f: merging the image regions in the image region set into two image regions, so as to divide the target image block into two image regions.

[0129] In some embodiments, the above step f (merging the image regions in the image region set into two image regions to divide the target image block into two image regions) includes the following steps f1 to f3:

[0130] Step f1, obtaining at least two merging schemes of the image region set.

[0131] The at least two merging schemes include various schemes for merging the image regions in the image region set into two image regions.

[0132] When obtaining the merging solution of the image region set, it may be based on the connectivity of each image region in the image region set. If two image regions have a common boundary, the two image regions may be merged into one image region.

[0133] For example, refer to Figure 8 As shown, the image region set includes: image region A, image region B and image region C, so there are 3 merging schemes. Merging scheme 1 is: merging image region A and image region B to obtain image region D, thereby merging the image regions in the image region set into two image regions, image region D and image region C; merging scheme 2 is: merging image region A and image region C to obtain image region E, thereby merging the image regions in the image region set into two image regions, image region E and image region B; merging scheme 3 is: merging image region B and image region C to obtain image region F, thereby merging the image regions in the image region set into two image regions, image region F and image region A.

[0134] Step f2: respectively obtain the difference between the at least two merging schemes.

[0135] Among them, the difference of any merging scheme is the absolute difference between the first similarity and the second similarity of the merging scheme; the first similarity and the second similarity of any merging scheme are respectively the sum of the absolute differences in the grayscale values ​​of each pixel point of the two image areas and the corresponding co-located blocks under the merging scheme.

[0136] That is, the difference of the merging solutions can be obtained through the following steps f21 to f23:

[0137] Step f21, respectively obtain the co-located blocks corresponding to the two image regions under the merging scheme.

[0138] Continuing with the above example, when obtaining the difference of merging scheme 1, the same-position blocks of image area D and image area C are obtained respectively; when obtaining the difference of merging scheme 2, the same-position blocks of image area E and image area B are obtained respectively; when obtaining the difference of merging scheme 3, the same-position blocks of image area F and image area A are obtained respectively.

[0139] There is a strong temporal correlation between video frames that are close in time domain, so the adjacent video coded frames can be used to estimate the current video frame, and the adjacent coded video frames of the current video frame are different in different coding structures. Coding structures include: All Intra (AI), Low Delay (LD), Low Latency (LP), and Random Access (RA). The co-located block corresponding to any image area refers to the image area in the adjacent coded video frame of the current video frame that has the same position as the image area.

[0140] Step f22: respectively calculate the sum of the absolute differences of the grayscale values ​​of each pixel of the two image regions under the merging scheme and the corresponding co-located blocks to obtain the first similarity and the second similarity of the merging scheme.

[0141] The first similarity of the merge scheme j is expressed as Δ j1 , then the first similarity Δ of the merge scheme j can be calculated by the following formula (9): j1 :

[0142]

[0143] Wherein, n1 is the number of pixels in the first image region under the merging scheme j, and n1 is also the number of pixels in the same-position block corresponding to the first image region under the merging scheme j; is the brightness value of pixel i in the first image area; is the brightness value of pixel i in the same-position block corresponding to the first image area.

[0144] The second similarity of the merge scheme j is expressed as Δ j2 , then the second similarity Δ of the merge scheme j can be calculated by the following formula (10): j2 :

[0145]

[0146] Wherein, n2 is the number of pixels in the second image area under the merging scheme j, and n2 is also the number of pixels in the same-position block corresponding to the second image area under the merging scheme j; is the brightness value of pixel i in the second image area; is the brightness value of pixel i in the same-position block corresponding to the second image area.

[0147] Step f23: Calculate the absolute difference between the first similarity and the second similarity of the merging scheme to obtain the difference of the merging scheme.

[0148] The difference of the merge scheme j is expressed as D j , then the difference D of the merge scheme j can be calculated by the following formula (11): j :

[0149] D j =|Δ j1 -Δ j2 | (11)

[0150] Step f3: merging the image regions in the image region set into two image regions by using a merging scheme with the largest difference among the at least two merging schemes.

[0151] As shown in the above example, the differences of merging scheme 1, merging scheme 2, and merging scheme 3 are D1, D2, and D3 respectively. If D1>D2 and D1>D3, the image areas in the image area set are merged into two image areas, image area D and image area C, through merging scheme 1. If D2>D1 and D2>D3, the image areas in the image area set are merged into two image areas, image area B and image area E, through merging scheme 2. If D3>D1 and D3>D2, the image areas in the image area set are merged into two image areas, image area A and image area F, through merging scheme 3.

[0152] Because each merging scheme merges the image regions in the image region set into two image regions, the difference D of the merging scheme jThe larger the value, the greater the difference in grayscale values ​​of the two image partitions obtained by the merging scheme, and the greater the possibility that the two image partitions obtained by the merging scheme belong to different image content objects. Therefore, by merging the image areas in the image area set into two image areas through the merging scheme with the largest difference among the at least two merging schemes, the target image block can be divided into two image areas more reasonably.

[0153] It should be noted that the purpose of step f is to find and construct an optimal edge line among multiple edges. In the above implementation, each image area of ​​the target image block is merged to achieve this purpose, but the embodiment of the present application is not limited to this. Other solutions that can solve the problem of selecting multiple edge lines can also be used as variant solutions, such as: using a path-finding algorithm to find the longest path. For example, the Astar image search algorithm can be used to find the optimal dividing line to obtain a discrete curve whose starting point and end point are both on the boundaries of the coding block.

[0154] S503: Determine whether spatially adjacent pixel points at four edges of the target image block can be obtained in the encoded video content.

[0155] Since there are no spatially adjacent pixels on one or more edges of the target image block when the target image block is located at the edge of the current video frame, the premise for obtaining the spatially adjacent pixels on the four edges of the target image block in the encoded video content is that the target image block is not located at the edge of the current video frame and the image content around the target image block has completed video encoding.

[0156] For example, see Fig. 9 As shown, Fig. 9 Taking the target image block 90 with a resolution of M*N as an example, it is determined whether the spatially adjacent pixel points at the four edges of the target image block 90 can be obtained in the encoded video content, that is, whether the M pixel points 91 directly above the target image block, the M pixel points 92 directly below the target image block, the N pixel points 93 directly to the left of the target image block, and the N pixel points 94 directly to the right of the target image block can be obtained in the encoded video content.

[0157] In the above step S503, if the spatially adjacent pixel points of the four edges of the target image block can be obtained in the encoded video content, the following step S504 is performed:

[0158] S504: Acquire a segmentation start point and a segmentation end point of a segmentation curve according to spatially adjacent pixel points at four edges of the target image block.

[0159] In some embodiments, the above step S504 (obtaining the segmentation start point and segmentation end point of the segmentation curve according to the spatially adjacent pixel points of the four edges of the target image block) includes the following steps 1 to 3:

[0160] Step 1: Obtain the gradient values ​​of spatially adjacent pixel points at four edges of the target image block in the direction of the corresponding edges.

[0161] Due to the consistency of pixel intervals and the target image block being a rectangular image block, the directions of the four edges of the target image block are horizontal or vertical, so obtaining the gradient values ​​of spatially adjacent pixel points at the four edges of the target image block in the direction of the corresponding edges can be simplified to the difference in pixel grayscale values ​​in the horizontal or vertical direction.

[0162] Step 2: Obtain at least two pixel value step points according to the gradient values ​​of spatially adjacent pixel points at four edges of the target image block in the direction of the corresponding edges.

[0163] The pixel value step point is a pixel point having the largest gradient value and a gradient value greater than a threshold gradient value among spatially adjacent pixel points at any edge of the target image block.

[0164] In some embodiments, according to the gradient values ​​of the spatially adjacent pixels of the four edges of the target image block in the direction of the corresponding edge, an implementation method for obtaining at least two pixel value step points may include: traversing the spatially adjacent pixels of the four edges of the target image block respectively, and determining whether the gradient value of the current pixel is greater than the current maximum gradient value; if so, updating the current maximum gradient value to the gradient value of the current pixel; if not, determining whether the gradient value of the next pixel is greater than the current maximum gradient value; after completing the traversal of the spatially adjacent pixels of the edge, determining whether the maximum gradient value is greater than the threshold gradient value; if so, determining the pixel corresponding to the maximum gradient value as a pixel value step point; if not, determining that no pixel value step point exists in the spatially adjacent pixels of the current edge.

[0165] Since the target image block has four edges, there is one pixel value step point or no pixel value step point in the spatial domain adjacent pixel points on each edge, so according to the gradient value of the spatial domain adjacent pixel points of the four edges of the target image block in the direction of the corresponding edge, at most four pixel value step points can be obtained, and at least 0 pixel value step points can be obtained. When the number of pixel value step points obtained is 0 or 1, it means that there is no obvious boundary inside the target image block, and continuing to divide the special-shaped coding block will not bring significant compression performance improvement. The encoder is also very likely not to select this mode when selecting the coding mode later, but affects the coding efficiency. Therefore, when the number of pixel value step points is 0 or 1 for the coding block, the special-shaped coding block mode is directly terminated in advance, when the number of pixel value step points is 2, the two pixel value step points are directly determined as the segmentation starting point and segmentation end point of the segmentation curve, and when the number of pixel value step points obtained is 3 or 4, it is necessary to further perform the following step 3 to determine the segmentation starting point and segmentation end point of the segmentation curve.

[0166] Step 3: respectively determine the two pixel value step points with the largest gradient values ​​among the at least two pixel value step points as the segmentation starting point and the segmentation end point.

[0167] For example, refer to Fig.10 As shown, Fig.10 In the example, the above step 3 is described by taking the case where four pixel value step points are obtained according to the gradient values ​​of the spatially adjacent pixels of the four edges of the target image block in the direction of the corresponding edges. Fig.10 As shown, the four pixel value step points are pixel point 101 among the spatially adjacent pixels at the upper edge, pixel point 102 among the spatially adjacent pixels at the lower edge, pixel point 103 among the spatially adjacent pixels at the left edge, and pixel point 103 among the spatially adjacent pixels at the right edge. The gradient value of pixel point 101 is e1, the gradient value of pixel point 102 is e2, the gradient value of pixel point 103 is e3, and the gradient value of pixel point 104 is e4. Then, the two pixel value step points with the largest values ​​are selected from e1, e2, e3, and e4, and the two selected pixel value step points are determined as the segmentation starting point and the segmentation end point.

[0168] In the above step S503, if the spatially adjacent pixel points of the four edges of the target image block cannot be obtained in the encoded video content, the following step S505 is performed:

[0169] S505 , obtaining a best matching block of the target image block in the encoded video content, and obtaining the segmentation start point and the segmentation end point according to edge pixel points of the best matching block of the target image block.

[0170] In some embodiments, the above step S505 (obtaining the best matching block of the target image block in the encoded video content, and obtaining the segmentation start point and the segmentation end point according to the edge pixel points of the best matching block of the target image block) includes the following steps ① to ④:

[0171] Step 1: Obtain the gradient values ​​of the pixels on the four edges of the best matching block of the target image block in the direction of the corresponding edges.

[0172] In some embodiments, the implementation of obtaining the best matching block of the target image block may include: performing motion estimation on the current video frame in a reference video frame of the current video frame to obtain the best matching block of the target image block.

[0173] The reference video frame of the current video frame is different in different coding structures. Coding structures include: All Intra (AI), Low Delay (LD), Low Latency (LP), and Random Access (RA).

[0174] The commonly used method for motion estimation is the block matching method, which focuses on the block matching criteria and search methods. There are currently three commonly used matching criteria, Sum of Absolute Difference (SAD), Mean Squared Error (MSE), and Normalized Cross Correlation Function (NCCF). Among them, the SAD criterion has the advantages of no multiplication operation and simple and convenient implementation, and is the most commonly used. The calculation formula of SAD is as follows (12):

[0175]

[0176] Where M×N is the resolution of the target image block, f(x,y) is the pixel value at (x,y) of the target image block, and g(x,y) is the reconstructed pixel value at (x,y).

[0177] Search methods include full search, cross search, two-dimensional logarithmic search and other fast search algorithms. The full search algorithm is extremely complex and cannot meet the requirements of real-time coding. The TZ search algorithm is a commonly used fast search algorithm. Compared with the full search algorithm, the performance is slightly reduced, while the search time is greatly reduced.

[0178] The motion estimation in the embodiment of the present application can use the TZ search fast algorithm to perform a diamond search, select SAD as the matching criterion, and finally obtain an optimal integer pixel motion vector.

[0179] Step ②, obtaining at least two pixel value step points according to the gradient values ​​of the pixel points on the four edges of the best matching block of the target image block in the direction of the corresponding edges.

[0180] The pixel value step point is a pixel point having the largest gradient value and a gradient value greater than a threshold gradient value among the pixel points on any edge of the best matching block of the target image block.

[0181] Step ③, obtaining two pixel value step points with the largest gradient values ​​among the at least two pixel value step points.

[0182] The implementation of step ② and step ③ is similar to the implementation of step 2 and step 3 above, and will not be repeated here to avoid redundancy.

[0183] Step ④, determining the segmentation starting point and the segmentation end point according to the two pixel value step points with the largest gradient values ​​and the motion vector of the best matching block of the target image block respectively.

[0184] Since the two pixel value step points with the largest gradient values ​​in step ④ are obtained on the best matching block, their position information needs to be matched to the target image coding block before being set as the segmentation starting point and segmentation end point of the segmentation curve. In some embodiments, the segmentation starting point and the segmentation end point are determined in the target image block according to the two pixel value step points with the largest gradient values ​​and the motion vector of the best matching block, respectively, including: subtracting the position coordinates of the two pixel value step points with the largest gradient values ​​from the motion vector of the best matching block in the horizontal and vertical directions, respectively, to obtain the position coordinates of the segmentation starting point and the segmentation end point.

[0185] S506 , performing third-order Bezier curve fitting on the boundary line with the segmentation start point and the segmentation end point as two end points of the third-order Bezier curve to obtain a first control point and a second control point.

[0186] The third-order Bezier curve is a curve defined by a starting point s, an end point e, a first control point p1, and a second control point p2. Therefore, the dividing line is fitted with a third-order Bezier curve using the segmentation starting point and the segmentation end point as the two endpoints of the third-order Bezier curve, that is, the segmentation starting point is used as the starting point s of the third-order Bezier curve, and the segmentation end point is used as the end point e of the third-order Bezier curve to obtain the first control point p1 and the second control point p2.

[0187] The formula of the third-order Bezier curve is shown in the following formula (13):

[0188] B(t)=(1-t) 3 ·s+3·(1-t)2 ·t·p1+3·(1-t)·t 2 p2+t 3 ·e (13)

[0189] Among them, t∈[0,1] is the endpoint interpolation variable of the Bezier curve function.

[0190] The boundary line is fitted with a third-order Bezier curve, that is, the coordinates of a set of points on the boundary line (x0, y0),…, (x m ,y m ) is substituted into the following formula (14), and the coordinates of the first control point p1 and the second control point p2 are solved according to the coefficients of the Bezier curve obtained when the value of formula (14) is minimum:

[0191]

[0192] Where t1 = (1-t) 3 , t2=3(1-t) 2 t, t3=3(1-t)t 2 , t4=t 3 ,s x is the x coordinate of the starting point of the dividing line, P 1x is the x coordinate of the first control point, P 2x is the x coordinate of the second control point, e x is the x-coordinate of the end point of the dividing line, s y is the y coordinate of the starting point of the dividing line, P 1y is the y coordinate of the first control point, P 2y is the y coordinate of the second control point, e y is the y coordinate of the end point of the dividing line, X i and Y i are the x-coordinate and y-coordinate of the i-th point on the dividing line, the t corresponding to the i-th point is (i-1) / (N-1), and N is the number of points extracted from the dividing line.

[0193] For P in the above formula (14), 1x , P 1y , P 2x , P 2y Derivatives are taken separately, and the partial derivatives are set to 0. The following matrix can be obtained after sorting:

[0194]

[0195] Solving the above matrix, we can obtain P 1x , P 1y , P 2x , P 2y The value of .

[0196] In addition, in the above-mentioned embodiments, the coordinate values ​​of the control points of the Bezier curve are solved by means of a matrix, but in other embodiments, the coordinate values ​​of the control points of the Bezier curve can also be solved by other methods, for example: the coordinate values ​​of the control points of the Bezier curve can also be solved by fitting method, numerical integration method, derivative solution method, etc. The embodiments of the present application do not limit this, and the coordinate values ​​of the control points of the Bezier curve can be solved.

[0197] For example, refer to Fig.11 As shown, Fig.11 The dividing line is Fig.11 As shown by line 111 in FIG. 1 , a third-order Bezier curve is obtained by fitting the boundary line with a third-order Bezier curve using the segmentation start point and the segmentation end point as two end points of the third-order Bezier curve. Fig.11 The line 112 in FIG. Fig.11 As shown, the starting points of the fitted third-order Bezier curves are all the segmentation starting points s, the starting points of the fitted third-order Bezier curves are all the segmentation end points e, and the first control points and the control points of the fitted third-order Bezier curves are p1 and p2 respectively.

[0198] S507: Determine whether the first control point and the second control point are located on the same side of a straight line connecting the segmentation start point and the segmentation end point.

[0199] If the first control point and the second control point of the fitted third-order Bezier curve are on the same side of the straight line connecting the segmentation starting point and the segmentation end point, it means that the segmentation curve is a single arch, so the second-order Bezier curve is selected for fitting to further reduce the bit rate overhead caused by representing the segmentation curve. If the first control point and the second control point of the fitted third-order Bezier curve are on both sides of the straight line connecting the segmentation starting point and the segmentation end point, it means that the segmentation curve is a double arch, and the fitting accuracy of the second-order Bezier curve is poor, so the third-order Bezier curve is selected for fitting to improve the fitting accuracy of the segmentation curve. That is, in the above step S507, if it is determined that the first control point and the second control point are not on the same side of the straight line connecting the segmentation starting point and the segmentation end point, the following step S508 is executed:

[0200] S508: Acquire the segmentation curve according to the segmentation starting point, the segmentation end point, the first control point, and the second control point.

[0201] In some embodiments, the above step S508 (obtaining the segmentation curve according to the segmentation starting point, the segmentation end point, the first control point and the second control point) includes the following steps 5081 to 5083:

[0202] Step 5081: Determine a first pixel point and a second pixel point according to the segmentation starting point, the segmentation end point, the first control point, and the second control point.

[0203] Among them, the first pixel point is a pixel point located on a one-third perpendicular line of a line segment with the segmentation starting point and the segmentation end point as endpoints and is closest to the first control point, and the second pixel point is a pixel point located on a two-thirds perpendicular line of a line segment with the segmentation starting point and the segmentation end point as endpoints and is closest to the second control point.

[0204] For example, refer to Fig.12 As shown, the line segment with the segmentation starting point s and the segmentation end point e as endpoints is line segment 121, the one-third perpendicular line of the line segment 121 is straight line 122, the pixel point located on straight line 122 and closest to the first control point p1 is pixel point a, and the two-thirds perpendicular line of the line segment 121 is straight line 123, the pixel point located on straight line 123 and closest to the second control point p2 is pixel point b. Therefore, the first pixel point a and the second pixel point b can be determined according to the segmentation starting point s, the segmentation end point e, the first control point p1 and the second control point p2.

[0205] Step 5082: quantize the position coordinates of the first pixel point and the position coordinates of the second pixel point respectively to obtain a first quantization control point and a second quantization control point.

[0206] In some embodiments, the quantization step size when quantizing the position coordinates of the first pixel point and the position coordinates of the second pixel point can be set according to the resolution of the target image block. For example, if the resolution of the target image block is M*N, the x value and y value of the position coordinate can be set to M and N respectively.

[0207] Quantizing the position coordinates of the first pixel point and the position coordinates of the second pixel point can reduce the bit rate overhead caused by the control information representing the segmentation curve.

[0208] Step 5083: Generate the segmentation curve according to the segmentation starting point, the segmentation end point, the first quantization control point, the second quantization control point and the third-order Bezier curve formula.

[0209] In the above step S507, if it is determined that the first control point and the second control point are located on the same side of the straight line connecting the segmentation starting point and the segmentation end point, the following step S509 is performed:

[0210] S509: Taking the segmentation start point and the segmentation end point as two endpoints of a second-order Bezier curve, perform second-order Bezier curve fitting on the boundary line to obtain control points.

[0211] The second-order Bezier curve is a curve defined by a starting point s, an end point e and a control point p. Therefore, the segmentation starting point and the segmentation end point are used as the two endpoints of the second-order Bezier curve, and the second-order Bezier curve is fitted to the dividing line, that is, the segmentation starting point is used as the starting point s of the second-order Bezier curve, and the segmentation end point is used as the end point e of the second-order Bezier curve to obtain the control point p.

[0212] The formula of the second-order Bezier curve is shown in the following formula (15):

[0213] B(t)=(1-t) 2 ·s+2·(1-t)·t·p+t 2 ·e (15)

[0214] Among them, t∈[0,1] is the endpoint interpolation variable of the Bezier curve function.

[0215] The dividing line is fitted with a second-order Bezier curve, that is, the coordinates of a set of points on the dividing line (x0, y0),…, (x m ,y m ) is substituted into the following formula (16), and the coordinates of the control point p are solved according to the coefficients of the Bezier curve obtained when the value of formula (16) is minimum:

[0216]

[0217] Among them, B ix and B iy is the x-coordinate and y-coordinate of the i-th point after fitting, X i and Y i are the x- and y-coordinates of the ith point on the dividing line.

[0218] For example, refer to Fig.13 As shown, Fig.13 The dividing line is Fig.13 The second-order Bezier curve obtained by fitting the line 131 in FIG. 1 can be shown as follows: Fig.13 The line 132 in FIG. Fig.13 As shown, the starting point of the second-order Bezier curve obtained by fitting is the segmentation starting point s, the end point of the second-order Bezier curve obtained by fitting is the segmentation end point e, the second-order Bezier curve obtained by fitting is a single arch curve, and the shape of the first fitting result is controlled by the control point p.

[0219] S510: Acquire the segmentation curve according to the segmentation starting point, the segmentation end point, and the control point.

[0220] In some embodiments, the above step S510 (obtaining the segmentation curve according to the segmentation starting point, the segmentation end point and the control point) includes the following steps 5101 to 5103:

[0221] Step 5101: determine the target pixel point according to the segmentation starting point, the segmentation end point and the control point.

[0222] The target pixel point is a pixel point located on the perpendicular bisector of a line segment with the segmentation start point and the segmentation end point as endpoints and which is closest to the control point.

[0223] For example, refer to Fig.14 As shown, the line segment with the segmentation starting point s and the segmentation end point e as endpoints is line segment 141, the perpendicular bisector of line segment 141 is straight line 142, and the pixel point located on straight line 142 and closest to the control point p is pixel point c. Therefore, the target pixel point c can be determined based on the segmentation starting point s, the segmentation end point e and the control point p.

[0224] Step 5102: quantize the position coordinates of the target pixel point to obtain a quantized control point.

[0225] Likewise, the quantization step size when quantizing the position coordinates of the target pixel point may be set according to the resolution of the target image block.

[0226] Step 5103: Generate the segmentation curve according to the segmentation starting point, the segmentation end point, the quantization control point and the second-order Bezier curve formula.

[0227] After obtaining the segmentation curve through the above step S508 or S510, the embodiment of the present application continues to perform the following step S511:

[0228] S511 . Sampling the segmentation curve of the target image block with integer pixel accuracy to obtain a sampling result of the segmentation curve.

[0229] Since the third-order Bezier curve generated in the above step S508 and the second Bezier curve generated in the step S510 are both continuous curves and cannot be directly applied to the division of digital images, it is necessary to sample with integer pixel accuracy to obtain the sampling results of discrete segmentation curves, and then segment the target image blocks.

[0230] In some embodiments, sampling the segmentation curve of the target image block with integer pixel accuracy may include: determining a boundary of pixels through which the segmentation curve passes as a sampling result of the segmentation curve.

[0231] For example, refer to Fig.15 As shown, the segmentation curve is Fig.15The sampling result of the segmentation curve obtained by sampling the segmentation curve of the target image block with integer pixel accuracy can be as follows: Fig.15 As shown by line 152 in FIG. Fig.15 As shown, the sampling results of the segmentation curve are all located on the boundary of the pixel.

[0232] S512: Divide the target image block into two sub-image blocks according to the sampling result of the segmentation curve.

[0233] For example, Fig.16 As shown, in Fig.15 Based on the sampling result 152 of the segmentation curve, the target image block can be segmented into two sub-image blocks, a sub-image block 161 and a sub-image block 162 .

[0234] S513: Encode the control information of the segmentation curve of the target image block to obtain first code stream data.

[0235] When the segmentation curve is a third-order Bezier curve obtained through step S508, the control information of the segmentation curve of the target image block is encoded to obtain the first bitstream data, including: entropy encoding the position information of the first control point of the third-order Bezier curve (the position information of the first quantization control point) and the position information of the second control point of the third-order Bezier curve (the position information of the second quantization control point) to obtain the first bitstream data.

[0236] When the segmentation curve is a second-order Bezier curve obtained through step S510, the control information of the segmentation curve of the target image block is encoded to obtain the first bitstream data, including: entropy encoding the position information of the control points of the second-order Bezier curve (the position information of the quantized control points) to obtain the first bitstream data.

[0237] S514: Encode the two sub-image blocks respectively to obtain second code stream data and third code stream data.

[0238] In some embodiments, encoding a sub-image block to obtain code stream data of the sub-image block may include: first, performing motion estimation on the sub-image block to obtain a reference block corresponding to the sub-image block, then performing motion compensation on the reference block to obtain a prediction block and motion vector corresponding to each coding block, and calculating the residual between the prediction block and the sub-image block to obtain a prediction residual, and finally transforming and quantizing the prediction residual to obtain a transform coefficient, and entropy encoding the motion vector and transform coefficient to obtain the code stream data of the sub-image block.

[0239] S515: Generate code stream data corresponding to the target image block according to the first code stream data, the second code stream data, and the third code stream data.

[0240] In some embodiments, generating the code stream data corresponding to the target image block according to the first code stream data, the second code stream data and the third code stream data includes: encapsulating the first code stream data, the second code stream data and the third code stream data to obtain the code stream data corresponding to the target image block.

[0241] The present application also provides a video decoding method, referring to Fig.17 As shown, the video decoding method includes the following steps S171 to S177:

[0242] S171. Obtain code stream data corresponding to the target image block.

[0243] The bitstream data includes: first bitstream data, second bitstream data and third bitstream data; the first bitstream data is bitstream data obtained by encoding control information of a segmentation curve, the segmentation curve is a curve for dividing the target image block into two sub-image blocks, and the second bitstream data and the third bitstream data are bitstream data obtained by encoding the two sub-image blocks respectively.

[0244] In some embodiments, obtaining code stream data corresponding to the target image block includes: receiving video stream data of a to-be-played video sent by a media resource server, and extracting code stream data corresponding to the target image block from the video stream data.

[0245] S172: Acquire control information of the segmentation curve according to the first bitstream data.

[0246] In some embodiments, obtaining the control information of the segmentation curve according to the first bitstream data includes: performing entropy decoding on the first bitstream data to obtain position information of a first control point and position information of a second control point of a third-order Bezier curve.

[0247] In some other embodiments, acquiring the control information of the segmentation curve according to the first bitstream data includes: performing entropy decoding on the first bitstream data to acquire position information of control points of the second-order Bezier curve.

[0248] S173. Obtain two endpoints of the segmentation curve according to the decoded video content to obtain a segmentation start point and a segmentation end point.

[0249] In some embodiments, the above step S173 can be implemented through steps S503 to S505 in the above embodiments, which will not be described in detail here to avoid redundancy.

[0250] It should be noted that since the decoding end and the encoding end obtain the segmentation start point and segmentation end point in the same implementation manner, the decoding end can obtain the segmentation start point and segmentation end point consistent with the encoding end, thereby ensuring the consistency between the encoding end and the decoding end.

[0251] S174: Acquire the segmentation curve according to the control information of the segmentation curve, the segmentation starting point, and the segmentation end point.

[0252] When the control information of the segmentation curve includes position information of two control points, the step S174 includes: generating the segmentation curve according to the position information of the first control point, the position information of the second control point, the segmentation start point, the segmentation end point and a third-order Bezier curve.

[0253] When the control information of the segmentation curve includes only the position information of one control point, the step S174 includes: generating the segmentation curve according to the position information of the control point, the segmentation start point, the segmentation end point and a second-order Bezier curve.

[0254] S175 . Segment the target image block based on the segmentation curve to determine the two sub-image blocks.

[0255] In some embodiments, the above step S175 (segmenting the target image block based on the segmentation curve to obtain the two sub-image blocks) includes the following steps 1751 and 1752:

[0256] Step 1751: Sample the segmentation curve of the target image block with integer pixel accuracy to obtain a sampling result of the segmentation curve.

[0257] Step 1752: divide the target image block into two sub-image blocks according to the sampling result of the segmentation curve.

[0258] The implementation of step 1751 and step 1752 may refer to the implementation of steps S511 and S512 in the above embodiment, and will not be repeated here to avoid redundancy.

[0259] It should be noted that what is obtained in step S175 is only the shape, size and position of the two sub-image blocks, but does not include the image content of the two sub-image blocks. The image content of the two sub-image blocks needs to be reconstructed through subsequent steps.

[0260] S176: Reconstruct the two sub-image blocks according to the second code stream data and the third code stream data.

[0261] In some embodiments, the second code stream data and the third code stream data respectively include prediction residuals and motion vectors of the two sub-image blocks, and reconstructing the two sub-image blocks according to the second code stream data and the third code stream data includes: respectively determining reference blocks corresponding to the two sub-image blocks, respectively performing motion compensation on the reference blocks of the two sub-image blocks according to the motion vectors of the two sub-image blocks, so as to respectively obtain prediction blocks corresponding to the to-be-reconstructed blocks of the two sub-image blocks; and respectively adding and fusing the prediction residuals of the two sub-image blocks with the prediction blocks, so as to obtain the reconstructed two sub-image blocks.

[0262] S177: Obtain the reconstructed target image block according to the two reconstructed sub-image blocks.

[0263] That is, the two reconstructed sub-image blocks are spliced ​​into the reconstructed target image block.

[0264] After obtaining the bitstream data corresponding to the target image block, the video decoding method provided in the embodiment of the present application includes the first bitstream data obtained by encoding the control information of the segmentation curve used to segment the target image block into two sub-image blocks in the bitstream data corresponding to the target image block. Therefore, the embodiment of the present application can obtain the control information of the segmentation curve according to the first bitstream data, obtain the segmentation start point and the segmentation end point according to the decoded video content, and obtain the segmentation curve according to the control information of the segmentation curve, the segmentation start point and the segmentation end point, and segment the target image block based on the segmentation curve to obtain the two sub-image blocks. Because the bitstream data corresponding to the target image block also includes the second bitstream data and the third bitstream data respectively obtained by encoding the two sub-image blocks, the embodiment of the present application can also reconstruct the two sub-image blocks according to the second bitstream data and the third bitstream data, and obtain the reconstructed target image block according to the reconstructed two sub-image blocks. Since the embodiment of the present application does not need to add the endpoint information of the segmentation curve in the bitstream data, the embodiment of the present application can realize normal decoding of the video while reducing the bitrate overhead caused by representing the segmentation curve.

[0265] In some embodiments, some embodiments of the present application provide a video encoding device, the video encoding device comprising:

[0266] a memory configured to store a computer program;

[0267] The processor is configured to enable the video encoding device to implement the video encoding method described in any of the above embodiments when calling the computer program.

[0268] In some embodiments, some embodiments of the present application provide a video decoding device, the video decoding device comprising:

[0269] a memory configured to store a computer program;

[0270] The processor is configured to enable the video decoding device to implement the video decoding method described in any of the above embodiments when calling the computer program.

[0271] In some embodiments, some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the video encoding method described in any of the above embodiments or the video decoding method described in any of the above embodiments.

[0272] In some embodiments, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the video encoding method described in any of the above embodiments or the video decoding method described in any of the above embodiments.

[0273] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0274] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A video encoding method, characterized in that: include: Obtaining a target image block, wherein the target image block is a rectangular image block obtained by dividing the video frame into blocks; Acquire a boundary line of the target image block according to a texture edge in the target image block, wherein the boundary line is used to divide the target image block into two image areas; Obtain two endpoints of the segmentation curve according to the encoded video content to obtain the segmentation start point and segmentation end point; Acquire the segmentation curve and control information of the segmentation curve according to the segmentation starting point, the segmentation end point and the dividing line; Splitting the target image block into two sub-image blocks based on a segmentation curve of the target image block; According to the control information of the segmentation curve and the two sub-image blocks, the code stream data corresponding to the target image block is obtained.

2. The method according to claim 1, characterized in that The step of obtaining two endpoints of the segmentation curve according to the encoded video content to obtain a segmentation start point and a segmentation end point includes: Determine whether spatially adjacent pixel points of four edges of the target image block can be obtained in the encoded video content; If yes, obtaining the segmentation start point and the segmentation end point according to the spatial adjacent pixel points of the four edges of the target image block; If not, then the best matching block of the target image block is obtained in the encoded video content, and the segmentation start point and the segmentation end point are obtained according to the edge pixel points of the best matching block of the target image block.

3. The method according to claim 2, characterized in that The acquiring the segmentation starting point and the segmentation end point according to the spatially adjacent pixel points of four edges of the target image block comprises: Obtaining gradient values ​​of spatially adjacent pixel points at four edges of the target image block in the direction of the corresponding edges; According to the gradient values ​​of the spatially adjacent pixel points at the four edges of the target image block in the direction of the corresponding edge, obtaining at least two pixel value step points, wherein the pixel value step points are pixel points having the largest gradient value and a gradient value greater than a threshold gradient value among the spatially adjacent pixel points at any edge of the target image block; Two pixel value step points with the largest gradient values ​​among the at least two pixel value step points are respectively determined as the segmentation starting point and the segmentation end point.

4. The method according to claim 2, characterized in that: The acquiring the segmentation starting point and the segmentation end point according to the edge pixel points of the best matching block of the target image block comprises: Obtaining gradient values ​​of pixel points on four edges of the best matching block of the target image block in the direction of the corresponding edges; According to the gradient values ​​of the pixel points on the four edges of the best matching block of the target image block in the direction of the corresponding edges, at least two pixel value step points are obtained, wherein the pixel value step points are pixel points with the largest gradient value and a gradient value greater than a threshold gradient value among the pixel points on any edge of the best matching block of the target image block; Obtaining two pixel value step points with the largest gradient values ​​among the at least two pixel value step points; The segmentation start point and the segmentation end point are determined respectively according to the two pixel value step points with the largest gradient values ​​and the motion vector of the best matching block of the target image block.

5. The method according to claim 1, characterized in that The acquiring the segmentation curve according to the segmentation starting point, the segmentation end point and the dividing line comprises: Performing a third-order Bezier curve fitting on the boundary line with the segmentation starting point and the segmentation end point as two endpoints of the third-order Bezier curve to obtain a first control point and a second control point; Determine whether the first control point and the second control point are located on the same side of a straight line connecting the segmentation starting point and the segmentation end point; If not, acquiring the segmentation curve according to the segmentation starting point, the segmentation end point, the first control point and the second control point; If so, the segmentation starting point and the segmentation end point are used as two endpoints of a second-order Bezier curve, a second-order Bezier curve is fitted on the boundary line to obtain control points, and the segmentation curve is obtained according to the segmentation starting point, the segmentation end point and the control points.

6. The method according to claim 5, characterized in that The acquiring the segmentation curve according to the segmentation starting point, the segmentation end point, the first control point, and the second control point includes: Determine a first pixel point and a second pixel point according to the segmentation starting point, the segmentation end point, the first control point, and the second control point, wherein the first pixel point is a pixel point located on a one-third perpendicular line of a line segment with the segmentation starting point and the segmentation end point as endpoints and is closest to the first control point, and the second pixel point is a pixel point located on a two-thirds perpendicular line of a line segment with the segmentation starting point and the segmentation end point as endpoints and is closest to the second control point; quantizing the position coordinates of the first pixel point and the position coordinates of the second pixel point respectively to obtain a first quantization control point and a second quantization control point; The segmentation curve is generated according to the segmentation starting point, the segmentation end point, the first quantization control point, the second quantization control point and a third-order Bezier curve formula.

7. The method according to claim 6, characterized in that The acquiring, according to the control information of the segmentation curve and the two sub-image blocks, code stream data corresponding to the target image block comprises: Encoding the position information of the first quantization control point and the position information of the second quantization control point to obtain first bitstream data; Encoding the two sub-image blocks respectively to obtain second code stream data and third code stream data; Generate code stream data corresponding to the target image block according to the first code stream data, the second code stream data and the third code stream data.

8. The method according to claim 6, characterized in that The acquiring the segmentation curve according to the segmentation starting point, the segmentation end point and the control point comprises: Determine a target pixel point according to the segmentation starting point, the segmentation end point and the control point, the target pixel point being a pixel point located on the perpendicular bisector of a line segment with the segmentation starting point and the segmentation end point as endpoints and being closest to the control point; Quantifying the position coordinates of the target pixel point to obtain a quantized control point; The segmentation curve is generated according to the segmentation starting point, the segmentation end point, the quantization control point and a second-order Bezier curve formula.

9. The method according to claim 8, characterized in that The acquiring, according to the control information of the segmentation curve and the two sub-image blocks, code stream data corresponding to the target image block comprises: Encoding the position information of the quantization control point to obtain first bitstream data; Encoding the two sub-image blocks respectively to obtain second code stream data and third code stream data; Generate code stream data corresponding to the target image block according to the first code stream data, the second code stream data and the third code stream data.

10. The method according to any one of claims 1 to 9, characterized in that: The step of dividing the target image block into two sub-image blocks based on the segmentation curve of the target image block comprises: Sampling the segmentation curve with integer pixel accuracy to obtain a sampling result of the segmentation curve; The target image block is divided into two sub-image blocks according to the sampling result of the segmentation curve.

11. The method according to any one of claims 1 to 9, characterized in that: The step of obtaining a boundary line of the target image block according to a texture edge in the target image block includes: The target image block is divided into two image areas based on a texture edge in the target image block, and a boundary line of the two image areas is obtained as a boundary line of the target image block.

12. A video decoding method, characterized in that: include: Acquire bitstream data corresponding to a target image block, the bitstream data comprising: first bitstream data, second bitstream data and third bitstream data; the first bitstream data is bitstream data obtained by encoding control information of a segmentation curve, the segmentation curve is a curve for dividing the target image block into two sub-image blocks, and the second bitstream data and the third bitstream data are bitstream data obtained by encoding the two sub-image blocks respectively; Acquire control information of the segmentation curve according to the first bitstream data; Obtain two endpoints of the segmentation curve according to the decoded video content to obtain a segmentation start point and a segmentation end point; Acquire the segmentation curve according to the control information of the segmentation curve, the segmentation starting point, and the segmentation end point; Segmenting the target image block based on the segmentation curve to determine the two sub-image blocks; Reconstructing the two sub-image blocks according to the second code stream data and the third code stream data; The reconstructed target image block is obtained according to the two reconstructed sub-image blocks.

13. A video encoding device, characterized in that: include: a memory configured to store a computer program; The processor is configured to enable the video encoding device to implement the video encoding method according to any one of claims 1 to 11 when calling a computer program.

14. A video decoding device, characterized in that: include: a memory configured to store a computer program; The processor is configured to enable the video decoding device to implement the video decoding method according to claim 12 when calling the computer program.

Images