Method and device for determining anchor block in super-resolution lifting and electronic equipment

Abstract

The invention relates to the technical field of computer vision, and provides a method and a device for determining an anchor block in super-resolution improvement and electronic equipment, and the method for determining the anchor block in super-resolution improvement comprises the steps: obtaining a video frame of a video to be subjected to super-resolution improvement; segmenting the video frame to obtain video blocks which are not overlapped in space and are combined to cover a space region of the video frame; the method comprises the following steps of: dividing a macro block into sub-macro blocks according to the overlapping condition of the macro block of a video frame and a video block, and only overlapping the sub-macro block with one video block; whether the video block is an anchor block or not is determined according to the texture complexity of the sub-macro block in the video block, and super-resolution improvement is carried out on the anchor block in the mode that the anchor block is not based on the super-resolution version of the multiplexing reference area. According to the method, the video blocks can be independent of other video blocks in the same video frame to select whether the video blocks are based on the super-resolution version of the multiplexing reference area to carry out super-resolution improvement, and the overhead of the super-resolution task is further optimized through finer decision granularity.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device and electronic device for determining an anchor block in super-resolution enhancement. Background Art

[0002] Super-resolution neural networks have many uses, such as running super-resolution neural networks on the server side to refurbish old low-resolution videos into high-resolution videos, and using the computing power of the client device to run super-resolution neural networks to restore the downloaded videos to high resolution when the client with adaptive bitrate function perceives that the network speed has slowed down and downloads a low-resolution version of the video. However, using super-resolution neural networks to process each frame of the video will bring a very high amount of computing, which results in a high computing cost when running super-resolution neural networks on the server side, and running super-resolution neural networks on the end device will cause high energy consumption and low battery life problems of the device, and the weak computing power end device cannot perform super-resolution neural network inference in real time to enhance the resolution in real time.

[0003] At present, the optimization scheme for super-resolution neural network overhead is to use the redundancy of video frames, that is, only some frames in the video (called anchor frames) are processed by independent super-resolution algorithms such as super-resolution neural networks to improve the resolution (called neural network-based super-resolution), and the remaining frames (called non-anchor frames) are demultiplexed with the super-resolution version of their reference frames for resolution improvement (called reuse-based super-resolution). Reuse-based super-resolution methods usually have lower computational overhead than independent super-resolution algorithms based on neural networks, and the more redundant the video frame content is in the temporal dimension compared to its reference frame, the closer the reuse-based super-resolution method is to the independent super-resolution algorithm based on neural networks in terms of restoration quality. By adopting a suitable anchor frame selection algorithm, it is possible to achieve high-quality super-resolution versions of all frames (including anchor frames and non-anchor frames) when only a small number of frames are selected as anchor frames. Compared with using neural networks for super-resolution enhancement on all frames, this scheme can greatly optimize the average processing overhead of all frames without significantly sacrificing the quality of super-resolution video.

[0004] However, there may still be some areas in an anchor frame whose resolution can be improved with high quality by multiplexing, and there may also be some areas in a non-anchor frame whose high-resolution version obtained by multiplexing is of very low quality and is more suitable for restoring its resolution using a neural network. Summary of the invention

[0005] The present invention provides a method, device and electronic device for determining anchor blocks in super-resolution enhancement, which is used to solve the defects in the prior art that an anchor frame may still have some areas whose resolution can be improved with high quality by multiplexing, and a non-anchor frame may also have some areas whose high-resolution version obtained by multiplexing is of very low quality and is more suitable for restoring its resolution by neural network and other methods, so as to realize that each video frame is divided into video blocks, and the specific super-resolution method to be adopted is determined by video blocks rather than frames. By adopting a finer block-level decision granularity, the total anchor area required is reduced while achieving the same or even higher super-resolution video quality, thereby further optimizing the overall super-resolution overhead.

[0006] The present invention provides a method for determining an anchor block in super-resolution enhancement, comprising: obtaining a video frame of a video to be subjected to super-resolution enhancement; dividing the video frame to obtain video blocks, wherein the video blocks do not overlap in space and are combined to cover the spatial area of ​​the video frame; dividing the macroblock into sub-macroblocks according to the overlap between a macroblock of the video frame and a video block, wherein the sub-macroblock overlaps only with one video block and two sub-macroblocks do not overlap with the same video block; determining whether the video block is an anchor block according to the texture complexity of the sub-macroblock in the video block, wherein the anchor block is not subjected to super-resolution enhancement based on a super-resolution version of a multiplexed reference area.

[0007] According to a method for determining an anchor block in super-resolution enhancement provided by the present invention, the macroblock includes an inter-frame prediction macroblock, the sub-macroblock includes an inter-frame prediction sub-macroblock, and the determining whether the video block is an anchor block according to the texture complexity of the sub-macroblock in the video block includes: determining whether the video block is an anchor block based on a constructed directed acyclic graph, a node in the directed acyclic graph represents one of the video blocks, at least one of the inter-frame prediction sub-macroblocks exists inside the video block corresponding to the destination node of the directed edge connecting the node, inter-frame prediction encoding is performed with reference to the pixel value of the video block corresponding to the source node of the directed edge, the attribute of the node includes a first attribute, the first attribute represents whether the video block corresponding to the node can be well super-resolution enhanced based on a super-resolution version of a reused reference area, and the first attribute is determined according to the texture complexity of the sub-macroblock in the video block.

[0008] According to a method for determining an anchor block in super-resolution enhancement provided by the present invention, the attribute of the node also includes a second attribute, the second attribute represents the area of ​​the video block corresponding to the node, the macroblock also includes an intra-frame prediction macroblock, the sub-macroblock also includes an intra-frame prediction sub-macroblock, the texture complexity of the sub-macroblock includes a first texture complexity of the inter-frame prediction sub-macroblock and a second texture complexity of the intra-frame prediction sub-macroblock, the first texture complexity includes the texture complexity of the residual of the inter-frame prediction sub-macroblock and the reference area when the inter-frame prediction sub-macroblock performs inter-frame prediction, and the second texture complexity includes the texture complexity of the intra-frame prediction residual or the pixel value of the intra-frame prediction sub-macroblock itself; the step of determining the first attribute includes: determining the first texture complexity of the inter-frame prediction sub-macroblock in the node and the second texture complexity of the intra-frame prediction sub-macroblock; calculating the sum of the first texture complexity of the inter-frame prediction sub-macroblock and the second texture complexity of the intra-frame prediction sub-macroblock in the node, and dividing it by the second attribute of the node to obtain the first attribute.

[0009] According to a method for determining an anchor block in super-resolution enhancement provided by the present invention, the attributes of the directed edge include a weight attribute, and the weight attribute is set by the following steps: based on the motion vector, reference index, and multi-reference frame weighted prediction weight of the macroblock extracted from the video encoding of the video to be super-resolution enhanced, determining the pixel ratio for inter-frame prediction in the video block corresponding to the destination node of the directed edge with reference to the pixel value of the video block corresponding to the source node of the directed edge as the weight attribute; or using an optical flow algorithm or a motion estimation algorithm between video frames associated with the directed edge, estimating the pixel ratio for inter-frame prediction in the video block corresponding to the destination node of the directed edge with reference to the pixel value of the video block corresponding to the source node of the directed edge as the weight attribute.

[0010] According to a method for determining an anchor block in super-resolution enhancement provided by the present invention, the determining whether the video block is an anchor block based on a constructed directed acyclic graph includes: executing the following anchor block determination steps until a preset end condition is reached: selecting a video block to add to an anchor block set according to a preset search strategy, so that the estimated super-resolution defect of the entire video is quickly reduced, wherein the super-resolution defect of the entire video is the sum of the super-resolution defects of the video blocks outside the anchor block set, and the super-resolution defect of the video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels of the video block The average per-pixel super-resolution quality defect of the video block is determined by the following steps: selecting a target video frame from the video frames in sequence according to a decoding order, and determining the average per-pixel super-resolution quality defect of the video block in the target video frame according to whether the video block in the target video frame is an anchor block or a non-anchor block, a first attribute of a node corresponding to the video block in the target video frame, and a quality defect transmitted through a directed edge; the quality defect attribute transmitted through a directed edge is determined based on the product of the average per-pixel super-resolution quality defect of the video block corresponding to a source node of the directed edge and a weight attribute of the directed edge.

[0011] According to a method for determining an anchor block in super-resolution enhancement provided by the present invention, the anchor block performs super-resolution enhancement based on a pre-built super-resolution neural network.

[0012] According to a method for determining an anchor block in super-resolution enhancement provided by the present invention, the determining whether the video block is an anchor block based on a constructed directed acyclic graph includes: sorting the video blocks corresponding to the nodes according to the first attributes of the nodes in the directed acyclic graph to obtain a video block sorting result; and determining whether the video block is an anchor block according to the video block sorting result.

[0013] According to a method for determining an anchor block in super-resolution enhancement provided by the present invention, determining whether the video block is an anchor block based on a constructed directed acyclic graph includes: sorting the nodes of the directed acyclic graph based on a pre-constructed node sorting model of a graph neural network to obtain a node sorting result; and determining whether the video block is an anchor block based on the node sorting result.

[0014] The present invention also provides a device for determining an anchor block in super-resolution enhancement, comprising: an acquisition module, configured to acquire a video frame of a video to be subjected to super-resolution enhancement; a segmentation module, configured to segment the video frame to obtain video blocks, the video blocks do not overlap in space, and are combined to cover the spatial area of ​​the video frame; a division module, configured to divide the macroblock into sub-macroblocks according to the overlap between the macroblock of the video frame and the video block, the sub-macroblock overlaps with only one of the video blocks, and two sub-macroblocks do not overlap with the same video block; a determination module, configured to determine whether the video block is an anchor block according to the texture complexity of the sub-macroblock in the video block, and the anchor block is not subjected to super-resolution enhancement based on a super-resolution version of a reused reference area.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for determining an anchor block in super-resolution enhancement as described in any one of the above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for determining an anchor point block in super-resolution enhancement as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for determining an anchor point block in super-resolution enhancement as described in any one of the above is implemented.

[0018] The method, device and electronic device for determining anchor blocks in super-resolution enhancement provided by the present invention obtain video blocks by segmenting video frames, and divide macroblocks into sub-macroblocks according to the overlap between macroblocks of video frames and video blocks, and finally determine whether a video block is an anchor block according to the texture complexity of the sub-macroblocks in the video block, and the anchor block is not super-resolution enhanced based on the super-resolution version of the reused reference area. Compared with the existing scheme of selecting the processing method in units of frames (i.e., anchor frames or non-anchor frames are super-resolution enhanced in different ways), the present invention accurately finds the anchor blocks that are most necessary to use the computational overhead through finer decision granularity, and only needs to use the multiplexing method with relatively lower overhead for super-resolution enhancement for other video areas, while ensuring the overall effect of super-resolution enhancement, reducing the overall overhead of super-resolution enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flowchart of a method for determining an anchor block in super-resolution enhancement provided by the present invention.

[0021] Figure 2 It is a schematic diagram of the process of determining the anchor block in the super-resolution enhancement provided by the present invention.

[0022] Figure 3 It is a schematic diagram of a video frame in the method for determining an anchor block in super-resolution enhancement provided by the present invention.

[0023] Figure 4 It is a structural schematic diagram of a device for determining an anchor point block in super-resolution enhancement provided by the present invention.

[0024] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The following is a brief explanation of the terms involved in the present invention.

[0027] Video: A continuous sequence of images, consisting of consecutive frames, where one frame is an image.

[0028] Video coding: A type of technology that compresses video files. Existing video coding technologies can usually significantly reduce the size of video files without significantly reducing the similarity between the encoded video and the original unencoded video, allowing users to download and play videos in real time at limited network speeds. There is usually strong redundancy between frames that are close in time within a video (for example, in a selfie video, the selfie taker's facial information will appear repeatedly in different frames, resulting in very strong temporal redundancy in the video), and there is also spatial redundancy within the same frame (for example, if a blue sky appears in a single video frame, then multiple pixels reflect the blue color of the sky, and the values ​​of these pixels are very close and redundant to each other). Video coding mainly compresses video files by eliminating redundant information in the temporal and spatial dimensions of unencoded videos.

[0029] Reference frame: In order to compress a frame and eliminate redundant information in the time dimension, video encoding technology will select several other frames as reference frames for the frame, and try to find out which information of the current video frame has appeared redundantly in its reference frame. After encoding, only the newly appearing information in the current video frame is stored in the video, so that the encoded frame has very low redundancy and a very small file size.

[0030] Macroblock, motion vector, reference frame index, residual: Video coding technology generally divides a video frame into multiple sub-regions called macroblocks in space. If a macroblock is redundant in time, the video coding technology will find its approximate region in the reference frame for the macroblock. The approximate region can be used as the prediction value of the current macroblock. The encoded video does not need to store all the pixel values ​​of the macroblock, but only needs to store the spatial displacement of the macroblock relative to the approximate region (i.e., motion vector), which reference frame the approximate region belongs to in the frame (i.e., reference frame index), and the difference between the original value and the predicted value of the macroblock (i.e., residual). The encoding method of this macroblock is inter-frame prediction; if a macroblock is redundant in space, the video coding technology will obtain the predicted value of the macroblock based on the pixel values ​​of the macroblocks near the macroblock in the same frame. The encoded video does not need to store all the pixel values ​​of the macroblock, but only needs to store the difference between the original value and the predicted value of the macroblock (i.e., residual) and other information. The encoding method of this macroblock is intra-frame prediction.

[0031] Super-resolution neural network: A neural network whose input is a low-resolution image and possible auxiliary information, and whose output is a high-resolution version of the image and possible additional information.

[0032] Image interpolation: In mathematics, interpolation refers to using the value of a function at a finite number of points to estimate the approximate value of a function at other points. Image interpolation can be used to restore low-resolution video frames to high resolution. Generally speaking, the computational complexity and restoration quality of image interpolation are lower than those of super-resolution neural networks.

[0033] Adaptive bitrate: When downloading and playing videos in real time over the Internet, the speed of the user's device will change constantly. In order to adapt to this change, the video file is usually converted into several versions, some of which have high resolution and some have low resolution. When the user's network speed increases, the client will automatically switch to downloading the high-resolution version to ensure that the user can see a clearer picture. When the network speed slows down, it will automatically switch to downloading the low-resolution version to prevent the download speed from being slower than the playback speed, causing playback to freeze.

[0034] Greedy search: is an algorithmic strategy used to solve optimization problems. It takes the currently optimal option at each step, hoping to reach the global optimal solution through a series of local optimal solutions. This strategy does not guarantee that the global optimal solution will always be obtained, but it is usually simple and fast.

[0035] Combine the following Figure 1-Figure 5 The present invention describes a method, device and electronic device for determining an anchor block in super-resolution enhancement.

[0036] Figure 1 It is one of the flow charts of the method for determining the anchor block in super-resolution enhancement provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Obtain a video frame of a video to be super-resolution enhanced.

[0037] In this embodiment, super-resolution enhancement may be performed by using a super-resolution version based on a reused reference area or other methods, such as a super-resolution neural network.

[0038] Step 102: Segment the video frame to obtain video blocks.

[0039] In this embodiment, the video blocks do not overlap in space and are combined to cover the spatial area of ​​the video frame. The specific video block segmentation method can be determined according to the complexity of the video to be super-resolution enhanced, content characteristics, and system computing power.

[0040] Step 103: Divide the macroblock into sub-macroblocks according to the overlap between the macroblock of the video frame and the video block.

[0041] In this embodiment, a sub-macroblock overlaps only one video block, and two sub-macroblocks do not overlap with the same video block. When a macroblock overlaps only one video block, the macroblock itself is a sub-macroblock. Through such division, each video block contains one or more sub-macroblocks from different macroblocks.

[0042] Step 104: Determine whether the video block is an anchor block according to the texture complexity of the sub-macroblock in the video block.

[0043] In this embodiment, the texture complexity of the sub-macroblock can be determined by calculating the Euclidean norm, variance, mean absolute deviation, etc. of the error of re-upsampling after downsampling. The anchor block is not super-resolution enhanced based on the super-resolution version of the reused reference area. The anchor block can be processed using an independent super-resolution algorithm instead of relying on the reuse of reference frames. As an example, the anchor block can be super-resolution enhanced using deep learning models, which can learn the mapping from low resolution to high resolution.

[0044] In addition, the video blocks can be directly sorted according to the texture complexity of the sub-macroblocks within the video blocks to determine whether the video blocks are anchor blocks. A directed acyclic graph can be established based on the texture complexity of the sub-macroblocks within the video blocks to determine whether the video blocks are anchor blocks. It can also be determined whether the video blocks are anchor blocks based on a graph neural network.

[0045] The method for determining anchor blocks in super-resolution enhancement provided by the present invention obtains video blocks by segmenting video frames, and divides macroblocks into sub-macroblocks according to the overlap between macroblocks of video frames and video blocks, and finally determines whether a video block is an anchor block according to the texture complexity of the sub-macroblocks in the video block, and further optimizes the overhead of the super-resolution task through fine-grained super-resolution method selection. The operation of the anchor block determination method provided by the present invention does not depend on the original high-resolution version of the video, and can be applied to application scenarios where the original high-resolution version of the video does not exist or cannot be obtained at the same time as the low-resolution version of the video (such as the enhancement of old videos). The operation delay of the anchor block determination method provided by the present invention is very low, which can meet the application scenarios with low delay requirements such as live broadcast.

[0046] In some optional implementations, the macroblock includes an inter-frame prediction macroblock, the sub-macroblock includes an inter-frame prediction sub-macroblock, and the video block is determined to be an anchor block according to the texture complexity of the sub-macroblock in the video block, including: determining whether the video block is an anchor block based on a constructed directed acyclic graph, a node in the directed acyclic graph represents a video block, and there is at least one inter-frame prediction sub-macroblock inside the video block corresponding to the destination node of the directed edge connecting the node, and inter-frame prediction coding is performed with reference to the pixel value of the video block corresponding to the source node of the directed edge, and the attribute of the node includes a first attribute, and the first attribute characterizes whether the video block corresponding to the node can be well super-resolution-enhanced based on the super-resolution version of the reused reference area, and the first attribute is determined according to the texture complexity of the sub-macroblock in the video block. The first attribute can be directly determined according to the texture complexity of the sub-macroblock in the video block, and the area of ​​the video block and other factors can be further considered to further improve the accuracy of the determined first attribute. The generation and transmission process of super-resolution video quality defects can be accurately modeled by the data structure of the directed acyclic graph, and this implementation further improves the accuracy of anchor block determination.

[0047] See also Figure 2 , Figure 2 The following is a schematic diagram of the process of determining anchor blocks in super-resolution enhancement provided by the present invention. The low-resolution video, i.e., the video to be super-resolution enhanced, is input into the first submodule of the device, the directed acyclic graph construction code submodule. This submodule will generate a code for each video block in the low-resolution video. Create a graph node. Consider two video blocks (the corresponding nodes are denoted as and ), if and only if there is at least one The internal inter-frame prediction sub-macroblock refers to To predict the pixel value in, there is a directed acyclic graph from Departure, direction Directed edge Therefore, there is a line from Departure, direction Directed edge The necessary condition is that The video frame is composed of The reference frame of the video frame. Since the reference relationship between frames is acyclic in common video coding standards, the constructed graph is a directed acyclic graph. To set the attribute values ​​of the nodes and edges in the graph, it is necessary to obtain the internal coding information of the low-resolution coded video such as residuals and motion vectors. The directed acyclic graph construction code submodule can add code based on the existing video decoder software, and use the information extracted by the decoder from the coded video to calculate the attribute values ​​of the nodes and edges.

[0048] Each node in the directed acyclic graph Contains a static first attribute, the name of the first attribute can be a non-reusable attribute , this non-reusability attribute reflects whether the node can be well processed by the reuse-based super-resolution method. In order to set the non-reusability attribute The value of needs to calculate the texture complexity of the sub-macroblock contained in the video block corresponding to the node.

[0049] In some optional implementations, the attributes of the node also include a second attribute, the second attribute represents the area of ​​the video block corresponding to the node, the macroblock also includes an intra-frame prediction macroblock, the sub-macroblock also includes an intra-frame prediction sub-macroblock, the texture complexity of the sub-macroblock includes a first texture complexity of the inter-frame prediction sub-macroblock and a second texture complexity of the intra-frame prediction sub-macroblock, the first texture complexity includes the texture complexity of the residual of the area referenced when the inter-frame prediction sub-macroblock and the inter-frame prediction sub-macroblock perform inter-frame prediction, and the second texture complexity includes the texture complexity of the intra-frame prediction residual or the pixel value of the intra-frame prediction sub-macroblock itself; the step of determining the first attribute includes: determining the first texture complexity of the inter-frame prediction sub-macroblock in the node and the second texture complexity of the intra-frame prediction sub-macroblock; calculating the sum of the first texture complexity of the inter-frame prediction sub-macroblock and the second texture complexity of the intra-frame prediction sub-macroblock in the node, and dividing it by the second attribute of the node to obtain the first attribute. The decoder extracts this information from the encoded video to calculate the attribute values ​​of the nodes and edges. The second attribute can be named as the area attribute, that is .

[0050] First, a super-resolution enhancement method based on multiplexing is introduced: For any inter-predicted macroblock in low-resolution video , its pixel value can be recorded as , the reference area of ​​the macroblock for inter-frame prediction is recorded as , the residual is recorded as , then

[0051] Using symbols Indicates that image a is interpolated and magnified by b times, and scale indicates the resolution ratio of the super-resolution video to the low-resolution video. When using the multiplexing-based super-resolution The super-resolution version of

[0052] in, Represents a super-resolution version of the reference region for inter prediction.

[0053] For any intra-predicted macroblock in low-resolution video , its pixel value can be recorded as , then when using multiplexing-based super-resolution The super-resolution version of

[0054] The texture complexity of the sub-macroblocks contained in the video block can be determined based on all its inter-frame prediction sub-macroblocks. The residual The higher the complexity, the more the residual is calculated according to the formula (2) in the previous article. The result of interpolation The lower the quality, the higher the super-resolution version of the sub-macroblock The quality compared to The greater the quality gap, the less suitable it is for the sub-macroblock to be processed by the multiplexing-based super-resolution method; at the same time, all intra-frame prediction sub-macroblocks contained in the video block corresponding to the node will be calculated. A low-resolution version of The higher the complexity, the more complex the texture is according to the formula (3) in the previous article. Interpolated super-resolution version The worse the quality; the sum of the texture complexity of the residuals of all inter-frame prediction sub-macroblocks and the texture complexity of all intra-frame prediction sub-macroblocks contained in the video block corresponding to the node, divided by the area attribute of the node, is the value of the non-reusability attribute of the node.

[0055] To calculate a two-dimensional array (Specifically it may be or ) texture complexity, the specific method adopted by the present invention is (Compute the L2 norm (also called Euclidean norm) of the error after downsampling and then re-upsampling), where Can be any real number greater than 1.

[0056] In addition, other methods can be used to calculate or The texture complexity of the video can be calculated by using the methods of calculating variance, mean absolute deviation, etc. In the case of the original high-resolution video, the interpolated video can also be obtained. or Compare with the relevant values ​​in the original high-resolution video and calculate the difference as the value of texture complexity.

[0057] In the above multiplexing method, the intra-frame prediction coded macroblocks are processed by interpolation, as shown in formula (3). An alternative multiplexing method is to interpolate the intra-frame coded residual scale times and add it to the predicted value. If this alternative multiplexing method is adopted, it is no longer necessary to calculate when constructing the directed acyclic graph. The texture complexity of low-resolution video can be calculated In fact, even if we still use formula (3) to demultiplex the intra-frame prediction coded macroblocks, using The residual texture complexity of is also a viable alternative, although its effect in selecting anchor blocks may not be as good as The texture complexity.

[0058] Each edge in a directed acyclic graph Can have a static weight attribute , which reflects the video block corresponding to the destination node of the edge (denoted as ) If the reuse-based super-resolution processing is used, how many proportions of pixels in the block reuse the source node of the edge corresponding to the video block (denoted as ). To set the weights, we need to extract the pixels from the encoded low-resolution video. The motion vector information of each inter-frame prediction sub-macroblock in the corresponding video block is used to determine the reference of each inter-frame prediction sub-macroblock based on the motion vector information. The number of pixels in the corresponding video block is predicted (if multiple reference frames are used for weighted prediction, the number of pixels needs to be multiplied by the corresponding weight), the sum of the number of reference pixels is divided by , the result of the division is the weight of this edge According to formula (2), the super-resolution version of the sub-macroblock when using the multiplexed super-resolution is The quality of the super-resolution version of the reference region The impact of quality, of quality defects will spread to The purpose of using the weight attribute is to quantify the propagation of quality defects at the sub-macroblock scale as the proportion of the quality defects at the video block scale.

[0059] In addition, other methods can also be used to calculate the weight attributes of the edges, such as calculating the optical flow between frames, or using motion estimation algorithms such as diamond search to obtain the area that each video block may refer to when doing inter-frame prediction encoding. The advantage of this method is that there is no need to add code to the decoder software to parse information such as motion vectors and weighted prediction weights for multi-reference frame predictions.

[0060] Figure 3is a schematic diagram of a video frame in the method for determining an anchor block in super-resolution enhancement provided by the present invention, Figure 3 Two consecutive frames from a low-resolution video are shown in , , assuming yes The only reference frame and the frame Pre-divided into four video blocks , , , , the frame Pre-divided into four video blocks , , , . It is a video block The only inter-frame prediction coded sub-macroblock in the video block There are multiple intra-frame prediction coded sub-macroblocks According to the above content, the video block The corresponding node Property value equals .because and , There is overlap, and , The overlapping areas are 0.105 and 0.323 times the area, so there exists Starting from and pointing to the corresponding node The edge corresponding to the node has a weight of 0.105, and there is also Starting from and pointing to the corresponding node The weight of the edge corresponding to the node is 0.323.

[0061] In some optional implementations, determining whether a video block is an anchor block based on a constructed directed acyclic graph includes: performing the following anchor block determination steps until a preset end condition is reached: selecting a video block to add to the anchor block set according to a preset search strategy, so that the estimated super-resolution defect of the entire video is rapidly reduced, wherein the super-resolution defect of the entire video is the sum of the super-resolution defects of the video blocks outside the anchor block set, the super-resolution defect of the video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels of the video block, and the average super-resolution quality defect per pixel of the video block is determined by the following steps: selecting a target video frame from the video frames in sequence according to a decoding order, determining the average super-resolution quality defect per pixel of the video block in the target video frame according to whether the video block in the target video frame is an anchor block or a non-anchor block, a first attribute of a node corresponding to the video block in the target video frame, and a quality defect transmitted through a directed edge; the quality defect attribute transmitted through a directed edge is determined based on the product of the average super-resolution quality defect per pixel of the video block corresponding to the source node of the directed edge and the weight attribute of the directed edge.

[0062] Continue to see Figure 2 The second submodule is the anchor frame selection code submodule based on directed acyclic graph. This module uses the super-resolution video quality estimation based on directed acyclic graph to select the anchor block set that makes the estimated quality defect low. Greedy or other arbitrary search strategies can be used during selection. The key here is how to estimate the quality of super-resolution video based on directed acyclic graph. Given a constructed directed acyclic graph and an anchor block set, the specific method for super-resolution video quality estimation is: Step 1: Sort the video frames according to the decoding order of the low-resolution video frames, and record them as , in this order, use the following step 2 to sequentially estimate the average per-pixel super-resolution quality defect level of each video frame.

[0063] Step 2: For the frame , whether each video block inside it belongs to a given anchor block set can be represented by a vector consisting of 0 and 1 To indicate (0 means it belongs to the anchor block set, 1 means it does not belong to the anchor block set), frame The non-reusability attribute and area attribute of the node corresponding to the internal video block can form another two vectors , Frame Recording The average per-pixel super-resolution quality defect attribute of the internal video block is , the reference frame of this frame is , ,…,connect The corresponding nodes of the inner video block and The weights of the edges of the corresponding nodes in the video block are combined into a weight matrix (if A video block within A certain video block has no edge connection, and the corresponding element in the weight matrix is ​​0), then there is ,in represents element-by-element multiplication. When the weight matrix When a large number of elements in are 0, sparse matrix multiplication can be used to calculate .

[0064] Step 3: Calculate the super-resolution quality defects of each video block within all frames , ,…, the super-resolution quality defect of the overall video is estimated to be , where the superscript Express Transpose, the quality can be used To characterize.

[0065] In order to reduce the delay of super-resolution video quality estimation based on directed acyclic graphs, and thus better support delay-sensitive applications such as live video, super-resolution video quality estimation under multiple anchor block sets (which can be considered as an anchor block set batch) can be performed in parallel. The specific method is to perform , An additional batch dimension is added, making , From a one-dimensional vector to a two-dimensional tensor.

[0066] In some optional implementations, the anchor block performs super-resolution enhancement based on a pre-built super-resolution neural network. Super-resolution enhancement through a super-resolution neural network can better capture image features and improve super-resolution quality. Pre-built super-resolution neural networks may include EDSR (Enhanced Deep Super-Resolution), SRGAN (Super-Resolution Generative Adversarial Network), and RCAN (Residual Channel Attention Network), etc., which are not limited in this application.

[0067] In some optional implementations, determining whether a video block is an anchor block based on a constructed directed acyclic graph includes: sorting the video blocks corresponding to the nodes in the directed acyclic graph according to the first attributes of the nodes to obtain a video block sorting result; and determining whether the video block is an anchor block according to the video block sorting result.

[0068] In some optional implementations, determining whether a video block is an anchor block based on a constructed directed acyclic graph includes: sorting the nodes of the directed acyclic graph based on a pre-constructed node sorting model of a graph neural network to obtain a node sorting result; and determining whether the video block is an anchor block based on the node sorting result. A graph neural network (GNN) is a neural network that can process graph structured data and learn the relationship between nodes. The graph neural network can be trained using pre-prepared sample data to obtain a model that can sort nodes.

[0069] In this embodiment, multiple anchor point selection methods can also be combined and used together. For example, some candidate anchor point blocks are first selected based on the non-reusability attribute value of the node, and then the final anchor point block set is selected from the candidate anchor point blocks using super-resolution video quality estimation based on a directed acyclic graph.

[0070] The present invention uses multiple videos published on the Internet to verify the effect, confirming that the method of the present invention can select suitable anchor blocks with very low delay. The verification method is to convert the video from the original high-resolution version to a low-resolution version, and then apply the anchor block selection algorithm proposed in the present invention and the existing anchor frame selection algorithm to select the anchor area. The experimental results show that the selection delay of the anchor block selection algorithm proposed in the present invention is in the order of ten milliseconds, which can be applied to low-latency scenarios such as live broadcasts, and compared with the existing anchor frame selection algorithm that can also be applied to low-latency scenarios such as live broadcasts, the anchor block selection algorithm of the present invention can reduce the area of ​​the anchor area by an average of about 38% and a maximum of about 80% without reducing the quality of the super-resolution video (by comparing with the original high-resolution version to calculate PSNR and measure quality).

[0071] Figure 4 A schematic diagram of the structure of a device for determining an anchor block in super-resolution enhancement provided in an embodiment of the present application, such as Figure 4As shown, it specifically includes: an acquisition module 401, which is configured to acquire a video frame of a video to be super-resolution enhanced; a segmentation module 402, which is configured to segment the video frame to obtain video blocks, the video blocks do not overlap in space, and are combined to cover the spatial area of ​​the video frame; a division module 403, which is configured to divide the macroblock into sub-macroblocks according to the overlap between the macroblock of the video frame and the video block, the sub-macroblock overlaps with only one video block, and two sub-macroblocks do not overlap with the same video block; a determination module 404, which is configured to determine whether the video block is an anchor block according to the texture complexity of the sub-macroblock in the video block, and the anchor block is not super-resolution enhanced based on the super-resolution version of the reused reference area.

[0072] In some optional implementations, the macroblock includes an inter-frame prediction macroblock, and the sub-macroblock includes an inter-frame prediction sub-macroblock. The determination module 404 is further configured to: determine whether the video block is an anchor block based on a constructed directed acyclic graph, a node in the directed acyclic graph represents a video block, and there is at least one inter-frame prediction sub-macroblock inside the video block corresponding to the destination node of the directed edge connecting the nodes, and inter-frame prediction encoding is performed with reference to the pixel value of the video block corresponding to the source node of the directed edge. The attributes of the node include a first attribute, and the first attribute represents whether the video block corresponding to the node can be well super-resolution enhanced based on a super-resolution version of the reused reference area. The first attribute is determined according to the texture complexity of the sub-macroblock in the video block.

[0073] In some optional implementations, the attributes of the node also include a second attribute, the second attribute represents the area of ​​the video block corresponding to the node, the macroblock also includes an intra-frame prediction macroblock, the sub-macroblock also includes an intra-frame prediction sub-macroblock, the texture complexity of the sub-macroblock includes a first texture complexity of the inter-frame prediction sub-macroblock and a second texture complexity of the intra-frame prediction sub-macroblock, the first texture complexity includes the texture complexity of the residual of the inter-frame prediction sub-macroblock and the reference area when the inter-frame prediction sub-macroblock performs inter-frame prediction, and the second texture complexity includes the texture complexity of the intra-frame prediction residual or the pixel value of the intra-frame prediction sub-macroblock itself; the step of determining the first attribute includes: determining the first texture complexity of the inter-frame prediction sub-macroblock in the node and the second texture complexity of the intra-frame prediction sub-macroblock; calculating the sum of the first texture complexity of the inter-frame prediction sub-macroblock and the second texture complexity of the intra-frame prediction sub-macroblock in the node, and dividing it by the second attribute of the node to obtain the first attribute.

[0074] In some optional implementations, the attributes of the directed edge include a weight attribute, and the weight attribute is set by the following steps: based on the motion vector, reference index, and multi-reference frame weighted prediction weight of the macroblock extracted from the video encoding of the video to be super-resolution enhanced, determining the pixel ratio of the video block corresponding to the destination node of the directed edge for inter-frame prediction with reference to the pixel value of the video block corresponding to the source node of the directed edge as the weight attribute; or using the optical flow algorithm or motion estimation algorithm between the video frames associated with the directed edge, estimating the pixel ratio of the video block corresponding to the destination node of the directed edge for inter-frame prediction with reference to the pixel value of the video block corresponding to the source node of the directed edge as the weight attribute.

[0075] In some optional implementations, the determination module 404 is further configured to: perform the following anchor block determination steps until a preset end condition is reached: select video blocks according to a preset search strategy and add them to the anchor block set, so that the estimated super-resolution defect of the entire video is quickly reduced, wherein the super-resolution defect of the entire video is the sum of the super-resolution defects of the video blocks outside the anchor block set, the super-resolution defect of the video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels of the video block, and the average super-resolution quality defect per pixel of the video block is determined by the following steps: select target video frames from the video frames in sequence according to the decoding order, determine the average super-resolution quality defect per pixel of the video block in the target video frame according to whether the video block in the target video frame is an anchor block or a non-anchor block, the first attribute of the node corresponding to the video block in the target video frame, and the quality defect transmitted through the directed edge; the quality defect attribute transmitted through the directed edge is determined based on the product of the average super-resolution quality defect per pixel of the video block corresponding to the source node of the directed edge and the weight attribute of the directed edge.

[0076] In some optional implementations, the anchor block performs super-resolution enhancement based on a pre-built super-resolution neural network.

[0077] In some optional implementations, the determination module 404 is further configured to: sort the video blocks corresponding to the nodes in the directed acyclic graph according to the first attributes of the nodes to obtain a video block sorting result; and determine whether the video block is an anchor block according to the video block sorting result.

[0078] In some optional implementations, the determination module 404 is further configured to: sort the nodes of the directed acyclic graph based on a pre-built node sorting model of the graph neural network to obtain a node sorting result; and determine whether the video block is an anchor block based on the node sorting result.

[0079] In this embodiment, the specific processing of the acquisition module 401, the segmentation module 402, the division module 403 and the determination module 404 of the apparatus for determining the anchor block in super-resolution enhancement can be referred to in Figure 1 This corresponds to step 101, step 102, step 103 and step 104 in the embodiment.

[0080] The device for determining anchor blocks in super-resolution enhancement provided by the present invention obtains video blocks by segmenting video frames, and divides the macroblocks into sub-macroblocks according to the overlap between macroblocks of the video frame and the video blocks, and finally determines whether the video block is an anchor block according to the texture complexity of the sub-macroblocks in the video block, thereby further optimizing the overhead of the super-resolution task.

[0081] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a method for determining an anchor block in super-resolution enhancement, the method comprising: obtaining a video frame of a video to be super-resolution enhanced; segmenting the video frame to obtain video blocks, the video blocks do not overlap in space, and are combined to cover the spatial area of ​​the video frame; according to the overlap between the macroblock of the video frame and the video block, the macroblock is divided into sub-macroblocks, the sub-macroblock overlaps only with one video block, and the two sub-macroblocks do not overlap with the same video block; according to the texture complexity of the sub-macroblock in the video block, it is determined whether the video block is an anchor block, and the anchor block is not super-resolution enhanced based on the super-resolution version of the multiplexed reference area.

[0082] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0083] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for determining the anchor block in super-resolution enhancement provided by the above-mentioned methods, and the method includes: obtaining a video frame of a video to be super-resolution enhanced; dividing the video frame to obtain video blocks, the video blocks do not overlap in space, and are combined to cover the spatial area of ​​the video frame; dividing the macroblock into sub-macroblocks according to the overlap between the macroblock of the video frame and the video block, the sub-macroblock overlaps with only one video block, and two sub-macroblocks do not overlap with the same video block; determining whether the video block is an anchor block according to the texture complexity of the sub-macroblock in the video block, and the anchor block is not super-resolution enhanced based on the super-resolution version of the reused reference area.

[0084] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for determining an anchor block in super-resolution enhancement provided by the above-mentioned methods, the method comprising: obtaining a video frame of a video to be super-resolution enhanced; dividing the video frame to obtain video blocks, the video blocks do not overlap in space, and are combined to cover the spatial area of ​​the video frame; dividing the macroblock into sub-macroblocks according to the overlap between the macroblock of the video frame and the video block, the sub-macroblock overlaps with only one video block, and two sub-macroblocks do not overlap with the same video block; determining whether the video block is an anchor block according to the texture complexity of the sub-macroblock in the video block, and the anchor block is not super-resolution enhanced based on the super-resolution version of the reused reference area.

[0085] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining anchor blocks in super-resolution enhancement, characterized in that: include: Obtain a video frame of a video to be super-resolution enhanced; Segmenting the video frame to obtain video blocks, wherein the video blocks do not overlap in space and are combined to cover a spatial region of the video frame; According to the overlap between the macroblock of the video frame and the video block, the macroblock is divided into sub-macroblocks, wherein the sub-macroblock overlaps only with one of the video blocks, and two sub-macroblocks do not overlap with the same video block; Determining whether the video block is an anchor block according to the texture complexity of the sub-macroblock in the video block, and the anchor block is not super-resolution enhanced based on a super-resolution version of a multiplexed reference area.

2. The method for determining anchor blocks in super-resolution enhancement according to claim 1, characterized in that: The macroblock includes an inter-frame prediction macroblock, the sub-macroblock includes an inter-frame prediction sub-macroblock, and determining whether the video block is an anchor block according to the texture complexity of the sub-macroblock in the video block includes: Based on a constructed directed acyclic graph, it is determined whether the video block is an anchor block, a node in the directed acyclic graph represents a video block, at least one inter-frame prediction sub-macroblock exists inside the video block corresponding to the destination node of the directed edge connecting the node, and inter-frame prediction encoding is performed with reference to the pixel value of the video block corresponding to the source node of the directed edge, and the attributes of the node include a first attribute, the first attribute characterizing whether the video block corresponding to the node can be well super-resolution enhanced based on a super-resolution version of a reused reference area, and the first attribute is determined according to the texture complexity of the sub-macroblock in the video block.

3. The method for determining anchor blocks in super-resolution enhancement according to claim 2, characterized in that: The attribute of the node also includes a second attribute, the second attribute represents the area of ​​the video block corresponding to the node, the macroblock also includes an intra-frame prediction macroblock, the sub-macroblock also includes an intra-frame prediction sub-macroblock, the texture complexity of the sub-macroblock includes a first texture complexity of an inter-frame prediction sub-macroblock and a second texture complexity of the intra-frame prediction sub-macroblock, the first texture complexity includes the texture complexity of the residual of the inter-frame prediction sub-macroblock and the reference area when the inter-frame prediction sub-macroblock performs inter-frame prediction, and the second texture complexity includes the texture complexity of the intra-frame prediction residual or the pixel value of the intra-frame prediction sub-macroblock itself; The step of determining the first attribute comprises: Determine a first texture complexity of an inter-frame prediction sub-macroblock and a second texture complexity of an intra-frame prediction sub-macroblock within a node; The first attribute is obtained by calculating the sum of the first texture complexity of the inter-frame prediction sub-macroblock and the second texture complexity of the intra-frame prediction sub-macroblock in the determined node and dividing the sum by the second attribute of the node.

4. The method for determining anchor blocks in super-resolution enhancement according to claim 3, characterized in that: The attributes of the directed edge include a weight attribute, and the weight attribute is set by the following steps: Based on the motion vector, reference index, and multi-reference frame weighted prediction weight of the macroblock extracted from the video encoding of the video to be super-resolution enhanced, determine, as the weight attribute, a pixel ratio in the video block corresponding to the destination node of the directed edge for inter-frame prediction with reference to the pixel value of the video block corresponding to the source node of the directed edge; Or, an optical flow algorithm or a motion estimation algorithm is used between video frames associated with directed edges to estimate the pixel ratio of the video block corresponding to the destination node of the directed edge for inter-frame prediction with reference to the pixel value of the video block corresponding to the source node of the directed edge, as the weight attribute.

5. The method for determining anchor blocks in super-resolution enhancement according to claim 2, characterized in that: The determining whether the video block is an anchor block based on the constructed directed acyclic graph includes: The following anchor block determination steps are performed until the preset end condition is reached: Selecting video blocks to be added to the anchor block set according to a preset search strategy, so that the estimated super-resolution defect of the entire video is quickly reduced, wherein the super-resolution defect of the entire video is the sum of the super-resolution defects of the video blocks outside the anchor block set, and the super-resolution defect of the video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels of the video block, and the average super-resolution quality defect per pixel of the video block is determined by the following steps: According to the decoding order, target video frames are selected from the video frames in turn, and the average per-pixel super-resolution quality defect of the video block in the target video frame is determined according to whether the video block in the target video frame is an anchor block or a non-anchor block, the first attribute of the node corresponding to the video block in the target video frame, and the quality defect transmitted through the directed edge; the quality defect attribute transmitted through the directed edge is determined based on the product of the average per-pixel super-resolution quality defect of the video block corresponding to the source node of the directed edge and the weight attribute of the directed edge.

6. The method for determining anchor blocks in super-resolution enhancement according to claim 1, characterized in that: The anchor block performs super-resolution enhancement based on a pre-built super-resolution neural network.

7. The method for determining anchor blocks in super-resolution enhancement according to claim 2, characterized in that: The determining whether the video block is an anchor block based on the constructed directed acyclic graph includes: Sorting the video blocks corresponding to the nodes according to the first attributes of the nodes in the directed acyclic graph to obtain a video block sorting result; Determine whether the video block is an anchor block according to the video block sorting result.

8. The method for determining anchor blocks in super-resolution enhancement according to claim 2, characterized in that: The determining whether the video block is an anchor block based on the constructed directed acyclic graph includes: Based on a pre-built node sorting model of a graph neural network, sort the nodes of the directed acyclic graph to obtain a node sorting result; It is determined whether the video block is an anchor block based on the node sorting result.

9. A device for determining an anchor block in super-resolution enhancement, characterized in that: include: An acquisition module, configured to acquire a video frame of a video to be super-resolution enhanced; a segmentation module configured to segment the video frame to obtain video blocks, wherein the video blocks do not overlap in space and are combined to cover a spatial area of ​​the video frame; a division module configured to divide the macroblock of the video frame into sub-macroblocks according to the overlap between the macroblock and the video block, wherein the sub-macroblock overlaps only one of the video blocks, and two sub-macroblocks do not overlap with the same video block; The determination module is configured to determine whether the video block is an anchor block according to the texture complexity of the sub-macroblock in the video block, and the anchor block is not super-resolution-enhanced based on the super-resolution version of the multiplexed reference area.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for determining an anchor block in super-resolution enhancement according to any one of claims 1 to 8 is implemented.

Images