Method and device for determining anchor block in super-resolution promotion, and electronic equipment

Abstract

The present application relates to the technical field of computer vision, and provides a method and device for determining anchor point blocks in super-resolution enhancement and electronic equipment, the method comprising: obtaining a video frame of a video to be subjected to super-resolution enhancement; segmenting the video frame to obtain video blocks, the video blocks being non-overlapping in space and combining to cover the spatial region of the video frame; dividing macroblocks of the video frame into sub-macroblocks according to the overlapping condition of the macroblocks and the video blocks, the sub-macroblocks only having overlap with one video block; determining whether the video block is an anchor point block according to the texture complexity of the sub-macroblocks in the video block, the anchor point block not being subjected to super-resolution enhancement in the manner of super-resolution version of a reused reference region. In the method, the video block can independently select whether to be subjected to super-resolution enhancement in the manner of super-resolution version of a reused reference region from other video blocks in the same video frame, and the super-resolution task overhead is further optimized through finer decision granularity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a method and device for determining anchor point blocks in super-resolution enhancement and an electronic device. BACKGROUND

[0002] Super-resolution neural networks have multiple uses, such as running a super-resolution neural network on a server side to update a low-resolution old video into a high-resolution video, and when a client side with adaptive bitrate function senses a slow network speed and downloads a low-resolution version of a video, using the computing power of the client device to run a super-resolution neural network to restore the downloaded video to a high resolution. However, using a super-resolution neural network to process each frame of a video will result in a very high computational load, which causes a high computing power cost overhead when running a super-resolution neural network on a server side, and a high energy consumption and low battery life problem when running a super-resolution neural network on an end device, and a problem that a weak computing power end device cannot perform real-time super-resolution neural network inference to enhance the resolution in real time.

[0003] At present, the super-resolution neural network overhead optimization scheme uses the redundancy of video frames, that is, only a part of the frames in the video (referred to as anchor frames) are processed by a super-resolution neural network or other independent super-resolution algorithm to improve the resolution (referred to as neural network-based super-resolution), and the remaining frames (referred to as non-anchor frames) are multiplexed to restore the super-resolution version of the reference frame to improve the resolution (referred to as multiplex-based super-resolution). The multiplex-based super-resolution method usually has lower computational overhead than the independent super-resolution algorithm based on a neural network or the like, and the more redundant the content of a video frame is compared to its reference frame in the time dimension, the closer the multiplex-based super-resolution method is to the independent super-resolution algorithm based on a neural network or the like in terms of restoration quality. By using a suitable anchor frame selection algorithm, the super-resolution versions of all frames (including anchor frames and non-anchor frames) can have very high quality while only a small number of frames are selected as anchor frames. Compared to using a neural network to perform super-resolution enhancement on all frames, this scheme can greatly optimize the average processing overhead of all frames without significantly sacrificing the quality of the super-resolution video.

[0004] However, there may still be some areas in an anchor frame that can be improved in resolution with high quality using a multiplexing method, and there may also be some areas in a non-anchor frame that have a low quality high-resolution version obtained by processing using a multiplexing method and are more suitable for using a neural network to restore their resolution. SUMMARY

[0005] The application provides a method and device for determining anchor blocks in super-resolution enhancement, and electronic equipment, to solve the defects in the prior art that there may still be some areas in an anchor frame that can be enhanced in high quality by multiplexing, and there may also be some areas in a non-anchor frame that are low in quality after being processed by multiplexing and are more suitable for recovering their resolution by a neural network, etc., so that each video frame is divided into video blocks, and the specific super-resolution method is determined based on the video blocks rather than the frames. By adopting a finer decision granularity at the block level, the total anchor area required is reduced under the premise of achieving the same or even higher super-resolution video quality, so as to further optimize the overall super-resolution overhead.

[0006] The application provides a method for determining anchor blocks in super-resolution enhancement, comprising: acquiring a video frame of a video to be subjected to super-resolution enhancement; segmenting the video frame to obtain video blocks, the video blocks being spatially non-overlapping and combining to cover the spatial area of the video frame; dividing macro blocks of the video frame into sub-macro blocks according to the overlap of the macro blocks and the video blocks, the sub-macro blocks only having overlap with one video block, and two sub-macro blocks not having overlap with the same video block; and determining whether the video block is an anchor block according to the texture complexity of the sub-macro blocks in the video block, the anchor block not being subjected to super-resolution enhancement based on the super-resolution version of the multiplexed reference area.

[0007] According to the method for determining anchor blocks in super-resolution enhancement provided by the application, the macro blocks comprise inter-frame prediction macro blocks, and the sub-macro blocks comprise inter-frame prediction sub-macro blocks, and the determination of whether the video block is an anchor block according to the texture complexity of the sub-macro blocks in the video block comprises: determining whether the video block is an anchor block based on a constructed directed acyclic graph, one node in the directed acyclic graph representing one video block, the destination node of the directed edge connecting the node corresponding to a video block in which there is at least one inter-frame prediction sub-macro block, the pixel value of the video block corresponding to the source node of the directed edge being used for inter-frame prediction encoding, the attribute of the node comprising a first attribute, the first attribute representing whether the video block corresponding to the node can be subjected to super-resolution enhancement based on the super-resolution version of the multiplexed reference area, and the first attribute being determined according to the texture complexity of the sub-macro blocks in the video block.

[0008] According to the method for determining anchor point block in super-resolution promotion provided by the application, the attribute of the node further comprises a second attribute, the second attribute represents the area of the video block corresponding to the node, the macro block further comprises an intra prediction macro block, the sub-macro block further comprises an intra prediction sub-macro block, the texture complexity of the sub-macro block comprises a first texture complexity of the inter prediction sub-macro block and a second texture complexity of the intra prediction sub-macro block, the first texture complexity comprises the texture complexity of the residual of the area referenced by the inter prediction sub-macro block when the inter prediction sub-macro block and the inter prediction sub-macro block perform inter prediction, and the second texture complexity comprises the texture complexity of the intra prediction residual or the pixel value of the intra prediction sub-macro block itself; the determination of the first attribute comprises: determining the first texture complexity of the inter prediction sub-macro block in the node and the second texture complexity of the intra prediction sub-macro block; and calculating the sum of the determined first texture complexity of the inter prediction sub-macro block in the node and the second texture complexity of the intra prediction sub-macro block, and dividing the second attribute of the node to obtain the first attribute.

[0009] According to the method for determining anchor point block in super-resolution promotion provided by the application, the attribute of the node further comprises a second attribute, the second attribute represents the area of the video block corresponding to the node, the macro block further comprises an intra prediction macro block, the sub-macro block further comprises an intra prediction sub-macro block, the texture complexity of the sub-macro block comprises a first texture complexity of the inter prediction sub-macro block and a second texture complexity of the intra prediction sub-macro block, the first texture complexity comprises the texture complexity of the residual of the area referenced by the inter prediction sub-macro block when the inter prediction sub-macro block and the inter prediction sub-macro block perform inter prediction, and the second texture complexity comprises the texture complexity of the intra prediction residual or the pixel value of the intra prediction sub-macro block itself; the determination of the first attribute comprises: determining the first texture complexity of the inter prediction sub-macro block in the node and the second texture complexity of the intra prediction sub-macro block; and calculating the sum of the determined first texture complexity of the inter prediction sub-macro block in the node and the second texture complexity of the intra prediction sub-macro block, and dividing the second attribute of the node to obtain the first attribute.

[0010] According to the anchor point block determination method in the super-resolution promotion provided by the application, whether the video block is an anchor point block is determined based on the constructed directed acyclic graph, and the following anchor point block determination steps are performed until a preset ending condition is reached: a video block is selected according to a preset search strategy and added to an anchor point block set, so that the super-resolution defect of the video whole is rapidly reduced, wherein the super-resolution defect of the video whole is the sum of the super-resolution defects of the video blocks outside the anchor point block set, the super-resolution defect of a video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels of the video block, and the average super-resolution quality defect per pixel of the video block is determined by the following steps: a target video frame is selected from the video frame in sequence according to the decoding order, and the average super-resolution quality defect per pixel of the video block in the target video frame is determined according to whether the video block in the target video frame is an anchor point block or a non-anchor point block, the first attribute of the node corresponding to the video block in the target video frame and the quality defect conducted through the directed edge; and the quality defect conducted through the directed edge is determined based on the product of the average super-resolution quality defect per pixel of the video block corresponding to the source node of the directed edge and the weight attribute of the directed edge.

[0011] According to the anchor point block determination method in the super-resolution promotion provided by the application, whether the video block is an anchor point block is determined based on the constructed directed acyclic graph, and the following anchor point block determination steps are performed until a preset ending condition is reached: a video block is selected according to a preset search strategy and added to an anchor point block set, so that the super-resolution defect of the video whole is rapidly reduced, wherein the super-resolution defect of the video whole is the sum of the super-resolution defects of the video blocks outside the anchor point block set, the super-resolution defect of a video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels of the video block, and the average super-resolution quality defect per pixel of the video block is determined by the following steps: a target video frame is selected from the video frame in sequence according to the decoding order, and the average super-resolution quality defect per pixel of the video block in the target video frame is determined according to whether the video block in the target video frame is an anchor point block or a non-anchor point block, the first attribute of the node corresponding to the video block in the target video frame and the quality defect conducted through the directed edge; and the quality defect conducted through the directed edge is determined based on the product of the average super-resolution quality defect per pixel of the video block corresponding to the source node of the directed edge and the weight attribute of the directed edge.

[0012] According to the anchor point block determination method in the super-resolution promotion provided by the application, whether the video block is an anchor point block is determined based on the constructed directed acyclic graph, and the following anchor point block determination steps are performed until a preset ending condition is reached: a video block is selected according to a preset search strategy and added to an anchor point block set, so that the super-resolution defect of the video whole is rapidly reduced, wherein the super-resolution defect of the video whole is the sum of the super-resolution defects of the video blocks outside the anchor point block set, the super-resolution defect of a video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels of the video block, and the average super-resolution quality defect per pixel of the video block is determined by the following steps: a target video frame is selected from the video frame in sequence according to the decoding order, and the average super-resolution quality defect per pixel of the video block in the target video frame is determined according to whether the video block in the target video frame is an anchor point block or a non-anchor point block, the first attribute of the node corresponding to the video block in the target video frame and the quality defect conducted through the directed edge; and the quality defect conducted through the directed edge is determined based on the product of the average super-resolution quality defect per pixel of the video block corresponding to the source node of the directed edge and the weight attribute of the directed edge.

[0013] According to the anchor point block determination method in the super-resolution promotion provided by the application, whether the video block is an anchor point block is determined based on the constructed directed acyclic graph, and the following anchor point block determination steps are performed until a preset ending condition is reached: a video block is selected according to a preset search strategy and added to an anchor point block set, so that the super-resolution defect of the video whole is rapidly reduced, wherein the super-resolution defect of the video whole is the sum of the super-resolution defects of the video blocks outside the anchor point block set, the super-resolution defect of a video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels of the video block, and the average super-resolution quality defect per pixel of the video block is determined by the following steps: a target video frame is selected from the video frame in sequence according to the decoding order, and the average super-resolution quality defect per pixel of the video block in the target video frame is determined according to whether the video block in the target video frame is an anchor point block or a non-anchor point block, the first attribute of the node corresponding to the video block in the target video frame and the quality defect conducted through the directed edge; and the quality defect conducted through the directed edge is determined based on the product of the average super-resolution quality defect per pixel of the video block corresponding to the source node of the directed edge and the weight attribute of the directed edge.

[0014] The application further provides a device for determining anchor point blocks in super-resolution enhancement, comprising: an acquisition module configured to acquire a video frame of a video to be subjected to super-resolution enhancement; a segmentation module configured to segment the video frame to obtain video blocks, the video blocks being spatially non-overlapping and combining to cover a spatial region of the video frame; a division module configured to divide macro blocks of the video frame into sub-macro blocks according to an overlap condition of the macro blocks and the video blocks, the sub-macro blocks only having overlap with one of the video blocks and two of the sub-macro blocks not having overlap with the same video block; and a determination module configured to determine whether the video blocks are anchor point blocks according to texture complexity of the sub-macro blocks in the video blocks, the anchor point blocks not being subjected to super-resolution enhancement in a manner of multiplexing super-resolution versions of reference regions.

[0015] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the method for determining anchor point blocks in super-resolution enhancement when executing the computer program.

[0016] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executable on a processor to implement the method for determining anchor point blocks in super-resolution enhancement.

[0017] The application further provides a computer program product comprising a computer program, the computer program being executable on a processor to implement the method for determining anchor point blocks in super-resolution enhancement.

[0018] The application provides a method, device and electronic device for determining anchor point blocks in super-resolution enhancement, which segments a video frame to obtain video blocks, divides macro blocks of the video frame into sub-macro blocks according to an overlap condition of the macro blocks and the video blocks, and finally determines whether the video blocks are anchor point blocks according to texture complexity of the sub-macro blocks in the video blocks, the anchor point blocks not being subjected to super-resolution enhancement in a manner of multiplexing super-resolution versions of reference regions. Compared with the existing scheme of selecting a processing mode in units of frames (i.e., anchor point frames or non-anchor point frames are subjected to super-resolution enhancement in different manners), the application accurately finds out anchor point blocks that most need to use computational overhead through a finer decision granularity, and only needs to use a multiplexing manner with relatively smaller overhead to perform super-resolution enhancement on other video regions, thereby reducing overall overhead of super-resolution enhancement while ensuring overall effect of super-resolution enhancement. BRIEF DESCRIPTION OF DRAWINGS

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the method for determining anchor blocks in super-resolution enhancement provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the process for determining anchor blocks in super-resolution enhancement provided by the present invention.

[0022] Figure 3 This is a schematic diagram of a video frame in the method for determining anchor blocks in super-resolution enhancement provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the structure of the device for determining anchor points in super-resolution enhancement provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] The terminology involved in this invention will be briefly explained below.

[0027] Video: A continuous sequence of images, consisting of consecutive frames, with each frame being an image.

[0028] Video encoding: A class of techniques for compressing video files. Existing video encoding technologies can typically significantly reduce the size of video files without significantly reducing the similarity between the encoded video and the original unencoded video, thus enabling users to download and play videos in real time even with limited network speeds. Frames that are temporally close within a video often exhibit strong redundancy (for example, in a selfie video, the subject's facial information appears repeatedly in different frames, causing very strong temporal redundancy), and there is also spatial redundancy within the same frame (for example, if a blue sky appears in a single video frame, multiple pixels reflect the blue color of that sky, and the values ​​of these pixels are very close and redundant). Video encoding primarily compresses video files by eliminating temporal and spatial redundancy within the unencoded video.

[0029] Reference frames: In order to compress a frame and eliminate redundant information in the temporal dimension, video coding technology selects several other frames as reference frames for the current video frame and tries to find out which information in the current video frame has already appeared redundantly in its reference frames. The encoded video only stores the information that appears in the current video frame, so that the encoded frame has very low redundancy and a very small file size.

[0030] Macroblocks, motion vectors, reference frame indexes, and residuals: Video coding techniques typically divide a video frame spatially into multiple sub-regions called macroblocks. If a macroblock is temporally redundant, the video coding technique will find its approximate region in the reference frame. The approximate region can be used as the prediction value of the current macroblock. The encoded video does not need to store all the pixel values ​​of the macroblock, but only needs to store the spatial displacement of the macroblock relative to the approximate region (i.e., motion vector), which reference frame the approximate region belongs to (i.e., reference frame index), and the difference between the original value and the predicted value of the macroblock (i.e., residual). This type of macroblock coding is called inter-frame prediction. If a macroblock is spatially redundant, the video coding technique will obtain the prediction value of the macroblock based on the pixel values ​​of macroblocks near the macroblock in the same frame. The encoded video does not need to store all the pixel values ​​of the macroblock, but only needs to store information such as the difference between the original value and the predicted value of the macroblock (i.e., residual). This type of macroblock coding is called intra-frame prediction.

[0031] Super-resolution neural network: A type of neural network that takes a low-resolution image and possible auxiliary information as input, and outputs a high-resolution version of the image and possible additional information.

[0032] Image interpolation: In mathematics, interpolation refers to estimating the approximate value of a function at other points by using the values ​​of the function at a finite number of points. Image interpolation can be used to restore high resolution from low-resolution video frames. Generally speaking, the computational complexity and restoration quality of image interpolation are lower than those of super-resolution neural networks.

[0033] Adaptive bitrate: When downloading and playing videos in real time over a network, the user's device network speed will constantly change. To adapt to this change, video files are usually converted into several versions, some with high resolution and others with low resolution. When the user's client has a faster network speed, it automatically switches to downloading the high-resolution version to ensure that the user can see a clearer picture. When the network speed is slow, it automatically switches to downloading the low-resolution version to prevent the download speed from being slower than the playback speed, which would cause stuttering during playback.

[0034] Greedy search is an algorithmic strategy used to solve optimization problems. At each step, it makes what appears to be the best choice at that moment, hoping to reach the global optimum through a series of locally optimal solutions. This strategy does not guarantee a global optimum, but it is typically simple and fast.

[0035] The following is combined with Figures 1-5 The present invention describes a method, apparatus, and electronic device for determining anchor blocks in super-resolution enhancement.

[0036] Figure 1 This is one of the flowcharts illustrating the method for determining anchor blocks in super-resolution enhancement provided by the present invention, such as... Figure 1 As shown, the method includes the following:

[0037] Step 101: Obtain video frames of the video to be super-resolution upscaled.

[0038] In this embodiment, super-resolution enhancement can be achieved by using a super-resolution version based on a reused reference region or by other methods, such as using a super-resolution neural network.

[0039] Step 102: Segment the video frames to obtain video blocks.

[0040] In this embodiment, the video blocks do not overlap spatially and are combined to cover the spatial region of the video frame. The specific segmentation method of the video blocks can be determined based on the complexity and content characteristics of the video to be super-resolution upgraded, as well as the system's computing power.

[0041] Step 103: Divide the macroblock into sub-macroblocks based on the overlap between the macroblock and video block of the video frame.

[0042] In this embodiment, a sub-macroblock overlaps with only one video block, and two sub-macroblocks do not overlap with the same video block. When a macroblock overlaps with only one video block, that macroblock itself is a sub-macroblock. Through this division, each video block contains one or more sub-macroblocks from different macroblocks.

[0043] Step 104: Determine whether a video block is an anchor block based on the texture complexity of the sub-macroblocks within the video block.

[0044] In this embodiment, the texture complexity of a sub-macroblock can be determined by calculating the Euclidean norm, variance, and mean absolute deviation of the error after downsampling and resampling. Anchor blocks are not super-resolution upscaled based on a super-resolution version of the reference region. Instead, anchor blocks can be processed using independent super-resolution algorithms, rather than relying on the reuse of the reference frame. As an example, deep learning models can be used to upscale anchor blocks, and these models can learn a low-resolution to high-resolution mapping.

[0045] In addition, video blocks can be sorted directly based on the texture complexity of sub-macroblocks within a video block to determine whether a video block is an anchor block. Alternatively, a directed acyclic graph can be built based on the texture complexity of sub-macroblocks within a video block to determine whether a video block is an anchor block. Graph neural networks can also be used to determine whether a video block is an anchor block.

[0046] The anchor block determination method for super-resolution enhancement provided by this invention obtains video blocks by segmenting video frames, and divides macroblocks into sub-macroblocks based on the overlap between macroblocks in the video frames and the video blocks. Finally, it determines whether a video block is an anchor block based on the texture complexity of the sub-macroblocks within the video block. This fine-grained super-resolution method further optimizes the overhead of the super-resolution task. The anchor block determination method provided by this invention does not depend on the original high-resolution version of the video, and can be applied to application scenarios where the original high-resolution version of the video is unavailable or cannot be obtained simultaneously with the low-resolution version (such as enhancing old videos). The anchor block determination method provided by this invention has very low latency, which can meet the requirements of applications with low latency requirements, such as live streaming.

[0047] In some optional implementations, macroblocks include inter-frame prediction macroblocks, and sub-macroblocks include inter-frame prediction sub-macroblocks. Determining whether a video block is an anchor block based on the texture complexity of the sub-macroblocks within the video block includes: determining whether a video block is an anchor block based on a constructed directed acyclic graph (DAG). In the DAG, each node represents a video block, and the video block corresponding to the destination node of the directed edge connecting the nodes contains at least one inter-frame prediction sub-macroblock. Inter-frame prediction coding is performed with reference to the pixel values ​​of the video blocks corresponding to the source nodes of the directed edges. Node attributes include a first attribute, which characterizes whether the video block corresponding to the node can effectively perform super-resolution enhancement based on the super-resolution version of the reused reference region. The first attribute is determined based on the texture complexity of the sub-macroblocks within the video block. The first attribute can be determined directly based on the texture complexity of the sub-macroblocks within the video block, or factors such as the area of ​​the video block can be further considered to improve the accuracy of the determined first attribute. The generation and transmission process of super-resolution video quality defects can be accurately modeled using the DAG data structure. This implementation further improves the accuracy of anchor block determination.

[0048] See Figure 2 , Figure 2 This is a schematic diagram of the process for determining anchor blocks in super-resolution upgrading provided by the present invention. The low-resolution video, i.e., the video to be super-resolution upgraded, is input to the first sub-module of the device, the directed acyclic graph construction code sub-module. This sub-module will process each video block in the low-resolution video... Create a graph node. Consider two video blocks (the corresponding nodes are denoted as follows). and ), if and only if there exists at least one The internal inter-frame prediction sub-macroblock references Predicting pixel values ​​in a directed acyclic graph (DAG) is performed, and a path exists from... Departure, direction directed edges Therefore, there exists a path from Departure, direction Directed edge The necessary condition is that it contains The video frame contains The reference frame of the video frame. Since the reference relationship between frames is acyclic in common video coding standards, the constructed graph is a directed acyclic graph. Setting the attribute values ​​of nodes and edges in the graph requires obtaining internal coding information of the low-resolution encoded video, such as residuals and motion vectors. The directed acyclic graph construction code submodule can add code to the existing video decoder software, using this information extracted from the encoded video by the decoder to calculate the attribute values ​​of nodes and edges.

[0049] Each node in a directed acyclic graph It includes a static first property, the name of which can be a non-reusability property. This non-reusability attribute reflects whether the node can be well processed using a reuse-based super-resolution approach. To set the non-reusability attribute... The value of needs to be calculated to determine the texture complexity of the sub-macroblocks contained in the video block corresponding to that node.

[0050] In some optional implementations, the node's attributes also include a second attribute, which characterizes the area of ​​the video block corresponding to the node. Macroblocks also include intra-prediction macroblocks, and sub-macroblocks also include intra-prediction sub-macroblocks. The texture complexity of a sub-macroblock includes a first texture complexity of the inter-prediction sub-macroblock and a second texture complexity of the intra-prediction sub-macroblock. The first texture complexity includes the texture complexity of the residual of the reference region when the inter-prediction sub-macroblock performs inter-frame prediction with the inter-prediction sub-macroblock. The second texture complexity includes the texture complexity of the intra-prediction residual or the pixel value of the intra-prediction sub-macroblock itself. The steps for determining the first attribute include: determining the first texture complexity and the second texture complexity of the inter-prediction sub-macroblock within the node; calculating the sum of the determined first and second texture complexities of the inter-prediction sub-macroblock within the node, and dividing by the node's second attribute to obtain the first attribute. The decoder extracts this information from the encoded video to calculate the attribute values ​​of nodes and edges. The second attribute can be named the area attribute, i.e. .

[0051] First, we introduce a reuse-based super-resolution enhancement method:

[0052] For macroblocks predicted by any inter-frame in low-resolution video Its pixel value can be denoted as The region referenced when this macroblock performs inter-frame prediction is denoted as The residual is recorded as Then there is

[0053]

[0054] Use symbols Let 'a' represent an image amplified by a factor of 'b' through interpolation, and 'scale' represent the ratio of the super-resolution video's resolution to the low-resolution video's resolution. Then, when using multiplexing-based super-resolution... The super-resolution version is

[0055]

[0056] in, This represents the super-resolution version of the reference region for inter-frame prediction.

[0057] For any intra-frame predicted macroblock in a low-resolution video Its pixel value can be denoted as Then, using a reuse-based super-resolution method The super-resolution version is

[0058]

[0059] The texture complexity of the sub-macroblocks contained in a video block can be predicted based on all its inter-frame sub-macroblocks. residual The texture complexity is determined, and the higher the complexity, the more the residual is treated according to formula (2) mentioned above. interpolation result The lower the quality, the lower the super-resolution version of the sub-macroblock. quality compared to The greater the quality difference, the less suitable the sub-macroblock is for processing using multiplexing-based super-resolution methods; at the same time, it will also calculate all intra-predicted sub-macroblocks contained in the video block corresponding to that node. Low-resolution version The texture complexity, the higher the complexity, the more it is determined according to formula (3) above. Interpolated super-resolution version The worse the quality, the more complex the texture complexity of the residuals of all inter-frame predicted sub-macroblocks contained in the video block corresponding to the node, and the sum of the texture complexity of all intra-frame predicted sub-macroblocks, divided by the area attribute of the node, which is the value of the unreusability attribute of the node.

[0060] To calculate a two-dimensional array (specifically may be) or The specific method adopted in this invention is to reduce the texture complexity of the texture. (Calculate the L2 norm (also known as the Euclidean norm) of the error after downsampling and then resampling), where It can be any real number greater than 1.

[0061] Alternatively, other methods can be used for calculation. or The texture complexity can be reduced by methods such as calculating variance and mean absolute deviation. Even with the original high-resolution video available, the interpolated texture can still be improved. or The difference is compared with the relevant values ​​in the original high-resolution video and used as a value for texture complexity.

[0062] In the above multiplexing method, the macroblocks of intra-frame predictive coding are processed by interpolation, as shown in formula (3). An alternative multiplexing method is to scale the residual of intra-frame coding and add it to the predicted value. If this alternative multiplexing method is adopted, it is not necessary to calculate the predicted value when constructing the directed acyclic graph. The texture complexity can be reduced, and the computation can be performed on low-resolution videos. The texture complexity of the residual. In fact, even if we still use formula (3) to demultiplex the macroblocks of intra-frame predictive coding, using Residual texture complexity is also a viable alternative, although its effect on selecting anchor blocks may not be as good as... Texture complexity.

[0063] Each edge in a directed acyclic graph It can have a static weight property. This reflects the video block corresponding to the destination node of the edge (denoted as ). If multiplexing-based super-resolution processing is used, what percentage of pixels within a block reuse the source node of the edge corresponding to the video block (denoted as...)? The pixels within ) are used to set weight values. These need to be extracted from the encoded low-resolution video. The motion vector information of each inter-frame prediction sub-macroblock within the corresponding video block is used to determine the references for each inter-frame prediction sub-macroblock based on the motion vector information. The prediction is performed on a number of pixels within the corresponding video block (if multiple reference frames are used for weighted prediction, the number of pixels needs to be multiplied by the corresponding weights), and the sum of these reference pixel counts is divided by [the specified number]. The result of the division is the weight of this edge. According to formula (2), the super-resolution version of the sub-macroblock is utilized when using the reused super-resolution. The quality is affected by the super-resolution version of the reference area. The impact on quality, Quality defects will spread to The purpose of using weighted attributes is to quantify the propagation of quality defects at the sub-macroblock scale into the proportion of quality defect propagation at the video block scale.

[0064] In addition, other methods can be used to calculate the weight attributes of the edges, such as calculating the optical flow between frames or using motion estimation algorithms such as diamond search to obtain the regions that may be referenced when performing inter-frame predictive coding for each video block. The advantage of this approach is that it does not require adding code to the decoder software to parse motion vectors, weighted prediction weights for multi-reference frame prediction, and other information.

[0065] Figure 3This is a schematic diagram of a video frame in the method for determining anchor blocks in super-resolution enhancement provided by the present invention. Figure 3 The image shows two consecutive frames from a low-resolution video. , Assuming yes The only reference frame, and the frame Pre-divided into four video blocks , , , , will frame Pre-divided into four video blocks , , , . It is a video block The only inter-frame predictive coded sub-macroblock within the video block, and all other video blocks. It also contains multiple intra-frame predictive coding sub-macroblocks. Based on the above, video block Corresponding node Attribute value equals .because and , They all overlap, and with , The areas of the overlapping regions are respectively The area is 0.105 and 0.323 times, so there exists a [missing information - likely a number]. Starting from and pointing to the corresponding node The edge corresponding to the node has a weight of 0.105, and there also exists from... Starting from and pointing to the corresponding node The edge corresponding to the node has a weight of 0.323.

[0066] In some optional implementations, determining whether a video block is an anchor block based on the constructed directed acyclic graph includes: performing the following anchor block determination steps until a preset termination condition is met: selecting video blocks to add to the anchor block set according to a pre-set search strategy, thereby rapidly reducing the estimated overall super-resolution defect of the video, where the overall super-resolution defect of the video is the sum of the super-resolution defects of video blocks outside the anchor block set, and the super-resolution defect of a video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels in the video block. The average super-resolution quality defect per pixel of the video block is determined by the following steps: selecting target video frames sequentially from the video frames according to the decoding order; determining the average super-resolution quality defect per pixel of the video blocks in the target video frame based on whether the video blocks in the target video frame are anchor blocks or non-anchor blocks, the first attribute of the node corresponding to the video block in the target video frame, and the quality defect propagated through the directed edge; and determining the quality defect attribute propagated through the directed edge based on the product of the average super-resolution quality defect per pixel of the video block corresponding to the source node of the directed edge and the weight attribute of the directed edge.

[0067] See also Figure 2 The second submodule is the anchor frame selection code submodule based on directed acyclic graphs (DAGs). This module utilizes DAG-based super-resolution video quality estimation to select a set of anchor blocks that result in low estimated quality defects. The selection can employ a greedy algorithm or other arbitrary search strategies. The key here is how to perform quality estimation for super-resolution video based on DAGs. Given a constructed DAG and a set of anchor blocks, the specific method for super-resolution video quality estimation is as follows:

[0068] Step 1: Sort the video frames according to the decoding order of the low-resolution video frames, and denot them as follows: Following this order, step 2 below is used to estimate the average super-resolution quality defect per pixel for each video frame.

[0069] Step 2, for frames Whether each video block within it belongs to a given set of anchor blocks can be determined using a vector of 0s and 1s. To represent (0 indicates belonging to the anchor block set, 1 indicates not belonging to the anchor block set), frame The non-reusability and area attributes of the corresponding nodes of the internal video block can form two other vectors. , Frame recording The average super-resolution quality defect attribute per pixel of the internal video block is The reference frame for this frame is , ,…,connect Internal video block corresponding nodes and The weights of the edges corresponding to the nodes of the inner video block are combined into a weight matrix. (if A certain video block and If a video block within a given area has no edge connections and the corresponding element in the weight matrix is ​​0, then we have: ,in This represents element-wise multiplication. When the weight matrix... When a large number of elements are 0, multiplication for sparse matrices can be used to calculate... .

[0070] Step 3: Calculate the super-resolution quality defects of each video block within all frames. , After that, the overall super-resolution quality defect of the video is estimated to be... superscript Indicates to Transpose, and the mass can be used To characterize.

[0071] To reduce the latency of super-resolution video quality estimation based on directed acyclic graphs (DAGs) and better support latency-sensitive applications such as live video streaming, super-resolution video quality estimation can be performed in parallel across multiple anchor block sets (which can be considered as a batch of anchor block sets). Specifically, in steps 2 and 3 above, the super-resolution video quality estimation is performed... , Add an extra batch dimension to each, making , Transform a one-dimensional vector into a two-dimensional tensor.

[0072] In some optional implementations, the anchor block performs super-resolution enhancement based on a pre-built super-resolution neural network. Super-resolution enhancement using a super-resolution neural network can better capture image features and improve super-resolution quality. The pre-built super-resolution neural network can include EDSR (Enhanced Deep Super-Resolution), SRGAN (Super-Resolution Generative Adversarial Network), and RCAN (Residual Channel Attention Network), etc., and this application does not limit the specific implementation.

[0073] In some optional implementations, determining whether a video block is an anchor block based on the constructed directed acyclic graph includes: sorting the video blocks corresponding to the nodes according to the first attribute of the nodes in the directed acyclic graph to obtain the video block sorting result; and determining whether a video block is an anchor block based on the video block sorting result.

[0074] In some optional implementations, determining whether a video block is an anchor block is based on a constructed directed acyclic graph (DAG). This includes: sorting the nodes of the DAG based on a pre-built graph neural network node ranking model to obtain the node ranking result; and determining whether a video block is an anchor block based on the node ranking result. A graph neural network (GNN) is a type of neural network capable of processing graph-structured data and learning the relationships between nodes. A graph neural network can be trained using pre-prepared sample data to obtain a model capable of ranking nodes.

[0075] In this embodiment, multiple anchor selection methods can also be combined. For example, some candidate anchor blocks can be selected based on the non-reusability attribute value of the node, and then the final set of anchor blocks can be selected from the candidate anchor blocks using super-resolution video quality estimation based on directed acyclic graph.

[0076] This invention utilizes multiple publicly available videos from the internet for effect verification, confirming that the method of this invention can select suitable anchor blocks with very low latency. The verification method involves converting the original high-resolution version of the video to a low-resolution version, and then applying the anchor block selection algorithm proposed in this invention and existing anchor frame selection algorithms to select anchor regions. Experimental results show that the selection latency of the anchor block selection algorithm proposed in this invention is on the order of tens of milliseconds, making it applicable to low-latency scenarios such as live streaming. Compared to existing anchor frame selection algorithms, which are also applicable to low-latency scenarios such as live streaming, the anchor block selection algorithm of this invention can reduce the area of ​​the anchor region by an average of approximately 38%, and at most approximately 80%, without reducing the quality of the super-resolution video (by comparing it with the original high-resolution version to calculate PSNR and measure quality).

[0077] Figure 4 This is a schematic diagram of the structure of the device for determining anchor blocks in super-resolution enhancement provided in the embodiments of this application, as shown below. Figure 4As shown, the specific components include: an acquisition module 401, configured to acquire video frames of the video to be super-resolution upscaled; a segmentation module 402, configured to segment the video frames into video blocks, wherein the video blocks do not overlap spatially and are combined to cover the spatial region of the video frames; a partitioning module 403, configured to divide macroblocks into sub-macroblocks based on the overlap between macroblocks of the video frames and video blocks, wherein a sub-macroblock overlaps with only one video block and two sub-macroblocks do not overlap with the same video block; and a determination module 404, configured to determine whether a video block is an anchor block based on the texture complexity of the sub-macroblocks within the video block, wherein anchor blocks are not super-resolution upscaled based on the super-resolution version of the reused reference region.

[0078] In some optional implementations, macroblocks include inter-frame prediction macroblocks, sub-macroblocks include inter-frame prediction sub-macroblocks, and the determination module 404 is further configured to: determine whether a video block is an anchor block based on a constructed directed acyclic graph (DAG). In the DAG, a node represents a video block, and the video block corresponding to the destination node of the directed edge connecting the node contains at least one inter-frame prediction sub-macroblock. Inter-frame prediction coding is performed with reference to the pixel values ​​of the video block corresponding to the source node of the directed edge. The node's attributes include a first attribute, which characterizes whether the video block corresponding to the node can be effectively upscaled based on the super-resolution version of the reused reference region. The first attribute is determined based on the texture complexity of the sub-macroblocks within the video block.

[0079] In some optional implementations, the node's attributes also include a second attribute, which represents the area of ​​the video block corresponding to the node. The macroblock also includes an intra-prediction macroblock, and the sub-macroblock also includes an intra-prediction sub-macroblock. The texture complexity of the sub-macroblock includes a first texture complexity of the inter-prediction sub-macroblock and a second texture complexity of the intra-prediction sub-macroblock. The first texture complexity includes the texture complexity of the residual of the reference region when the inter-prediction sub-macroblock and the inter-prediction sub-macroblock perform inter-frame prediction. The second texture complexity includes the texture complexity of the intra-prediction residual or the pixel value of the intra-prediction sub-macroblock itself. The steps for determining the first attribute include: determining the first texture complexity and the second texture complexity of the inter-prediction sub-macroblock within the node; calculating the sum of the determined first texture complexity and the second texture complexity of the inter-prediction sub-macroblock within the node, and dividing by the second attribute of the node to obtain the first attribute.

[0080] In some alternative implementations, the directed edge attributes include a weight attribute, which is set through the following steps: based on the motion vectors, reference indices, and multi-reference frame weighted prediction weights of the macroblocks extracted from the video coding of the video to be super-resolution upscaled, determine the pixel ratio for inter-frame prediction of the video block corresponding to the source node of the reference directed edge within the video block corresponding to the destination node of the directed edge, and use this as the weight attribute; or use an optical flow algorithm or motion estimation algorithm between the video frames associated with the directed edge to estimate the pixel ratio for inter-frame prediction of the video block corresponding to the source node of the reference directed edge within the video block corresponding to the destination node of the directed edge, and use this as the weight attribute.

[0081] In some optional implementations, the determination module 404 is further configured to: perform the following anchor block determination steps until a preset termination condition is met: select video blocks according to a pre-set search strategy and add them to the anchor block set, so that the estimated overall super-resolution defect of the video is rapidly reduced, wherein the overall super-resolution defect of the video is the sum of the super-resolution defects of video blocks outside the anchor block set, and the super-resolution defect of a video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels of the video block. The average super-resolution quality defect per pixel of the video block is determined by the following steps: select target video frames sequentially from the video frames according to the decoding order, determine the average super-resolution quality defect per pixel of the video block in the target video frame according to whether the video block in the target video frame is an anchor block or a non-anchor block, the first attribute of the node corresponding to the video block in the target video frame, and the quality defect propagated through the directed edge; determine the quality defect attribute propagated through the directed edge based on the product of the average super-resolution quality defect per pixel of the video block corresponding to the source node of the directed edge and the weight attribute of the directed edge.

[0082] In some alternative implementations, anchor blocks perform super-resolution enhancement based on a pre-built super-resolution neural network.

[0083] In some optional implementations, the determination module 404 is further configured to: sort the video blocks corresponding to the nodes according to the first attribute of the nodes in the directed acyclic graph to obtain the video block sorting result; and determine whether the video block is an anchor block based on the video block sorting result.

[0084] In some optional implementations, the determination module 404 is further configured to: sort the nodes of the directed acyclic graph based on a pre-built node sorting model of a graph neural network to obtain the node sorting result; and determine whether a video block is an anchor block based on the node sorting result.

[0085] In this embodiment, the specific processing of the acquisition module 401, segmentation module 402, partitioning module 403, and determination module 404 of the anchor block determination device in super-resolution enhancement can be referred to Figure 1 The corresponding steps are 101, 102, 103 and 104 in the embodiment.

[0086] The device for determining anchor blocks in super-resolution enhancement provided by the present invention obtains video blocks by segmenting video frames, divides macroblocks into sub-macroblocks according to the overlap between macroblocks of video frames and video blocks, and finally determines whether a video block is an anchor block based on the texture complexity of the sub-macroblocks within the video block, thereby further optimizing the overhead of the super-resolution task.

[0087] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a method for determining anchor blocks in super-resolution upscaling. This method includes: acquiring video frames of the video to be upscaled; segmenting the video frames into video blocks, where the video blocks do not overlap spatially and, when combined, cover the spatial region of the video frames; dividing macroblocks into sub-macroblocks based on the overlap between macroblocks and video blocks, where each sub-macroblock overlaps only with one video block, and no two sub-macroblocks overlap with the same video block; determining whether a video block is an anchor block based on the texture complexity of the sub-macroblocks within the video block, where anchor blocks are not upscaled based on a super-resolution version of a reused reference region.

[0088] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the method for determining anchor blocks in super-resolution enhancement provided by the above methods. The method includes: acquiring video frames of the video to be super-resolution enhanced; segmenting the video frames to obtain video blocks, wherein the video blocks do not overlap spatially and are combined to cover the spatial region of the video frames; dividing the macroblocks into sub-macroblocks according to the overlap between the macroblocks of the video frames and the video blocks, wherein each sub-macroblock overlaps only with one video block and two sub-macroblocks do not overlap with the same video block; determining whether a video block is an anchor block based on the texture complexity of the sub-macroblocks within the video block, wherein the anchor block is not super-resolution enhanced based on a super-resolution version of a reused reference region.

[0090] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for determining anchor blocks in super-resolution upscaling provided by the methods described above. This method includes: acquiring video frames of a video to be upscaled; segmenting the video frames to obtain video blocks, wherein the video blocks do not overlap spatially and are combined to cover the spatial region of the video frames; dividing macroblocks into sub-macroblocks based on the overlap between macroblocks of the video frames and video blocks, wherein each sub-macroblock overlaps only with one video block, and two sub-macroblocks do not overlap with the same video block; determining whether a video block is an anchor block based on the texture complexity of the sub-macroblocks within the video block, wherein the anchor block is not upscaled based on a super-resolution version of a reused reference region.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining anchor blocks in super-resolution enhancement, characterized in that, include: Acquire video frames from the video to be super-resolution upscaled; The video frame is segmented to obtain video blocks, which do not overlap in space and are combined to cover the spatial area of ​​the video frame; Based on the overlap between macroblocks and video blocks in the video frame, the macroblock is divided into sub-macroblocks. Each sub-macroblock overlaps with only one video block, and no two sub-macroblocks overlap with the same video block. Whether a video block is an anchor block is determined based on the texture complexity of the sub-macroblocks within the video block. The anchor block does not perform super-resolution enhancement based on a super-resolution version of a reused reference region.

2. The method for determining anchor blocks in super-resolution enhancement according to claim 1, characterized in that, The macroblock includes inter-frame prediction macroblocks, and the sub-macroblock includes inter-frame prediction sub-macroblocks. Determining whether a video block is an anchor block based on the texture complexity of the sub-macroblocks within the video block includes: The video block is determined to be an anchor block based on the constructed directed acyclic graph. In the directed acyclic graph, each node represents a video block. The video block corresponding to the destination node of the directed edge connecting the node contains at least one inter-frame prediction sub-macroblock. Inter-frame prediction coding is performed with reference to the pixel values ​​of the video block corresponding to the source node of the directed edge. The node's attributes include a first attribute, which characterizes whether the video block corresponding to the node can be effectively upscaled based on the super-resolution version of the reused reference region. The first attribute is determined based on the texture complexity of the sub-macroblocks within the video block.

3. The method for determining anchor blocks in super-resolution enhancement according to claim 2, characterized in that, The node's attributes also include a second attribute, which represents the area of ​​the video block corresponding to the node. The macroblock also includes an intra-prediction macroblock, and the sub-macroblock also includes an intra-prediction sub-macroblock. The texture complexity of the sub-macroblock includes a first texture complexity of the inter-prediction sub-macroblock and a second texture complexity of the intra-prediction sub-macroblock. The first texture complexity includes the texture complexity of the residual of the reference region when the inter-prediction sub-macroblock and the inter-prediction sub-macroblock perform inter-frame prediction. The second texture complexity includes the texture complexity of the intra-prediction residual or the pixel value of the intra-prediction sub-macroblock itself. The steps for determining the first attribute include: Determine the first texture complexity of the inter-frame prediction sub-macroblock within a node and the second texture complexity of the intra-frame prediction sub-macroblock; The sum of the first texture complexity of the determined inter-frame prediction sub-macroblock within a node and the second texture complexity of the intra-frame prediction sub-macroblock is calculated and divided by the second attribute of the node to obtain the first attribute.

4. The method for determining anchor blocks in super-resolution enhancement according to claim 3, characterized in that, The directed edge has a weight attribute, which is set through the following steps: Based on the motion vector, reference index, and multi-reference frame weighted prediction weight of the macroblock extracted from the video encoding of the video to be super-resolution upgraded, the proportion of pixels in the video block corresponding to the destination node of the directed edge that are referenced to the video block corresponding to the source node of the directed edge for inter-frame prediction is determined as the weight attribute. Alternatively, an optical flow algorithm or motion estimation algorithm can be used between video frames associated with directed edges to estimate the proportion of pixels in the video block corresponding to the destination node of the directed edge that are used to predict the inter-frame ratio by referencing the pixel values ​​of the video block corresponding to the source node of the directed edge, and this ratio can be used as the weight attribute.

5. The method for determining anchor blocks in super-resolution enhancement according to claim 2, characterized in that, The determination of whether a video block is an anchor block based on the constructed directed acyclic graph includes: Perform the following anchor point block determination steps until the preset termination condition is met: Video blocks are selected and added to the anchor block set according to a pre-set search strategy, thereby rapidly reducing the estimated overall super-resolution defect of the video. The overall super-resolution defect of the video is the sum of the super-resolution defects of video blocks outside the anchor block set. The super-resolution defect of a video block is the product of the average super-resolution quality defect per pixel of the video block and the number of pixels in the video block. The average super-resolution quality defect per pixel of the video block is determined through the following steps: According to the decoding order, target video frames are selected sequentially from the video frames. Based on whether the video block in the target video frame is an anchor block or a non-anchor block, the first attribute of the node corresponding to the video block in the target video frame, and the quality defects propagated through the directed edge, the average super-resolution quality defect per pixel of the video block in the target video frame is determined. The quality defect attribute propagated through the directed edge is determined based on the product of the average super-resolution quality defect per pixel of the video block corresponding to the source node of the directed edge and the weight attribute of the directed edge.

6. The method for determining anchor blocks in super-resolution enhancement according to claim 1, characterized in that, The anchor block is used for super-resolution enhancement based on a pre-built super-resolution neural network.

7. The method for determining anchor blocks in super-resolution enhancement according to claim 2, characterized in that, The determination of whether a video block is an anchor block based on the constructed directed acyclic graph includes: The video blocks corresponding to the nodes are sorted according to the first attribute of the nodes in the directed acyclic graph to obtain the video block sorting result; Determine whether the video block is an anchor block based on the video block sorting result.

8. The method for determining anchor blocks in super-resolution enhancement according to claim 2, characterized in that, The determination of whether a video block is an anchor block based on the constructed directed acyclic graph includes: Based on a pre-built node sorting model of a graph neural network, the nodes of the directed acyclic graph are sorted to obtain the node sorting result. Based on the node sorting results, determine whether the video block is an anchor block.

9. A device for determining anchor point blocks in super-resolution enhancement, characterized in that, include: The acquisition module is configured to acquire video frames of the video to be super-resolution upscaled. The segmentation module is configured to segment the video frame into video blocks, the video blocks not overlapping in space, and combined to cover the spatial region of the video frame; The segmentation module is configured to divide the macroblock into sub-macroblocks based on the overlap between the macroblocks and video blocks of the video frame, wherein the sub-macroblock overlaps with only one video block and no two sub-macroblocks overlap with the same video block; The determination module is configured to determine whether a video block is an anchor block based on the texture complexity of sub-macroblocks within the video block, wherein the anchor block does not perform super-resolution enhancement based on a super-resolution version of a reused reference region.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for determining anchor blocks in super-resolution enhancement as described in any one of claims 1 to 8.

Images