Video coding method, apparatus, device, storage medium and computer program

Abstract

Embodiments of the present application disclose a video coding method, device, equipment, storage medium and computer program, and belong to the technical field of coding and decoding. The method comprises: encoding a plurality of coding blocks included in a first video frame to be encoded to obtain coding results of the plurality of coding blocks and a second video frame; generating a first texture edge residual map based on the first video frame and the second video frame, the first texture edge residual map being used to represent edge residuals between texture regions in the first video frame and the second video frame; encoding the first texture edge residual map; and writing the coding results of the plurality of coding blocks and the coding results of the first texture edge residual map into a bitstream. According to the embodiments of the present application, the texture in the texture region in the second video frame is synthesized by using the first texture edge residual map, so that a realistic high-quality texture is formed, and the quality of the texture region in the reconstructed first video frame is improved.

Description

Technical Field

[0001] This application relates to the field of encoding and decoding technology, and in particular to a video encoding and decoding method, apparatus, device, storage medium, and computer program. Background Technology

[0002] With the rapid development of multimedia technology, the amount of video data is increasing dramatically. This is especially true when the textures in the video image are complex. However, transmitting and storing such large amounts of data presents challenges, making video encoding and decoding a key focus in the industry.

[0003] A proposed video encoding / decoding method involves grouping video frames into sets of T (where T is an integer greater than 2). Optical flow estimation is performed on every two adjacent video frames within this group to obtain T-1 optical flow fields. These T-1 optical flow fields correspond one-to-one with the following T-1 video frames in the group, and each optical flow field includes the motion vectors of all pixels within the corresponding video frame. A motion co-occurrence matrix is ​​determined based on these T-1 optical flow fields. The first and last video frames in the group, along with the motion co-occurrence matrix, are then encoded into the bitstream. The decoding end parses the bitstream to obtain the first and last video frames in the group, as well as the motion co-occurrence matrix. Optical flow estimation is performed on the first video frame in the group and the last video frame in the previous group to obtain the optical flow field of the first video frame in the group. Based on the optical flow field and motion co-occurrence matrix of the first video frame, the optical flow field of each intermediate video frame in the group, excluding the first and last frames, is predicted. Then, based on the optical flow field of each intermediate video frame, motion compensation is performed on the reference frame of each intermediate video frame to reconstruct these intermediate video frames.

[0004] However, when encoding and decoding using the above method, errors in optical flow estimation and determination of the motion co-occurrence matrix may lead to texture errors and unrealistic effects in the final reconstructed intermediate video frames. Furthermore, the first video frame in a set of video frames is usually used as a reference frame, i.e., an I-frame. I-frames typically require a higher bitrate, so directly encoding the first and last video frames into the bitstream would consume a significant amount of bitrate. Summary of the Invention

[0005] This application provides a video encoding / decoding method, apparatus, device, storage medium, and computer program, which can solve the problems of texture errors, unrealistic effects, and high bitrates in reconstructed video frames in related technologies. In this application, the first video frame is the current video frame to be encoded, the second video frame is the base reconstructed frame of the first video frame, the third video frame is the reference frame of the first video frame, and the fourth video frame is the base reconstructed frame of the third video frame. The technical solution is as follows:

[0006] In a first aspect, a video coding method is provided. In this method, a first video frame to be encoded comprises multiple coding blocks to obtain the encoding results of the multiple coding blocks and a second video frame. Based on the first and second video frames, a first texture edge residual map is generated, which characterizes the edge residuals between texture regions in the first and second video frames. The first texture edge residual map is encoded; the encoding results of the multiple coding blocks and the encoding results of the first texture edge residual map are written into the bitstream.

[0007] In this embodiment, the multiple coding blocks can be encoded using conventional encoding methods, or they can be encoded based on the mask label of the first video frame, where the mask label of the first video frame is used to distinguish between textured and non-textured regions in the first video frame. Of course, the multiple coding blocks can also be encoded in other ways, and this embodiment does not limit this approach.

[0008] When encoding the multiple coding blocks based on the mask label of the first video frame, the multiple coding blocks included in the first video frame can be encoded according to the following steps (1)-(3).

[0009] (1) Determine the mask label of the first video frame based on the first video frame.

[0010] There are several ways to determine the mask label of the first video frame; one method will be described below. Specifically, the first video frame is semantically segmented to obtain a semantic segmentation map. This map includes multiple semantic regions, each corresponding to a semantic category. Based on the semantic categories corresponding to these semantic regions, mask labels are determined for each of these semantic regions. Finally, the mask label for the first video frame is determined based on these mask labels.

[0011] Since the mask label for each of these multiple semantic regions is determined in the same way, we will now take one of these semantic regions as an example. That is, a semantic region is selected from these multiple semantic regions. If there is no texture category corresponding to the first semantic category in the correspondence between the semantic category and texture category, then the mask label for the selected semantic region is determined to be a first value. The first semantic category is the semantic category corresponding to the selected semantic region, and the first value indicates that the corresponding region is a non-texture region. If there is a texture category corresponding to the first semantic category in the stored correspondence between semantic categories and texture categories, then the mask label for the selected semantic region is determined to be a second value, and the second value indicates that the corresponding region is a texture region.

[0012] For some typical and special regions, these regions often possess rich textures, such as grasslands, buildings, mountains, and plants. Therefore, embodiments of this application predetermine the semantic categories and texture categories of these regions and store the correspondence between semantic categories and texture categories. Thus, for a selected semantic region, if the stored correspondence between semantic categories and texture categories does not contain a texture category corresponding to the first semantic category, it indicates that the selected semantic region is a non-texture region, and the mask label corresponding to the selected semantic region is determined to be a first value. If the correspondence between semantic categories and texture categories contains a texture category corresponding to the first semantic category, it indicates that the selected semantic region is a texture region, and the mask label corresponding to the selected semantic region is determined to be a second value.

[0013] It should be noted that a texture category may correspond to one semantic category, and conversely, a texture category may correspond to multiple semantic categories. For example, the semantic categories of "trees" and "grass" correspond to the same texture category, namely "plants." Similarly, the semantic categories of "mountains" and "hills" correspond to the same texture category, namely "mountains."

[0014] (2) Based on the mask label of the first video frame, determine the quantization parameters corresponding to the multiple coding blocks respectively. The quantization parameters corresponding to the coding blocks belonging to the texture region are greater than the quantization parameters corresponding to the coding blocks in the non-texture region.

[0015] Since the mask labels of the first video frame are determined on a semantic region basis, that is, the same semantic region in the mask labels of the first video frame corresponds to the same mask label, or all pixels in the same semantic region correspond to the same mask label. However, the quantization parameters are determined on a coding block basis. Therefore, it is necessary to determine the mask label corresponding to each coding block in the multiple coding blocks based on the mask labels of the first video frame, and then determine the quantization parameters corresponding to the multiple coding blocks based on the mask labels corresponding to the multiple coding blocks respectively.

[0016] As an example, for any one of the multiple coding blocks, based on the mask label of the first video frame, the number of pixels within that coding block with a mask label of a first value is counted to obtain a first count. A ratio is determined between the first count and a second count, where the second count is the total number of pixels in the coding block. If the ratio is greater than or equal to a ratio threshold, the mask label corresponding to that coding block is determined to be the first value; if the ratio is less than the ratio threshold, the mask label corresponding to that coding block is determined to be the second value. That is, if the ratio is greater than or equal to the ratio threshold, it indicates that most of the area within the coding block is a non-texture region, and the mask label corresponding to that coding block is determined to be the first value. If the ratio is less than the ratio threshold, it indicates that most of the area within the coding block is a texture region, and the mask label corresponding to that coding block is determined to be the second value.

[0017] After determining the mask label corresponding to each coding block, the quantization parameter corresponding to each coding block can be determined based on the mask label. For any coding block among the multiple coding blocks, if the mask label corresponding to the coding block is a first value, then the quantization parameter corresponding to the coding block is determined as the first quantization parameter. If the mask label corresponding to the coding block is a second value, then the first quantization parameter is added to the set quantization parameter to obtain the second quantization parameter, and the quantization parameter corresponding to the coding block is determined as the second quantization parameter.

[0018] Because the texture area is particularly rich in texture, encoding the texture area using conventional encoding methods would consume a large amount of encoding bitrate. However, users are not sensitive to distortion or changes in texture areas. Therefore, in this embodiment, non-texture areas are encoded using conventional encoding methods, while texture areas are encoded using higher quantization parameters to reduce the encoding quality of texture areas and thus reduce the encoding bitrate occupied by texture areas.

[0019] (3) Encode the multiple coding blocks based on the quantization parameters corresponding to each of the multiple coding blocks.

[0020] Based on the above description, the quantization parameters of these multiple coding blocks may be different. Therefore, each coding block can be encoded according to its corresponding quantization parameters.

[0021] It should be noted that the encoding method of these multiple encoding blocks can be any encoding method, and this application embodiment does not limit it.

[0022] As an example, for steps (2) and (3) above, an index table can be formed from the mask labels and position information of the multiple coding blocks, and the index table can be input into the conventional encoder. When the conventional encoder encodes a coding block at a certain position, it retrieves the mask label corresponding to the coding block from the index table, determines the quantization parameters of the coding block according to the above method based on the mask label corresponding to the coding block, and then encodes the coding block according to the quantization parameters corresponding to the coding block.

[0023] The embodiments of this application can improve the conventional encoder, so that during the encoding of the multiple coding blocks by the conventional encoder, not only the encoding results of the multiple coding blocks can be obtained, but also the second video frame can be obtained.

[0024] Since the edge residuals between the texture regions in the first and second video frames are subsequently encoded into the bitstream, and the decoding end parses the encoding results of these multiple coded blocks to obtain the second video frame, and then synthesizes the texture based on the edge residuals in the bitstream to obtain the reconstructed first video frame, the second video frame can also be called the basic reconstructed frame.

[0025] In this embodiment of the application, a first texture edge residual map is generated based on the first video frame and the second video frame according to the following steps (1)-(3).

[0026] (1) Generate a texture edge map based on the first video frame. The texture edge map is used to represent the edge information within the texture region in the first video frame.

[0027] In the embodiments of this application, the first video frame may be an I-frame or a non-I-frame, such as a P-frame or a B-frame. The method of generating texture edge maps differs for I-frames and non-I-frames; therefore, I-frames and non-I-frames will be described separately below.

[0028] The first video frame is an I-frame.

[0029] In this case, the texture edge map is used to represent the edge information in all texture regions in the first video frame. The process of generating the texture edge map based on the first video frame includes: detecting the edge information in all texture regions in the first video frame based on the mask label of the first video frame to generate the texture edge map.

[0030] Since the mask label of the first video frame is used to distinguish between textured and non-textured regions in the first video frame, and I-frames only perform intra-frame predictive coding and do not depend on other video frames, all textured regions in the first video frame can be determined through the mask label. Then, edge information within all textured regions in the first video frame is detected to generate a texture edge map.

[0031] Based on the above description, the mask label of the first video frame includes a binarized mask label. The region with a mask label of a first value is used to represent a non-textured region, and the region with a mask label of a second value is used to represent a textured region. Therefore, the coordinates of the pixels with the mask label of the second value can be determined from the first video frame, thereby obtaining the textured region within the first video frame. Then, the edge information within the textured region can be detected to generate a texture edge map.

[0032] The first video frame is a non-I-frame.

[0033] In this scenario, the texture edge map is used to represent edge information within a local texture region in the first video frame. The process of generating the texture edge map based on the first video frame includes: determining a target location range, where the target location range is the position of the newly added region in the first video frame relative to the third video frame, and the third video frame serves as the reference frame for the first video frame; and generating the texture edge map based on the first video frame and the target location range.

[0034] Since non-I-frames require intra-frame predictive coding and inter-frame predictive coding, and inter-frame predictive coding relies on other video frames, i.e., reference frames, it is not necessary to detect edge information within all texture regions to generate texture edge maps for non-I-frames. Instead, edge information within local texture regions is sufficient. These local texture regions are the texture regions added to the first video frame relative to the third video frame.

[0035] The target location range refers to the location of the newly added region. The process of determining the target location range includes: determining global motion parameters, which characterize the amount of motion of pixels in the third video frame moving to the first video frame; performing motion compensation on the third video frame based on the global motion parameters to obtain a first motion-compensated frame, which includes the newly added region of the first video frame relative to the third video frame; and traversing the coordinates of pixels within this newly added region in the first motion-compensated frame to obtain the target location range.

[0036] Since global motion parameters are used to characterize the amount of motion of pixels in the third video frame moving to the first video frame, in some embodiments, global motion parameters can be determined based on the first and third video frames. In other embodiments, the second video frame can be obtained during the encoding of multiple coded blocks included in the first video frame, and similarly, the fourth video frame can be obtained during the encoding of multiple coded blocks included in the third video frame. Therefore, global motion parameters can also be determined based on the second and fourth video frames.

[0037] The process of generating a texture edge map based on a first video frame and a target location range includes: extracting an image region located within the target location range from the first video frame to obtain a first image region; and detecting edge information within all texture regions of the first image region based on the mask labels of the first video frame to generate a texture edge map.

[0038] Since the mask label of the first video frame is used to distinguish between textured and non-textured regions in the first video frame, and the first video frame includes a first image region, after extracting the first image region from the first video frame, the textured regions in the first image region can be determined using the mask label of the first video frame. Then, edge information within all textured regions of the first image region is detected to generate a texture edge map.

[0039] (2) Generate a first edge map based on the second video frame. The first edge map is used to characterize the edge information in the second video frame.

[0040] Based on the above description, the first video frame may be an I-frame or a non-I-frame. The method for generating the first edge map differs for I-frames and non-I-frames; therefore, I-frames and non-I-frames will be introduced separately below.

[0041] The first video frame is an I-frame.

[0042] In this case, the first edge map is used to characterize the edge information of the second video frame. The process of generating the first edge map based on the second video frame includes: detecting the edge information of the second video frame to generate the first edge map.

[0043] For details on edge detection of the second video frame, please refer to the previous description; it will not be repeated here.

[0044] The first video frame is a non-I-frame.

[0045] In this case, the first edge map is used to characterize the edge information in a local area in the second video frame. The process of generating the first edge map based on the second video frame includes: generating the first edge map based on the second video frame and the target location range.

[0046] Since non-I-frames require intra-frame predictive coding and inter-frame predictive coding, and inter-frame predictive coding depends on other video frames, i.e., reference frames, it is not necessary to detect edge information in all regions to generate the first edge map for non-I-frames. Instead, edge information in local regions only needs to be detected to generate the first edge map.

[0047] The process of generating a first edge map based on a second video frame and a target location range includes: extracting an image region located within the target location range from the second video frame to obtain a second image region; and detecting the edge information of the second image region to generate the first edge map.

[0048] (3) Generate a first texture edge residual map based on the texture edge map and the first edge map.

[0049] Since there is overlapping edge information in the texture edge map and the first edge map, in order to reduce the edge information transmitted to the decoding end, this embodiment of the application needs to generate a first texture edge residual map based on the texture edge map and the first edge map.

[0050] As an example, the first texture edge residual map is generated based on the texture edge map and the first edge map according to the following formula.

[0051]

[0052] In the above formula, I residual,i,j Let I be the pixel value of the pixel at coordinates (i,j) in the first texture edge residual map. ori,i,j Let I be the pixel value of the pixel at coordinates (i,j) in the texture edge map. encode,i,j This represents the pixel value of the pixel at coordinates (i,j) in the first edge map. 255 indicates that the pixel is an edge point, and 0 indicates that the pixel is not an edge point.

[0053] As can be seen from the above formula, the first texture edge residual map only includes edge information that exists in the first video frame but disappears in the second video frame due to encoding distortion. In other words, the first texture edge residual map only includes edge information that exists in the texture edge map but not in the first edge map. Therefore, subsequently, only the edge information that exists in the first video frame but not in the second video frame needs to be transmitted to the decoding end, without needing to transmit edge information that exists in the second video frame but not in the first video frame, nor the overlapping edge information between the first and second video frames, thus reducing the amount of edge information that needs to be transmitted.

[0054] Optionally, after generating the texture edge map based on the first video frame, the method further includes: downsampling the texture edge map according to a first sampling ratio, where the first sampling ratio is proportional to the quantization parameters of the non-texture regions in the first video frame; and upsampling the downsampled texture edge map according to the first sampling ratio. Similarly, after generating the first edge map based on the second video frame, the method further includes: downsampling the first edge map according to the first sampling ratio; and upsampling the downsampled first edge map according to the first sampling ratio.

[0055] Since there is overlapping edge information in the texture edge map and the first edge map, and the edge information extracted by the above method is relatively rich (i.e., the texture edge map and the first edge map contain relatively rich content), this embodiment of the application can further downsample and upsample the texture edge map and the first edge map to reduce the edge information in the texture edge map and the first edge map, while balancing the coding rate and image distortion, and considering the sparsity and binarization characteristics of the texture edge map. Thus, the first texture edge residual map determined by the sampled texture edge map and the sampled first edge map contains even less edge information, thereby further reducing the edge information required for encoding and transmission.

[0056] Based on the above description, the texture edge map and the first edge map can be downsampled and then upsampled according to the first sampling ratio. Of course, the texture edge map and the first edge map can also be left unsampled. However, regardless of whether the texture edge map and the first edge map are sampled, after the first texture edge residual map is determined, it can be downsampled according to the second sampling ratio to reduce the edge information that needs to be transmitted.

[0057] It should be noted that the first sampling ratio and the second sampling ratio can be the same or different, and this application embodiment does not limit this.

[0058] To improve the quality of texture regions reconstructed in the first video frame by the decoder, the encoder obtains a second video frame during the encoding process of the multiple coding blocks. This allows the encoder to determine the edge residuals between the texture regions in the first and second video frames, i.e., a first texture edge residual map, which is then encoded into the bitstream. After obtaining the second video frame, the decoder synthesizes the textures within the texture regions of the second video frame based on the first texture edge residual map, thereby forming realistic, high-quality textures and improving the quality of the reconstructed texture regions in the first video frame.

[0059] Because textured regions are particularly rich in texture, directly encoding these regions using conventional encoding methods would consume a significant amount of bitrate. However, the human eye is not sensitive to distortions or changes in these textured regions. Therefore, in this embodiment, the quantization parameters of the textured regions are increased to encode the multiple coding blocks included in the first video frame, thereby reducing the bitrate occupied by the coding blocks belonging to the textured regions and thus reducing the overall bitrate of the video frame. In this way, while reducing the bitrate, the quality of the textured regions in the first video frame reconstructed at the decoding end can still be guaranteed.

[0060] In addition, the embodiments of this application use edge information as texture features for texture synthesis, mainly because edge information has the following characteristics and advantages: (1) Edge information is a low-level image feature and is easy to extract; (2) Complex texture patterns have rich edges, and edges are crucial for the restoration of the content structure in the image; (3) Edges are simple and easy to compress compared to other features; (4) Edges are pixel domain features and are easy to use as input for training and use in texture synthesis networks. At the same time, texture synthesis in video also needs to take into account the characteristics of the spatiotemporal domain, and pixel domain features are easy to perform spatiotemporal domain operations.

[0061] Secondly, a video decoding method is provided, in which: a bitstream is parsed to obtain a second video frame; a first edge map is generated based on the second video frame, the first edge map being used to characterize the edge information of the second video frame; a second texture edge residual map is obtained, the second texture edge residual map being used to characterize the edge residual between all texture regions in the first video frame to be reconstructed and the second video frame; the second texture edge residual map is superimposed on the first edge map to obtain a second edge map; and the second video frame and the second edge map are input into a texture synthesis network to obtain the reconstructed first video frame.

[0062] Based on the preceding description, the encoding end can either downsample and then upsample the texture edge map and the first edge map, or it can choose not to sample at all. If the encoding end downsamples and then upsamples the texture edge map and the first edge map, the decoding end, after generating the first edge map based on the second video frame, can also downsample and then upsample the first edge map. That is, the first edge map is downsampled according to a first sampling ratio, which is proportional to the quantization parameters of the non-texture regions in the first video frame. The downsampled first edge map is then upsampled according to the first sampling ratio.

[0063] Since the encoding end encodes multiple coded blocks based on their quantization parameters in the first video frame, and then writes the encoding results of these blocks into the bitstream and transmits them to the decoding end, the encoding end also instructs the decoding end on the quantization parameters of these multiple coded blocks to ensure that the decoding end can accurately parse the second video frame. In this way, the decoding end can obtain the quantization parameters of the non-texture regions in the first video frame, i.e., the first quantization parameters, and thus determine the first sampling ratio based on the first quantization parameters.

[0064] Based on the preceding description, the first video frame may be an I-frame or a non-I-frame. The methods for obtaining the second texture edge residual map differ for I-frames and non-I-frames; therefore, I-frames and non-I-frames will be described separately below.

[0065] The first video frame is an I-frame.

[0066] In this case, the process of obtaining the second texture edge residual map includes: parsing the bitstream to obtain the reconstructed first texture edge residual map, and the second texture edge residual map being the reconstructed first texture edge residual map.

[0067] For an I-frame, the first texture edge residual map is used to characterize the edge residuals between all texture regions in the first video frame and the second video frame, not just the edge residuals between texture regions in the newly added region. Therefore, the reconstructed first texture edge residual map can be directly used as the second texture edge residual map.

[0068] The first video frame is a non-I-frame.

[0069] In this case, the process of obtaining the second texture edge residual map includes: parsing the bitstream to obtain the reconstructed first texture edge residual map; determining the third texture edge residual map based on the second video frame and global motion parameters, where the global motion parameters characterize the amount of motion of pixels in the fourth video frame moving to the second video frame, and the fourth video frame serves as the reference frame for the first video frame to be reconstructed; and superimposing the reconstructed first texture edge residual map with the third texture edge residual map to obtain the second texture edge residual map.

[0070] The third texture edge residual map is used to characterize the texture edge residuals of non-newly added regions in the first video frame. Since non-I-frames do not encode the edge residuals of all texture regions into the bitstream, but only the edge residuals of newly added regions, the texture edge residuals of non-newly added regions can be determined by using a reference frame for non-I-frames. That is, the texture edge residuals of non-newly added regions can be determined by using the texture edge residuals of the fourth video frame.

[0071] The process of determining the third texture edge residual map based on the second video frame and global motion parameters includes: performing motion compensation on the texture edge residual map of the fourth video frame based on the global motion parameters to obtain a motion-compensated edge residual map; performing motion compensation on the fourth video frame based on the global motion parameters to obtain a second motion-compensated frame; performing optical flow estimation on the second video frame and the second motion-compensated frame to obtain a first optical flow field; and performing optical flow transformation on the motion-compensated edge residual map based on the first optical flow field to obtain the third texture edge residual map.

[0072] Based on the preceding description, the encoder may or may not encode the global motion parameters into the bitstream. If the encoder encodes the global motion parameters into the bitstream, the decoder can directly parse the bitstream to obtain the global motion parameters. If the encoder does not encode the global motion parameters into the bitstream, the decoder can determine the global motion parameters based on the second and fourth video frames.

[0073] Based on the terminology explained above, the optical flow field refers to the amount of motion of all pixels in one video frame moving to another video frame. Moreover, the optical flow field includes the motion vector corresponding to each pixel in the video frame. Therefore, after performing optical flow estimation on the second video frame and the second motion-compensated frame to obtain the first optical flow field, the motion vector of the corresponding pixel in the first optical flow field can be added to the coordinates of each pixel in the motion-compensated edge residual map, and then the pixel values ​​can be copied to obtain the third texture edge residual map, thereby realizing the optical flow transformation of the motion-compensated edge residual map.

[0074] Based on the preceding description, the encoder can downsample the first texture edge residual map according to the second sampling ratio, or it can choose not to downsample it. If the encoder downsamples the first texture edge residual map, in order to ensure that the resolution of the first texture edge residual map is the same, the decoder needs to upsample the reconstructed first texture edge residual map according to the second sampling ratio after obtaining it.

[0075] It should be noted that when the first video frame is not an I-frame, the second texture edge residual map can be obtained not only in the manner described above, but also in other ways. For example, optical flow estimation can be performed on the second and fourth video frames to obtain the second optical flow field. Based on the second optical flow field, optical flow transformation can be performed on the texture edge residual map of the fourth video frame to obtain the second texture edge residual map.

[0076] In other words, the encoding end only needs to encode the multiple coding blocks included in the first video frame into the bitstream, without needing to encode the texture edge residual map of the newly added region into the bitstream. Thus, for the decoding end, the texture edge residual map of the current frame can be obtained by performing optical flow transformation on the texture edge residual map of the reference frame.

[0077] Since the second texture edge residual map is used to characterize the edge residual between all texture regions in the second video frame and the first video frame to be reconstructed, and the first edge map is used to characterize the edge information of the second video frame, the edge information of the first video frame to be reconstructed, i.e., the second edge map, can be obtained by superimposing the second texture edge residual map and the first edge map.

[0078] Overlay refers to fusing edge information from two images. That is, for a pixel at the same location in two images, if at least one of the two images represents the pixel as an edge point, then the pixel is determined to be an edge point; if both images represent the pixel as a non-edge point, then the pixel is determined to be a non-edge point.

[0079] In this application, to improve the quality of the texture region in the first video frame reconstructed by the decoder, the encoder needs to determine the edge residual between the texture regions in the first and second video frames, i.e., the first texture edge residual map, and then encode the first texture edge residual map into the bitstream. Thus, after the decoder obtains the second video frame, it synthesizes the texture within the texture region in the second video frame based on the first texture edge residual map, thereby forming a realistic, high-quality texture and improving the quality of the reconstructed texture region in the first video frame. Furthermore, because the texture regions are particularly rich in texture, directly encoding these texture regions using conventional encoding methods would consume a significant amount of bitrate, but the human eye is not sensitive to distortions or changes in these texture regions. Therefore, the encoder increases the quantization parameters of the texture regions to encode multiple coding blocks included in the first video frame, thereby reducing the bitrate occupied by the coding blocks belonging to the texture regions, and thus reducing the overall bitrate of the video frame. In this way, while reducing the bitrate, the quality of the texture region in the first video frame reconstructed by the decoder can still be guaranteed.

[0080] Thirdly, a video encoding apparatus is provided, which has the function of implementing the video encoding method described in the first aspect. The video encoding apparatus includes at least one module for implementing the video encoding method provided in the first aspect.

[0081] Fourthly, a video decoding apparatus is provided, which has the function of implementing the video decoding method described in the second aspect above. The video decoding apparatus includes at least one module for implementing the video decoding method provided in the second aspect above.

[0082] Fifthly, an encoding end device is provided, the encoding end device including a processor and a memory, the memory being used to store a computer program for executing the video encoding method provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the video encoding method described in the first aspect.

[0083] Optionally, the encoding end device may further include a communication bus for establishing a connection between the processor and the memory.

[0084] In a sixth aspect, a decoding device is provided, the decoding device including a processor and a memory, the memory being used to store a computer program for executing the video decoding method provided in the second aspect above. The processor is configured to execute the computer program stored in the memory to implement the video decoding method described in the second aspect above.

[0085] Optionally, the decoding device may further include a communication bus for establishing a connection between the processor and the memory.

[0086] In a seventh aspect, a computer-readable storage medium is provided, the storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the method described in the first or second aspect.

[0087] Eighthly, a computer-readable storage medium is provided, comprising the bitstream obtained by the encoding method described in the first aspect above.

[0088] A ninth aspect provides a computer program product comprising instructions that, when executed on a computer, cause the computer to perform the steps of the methods described in the first or second aspect. Alternatively, a computer program is provided that, when executed on a computer, causes the computer to perform the steps of the methods described in the first or second aspect.

[0089] The technical effects achieved by the third to ninth aspects mentioned above are similar to those achieved by the corresponding technical means in the first or second aspects, and will not be repeated here.

[0090] The technical solutions provided in this application can bring at least the following beneficial effects:

[0091] To improve the quality of texture regions reconstructed in the first video frame by the decoder, the encoder needs to determine the edge residuals between the texture regions in the first and second video frames, i.e., the first texture edge residual map, and then encode the first texture edge residual map into the bitstream. Thus, after obtaining the second video frame, the decoder synthesizes the textures within the texture regions of the second video frame based on the first texture edge residual map, thereby forming realistic, high-quality textures and improving the quality of the reconstructed texture regions in the first video frame. According to the method provided in this application, there is no need to divide the video frames, nor is it necessary to encode the first and last video frames into the bitstream, reducing the encoding bitrate. Attached Figure Description

[0092] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0093] Figure 2 This is a schematic diagram of an exemplary implementation environment provided in an embodiment of this application;

[0094] Figure 3 This is an exemplary block diagram of a video encoding and decoding method provided in an embodiment of this application;

[0095] Figure 4 This is a flowchart of a video encoding method provided in an embodiment of this application;

[0096] Figure 5 This is a schematic diagram illustrating how to determine the mask label of a first video frame according to an embodiment of this application;

[0097] Figure 6 This is a schematic diagram illustrating the determination of a target location range according to an embodiment of this application;

[0098] Figure 7 This is a flowchart of a video decoding method provided in an embodiment of this application;

[0099] Figure 8 This is a schematic diagram of the structure of a generator provided in an embodiment of this application;

[0100] Figure 9 This is a schematic diagram of the structure of a Resblock provided in an embodiment of this application;

[0101] Figure 10 This is a schematic diagram of the structure of an SFT layer provided in an embodiment of this application;

[0102] Figure 11 This is a schematic diagram of the structure of a discriminator provided in an embodiment of this application;

[0103] Figure 12 This is an exemplary block diagram of a video encoding and decoding method for an I-frame provided in an embodiment of this application;

[0104] Figure 13 This is an exemplary block diagram of a non-I-frame video encoding and decoding method provided in an embodiment of this application;

[0105] Figure 14 This is an exemplary block diagram of another non-I-frame video encoding and decoding method provided in the embodiments of this application;

[0106] Figure 15 This is a schematic diagram of the structure of a video encoding device provided in an embodiment of this application;

[0107] Figure 16 This is a schematic diagram of the structure of a video decoding device provided in an embodiment of this application;

[0108] Figure 17 This is a schematic block diagram of an encoding / decoding device provided in an embodiment of this application. Detailed Implementation

[0109] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0110] Before providing a detailed explanation of the video encoding and decoding methods provided in the embodiments of this application, the terminology and implementation environment involved in the embodiments of this application will be introduced first.

[0111] To facilitate understanding, the terms used in the embodiments of this application will be explained first.

[0112] Encoding refers to the process of compressing a video to be encoded into a bitstream. A video consists of multiple video frames, and video encoding is the process of compressing the sequence of video frames into a bitstream.

[0113] It should be noted that a video that has been compressed into a bitstream can be called an encoded video, and a video frame that has been compressed into a bitstream can be called an encoded video frame.

[0114] Decoding refers to the process of restoring an encoded bitstream into a reconstructed video according to specific syntax rules and processing methods. Specifically, video bitstream decoding refers to the process of restoring a video bitstream into a reconstructed sequence of video frames.

[0115] A coding block is a region of data obtained by dividing a video frame into its corresponding coding blocks. A video frame can be divided into multiple coding blocks, which together constitute the video frame. Each coding block can be encoded independently, and its size can be 128*128.

[0116] Quantization is the process of mapping continuous values ​​of a signal to multiple discrete amplitude values. Quantization can effectively reduce the range of signal values, thereby achieving better compression, and it is also the root cause of distortion.

[0117] The quantization parameter (QP) is a crucial parameter controlling the degree of quantization and reflects the compression level of a video frame. Generally, a smaller QP results in finer quantization, preserving more image details and producing higher encoded quality, but also consuming more bitrate. Conversely, a larger QP results in coarser quantization, greater loss of image details, lower encoded quality, more noticeable distortion, and less bitrate. In other words, the quantization parameter and bitrate are negatively correlated.

[0118] Optical flow refers to the motion of all pixels in one video frame to another. In other words, the optical flow includes the motion vector corresponding to each pixel in the video frame, which is (Δx, Δy). Thus, by adding this motion vector to the pixel's coordinates (x, y), the pixel can be mapped to another video frame.

[0119] Mask: Typically used for semantic differentiation. This application extends the mask to distinguish between textured and non-textured regions in a video frame, and optionally, also to distinguish the texture category to which the textured region belongs. In this application, the extended mask is referred to as a mask label.

[0120] Edge map: This refers to the edge information detected in a video frame. In other words, the edge map is used to represent the edge information in a video frame.

[0121] Texture edge map: This refers to the detection of edge information within texture regions in a video frame. In other words, the texture edge map is used to represent the edge information within texture regions in a video frame.

[0122] Texture edge residual map: This refers to the residual between edge information within texture regions in two video frames. In other words, the texture edge residual map is used to characterize the residual between edge information within texture regions in two video frames.

[0123] The implementation environment involved in the embodiments of this application will be described next.

[0124] Please refer to Figure 1 , Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. The implementation environment includes a source device 10, a destination device 20, a link 30, and a storage device 40. The source device 10 can generate encoded video. Therefore, the source device 10 can also be referred to as a video encoding device. The destination device 20 can decode the encoded video generated by the source device 10. Therefore, the destination device 20 can also be referred to as a video decoding device. The link 30 can receive the encoded video generated by the source device 10 and can transmit the encoded video to the destination device 20. The storage device 40 can receive the encoded video generated by the source device 10 and can store the encoded video. Under such conditions, the destination device 20 can directly obtain the encoded video from the storage device 40. Alternatively, the storage device 40 can correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device 10. Under such conditions, the destination device 20 can stream or download the encoded video stored in the storage device 40.

[0125] Both source device 10 and destination device 20 may include one or more processors and memory coupled to the one or more processors. This memory may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or any other media that can be used to store desired program code in the form of computer-accessible instructions or data structures. For example, both source device 10 and destination device 20 may include desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, handsets such as so-called "smartphones," televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, or the like.

[0126] Link 30 may include one or more media or devices capable of transmitting encoded video from source device 10 to destination device 20. In one possible implementation, link 30 may include one or more communication media enabling source device 10 to directly transmit encoded video to destination device 20 in real time. In this embodiment, source device 10 may modulate the encoded video according to a communication standard, such as a wireless communication protocol, and transmit the modulated video to destination device 20. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, wide area network, or global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices facilitating communication from source device 10 to destination device 20, etc., which are not specifically limited in this embodiment.

[0127] In one possible implementation, storage device 40 can store the received encoded video sent by source device 10, and destination device 20 can directly retrieve the encoded video from storage device 40. Under such conditions, storage device 40 can include any of a variety of distributed or locally accessed data storage media. For example, any of these distributed or locally accessed data storage media can be a hard disk drive, Blu-ray disc, digital versatile disc (DVD), compact disc read-only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video.

[0128] In one possible implementation, storage device 40 may correspond to a file server or another intermediate storage device capable of storing the encoded video generated by source device 10, and destination device 20 may stream or download the video stored on storage device 40. The file server may be any type of server capable of storing and sending the encoded video to destination device 20. In one possible implementation, the file server may include a web server, a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive, etc. Destination device 20 may acquire the encoded video via any standard data connection (including an Internet connection). Any standard data connection may include a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both suitable for acquiring the encoded video stored on a file server. The transmission of the encoded video from storage device 40 may be streaming, downloading, or a combination of both.

[0129] Figure 1 The implementation environment shown is only one possible implementation, and the technology of this application embodiment can be applied to more than just this. Figure 1 The source device 10 shown can encode video, and the destination device 20 can decode the encoded video. It can also be applied to other devices that can encode video and decode encoded video. This application embodiment does not specifically limit them.

[0130] exist Figure 1In the illustrated implementation environment, source device 10 includes a data source 120, an encoder 100, and an output interface 140. In some embodiments, the output interface 140 may include a modem / demodulator and / or a transmitter, wherein the transmitter may also be referred to as a transmitter. Data source 120 may include an image capture device (e.g., a camera, etc.), an archive containing previously captured video, a feed interface for receiving video from a video content provider, and / or a computer graphics system for generating video, or a combination of these sources of video.

[0131] Data source 120 can send video to encoder 100, which can encode the received video from data source 120 to obtain encoded video. The encoder can then send the encoded video to an output interface. In some embodiments, source device 10 directly sends the encoded video to destination device 20 via output interface 140. In other embodiments, the encoded video can also be stored on storage device 40 for later retrieval by destination device 20 for decoding and / or display.

[0132] exist Figure 1 In the illustrated implementation environment, destination device 20 includes an input interface 240, a decoder 200, and a display device 220. In some embodiments, the input interface 240 includes a receiver and / or a modem. The input interface 240 may receive encoded video via link 30 and / or from storage device 40, and then send it to the decoder 200, which may decode the received encoded video to obtain decoded video. The decoder may send the decoded video to the display device 220. The display device 220 may be integrated with destination device 20 or may be external to destination device 20. Generally, the display device 220 displays the decoded video. The display device 220 may be any type of display device, for example, a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0133] although Figure 1Not shown, but in some respects, encoder 100 and decoder 200 may be integrated with audio encoder and decoder respectively, and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software for encoding both audio and video in a common data stream or separate data streams. In some embodiments, the MUX-DEMUX unit may conform to the ITU H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP), if applicable.

[0134] Encoder 100 and decoder 200 may each be any of the following circuits: one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the techniques of the embodiments of this application are implemented in part in software, the apparatus may store instructions for software in a suitable non-volatile computer-readable storage medium, and the instructions may be executed in hardware using one or more processors to implement the techniques of the embodiments of this application. Any of the foregoing (including hardware, software, combinations of hardware and software, etc.) may be considered as one or more processors. Each of encoder 100 and decoder 200 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder / decoder (encoder-decoder) in the respective apparatus.

[0135] In this application embodiment, encoder 100 may be generally referred to as an apparatus that “signals” or “sends” certain information to, for example, decoder 200. The terms “signals” or “sends” may generally refer to the transmission of syntax elements and / or other data for decoding encoded video. This transmission may occur in real-time or near real-time. Alternatively, this communication may occur after a period of time, for example, during encoding when syntax elements are stored in a computer-readable storage medium in the encoded bitstream, and the decoding apparatus may then retrieve the syntax elements at any time after they have been stored in this medium.

[0136] The video encoding and decoding method provided in this application embodiment can be applied to various scenarios. For example, in a smart city scenario, there are a large number of complex texture areas such as trees and grass in the captured images. Therefore, in order to improve the quality of texture areas in the reconstructed images, the video encoding and decoding method provided in this application embodiment can be used for smart city scenarios.

[0137] Please refer to Figure 2 , Figure 2 This is a schematic diagram of an exemplary implementation environment provided in this application embodiment. Taking a smart city scenario as an example, the implementation environment includes a data acquisition terminal 101 and a terminal device 201, and the data acquisition terminal 101 and the terminal device 201 are communicatively connected. This communication connection can be a wireless connection or a wired connection, and this application embodiment does not limit it in this way.

[0138] The acquisition terminal 101 can be the above Figure 1 The source device 10 in the implementation environment shown. The terminal device 201 can be the aforementioned. Figure 1 The destination device 20 in the implementation environment shown.

[0139] Alternatively, please refer to Figure 3 The acquisition terminal 101 is used to acquire video frames to obtain video. For each video frame in the video, taking the first video frame as an example, the acquisition terminal 101 performs texture processing on the first video frame, and based on the texture processing results, encodes multiple coding blocks included in the first video frame, and encodes a first texture edge residual map based on the texture processing results. The first texture edge residual map is used to characterize the edge residual between the texture regions in the first video frame and the second video frame. The second video frame is obtained by parsing the encoding results of the multiple coding blocks included in the first video frame. Then, the acquisition terminal 101 writes the encoding results of the multiple coding blocks and the encoding results of the first texture edge residual map into the bitstream and transmits them to the terminal device 201 through the channel.

[0140] Terminal device 201 parses the bitstream transmitted by acquisition terminal 101 to obtain a second video frame, and generates a first edge map based on the second video frame. The first edge map is used to represent the edge information of the second video frame. A second texture edge residual map is obtained, which is used to represent the edge residual between all texture regions in the first video frame to be reconstructed and the second video frame. The second texture edge residual map is superimposed on the first edge map to obtain a second edge map. The second video frame and the second edge map are then combined to obtain the reconstructed first video frame.

[0141] Among them, the acquisition end 101 can be a camera, monitoring equipment, etc., and the terminal device 201 can be any electronic product that can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as personal computer (PC), mobile phone, smartphone, personal digital assistant (PDA), wearable device, handheld PC (PPC), tablet computer, smart car system, smart TV, smart speaker, etc.

[0142] Those skilled in the art should understand that the above-described acquisition terminal 101 and terminal device 201 are merely examples. Other existing or future devices that are applicable to the embodiments of this application should also be included within the scope of protection of the embodiments of this application, and are hereby incorporated by reference.

[0143] The video encoding and decoding method provided in the embodiments of this application will be described next. It should be noted that, in conjunction with... Figure 1 In the implementation environment shown, any of the video encoding methods described below can be performed by the encoder 100 in the source device 10. Figure 2 For example, any of the video encoding methods mentioned below can be Figure 2 The acquisition end 101 in the process is used for execution. Any of the video decoding methods described below can be executed by the decoder 200 in the destination device 20. Figure 2 For example, any of the video decoding methods mentioned below can be Figure 2 The terminal device 201 in the process is executed.

[0144] Furthermore, for a video segment, the encoding end can encode all video frames in that segment one by one. Therefore, the following explanation will use one video frame as an example. For ease of description, this video frame will be referred to as the first video frame. Please refer to... Figure 4 , Figure 4 This is a flowchart of a video encoding method provided in an embodiment of this application. The method includes the following steps.

[0145] Step 401: Encode the multiple coding blocks included in the first video frame to be encoded to obtain the encoding result of the multiple coding blocks and the second video frame.

[0146] In this embodiment, the multiple coding blocks can be encoded using conventional encoding methods, or they can be encoded based on the mask label of the first video frame, where the mask label of the first video frame is used to distinguish between textured and non-textured regions in the first video frame. Of course, the multiple coding blocks can also be encoded in other ways, and this embodiment does not limit this approach.

[0147] When encoding the multiple coding blocks based on the mask label of the first video frame, the multiple coding blocks included in the first video frame can be encoded according to the following steps (1)-(3).

[0148] (1) Determine the mask label of the first video frame based on the first video frame.

[0149] There are several ways to determine the mask label of the first video frame; one method will be described below. Specifically, the first video frame is semantically segmented to obtain a semantic segmentation map. This map includes multiple semantic regions, each corresponding to a semantic category. Based on the semantic categories corresponding to these semantic regions, mask labels are determined for each of these semantic regions. Finally, the mask label for the first video frame is determined based on these mask labels.

[0150] In this embodiment, the first video frame is input into a semantic segmentation network to obtain a semantic segmentation map. That is, the first video frame is semantically segmented using the semantic segmentation network. Of course, other methods can also be used to semantically segment the first video frame, and this embodiment does not limit this method.

[0151] Since the mask label for each of these multiple semantic regions is determined in the same way, we will now take one of these semantic regions as an example. That is, a semantic region is selected from these multiple semantic regions. If there is no texture category corresponding to the first semantic category in the correspondence between the semantic category and texture category, then the mask label for the selected semantic region is determined to be a first value. The first semantic category is the semantic category corresponding to the selected semantic region, and the first value indicates that the corresponding region is a non-texture region. If there is a texture category corresponding to the first semantic category in the stored correspondence between semantic categories and texture categories, then the mask label for the selected semantic region is determined to be a second value, and the second value indicates that the corresponding region is a texture region.

[0152] For some typical and special regions, these regions typically possess rich textures, such as grasslands, buildings, mountains, and plants. Therefore, embodiments of this application predetermine the semantic categories and texture categories of these regions and store the correspondence between semantic categories and texture categories. Thus, for a selected semantic region, if the stored correspondence between semantic categories and texture categories does not contain a texture category corresponding to the first semantic category, it indicates that the selected semantic region is a non-texture region, and the mask label corresponding to the selected semantic region is determined to be a first value. If the correspondence between semantic categories and texture categories contains a texture category corresponding to the first semantic category, it indicates that the selected semantic region is a texture region, and the mask label corresponding to the selected semantic region is determined to be a second value.

[0153] It should be noted that a texture category may correspond to one semantic category, and conversely, a texture category may correspond to multiple semantic categories. For example, the semantic categories of "trees" and "grass" correspond to the same texture category, namely "plants." Similarly, the semantic categories of "mountains" and "hills" correspond to the same texture category, namely "mountains."

[0154] The mask labels corresponding to semantic regions are used to indicate whether the corresponding region is a textured region or a non-textured region. In some embodiments, the mask label value is a value between 0 and 255, with the first value and the second value being two different values, for example, the first value is 255 and the second value is 0. Thus, after determining the mask label corresponding to each semantic region, the mask labels corresponding to these multiple semantic regions are used as the mask labels for the first video frame. That is, the mask labels for the first video frame include the mask labels corresponding to these multiple semantic regions, and the mask labels for the first video frame include binarized mask labels.

[0155] For example, please refer to Figure 5 , Figure 5 Figure (a) shows the first video frame. Semantic segmentation of the first video frame yields the semantic segmentation map shown below. Figure 5 As shown in Figure (b), the semantic segmentation map includes multiple semantic regions, each corresponding to a semantic category. Based on the semantic categories corresponding to these multiple semantic regions, mask labels corresponding to these multiple semantic regions are determined, resulting in the mask labels for the first video frame as shown in Figure (b). Figure 5 As shown in Figure (c), the mask label of the first video frame includes a binarized mask label to distinguish between textured and non-textured regions in the first video frame.

[0156] (2) Based on the mask label of the first video frame, determine the quantization parameters corresponding to the multiple coding blocks respectively. The quantization parameters corresponding to the coding blocks belonging to the texture region are greater than the quantization parameters corresponding to the coding blocks in the non-texture region.

[0157] Since the mask labels of the first video frame are determined on a semantic region basis, that is, the same semantic region in the mask labels of the first video frame corresponds to the same mask label, or all pixels in the same semantic region correspond to the same mask label. However, the quantization parameters are determined on a coding block basis. Therefore, it is necessary to determine the mask label corresponding to each coding block in the multiple coding blocks based on the mask labels of the first video frame, and then determine the quantization parameters corresponding to the multiple coding blocks based on the mask labels corresponding to the multiple coding blocks respectively.

[0158] As an example, for any one of the multiple coding blocks, based on the mask label of the first video frame, the number of pixels within that coding block with a mask label of a first value is counted to obtain a first count. A ratio is determined between the first count and a second count, where the second count is the total number of pixels in the coding block. If the ratio is greater than or equal to a ratio threshold, the mask label corresponding to that coding block is determined to be the first value; if the ratio is less than the ratio threshold, the mask label corresponding to that coding block is determined to be the second value. That is, if the ratio is greater than or equal to the ratio threshold, it indicates that most of the area within the coding block is a non-texture region, and the mask label corresponding to that coding block is determined to be the first value. If the ratio is less than the ratio threshold, it indicates that most of the area within the coding block is a texture region, and the mask label corresponding to that coding block is determined to be the second value.

[0159] After determining the mask label corresponding to each coding block, the quantization parameter corresponding to each coding block can be determined based on the mask label. For any coding block among the multiple coding blocks, if the mask label corresponding to the coding block is a first value, then the quantization parameter corresponding to the coding block is determined as the first quantization parameter. If the mask label corresponding to the coding block is a second value, then the first quantization parameter is added to the set quantization parameter to obtain the second quantization parameter, and the quantization parameter corresponding to the coding block is determined as the second quantization parameter.

[0160] The determination of the quantization parameters corresponding to the coding block belonging to the texture region can also be expressed by the following formula (1).

[0161] QP new =QP old +QP adjust (1)

[0162] In the above formula (1), QP new The quantization parameter corresponding to the encoded block belonging to the texture region, i.e., the second quantization parameter, QP. old QP is the first quantization parameter. adjust To set quantization parameters.

[0163] It should be noted that the ratio threshold is preset, for example, 0.8. During application, this ratio threshold can be adjusted according to different needs. Similarly, the quantization parameters are also preset and can be adjusted according to different requirements.

[0164] Furthermore, for any video segment, there exists a target bitrate, which is the average bitrate after encoding the video segment. Since a video segment comprises multiple frames, the target bitrate allows us to determine the encoding bitrate of the first video frame, and subsequently, the first quantization parameter is determined based on this bitrate. In other words, the first quantization parameter can be determined using conventional encoding methods.

[0165] Because the texture area is particularly rich in texture, encoding the texture area using conventional encoding methods would consume a large amount of encoding bitrate. However, users are not sensitive to distortion or changes in texture areas. Therefore, in this embodiment, non-texture areas are encoded using conventional encoding methods, while texture areas are encoded using higher quantization parameters to reduce the encoding quality of texture areas and thus reduce the encoding bitrate occupied by texture areas.

[0166] (3) Encode the multiple coding blocks based on the quantization parameters corresponding to each of the multiple coding blocks.

[0167] Based on the above description, the quantization parameters of these multiple coding blocks may be different. Therefore, each coding block can be encoded according to its corresponding quantization parameters.

[0168] It should be noted that the encoding method of these multiple encoding blocks can be any encoding method, and this application embodiment does not limit it.

[0169] As an example, for steps (2) and (3) above, an index table can be formed from the mask labels and position information of the multiple coding blocks, and the index table can be input into the conventional encoder. When the conventional encoder encodes a coding block at a certain position, it retrieves the mask label corresponding to the coding block from the index table, determines the quantization parameters of the coding block according to the above method based on the mask label corresponding to the coding block, and then encodes the coding block according to the quantization parameters corresponding to the coding block.

[0170] The embodiments of this application can improve the conventional encoder, so that during the encoding of the multiple coding blocks by the conventional encoder, not only the encoding results of the multiple coding blocks can be obtained, but also the second video frame can be obtained.

[0171] Since the edge residuals between the texture regions in the first and second video frames are subsequently encoded into the bitstream, and the decoding end parses the encoding results of these multiple coded blocks to obtain the second video frame, and then synthesizes the texture based on the edge residuals in the bitstream to obtain the reconstructed first video frame, the second video frame can also be called the basic reconstructed frame.

[0172] Step 402: Based on the first video frame and the second video frame, generate a first texture edge residual map. The first texture edge residual map is used to characterize the edge residual between the texture regions in the first video frame and the second video frame.

[0173] In this embodiment of the application, a first texture edge residual map is generated based on the first video frame and the second video frame according to the following steps (1)-(3).

[0174] (1) Generate a texture edge map based on the first video frame. The texture edge map is used to represent the edge information within the texture region in the first video frame.

[0175] In the embodiments of this application, the first video frame may be an I-frame or a non-I-frame, such as a P-frame or a B-frame. The method of generating texture edge maps differs for I-frames and non-I-frames; therefore, I-frames and non-I-frames will be described separately below.

[0176] The first video frame is an I-frame.

[0177] In this case, the texture edge map is used to represent the edge information in all texture regions in the first video frame. The process of generating the texture edge map based on the first video frame includes: detecting the edge information in all texture regions in the first video frame based on the mask label of the first video frame to generate the texture edge map.

[0178] Since the mask label of the first video frame is used to distinguish between textured and non-textured regions in the first video frame, and I-frames only perform intra-frame predictive coding and do not depend on other video frames, all textured regions in the first video frame can be determined through the mask label. Then, edge information within all textured regions in the first video frame is detected to generate a texture edge map.

[0179] Based on the above description, the mask label of the first video frame includes a binarized mask label. The region with a mask label of a first value is used to represent a non-textured region, and the region with a mask label of a second value is used to represent a textured region. Therefore, the coordinates of the pixels with the mask label of the second value can be determined from the first video frame, thereby obtaining the textured region within the first video frame. Then, the edge information within the textured region can be detected to generate a texture edge map.

[0180] In this application embodiment, there are various methods for detecting edge information within the textured region. For example, the Canny edge detection method, the Sobel edge detection method, the Prewitt edge detection method, and so on.

[0181] Taking the Canny edge detection method as an example, the gradient value of each pixel within all texture regions in the first video frame is determined. Pixels with gradient values ​​greater than or equal to a gradient threshold are identified as edge points, while pixels with gradient values ​​less than the threshold are identified as non-edge points. The pixel values ​​corresponding to edge points are determined as the third value, and the pixel values ​​corresponding to non-edge points are determined as the fourth value, thus generating a texture edge map. That is, in the texture edge map, pixels with the third value are edge points, and pixels with the fourth value are non-edge points.

[0182] It should be noted that the gradient threshold is preset, and it can be adjusted according to different needs during application. Furthermore, since the gradient of edge points within a textured region is lower than that of edge points within a non-textured region, and the gradient threshold is used to determine edge points within the textured region, the gradient threshold is set relatively small in this embodiment. For example, the gradient threshold is 50.

[0183] In addition, the third and fourth values ​​can be any two different values ​​between 0 and 255. For example, the third value is 255 and the fourth value is 0.

[0184] The first video frame is a non-I-frame.

[0185] In this scenario, the texture edge map is used to represent edge information within a local texture region in the first video frame. The process of generating the texture edge map based on the first video frame includes: determining a target location range, where the target location range is the position of the newly added region in the first video frame relative to the third video frame, and the third video frame serves as the reference frame for the first video frame; and generating the texture edge map based on the first video frame and the target location range.

[0186] Since non-I-frames require intra-frame predictive coding and inter-frame predictive coding, and inter-frame predictive coding relies on other video frames, i.e., reference frames, it is not necessary to detect edge information within all texture regions to generate texture edge maps for non-I-frames. Instead, edge information within local texture regions is sufficient. These local texture regions are the texture regions added to the first video frame relative to the third video frame.

[0187] The target location range refers to the location of the newly added region. The process of determining the target location range includes: determining global motion parameters, which characterize the amount of motion of pixels in the third video frame moving to the first video frame; performing motion compensation on the third video frame based on the global motion parameters to obtain a first motion-compensated frame, which includes the newly added region of the first video frame relative to the third video frame; and traversing the coordinates of pixels within this newly added region in the first motion-compensated frame to obtain the target location range.

[0188] Since global motion parameters are used to characterize the amount of motion of pixels in the third video frame moving to the first video frame, in some embodiments, global motion parameters can be determined based on the first and third video frames. In other embodiments, the second video frame can be obtained during the encoding of multiple coded blocks included in the first video frame, and similarly, the fourth video frame can be obtained during the encoding of multiple coded blocks included in the third video frame. Therefore, global motion parameters can also be determined based on the second and fourth video frames.

[0189] The method for determining global motion parameters based on the first and third video frames is the same as the method for determining global motion parameters based on the second and fourth video frames; therefore, the latter will be used as an example in the following description. In this embodiment, the global motion parameters can be parameters corresponding to any transformation method. The affine transformation will be used as an example in the following description.

[0190] That is, multiple keypoints are determined from the key region of the second video frame, and multiple keypoints are determined from the key region of the fourth video frame. The keypoints in the second video frame are matched with the keypoints in the fourth video frame to obtain multiple keypoint pairs. Based on these multiple keypoint pairs, global motion parameters are determined according to the random sample consensus algorithm.

[0191] In this embodiment, the key region can be determined in any way. After determining the key region, multiple key points can be determined from the key regions of the second and fourth video frames using the Oriented Fast and Rotated Brief (ORB) algorithm. Of course, other algorithms can also be used to determine them, and this embodiment does not limit this.

[0192] It should be noted that, to improve the efficiency of determining global motion parameters, the number of keypoints determined from the second and fourth video frames is less than or equal to a first quantity threshold. This first quantity threshold is preset, for example, it is 1000. During application, this first quantity threshold can also be adjusted according to different needs.

[0193] The method for determining global motion parameters based on these multiple key point pairs using a random sampling consensus algorithm is a cyclic iterative approach. The i-th iteration of this cyclic iterative approach includes the following steps ag, where i is an integer greater than or equal to 1:

[0194] a. Randomly select three keypoint pairs from these multiple keypoint pairs.

[0195] b. Based on these three key point pairs, the affine transformation parameters for the i-th iteration are determined using the affine transformation model.

[0196] The affine transformation model is shown in formula (2) below.

[0197]

[0198] In formula (2) above, (x,y) and Let a0, a1, a2, b0, b1, and b2 be the coordinates before and after the affine transformation, respectively. Formula (2) above represents the mapping of the pixel (x, y) in the fourth video frame to the pixel in the second video frame after calculation using the affine transformation model.

[0199] Since a keypoint includes two matching keypoints, one of which is a pixel in the second video frame and the other is a pixel in the fourth video frame, after selecting three keypoint pairs, a set of affine transformation parameters can be determined based on these three keypoint pairs according to the above formula (2).

[0200] c. Based on the affine transformation parameters processed in the i-th iteration, determine the set of key points from key point pairs other than the three key point pairs.

[0201] As an example, the multiple keypoint pairs are traversed. Whenever a keypoint pair is encountered, if it is not one of the three keypoint pairs, the predicted coordinates of the second keypoint are determined using the affine transformation model based on the affine transformation parameters processed in the i-th iteration and the coordinates of the first keypoint. The first keypoint is the keypoint belonging to the fourth video frame in this keypoint pair, and the second keypoint is the keypoint belonging to the second video frame in this keypoint pair. The distance between the actual coordinates and the predicted coordinates of the second keypoint is determined. If this distance is less than a distance threshold, the keypoint pair is added to the keypoint set. It is then determined whether the number of keypoint pairs in the keypoint set is greater than a second number threshold. If the number of keypoint pairs in the keypoint set is greater than the second number threshold, the traversal stops. If the number of keypoint pairs in the keypoint set is less than or equal to the second number threshold, the next keypoint pair is traversed. If the distance is greater than or equal to the distance threshold, the next keypoint pair is traversed.

[0202] It should be noted that the second quantity threshold is preset, and it can be adjusted according to different needs during application.

[0203] d. Based on this set of key points, determine the error of the i-th iteration.

[0204] Based on the description of step c above, when determining the key point set, the distance of each key point pair in the key point set has been determined according to the affine transformation parameters of the i-th iteration. Therefore, the average distance of each key point pair in the key point set can be taken to obtain the error of the i-th iteration.

[0205] e. If the error of the i-th iteration is less than the minimum error, then update the minimum error to the error of the i-th iteration.

[0206] It should be noted that the initial value of the minimum error is predetermined, and during application, the initial value of the minimum error can be adjusted according to different needs.

[0207] f. If i is less than the maximum number of iterations, then execute the (i+1)th iteration of this loop iteration method.

[0208] It should be noted that the maximum number of iterations is predetermined, and can be adjusted according to different needs during application.

[0209] g. If i is greater than or equal to the maximum number of iterations, then terminate the execution of this loop iteration method and determine the affine transformation parameters processed in the i-th loop as global motion parameters.

[0210] For example, the pseudocode for determining global motion parameters using the random sampling consensus algorithm is shown below.

[0211] enter:

[0212] data: multiple key points

[0213] model: Affine transformation model

[0214] k: Maximum number of iterations

[0215] t: Distance threshold, i.e., the distance between the actual coordinates of the keypoint and the coordinates estimated by the affine transformation model.

[0216] d: The second quantity threshold, which is the minimum number of keypoint pairs required for a good fit.

[0217] Output:

[0218] bestfit: Affine transformation parameters

[0219] initialization:

[0220] Current iteration count - iterations = 0

[0221] Optimal model - bestfit = null

[0222] Minimum error - bester = inf

[0223] Iterative search for the optimal solution:

[0224]

[0225]

[0226] Optionally, after determining the global motion parameters, the method further includes: encoding the global motion parameters and writing the encoded results of the global motion parameters into the bitstream.

[0227] In some embodiments, the global motion parameters can be encoded using signed exponential Golomb coding. Of course, in other embodiments, the global motion parameters can also be encoded in other ways.

[0228] The process of motion compensation for the third video frame based on global motion parameters includes: For a first pixel in the third video frame, based on the global motion parameters and the coordinates of the first pixel, the coordinates of a second pixel in the first motion compensation frame that matches the first pixel are determined. The first pixel can be any pixel in the third video frame. If the coordinates of the second pixel do not exceed the boundary of the third video frame, the pixel value of the first pixel is used as the pixel value of the second pixel in the first motion compensation frame, that is, the pixel value of the first pixel is directly copied to the second pixel in the first motion compensation frame. If the coordinates of the second pixel exceed the boundary of the third video frame, the pixel value of the second pixel in the first motion compensation frame is determined to be the fifth value. After traversing every pixel in the third video frame, the first motion compensation frame is obtained.

[0229] It should be noted that the fifth value can be any value between 0 and 255; for example, the fifth value is 0.

[0230] For example, please refer to Figure 6 , Figure 6 Figure (a) shows the third video frame, with the fifth value being 0. After motion compensation of the third video frame, the resulting first motion-compensated frame is... Figure 6 As shown in Figure (b) of the document. Figure 6 The black area on the right side of Figure (b) represents the newly added area in the first video frame compared to the third video frame. Thus, by traversing the coordinates of the pixels within the black area of ​​the first motion-compensated frame, the target location range can be obtained.

[0231] The process of generating a texture edge map based on a first video frame and a target location range includes: extracting an image region located within the target location range from the first video frame to obtain a first image region; and detecting edge information within all texture regions of the first image region based on the mask labels of the first video frame to generate a texture edge map.

[0232] Since the mask label of the first video frame is used to distinguish between textured and non-textured regions in the first video frame, and the first video frame includes a first image region, after extracting the first image region from the first video frame, the textured regions in the first image region can be determined using the mask label of the first video frame. Then, edge information within all textured regions of the first image region is detected to generate a texture edge map.

[0233] The process of detecting edge information in all texture regions of the first image region is the same as the process of detecting edge information in all texture regions of the first video frame described above. Please refer to the above description, and it will not be repeated here.

[0234] (2) Generate a first edge map based on the second video frame. The first edge map is used to characterize the edge information in the second video frame.

[0235] Based on the above description, the first video frame may be an I-frame or a non-I-frame. The method for generating the first edge map differs for I-frames and non-I-frames; therefore, I-frames and non-I-frames will be introduced separately below.

[0236] The first video frame is an I-frame.

[0237] In this case, the first edge map is used to characterize the edge information of the second video frame. The process of generating the first edge map based on the second video frame includes: detecting the edge information of the second video frame to generate the first edge map.

[0238] For details on edge detection of the second video frame, please refer to the previous description; it will not be repeated here.

[0239] The first video frame is a non-I-frame.

[0240] In this case, the first edge map is used to characterize the edge information in a local area in the second video frame. The process of generating the first edge map based on the second video frame includes: generating the first edge map based on the second video frame and the target location range.

[0241] Since non-I-frames require intra-frame predictive coding and inter-frame predictive coding, and inter-frame predictive coding depends on other video frames, i.e., reference frames, it is not necessary to detect edge information in all regions to generate the first edge map for non-I-frames. Instead, edge information in local regions only needs to be detected to generate the first edge map.

[0242] The process of generating a first edge map based on a second video frame and a target location range includes: extracting an image region located within the target location range from the second video frame to obtain a second image region; and detecting the edge information of the second image region to generate the first edge map.

[0243] The process of detecting edge information in the second image region is the same as the process of detecting edge information in the second video frame described above. Please refer to the above description, and it will not be repeated here.

[0244] (3) Generate a first texture edge residual map based on the texture edge map and the first edge map.

[0245] Since there is overlapping edge information in the texture edge map and the first edge map, in order to reduce the edge information transmitted to the decoding end, this embodiment of the application needs to generate a first texture edge residual map based on the texture edge map and the first edge map.

[0246] As an example, the first texture edge residual map is generated based on the texture edge map and the first edge map according to the following formula (3).

[0247]

[0248] In formula (3) above, I residual,i,j Let I be the pixel value of the pixel at coordinates (i,j) in the first texture edge residual map. ori,i,j Let I be the pixel value of the pixel at coordinates (i,j) in the texture edge map. encode,i,j This represents the pixel value of the pixel at coordinates (i,j) in the first edge map. 255 indicates that the pixel is an edge point, and 0 indicates that the pixel is not an edge point.

[0249] As can be seen from the above formula (3), the first texture edge residual map only includes edge information that exists in the first video frame but disappears in the second video frame due to encoding distortion. That is, the first texture edge residual map only includes edge information that exists in the texture edge map but does not exist in the first edge map. In this way, in the subsequent process, only the edge information that exists in the first video frame but does not exist in the second video frame needs to be transmitted to the decoding end, without transmitting the edge information that exists in the second video frame but does not exist in the first video frame to the decoding end, and without transmitting the edge information that overlaps between the first and second video frames to the decoding end, thereby reducing the edge information that needs to be transmitted.

[0250] Optionally, after generating the texture edge map based on the first video frame, the method further includes: downsampling the texture edge map according to a first sampling ratio, where the first sampling ratio is proportional to the quantization parameters of the non-texture regions in the first video frame; and upsampling the downsampled texture edge map according to the first sampling ratio. Similarly, after generating the first edge map based on the second video frame, the method further includes: downsampling the first edge map according to the first sampling ratio; and upsampling the downsampled first edge map according to the first sampling ratio.

[0251] Since there is overlapping edge information in the texture edge map and the first edge map, and the edge information extracted by the above method is relatively rich (i.e., the texture edge map and the first edge map contain relatively rich content), this embodiment of the application can further downsample and upsample the texture edge map and the first edge map to reduce the edge information in the texture edge map and the first edge map, while balancing the coding rate and image distortion, and considering the sparsity and binarization characteristics of the texture edge map. Thus, the first texture edge residual map determined by the sampled texture edge map and the sampled first edge map contains even less edge information, thereby further reducing the edge information required for encoding and transmission.

[0252] In some embodiments, the correspondence between quantization parameters and sampling ratios can be stored in advance. Thus, after determining the quantization parameters (i.e., the first quantization parameters) for the non-texture regions in the first video frame, the first sampling ratio is obtained from the correspondence between these quantization parameters and sampling ratios based on the quantization parameters for the non-texture regions.

[0253] For example, the correspondence between quantization parameters and sampling ratios is shown in Table 1 below.

[0254] Table 1

[0255] Quantization parameters 26 32 38 ...... Sampling ratio 1.5 1.5 2 ......

[0256] It should be noted that the correspondence between quantization parameters and sampling ratios shown in Table 1 above is only one example, and the correspondence between quantization parameters and sampling ratios is not unique. Generally speaking, the larger the quantization parameter, the smaller the coding bitrate of the coding block, and the smaller the coding bitrate of edge information should also be. Therefore, the sampling ratio should be larger to reduce the transmitted edge information. That is, the sampling ratio is directly proportional to the quantization parameter of non-texture regions. Of course, the sampling ratio and the quantization parameter of non-texture regions can also have other relationships, as long as the coding bitrate after encoding the coding block and edge residuals according to the method provided in this application is less than the coding bitrate of conventional coding methods, and the texture quality of the finally reconstructed video frame is high.

[0257] Based on the above description, the texture edge map and the first edge map can be downsampled and then upsampled according to the first sampling ratio. Of course, the texture edge map and the first edge map can also be left unsampled. However, regardless of whether the texture edge map and the first edge map are sampled, after the first texture edge residual map is determined, it can be downsampled according to the second sampling ratio to reduce the edge information that needs to be transmitted.

[0258] It should be noted that the first sampling ratio and the second sampling ratio can be the same or different, and this application embodiment does not limit this.

[0259] Step 403: Encode the first texture edge residual map.

[0260] There are various methods for encoding the first texture edge residual map, such as screen content coding (SCC), JPEG, and JBIG. However, because SCC has good compression capabilities for a small number of color images or videos, this embodiment of the application can use SCC to encode the first texture edge residual map.

[0261] In practical applications, the HM16.20 software under the High Efficiency Video Coding (HEVC) standard can be used to perform lossless encoding on the first texture edge residual map with the encoding configuration of ALL Intra QP=0.

[0262] Step 404: Write the encoding results of the multiple encoding blocks and the encoding results of the first texture edge residual map into the bitstream.

[0263] To improve the quality of texture regions reconstructed in the first video frame by the decoder, the encoder obtains a second video frame during the encoding process of the multiple coding blocks. This allows the encoder to determine the edge residuals between the texture regions in the first and second video frames, i.e., a first texture edge residual map, which is then encoded into the bitstream. After obtaining the second video frame, the decoder synthesizes the textures within the texture regions of the second video frame based on the first texture edge residual map, thereby forming realistic, high-quality textures and improving the quality of the reconstructed texture regions in the first video frame.

[0264] Because textured regions are particularly rich in texture, directly encoding these regions using conventional encoding methods would consume a significant amount of bitrate. However, the human eye is not sensitive to distortions or changes in these textured regions. Therefore, in this embodiment, the quantization parameters of the textured regions are increased to encode the multiple coding blocks included in the first video frame, thereby reducing the bitrate occupied by the coding blocks belonging to the textured regions and thus reducing the overall bitrate of the video frame. In this way, while reducing the bitrate, the quality of the textured regions in the first video frame reconstructed at the decoding end can still be guaranteed.

[0265] In addition, the embodiments of this application use edge information as texture features for texture synthesis, mainly because edge information has the following characteristics and advantages: (1) Edge information is a low-level image feature and is easy to extract; (2) Complex texture patterns have rich edges, and edges are crucial for the restoration of the content structure in the image; (3) Edges are simple and easy to compress compared to other features; (4) Edges are pixel domain features and are easy to use as input for training and use in texture synthesis networks. At the same time, texture synthesis in video also needs to take into account the characteristics of the spatiotemporal domain, and pixel domain features are easy to perform spatiotemporal domain operations.

[0266] Please refer to Figure 7 , Figure 7 This is a flowchart of a video decoding method provided in an embodiment of this application. The method includes the following steps.

[0267] Step 701: Parse the bitstream to obtain the second video frame.

[0268] Since the bitstream includes the encoding results of multiple coded blocks in the first video frame, the decoding end can parse the encoding results of these multiple coded blocks in the bitstream to obtain the second video frame.

[0269] Based on the above description, these multiple coded blocks can be encoded using a conventional encoder, so the decoding end can also be decoded using a conventional decoder.

[0270] Step 702: Generate a first edge map based on the second video frame. The first edge map is used to represent the edge information of the second video frame.

[0271] In some embodiments, edge information of the second video frame is detected to generate a first edge map.

[0272] The implementation method for detecting the edge information of the second video frame to generate the first edge map is described in step 402 above, and will not be repeated here.

[0273] Based on the preceding description, the encoding end can either downsample and then upsample the texture edge map and the first edge map, or it can choose not to sample at all. If the encoding end downsamples and then upsamples the texture edge map and the first edge map, the decoding end, after generating the first edge map based on the second video frame, can also downsample and then upsample the first edge map. That is, the first edge map is downsampled according to a first sampling ratio, which is proportional to the quantization parameters of the non-texture regions in the first video frame. The downsampled first edge map is then upsampled according to the first sampling ratio.

[0274] Since the encoding end encodes multiple coded blocks based on their quantization parameters in the first video frame, and then writes the encoding results of these blocks into the bitstream and transmits them to the decoding end, the encoding end also instructs the decoding end on the quantization parameters of these multiple coded blocks to ensure that the decoding end can accurately parse the second video frame. In this way, the decoding end can obtain the quantization parameters of the non-texture regions in the first video frame, i.e., the first quantization parameters, and thus determine the first sampling ratio based on the first quantization parameters.

[0275] The method for determining the first sampling ratio at the decoding end is described in step 402 above and will not be repeated here.

[0276] Step 703: Obtain the second texture edge residual map, which is used to characterize the edge residual between all texture regions in the first video frame to be reconstructed and the second video frame.

[0277] Based on the preceding description, the first video frame may be an I-frame or a non-I-frame. The methods for obtaining the second texture edge residual map differ for I-frames and non-I-frames; therefore, I-frames and non-I-frames will be described separately below.

[0278] The first video frame is an I-frame.

[0279] In this case, the process of obtaining the second texture edge residual map includes: parsing the bitstream to obtain the reconstructed first texture edge residual map, and the second texture edge residual map being the reconstructed first texture edge residual map.

[0280] For an I-frame, the first texture edge residual map is used to characterize the edge residuals between all texture regions in the first video frame and the second video frame, not just the edge residuals between texture regions in the newly added region. Therefore, the reconstructed first texture edge residual map can be directly used as the second texture edge residual map.

[0281] The first video frame is a non-I-frame.

[0282] In this case, the process of obtaining the second texture edge residual map includes: parsing the bitstream to obtain the reconstructed first texture edge residual map; determining the third texture edge residual map based on the second video frame and global motion parameters, where the global motion parameters characterize the amount of motion of pixels in the fourth video frame moving to the second video frame, and the fourth video frame serves as the reference frame for the first video frame to be reconstructed; and superimposing the reconstructed first texture edge residual map with the third texture edge residual map to obtain the second texture edge residual map.

[0283] The third texture edge residual map is used to characterize the texture edge residuals of non-newly added regions in the first video frame. Since non-I-frames do not encode the edge residuals of all texture regions into the bitstream, but only the edge residuals of newly added regions, the texture edge residuals of non-newly added regions can be determined by using a reference frame for non-I-frames. That is, the texture edge residuals of non-newly added regions can be determined by using the texture edge residuals of the fourth video frame.

[0284] The process of determining the third texture edge residual map based on the second video frame and global motion parameters includes: performing motion compensation on the texture edge residual map of the fourth video frame based on the global motion parameters to obtain a motion-compensated edge residual map; performing motion compensation on the fourth video frame based on the global motion parameters to obtain a second motion-compensated frame; performing optical flow estimation on the second video frame and the second motion-compensated frame to obtain a first optical flow field; and performing optical flow transformation on the motion-compensated edge residual map based on the first optical flow field to obtain the third texture edge residual map.

[0285] Based on the preceding description, the encoder may or may not encode the global motion parameters into the bitstream. If the encoder encodes the global motion parameters into the bitstream, the decoder can directly parse the bitstream to obtain the global motion parameters. If the encoder does not encode the global motion parameters into the bitstream, the decoder can determine the global motion parameters based on the second and fourth video frames.

[0286] The method by which the decoding end determines the global motion parameters based on the second and fourth video frames is described in step 402 above and will not be repeated here.

[0287] The texture edge residual map of the fourth video frame is used to characterize the edge residuals between all texture regions in the third and fourth video frames. The method for motion compensation of the texture edge residual map of the fourth video frame and the fourth video frame based on global motion parameters is the same as the method for motion compensation of the third video frame described above. Please refer to the relevant descriptions above for details, which will not be repeated here.

[0288] Based on the terminology explained above, the optical flow field refers to the amount of motion of all pixels in one video frame moving to another video frame. Moreover, the optical flow field includes the motion vector corresponding to each pixel in the video frame. Therefore, after performing optical flow estimation on the second video frame and the second motion-compensated frame to obtain the first optical flow field, the motion vector of the corresponding pixel in the first optical flow field can be added to the coordinates of each pixel in the motion-compensated edge residual map, and then the pixel values ​​can be copied to obtain the third texture edge residual map, thereby realizing the optical flow transformation of the motion-compensated edge residual map.

[0289] Based on the preceding description, the encoder can downsample the first texture edge residual map according to the second sampling ratio, or it can choose not to downsample it. If the encoder downsamples the first texture edge residual map, in order to ensure that the resolution of the first texture edge residual map is the same, the decoder needs to upsample the reconstructed first texture edge residual map according to the second sampling ratio after obtaining it.

[0290] It should be noted that when the first video frame is not an I-frame, the second texture edge residual map can be obtained not only in the manner described above, but also in other ways. For example, optical flow estimation can be performed on the second and fourth video frames to obtain the second optical flow field. Based on the second optical flow field, optical flow transformation can be performed on the texture edge residual map of the fourth video frame to obtain the second texture edge residual map.

[0291] In other words, the encoding end only needs to encode the multiple coding blocks included in the first video frame into the bitstream, without needing to encode the texture edge residual map of the newly added region into the bitstream. Thus, for the decoding end, the texture edge residual map of the current frame can be obtained by performing optical flow transformation on the texture edge residual map of the reference frame.

[0292] Step 704: Overlay the second texture edge residual map with the first edge map to obtain the second edge map.

[0293] Since the second texture edge residual map is used to characterize the edge residual between all texture regions in the second video frame and the first video frame to be reconstructed, and the first edge map is used to characterize the edge information of the second video frame, the edge information of the first video frame to be reconstructed, i.e., the second edge map, can be obtained by superimposing the second texture edge residual map and the first edge map.

[0294] Overlay refers to fusing edge information from two images. That is, for a pixel at the same location in two images, if at least one of the two images represents the pixel as an edge point, then the pixel is determined to be an edge point; if both images represent the pixel as a non-edge point, then the pixel is determined to be a non-edge point.

[0295] In some embodiments, the superposition of the second texture edge residual map and the first edge map can also be called the addition of the pixel domain, which can be represented by the following formula (4).

[0296] I ts =I encode +I residual (4)

[0297] In formula (4) above, I ts For the second edge map, I encode For the first edge map, I residual This is the edge residual map of the second texture.

[0298] Step 705: Input the second video frame and the second edge map into the texture synthesis network to obtain the reconstructed first video frame.

[0299] Based on the second video frame, by synthesizing the edge information of all texture regions in the first video frame, the textures in the reconstructed texture regions of the first video frame can be made more realistic.

[0300] This application's embodiments design a Generative Adversarial Network (GAN) that uses edge information as feature input for texture synthesis. The GAN includes a generator and a discriminator. The discriminator distinguishes between unencoded video frames (e.g., the first video frame) and reconstructed video frames (e.g., the reconstructed first video frame), and tries to classify the reconstructed video frames as "fake" and the unencoded video frames as "real." The generator tries to "deceive" the discriminator by generating realistic images through adversarial learning. That is, for texture synthesis, due to the presence of edge information, the GAN needs to simultaneously receive the base reconstructed frame to be synthesized and the edge information as input, and can effectively fuse the edge information to generate realistic textures. The aforementioned texture synthesis network is the generator in the GAN.

[0301] Please refer to Figure 8 , Figure 8This is a schematic diagram of the structure of a generator provided in an embodiment of this application. The generator uses super-resolution generative adversarial network (SRGAN) in the field of video super-resolution as its basic architecture, and removes the upsampling at the end of the network and subsequent layers. The generator includes five convolutional (Conv) layers and 19 cascaded residual blocks (Resblocks).

[0302] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the Resblock structure provided in an embodiment of this application. The Resblock includes two convolutional layers, each with a 3x3 kernel and 48 channels. Each of these convolutional layers is preceded by a spatial feature transform (SFT) layer, and... Figure 8 The last Resblock also includes an SFT layer.

[0303] Please refer to Figure 10 , Figure 10 This is a schematic diagram of an SFT layer provided in an embodiment of this application. The SFT layer can effectively train the modulation parameters (γ, β) in the network based on prior conditions. The modulation parameters (γ, β) can adaptively transform the input features of the SFT layer, and can fuse prior conditions into the input features of the SFT layer.

[0304] Combination Figures 8 to 10 As can be seen, the generator designed in this application embodiment performs texture synthesis step by step to obtain high-quality textures. That is, the features of the texture edge map after three convolutional layers are used as prior conditions input to each SFT layer, and the input features of each SFT layer are the current features of the base reconstructed frame after passing through the network. In this way, in each SFT layer, the network can continuously fuse / enhance edge information, influencing the generation process through relevant network parameters, and finally recovering the texture structure information under the guidance of the texture edge map to synthesize high-quality textures.

[0305] Please refer to Figure 11 , Figure 11This is a schematic diagram of a discriminator provided in an embodiment of this application. The discriminator adopts the structure of the Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) in the video super-resolution field, including four discriminator blocks. Each discriminator block includes two sets of basic convolutional units, which include convolutional (Conv) layers, batch normalization (BN) layers, and LeakyReLU (with leakage correction linear units, a typical activation function) layers. To save computation and reduce computational complexity, the second convolutional layer of each discriminator block downsamples the input features, reducing the feature size to 1 / 4 of the original.

[0306] In this embodiment of the application, the generator and the discriminator can be trained simultaneously, and the training loss of the generator consists of the following three parts:

[0307] (1) The mean square error (MSE) loss representing the signal fidelity is denoted as L. rec It is determined by the following formula (5).

[0308] L rec =MSE(x r ,x f (5)

[0309] Where, x r For the real samples used in network training, i.e., unencoded video frame samples, x f This is to generate samples, which are video frame samples reconstructed after texture synthesis.

[0310] (2) Calculate the perceptual loss for the feature domain distance, denoted as L. P It is determined by the following formula (6).

[0311] L P =VGG(x r ,x f (6)

[0312] Since the visual geometry group (VGG) is a high-performance feature extractor, this embodiment uses VGG loss as the perceptual loss for network training. That is, real and generated samples are input into a pre-trained VGG19 network, features are extracted through the Conv5_4 layer in the VGG19 network, and then the perceptual loss is calculated using these features.

[0313] (3) Countermeasure loss, denoted as L G It is determined by the following formula (7).

[0314]

[0315] In formula (7) above, D(x) r ) and D(x f E[D(x)] represents the value of the sample after passing through the discriminator. f E[D(x)] represents the expected value of multiple generated samples after passing through the discriminator. r [)] represents the expected value of multiple real samples after passing through the discriminator. Represents log(1-D(x)) of multiple real samples r ,x f The expected value of ) Represents the log(D(x) of multiple generated samples f ,x r The expected value of ).

[0316] In summary, the training loss of the generator is obtained by summing the above three losses according to their weights, and can be expressed by the following formula (8).

[0317] L G,total =μ rec L rec +μ p L P +μ GAN L G (8)

[0318] In the above formula (8), μ rec μ p μ GAN These are the weights corresponding to the three parts of loss mentioned above.

[0319] The training loss of the discriminator is denoted as , and this training loss is related to the aforementioned adversarial loss L. G They are opposites and antagonistic, which can be represented by the following formula (9).

[0320]

[0321] After training the generator and discriminator using the methods described above, the base reconstructed frame and edge map can be input into the generator to obtain the texture-synthesized frame. That is, the second video frame and the second edge map are input into the texture synthesis network to obtain the reconstructed first video frame, which can also be called the texture-synthesized frame.

[0322] In this embodiment, to improve the quality of the texture region in the first video frame reconstructed by the decoder, the encoder needs to determine the edge residual between the texture regions in the first and second video frames, i.e., the first texture edge residual map, and then encode the first texture edge residual map into the bitstream. Thus, after obtaining the second video frame, the decoder synthesizes the texture within the texture region in the second video frame based on the first texture edge residual map, thereby forming a realistic, high-quality texture and improving the quality of the reconstructed texture region in the first video frame. Furthermore, since the texture regions are particularly rich in texture, directly encoding these texture regions using conventional encoding methods would consume a significant amount of bitrate, but the human eye is not sensitive to distortions or changes in these texture regions. Therefore, the encoder increases the quantization parameters of the texture regions to encode multiple coding blocks included in the first video frame, thereby reducing the bitrate occupied by the coding blocks belonging to the texture regions, and thus reducing the overall bitrate of the video frame. In this way, while reducing the bitrate, the quality of the texture region in the first video frame reconstructed by the decoder can still be guaranteed.

[0323] In combination with the above Figure 4 The encoding method shown and Figure 7 The decoding method shown below will be followed by an exemplary explanation of the encoding and decoding methods for I-frames and non-I-frames.

[0324] Please refer to Figure 12 , Figure 12 This is an exemplary block diagram of a video encoding / decoding method for I-frames provided in an embodiment of this application. Figure 12 In the process, the encoding end performs texture region discrimination on the first video frame to determine the textured and non-texture regions within it. Then, based on the texture region discrimination results, the quantization parameters of the coding blocks belonging to textured regions within the multiple coding blocks included in the first video frame are adjusted. Furthermore, the texture edge residuals of all textured regions can be extracted using the texture region discrimination results to obtain a first texture edge residual map. The first texture edge residual map is downsampled, and then encoded. Afterward, the encoding end encodes these multiple coding blocks, writes the encoding results of these multiple coding blocks and the encoding result of the downsampled first texture edge residual map into the bitstream, and transmits it to the decoding end.

[0325] After receiving the bitstream transmitted from the encoding end, the decoding end parses the bitstream to obtain a downsampled first texture edge residual map, and then upsamples this first texture edge residual map. Furthermore, it can also parse the encoding results of these multiple coding blocks using conventional video decoding to obtain a second video frame. Then, using the upsampled first texture edge residual map, texture synthesis is performed on the second video frame to obtain a texture-synthesized frame, i.e., the reconstructed first video frame.

[0326] Because temporal techniques such as motion compensation can be used in non-I-frames, the bitrate of non-I-frames is much lower than that of I-frames. If the same texture synthesis process as I-frames is used for non-I-frames, the bitrate for edge feature encoding will be much higher than that of conventional encoding, failing to achieve the goal of bitrate saving. Therefore, different methods are needed to synthesize non-I-frames. Please refer to [reference needed]. Figure 13 , Figure 13 This is an exemplary block diagram of a non-I-frame video encoding / decoding method provided in an embodiment of this application. Figure 13 In the process, the encoder performs texture region discrimination on the first video frame to identify textured and non-texture regions within it. Then, based on the texture region discrimination results, the quantization parameters of the coding blocks belonging to textured regions within the multiple coding blocks included in the first video frame are adjusted. Furthermore, global motion parameters can be determined, and motion compensation is performed on the third video frame based on these parameters to obtain a first motion-compensated frame. The first motion-compensated frame is used to determine newly added regions in the first video frame relative to the third video frame, and texture edge residuals are extracted from these newly added regions based on the texture region discrimination results to obtain a first texture edge residual map. The first texture edge residual map is downsampled and then encoded. Afterward, the encoder encodes the global motion parameters and the multiple coding blocks, writing the encoding results of the global motion parameters, the multiple coding blocks, and the downsampled first texture edge residual map into the bitstream and transmitting it to the decoder.

[0327] After receiving the bitstream transmitted from the encoding end, the decoding end parses the bitstream to obtain the downsampled first texture edge residual map and global motion parameters, and then upsamples the first texture edge residual map. Furthermore, it can also parse the encoding results of multiple coding blocks using conventional video decoding to obtain the second video frame. Then, using the global motion parameters, motion compensation is performed on the texture edge residual map of the fourth video frame and the fourth video frame itself. Based on the second video frame, the motion compensation result, and the upsampled first texture edge residual map, the second texture edge residual map is determined. Finally, using the second texture edge residual map, texture synthesis is performed on the second video frame to obtain the texture-synthesized frame, i.e., the reconstructed first video frame.

[0328] Please refer to Figure 14 , Figure 14 This is an exemplary block diagram of another non-I-frame video encoding / decoding method provided in this application embodiment. Figure 14 In the process, the encoding end performs texture region discrimination on the first video frame to determine the textured and non-texture regions within it. Then, based on the texture region discrimination results, the quantization parameters of the coded blocks belonging to textured regions within the multiple coded blocks included in the first video frame are adjusted. The encoding end encodes these multiple coded blocks, writes the encoding results of these multiple coded blocks into the bitstream, and transmits them to the decoding end.

[0329] After receiving the bitstream transmitted from the encoding end, the decoding end parses the encoding results of the multiple coded blocks using conventional video decoding to obtain the second video frame. Then, optical flow estimation is performed based on the second and fourth video frames to obtain the second optical flow field. Using the second optical flow field, optical flow transformation is applied to the texture edge residual map of the fourth video frame to obtain the second texture edge residual map. Finally, texture synthesis is performed on the second video frame using the second texture edge residual map to obtain the texture-synthesized frame, i.e., the reconstructed first video frame.

[0330] The following is through Figure 15 and Figure 16 A brief introduction is given to the video encoding and decoding apparatus provided in the embodiments of this application. Figure 15 The video encoding apparatus shown is capable of performing the video encoding method provided in the embodiments described above. Figure 16 The video decoding device shown is capable of performing the video decoding method provided in the embodiments described above.

[0331] Figure 15 This is a schematic diagram of the structure of a video encoding device provided in an embodiment of this application. The video encoding device can be implemented as part or all of an encoding end device by software, hardware, or a combination of both. The encoding end device can be... Figure 1 The source device shown may be... Figure 2 The acquisition terminal shown. See also... Figure 15 The device includes: a first encoding module 1501, an edge residual map generation module 1502, a second encoding module 1503, and a writing module 1504.

[0332] The first encoding module 1501 is used to encode multiple encoding blocks included in the first video frame to be encoded, so as to obtain the encoding result of the multiple encoding blocks and the second video frame.

[0333] The edge residual map generation module 1502 is used to generate a first texture edge residual map based on a first video frame and a second video frame. The first texture edge residual map is used to characterize the edge residual between texture regions in the first video frame and the second video frame.

[0334] The second encoding module 1503 is used to encode the first texture edge residual map;

[0335] The writing module 1504 is used to write the encoding results of multiple encoding blocks and the encoding results of the first texture edge residual map into the bitstream.

[0336] Optionally, the first encoding module 1501 includes:

[0337] The label determination submodule is used to determine the mask label of the first video frame based on the first video frame. The mask label of the first video frame is used to distinguish between textured regions and non-textured regions in the first video frame.

[0338] The quantization parameter determination submodule is used to determine the quantization parameters corresponding to multiple coding blocks based on the mask labels of the first video frame. Among the multiple coding blocks, the quantization parameters corresponding to the coding blocks belonging to the texture region are greater than the quantization parameters corresponding to the coding blocks belonging to the non-texture region.

[0339] The encoding submodule is used to encode multiple encoding blocks based on the quantization parameters corresponding to each of the multiple encoding blocks.

[0340] Optionally, the edge residual map generation module 1502 includes:

[0341] The texture edge map generation submodule is used to generate a texture edge map based on the first video frame. The texture edge map is used to represent the edge information within the texture region in the first video frame.

[0342] The edge map generation submodule is used to generate a first edge map based on the second video frame. The first edge map is used to represent the edge information in the second video frame.

[0343] The edge residual map generation submodule is used to generate a first texture edge residual map based on the texture edge map and the first edge map.

[0344] Optionally, the first video frame is an I-frame, the texture edge map is used to represent the edge information in all texture regions in the first video frame, and the first edge map is used to represent the edge information in the second video frame;

[0345] The texture edge map generation submodule is specifically used for:

[0346] Based on the mask label of the first video frame, edge information in all texture regions in the first video frame is detected to generate a texture edge map. The mask label of the first video frame is used to distinguish between texture regions and non-texture regions in the first video frame.

[0347] The edge graph generation submodule is specifically used for:

[0348] Edge information of the second video frame is detected to generate a first edge map.

[0349] Optionally, the first video frame is a non-I-frame, the texture edge map is used to represent the edge information in a local texture region in the first video frame, and the first edge map is used to represent the edge information in a local region in the second video frame;

[0350] The texture edge map generation submodule includes:

[0351] The location range determination unit is used to determine the target location range, which is the location of the newly added area of ​​the first video frame relative to the third video frame, and the third video frame is the reference frame of the first video frame.

[0352] The texture edge map generation unit is used to generate a texture edge map based on the first video frame and the target location range;

[0353] The edge graph generation submodule is specifically used for:

[0354] A first edge map is generated based on the second video frame and the target location range.

[0355] Optionally, the location range determination unit is specifically used for:

[0356] Determine the global motion parameters, which are used to characterize the amount of motion of pixels in the third video frame moving to the first video frame;

[0357] Based on global motion parameters, motion compensation is performed on the third video frame to obtain the first motion-compensated frame, which includes the newly added region.

[0358] Iterate through the coordinates of pixels within the newly added region in the first motion-compensated frame to obtain the target location range.

[0359] Optionally, the texture edge map generation submodule also includes:

[0360] The parameter encoding unit is used to encode global motion parameters;

[0361] The write unit is used to write the encoded results of global motion parameters into the bitstream.

[0362] Optionally, the texture edge map generation unit is specifically used for:

[0363] The first image region is obtained by extracting an image region located within the target location range from the first video frame;

[0364] Based on the mask label of the first video frame, edge information in all texture regions of the first image region is detected to generate a texture edge map. The mask label of the first video frame is used to distinguish between texture regions and non-texture regions in the first video frame.

[0365] Based on the second video frame and the target location range, a first edge map is generated, including:

[0366] The second image region is obtained by extracting an image region located within the target location range from the second video frame;

[0367] Edge information of the second image region is detected to generate a first edge map.

[0368] Optionally, the edge residual map generation module 1502 further includes:

[0369] The first downsampling submodule is used to downsample the texture edge map according to a first sampling ratio, wherein the first sampling ratio is proportional to the quantization parameters of the non-texture region in the first video frame.

[0370] The first upsampling submodule is used to upsample the downsampled texture edge map according to the first sampling ratio;

[0371] The second downsampling submodule is used to downsample the first edge map according to the first sampling ratio;

[0372] The second upsampling submodule is used to upsample the downsampled first edge map according to the first sampling ratio.

[0373] Optionally, the device further includes:

[0374] The downsampling module is used to downsample the first texture edge residual map according to the second sampling ratio.

[0375] In this embodiment, to improve the quality of the texture region in the first video frame reconstructed by the decoder, the encoder can obtain a second video frame during the encoding process of the multiple coding blocks. This allows the encoder to determine the edge residual between the texture regions in the first and second video frames, i.e., a first texture edge residual map, and then encode the first texture edge residual map into the bitstream. Thus, after obtaining the second video frame, the decoder synthesizes the texture within the texture region of the second video frame based on the first texture edge residual map, thereby forming a realistic, high-quality texture and improving the quality of the texture region in the reconstructed first video frame.

[0376] Because textured regions are particularly rich in texture, directly encoding these regions using conventional encoding methods would consume a significant amount of bitrate. However, the human eye is not sensitive to distortions or changes in these textured regions. Therefore, in this embodiment, the quantization parameters of the textured regions are increased to encode the multiple coding blocks included in the first video frame, thereby reducing the bitrate occupied by the coding blocks belonging to the textured regions and thus reducing the overall bitrate of the video frame. In this way, while reducing the bitrate, the quality of the textured regions in the first video frame reconstructed at the decoding end can still be guaranteed.

[0377] It should be noted that the video encoding device provided in the above embodiments is only illustrated by the division of the above functional modules during video encoding. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video encoding device and the video encoding method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0378] Figure 16 This is a schematic diagram of the structure of a video decoding device provided in an embodiment of this application. The video decoding device can be implemented as part or all of a decoding end device by software, hardware, or a combination of both. The decoding end device can be... Figure 1 The destination device shown may be... Figure 2 The terminal device shown. See also Figure 16 The device includes: a bitstream parsing module 1601, a first edge map generation module 1602, an edge residual map acquisition module 1603, a second edge map generation module 1604, and a texture synthesis module 1605.

[0379] The bitstream parsing module 1601 is used to parse the bitstream to obtain the second video frame;

[0380] The first edge map generation module 1602 is used to generate a first edge map based on the second video frame. The first edge map is used to represent the edge information of the second video frame.

[0381] The edge residual map acquisition module 1603 is used to acquire a second texture edge residual map, which is used to characterize the edge residual between all texture regions in the first video frame to be reconstructed and the second video frame.

[0382] The second edge map generation module 1604 is used to superimpose the second texture edge residual map with the first edge map to obtain the second edge map;

[0383] The texture synthesis module 1605 is used to input the second video frame and the second edge map into the texture synthesis network to obtain the reconstructed first video frame.

[0384] Optionally, the first video frame is an I-frame;

[0385] The edge residual map acquisition module 1603 is specifically used for:

[0386] The bitstream is parsed to obtain the reconstructed first texture edge residual map, and the second texture edge residual map is the reconstructed first texture edge residual map.

[0387] Optionally, the first video frame is a non-I-frame;

[0388] The edge residual map acquisition module 1603 includes:

[0389] The first bitstream parsing submodule is used to parse the bitstream to obtain the reconstructed first texture edge residual map;

[0390] The first edge residual map generation submodule is used to determine the third texture edge residual map based on the second video frame and global motion parameters. The global motion parameters are used to characterize the amount of motion of the pixels in the fourth video frame moving to the second video frame. The fourth video frame is the reference frame of the first video frame to be reconstructed.

[0391] The second edge residual map generation submodule is used to overlay the reconstructed first texture edge residual map with the third texture edge residual map to obtain the second texture edge residual map.

[0392] Optionally, the first edge residual map generation submodule is specifically used for:

[0393] Based on global motion parameters, motion compensation is performed on the texture edge residual map of the fourth video frame to obtain a motion-compensated edge residual map.

[0394] Based on global motion parameters, motion compensation is performed on the fourth video frame to obtain the second motion-compensated frame;

[0395] Optical flow estimation is performed on the second video frame and the second motion-compensated frame to obtain the first optical flow field;

[0396] Based on the first optical flow field, optical flow transformation is performed on the motion-compensated edge residual map to obtain the third texture edge residual map.

[0397] Optionally, the edge residual map acquisition module 1603 further includes:

[0398] The second bitstream parsing submodule is used to parse the bitstream to obtain global motion parameters.

[0399] Optionally, the device further includes:

[0400] The first upsampling module is used to upsample the reconstructed first texture edge residual map according to the second sampling ratio.

[0401] Optionally, the device further includes:

[0402] The downsampling module is used to downsample the first edge map according to a first sampling ratio, wherein the first sampling ratio is proportional to the quantization parameters of the non-texture region in the first video frame;

[0403] The second upsampling module is used to upsample the downsampled first edge map according to the first sampling ratio.

[0404] Optionally, the first video frame is a non-I-frame;

[0405] The edge residual map acquisition module 1603 is specifically used for:

[0406] Optical flow estimation is performed on the second and fourth video frames to obtain the second optical flow field. The fourth video frame is the reference frame for the first video frame to be reconstructed.

[0407] Based on the second optical flow field, optical flow transformation is performed on the texture edge residual map of the fourth video frame to obtain the second texture edge residual map.

[0408] In this embodiment, to improve the quality of the texture region in the first video frame reconstructed by the decoder, the encoder needs to determine the edge residual between the texture regions in the first and second video frames, i.e., the first texture edge residual map, and then encode the first texture edge residual map into the bitstream. Thus, after obtaining the second video frame, the decoder synthesizes the texture within the texture region in the second video frame based on the first texture edge residual map, thereby forming a realistic, high-quality texture and improving the quality of the reconstructed texture region in the first video frame. Furthermore, since the texture regions are particularly rich in texture, directly encoding these texture regions using conventional encoding methods would consume a significant amount of bitrate, but the human eye is not sensitive to distortions or changes in these texture regions. Therefore, the encoder increases the quantization parameters of the texture regions to encode multiple coding blocks included in the first video frame, thereby reducing the bitrate occupied by the coding blocks belonging to the texture regions, and thus reducing the overall bitrate of the video frame. In this way, while reducing the bitrate, the quality of the texture region in the first video frame reconstructed by the decoder can still be guaranteed.

[0409] It should be noted that the video decoding device provided in the above embodiments is only illustrated by the division of the above functional modules during video decoding. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video decoding device and the video decoding method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0410] Figure 17 This is a schematic block diagram of an encoding / decoding apparatus 1700 used in an embodiment of this application. The encoding / decoding apparatus 1700 may include a processor 1701, a memory 1702, and a bus system 1703. The processor 1701 and the memory 1702 are connected via the bus system 1703. The memory 1702 stores instructions, and the processor 1701 executes the instructions stored in the memory 1702 to perform various encoding or decoding methods described in the embodiments of this application. To avoid repetition, detailed descriptions are omitted here.

[0411] In this embodiment, the processor 1701 may be a central processing unit (CPU), or it may be other general-purpose processors, DSPs, ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0412] The memory 1702 may include a ROM device or a RAM device. Any other suitable type of storage device may also be used as memory 1702. Memory 1702 may include code and data 17021 accessed by processor 1701 using bus 1703. Memory 1702 may further include an operating system 17023 and an application program 17022, which includes at least one program that allows processor 1701 to execute the encoding or decoding methods described in the embodiments of this application. For example, application program 17022 may include applications 1 to N, which further include encoding or decoding applications (hereinafter referred to as encoding / decoding applications) that execute the encoding or decoding methods described in the embodiments of this application.

[0413] In addition to the data bus, the bus system 1703 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 1703 in the diagram.

[0414] Optionally, the codec device 1700 may also include one or more output devices, such as a display 1704. In one example, the display 1704 may be a haptic display that combines a display with a haptic unit capable of operatively sensing touch input. The display 1704 may be connected to the processor 1701 via a bus 1703.

[0415] It should be noted that the encoding / decoding device 1700 can execute the encoding method in the embodiments of this application, and can also execute the decoding method in the embodiments of this application.

[0416] Those skilled in the art will appreciate that the functionality described in conjunction with the various illustrative logic blocks, modules, and algorithmic steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium facilitating the transfer of a computer program from one place to another (e.g., based on a communication protocol). In this way, the computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may comprise a computer-readable medium.

[0417] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are specifically directed to non-temporary tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, DVDs, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0418] Instructions can be executed by one or more processors, such as digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the various illustrative logic blocks, modules, and steps described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be fully implemented in one or more circuit or logic elements. In one example, the various illustrative logic blocks, units, and modules in the encoder and decoder can be understood as corresponding circuit devices or logic elements.

[0419] The techniques described in this application can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The various components, modules, or units described in this application are intended to emphasize functional aspects of the apparatus for performing the disclosed techniques, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units can be combined with suitable software and / or firmware within a codec hardware unit, or provided via interoperable hardware units (including one or more processors as described above).

[0420] In other words, the above embodiments can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0421] In some embodiments, an encoding end device is provided, the encoding end device including a memory and a processor;

[0422] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the video encoding method described above.

[0423] In some embodiments, a decoding end device is provided, the decoding end device including a memory and a processor;

[0424] The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the video decoding method described above.

[0425] In some embodiments, a computer-readable storage medium is provided that stores instructions which, when executed on the computer, cause the computer to perform the steps of the method described above.

[0426] In some embodiments, a computer-readable storage medium is provided, comprising the bitstream obtained by the encoding method described above.

[0427] In some embodiments, a computer program is provided that, when executed, implements the methods described above.

[0428] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.

[0429] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A video encoding method, characterized in that, The method includes: The first video frame to be encoded comprises multiple coding blocks to obtain the encoding result of the multiple coding blocks and a second video frame. The second video frame is the base reconstructed frame obtained by decoding the encoding result of the multiple coding blocks. A texture edge map is generated based on the first video frame, the texture edge map being used to represent edge information within a texture region in the first video frame; a first edge map is generated based on the second video frame, the first edge map being used to represent edge information in the second video frame; a first texture edge residual map is generated based on the texture edge map and the first edge map, the first texture edge residual map being used to represent the residual between edge information within texture regions in the first video frame and the second video frame. Encode the first texture edge residual map; The encoding results of the multiple encoding blocks and the encoding results of the first texture edge residual map are written into the bitstream.

2. The method as described in claim 1, characterized in that, The encoding of the multiple coding blocks comprising the first video frame to be encoded includes: The mask label of the first video frame is determined based on the first video frame. The mask label of the first video frame is used to distinguish between textured regions and non-texture regions in the first video frame. The quantization parameters corresponding to the plurality of coding blocks are determined based on the mask labels of the first video frame. The quantization parameters corresponding to the coding blocks belonging to the texture region are greater than the quantization parameters corresponding to the coding blocks belonging to the non-texture region. The multiple coding blocks are encoded based on the quantization parameters corresponding to each of the multiple coding blocks.

3. The method as described in claim 1, characterized in that, The first video frame is an I-frame, and the texture edge map is used to represent the edge information of all texture regions in the first video frame. The first edge map is used to represent the edge information of the second video frame. The step of generating a texture edge map based on the first video frame includes: Based on the mask label of the first video frame, edge information in all texture regions of the first video frame is detected to generate the texture edge map. The mask label of the first video frame is used to distinguish between texture regions and non-texture regions in the first video frame. The generation of the first edge map based on the second video frame includes: Edge information of the second video frame is detected to generate the first edge map.

4. The method as described in claim 1, characterized in that, The first video frame is a non-I-frame, and the texture edge map is used to characterize the edge information within a local texture region in the first video frame. The first edge map is used to characterize the edge information within a local region in the second video frame. The step of generating a texture edge map based on the first video frame includes: Determine the target location range, which is the location of the newly added area of ​​the first video frame relative to the third video frame, where the third video frame is the reference frame of the first video frame; The texture edge map is generated based on the first video frame and the target location range; The generation of the first edge map based on the second video frame includes: The first edge map is generated based on the second video frame and the target location range.

5. The method as described in claim 4, characterized in that, Determining the target location range includes: Determine global motion parameters, which are used to characterize the amount of motion of a pixel in the third video frame moving to the first video frame; Based on the global motion parameters, motion compensation is performed on the third video frame to obtain a first motion-compensated frame, which includes the newly added region. The coordinates of the pixels in the newly added region in the first motion-compensated frame are traversed to obtain the target location range.

6. The method as described in claim 5, characterized in that, After determining the global motion parameters, the process also includes: The global motion parameters are encoded; The encoding results of the global motion parameters are written into the bitstream.

7. The method as described in claim 4, characterized in that, The step of generating the texture edge map based on the first video frame and the target location range includes: The first image region is obtained by extracting an image region located within the target location range from the first video frame; Based on the mask label of the first video frame, edge information in all texture regions of the first image region is detected to generate the texture edge map. The mask label of the first video frame is used to distinguish between texture regions and non-texture regions in the first video frame. The step of generating the first edge map based on the second video frame and the target location range includes: The second image region is obtained by extracting an image region located within the target location range from the second video frame; Edge information of the second image region is detected to generate the first edge map.

8. The method according to any one of claims 1-7, characterized in that, After generating the texture edge map based on the first video frame, the method further includes: The texture edge map is downsampled according to a first sampling ratio, where the first sampling ratio is proportional to the quantization parameters of the non-texture region in the first video frame. Upsample the downsampled texture edge map according to the first sampling ratio; After generating the first edge map based on the second video frame, the method further includes: The first edge map is downsampled according to the first sampling ratio; According to the first sampling ratio, the downsampled first edge map is upsampled.

9. The method according to any one of claims 1-8, characterized in that, After generating a first texture edge residual map based on the texture edge map and the first edge map, the method further includes: The first texture edge residual map is downsampled according to the second sampling ratio.

10. A video decoding method, characterized in that, The method includes: Parse the bitstream to obtain the second video frame; A first edge map is generated based on the second video frame, and the first edge map is used to characterize the edge information of the second video frame. Obtain a second texture edge residual map, which is used to characterize the residual between the edge information in all texture regions of the first video frame to be reconstructed and the second video frame; The second texture edge residual map is superimposed on the first edge map to obtain the second edge map; The second video frame and the second edge map are input into the texture synthesis network to obtain the reconstructed first video frame.

11. The method as described in claim 10, characterized in that, The first video frame is an I-frame; The step of obtaining the second texture edge residual map includes: The bitstream is parsed to obtain a reconstructed first texture edge residual map, and the second texture edge residual map is the reconstructed first texture edge residual map.

12. The method as described in claim 10, characterized in that, The first video frame is a non-I-frame; The step of obtaining the second texture edge residual map includes: The bitstream is parsed to obtain the reconstructed first texture edge residual map; Based on the second video frame and global motion parameters, a third texture edge residual map is determined. The global motion parameters are used to characterize the amount of motion of pixels in the fourth video frame moving to the second video frame. The fourth video frame is a reference frame for the first video frame to be reconstructed. The reconstructed first texture edge residual map is superimposed with the third texture edge residual map to obtain the second texture edge residual map.

13. The method as described in claim 12, characterized in that, The step of determining the third texture edge residual map based on the second video frame and global motion parameters includes: Based on the global motion parameters, motion compensation is performed on the texture edge residual map of the fourth video frame to obtain a motion-compensated edge residual map. Based on the global motion parameters, motion compensation is performed on the fourth video frame to obtain the second motion-compensated frame; Optical flow estimation is performed on the second video frame and the second motion-compensated frame to obtain the first optical flow field; Based on the first optical flow field, optical flow transformation is performed on the motion-compensated edge residual map to obtain the third texture edge residual map.

14. The method as described in claim 12 or 13, characterized in that, Before determining the third texture edge residual map based on the second video frame and global motion parameters, the method further includes: The bitstream is parsed to obtain the global motion parameters.

15. The method as described in any one of claims 11-14, characterized in that, After parsing the bitstream to obtain the reconstructed first texture edge residual map, the process further includes: The reconstructed first texture edge residual map is upsampled according to the second sampling ratio.

16. The method according to any one of claims 10-15, characterized in that, After generating the first edge map based on the second video frame, the method further includes: The first edge map is downsampled according to a first sampling ratio, and the first sampling ratio is proportional to the quantization parameter of the non-texture region in the first video frame. According to the first sampling ratio, the downsampled first edge map is upsampled.

17. The method as described in claim 10, characterized in that, The first video frame is a non-I-frame; The step of obtaining the second texture edge residual map includes: Optical flow estimation is performed on the second video frame and the fourth video frame to obtain the second optical flow field, wherein the fourth video frame is the reference frame of the first video frame to be reconstructed; Based on the second optical flow field, optical flow transformation is performed on the texture edge residual map of the fourth video frame to obtain the second texture edge residual map.

18. A video encoding device, characterized in that, The device includes: The first encoding module is used to encode multiple encoding blocks included in the first video frame to be encoded, so as to obtain the encoding result of the multiple encoding blocks and the second video frame, wherein the second video frame is the basic reconstructed frame obtained by decoding the encoding result of the multiple encoding blocks; An edge residual map generation module is used to generate a first texture edge residual map based on the first video frame and the second video frame. The first texture edge residual map is used to characterize the residual between edge information within the texture region in the first video frame and the second video frame. The second encoding module is used to encode the first texture edge residual map; The writing module is used to write the encoding results of the multiple encoding blocks and the encoding results of the first texture edge residual map into the bitstream; The edge residual map generation module includes: The texture edge map generation submodule is used to generate a texture edge map based on the first video frame, wherein the texture edge map is used to characterize the edge information within the texture region in the first video frame. An edge map generation submodule is used to generate a first edge map based on the second video frame, wherein the first edge map is used to characterize the edge information in the second video frame; The edge residual map generation submodule is used to generate the first texture edge residual map based on the texture edge map and the first edge map.

19. The apparatus as claimed in claim 18, characterized in that, The first encoding module includes: The label determination submodule is used to determine the mask label of the first video frame based on the first video frame. The mask label of the first video frame is used to distinguish between textured regions and non-texture regions in the first video frame. The quantization parameter determination submodule is used to determine the quantization parameters corresponding to the plurality of coding blocks based on the mask labels of the first video frame. The quantization parameters corresponding to the coding blocks belonging to the texture region are greater than the quantization parameters corresponding to the coding blocks belonging to the non-texture region. The encoding submodule is used to encode the multiple encoding blocks based on the quantization parameters corresponding to the multiple encoding blocks respectively.

20. The apparatus as claimed in claim 18, characterized in that, The first video frame is an I-frame, and the texture edge map is used to represent the edge information of all texture regions in the first video frame. The first edge map is used to represent the edge information of the second video frame. The texture edge map generation submodule is specifically used for: Based on the mask label of the first video frame, edge information in all texture regions of the first video frame is detected to generate the texture edge map. The mask label of the first video frame is used to distinguish between texture regions and non-texture regions in the first video frame. The edge map generation submodule is specifically used for: Edge information of the second video frame is detected to generate the first edge map.

21. The apparatus as claimed in claim 18, characterized in that, The first video frame is a non-I-frame, and the texture edge map is used to characterize the edge information within a local texture region in the first video frame. The first edge map is used to characterize the edge information within a local region in the second video frame. The texture edge map generation submodule includes: A location range determination unit is used to determine a target location range, wherein the target location range is the location of the newly added area of ​​the first video frame relative to the third video frame, and the third video frame is a reference frame of the first video frame. The texture edge map generation unit is used to generate the texture edge map based on the first video frame and the target location range; The edge map generation submodule is specifically used for: The first edge map is generated based on the second video frame and the target location range.

22. The apparatus as claimed in claim 21, characterized in that, The location range determination unit is specifically used for: Determine global motion parameters, which are used to characterize the amount of motion of a pixel in the third video frame moving to the first video frame; Based on the global motion parameters, motion compensation is performed on the third video frame to obtain a first motion-compensated frame, which includes the newly added region. The coordinates of the pixels in the newly added region in the first motion-compensated frame are traversed to obtain the target location range.

23. The apparatus as claimed in claim 22, characterized in that, The texture edge map generation submodule also includes: A parameter encoding unit is used to encode the global motion parameters; The writing unit is used to write the encoding result of the global motion parameters into the bitstream.

24. The apparatus as claimed in claim 21, characterized in that, The texture edge map generation unit is specifically used for: The first image region is obtained by cropping an image region located within the target location range from the first video frame; Based on the mask label of the first video frame, edge information in all texture regions of the first image region is detected to generate the texture edge map. The mask label of the first video frame is used to distinguish between texture regions and non-texture regions in the first video frame. The step of generating the first edge map based on the second video frame and the target location range includes: The second image region is obtained by extracting an image region located within the target location range from the second video frame; Edge information of the second image region is detected to generate the first edge map.

25. The apparatus according to any one of claims 18-24, characterized in that, The edge residual map generation module further includes: The first downsampling submodule is used to downsample the texture edge map according to a first sampling ratio, wherein the first sampling ratio is proportional to the quantization parameters of the non-texture region in the first video frame. The first upsampling submodule is used to upsample the downsampled texture edge map according to the first sampling ratio; The second downsampling submodule is used to downsample the first edge map according to the first sampling ratio; The second upsampling submodule is used to upsample the downsampled first edge map according to the first sampling ratio.

26. The apparatus according to any one of claims 18-25, characterized in that, The device further includes: The downsampling module is used to downsample the first texture edge residual map according to the second sampling ratio.

27. A video decoding device, characterized in that, The device includes: The bitstream parsing module is used to parse the bitstream to obtain the second video frame; The first edge map generation module is used to generate a first edge map based on the second video frame, wherein the first edge map is used to characterize the edge information of the second video frame; The edge residual map acquisition module is used to acquire a second texture edge residual map, which is used to characterize the residual between the edge information in all texture regions of the first video frame to be reconstructed and the second video frame. The second edge map generation module is used to overlay the second texture edge residual map with the first edge map to obtain the second edge map; The texture synthesis module is used to input the second video frame and the second edge map into the texture synthesis network to obtain the reconstructed first video frame.

28. The apparatus as claimed in claim 27, characterized in that, The first video frame is an I-frame; The edge residual map acquisition module is specifically used for: The bitstream is parsed to obtain a reconstructed first texture edge residual map, and the second texture edge residual map is the reconstructed first texture edge residual map.

29. The apparatus as claimed in claim 27, characterized in that, The first video frame is a non-I-frame; The edge residual map acquisition module includes: The first bitstream parsing submodule is used to parse the bitstream to obtain the reconstructed first texture edge residual map; The first edge residual map generation submodule is used to determine the third texture edge residual map based on the second video frame and global motion parameters. The global motion parameters are used to characterize the amount of motion of the pixels in the fourth video frame moving to the second video frame. The fourth video frame is the reference frame of the first video frame to be reconstructed. The second edge residual map generation submodule is used to superimpose the reconstructed first texture edge residual map with the third texture edge residual map to obtain the second texture edge residual map.

30. The apparatus as claimed in claim 29, characterized in that, The first edge residual map generation submodule is specifically used for: Based on the global motion parameters, motion compensation is performed on the texture edge residual map of the fourth video frame to obtain a motion-compensated edge residual map. Based on the global motion parameters, motion compensation is performed on the fourth video frame to obtain the second motion-compensated frame; Optical flow estimation is performed on the second video frame and the second motion-compensated frame to obtain the first optical flow field; Based on the first optical flow field, optical flow transformation is performed on the motion-compensated edge residual map to obtain the third texture edge residual map.

31. The apparatus as claimed in claim 29 or 30, characterized in that, The edge residual map acquisition module further includes: The second bitstream parsing submodule is used to parse the bitstream to obtain the global motion parameters.

32. The apparatus according to any one of claims 28-31, characterized in that, The device further includes: The first upsampling module is used to upsample the reconstructed first texture edge residual map according to the second sampling ratio.

33. The apparatus according to any one of claims 27-32, characterized in that, The device further includes: The downsampling module is used to downsample the first edge map according to a first sampling ratio, wherein the first sampling ratio is proportional to the quantization parameters of the non-texture region in the first video frame. The second upsampling module is used to upsample the downsampled first edge map according to the first sampling ratio.

34. The apparatus as claimed in claim 27, characterized in that, The first video frame is a non-I-frame; The edge residual map acquisition module is specifically used for: Optical flow estimation is performed on the second video frame and the fourth video frame to obtain the second optical flow field, wherein the fourth video frame is the reference frame of the first video frame to be reconstructed; Based on the second optical flow field, optical flow transformation is performed on the texture edge residual map of the fourth video frame to obtain the second texture edge residual map.

35. An encoding terminal device, characterized in that, The encoding device includes a memory and a processor; The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the video encoding method according to any one of claims 1-9.

36. A decoding device, characterized in that, The decoding device includes a memory and a processor; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the video decoding method according to any one of claims 10-17.

37. A computer-readable storage medium, characterized in that, The storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the method described in any one of claims 1-17.

38. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer, cause the computer to perform the steps of the method as claimed in any one of claims 1-17.

Images