Resolution up-stage reference picture list enhancement method, apparatus and device for video coding and medium

Abstract

The application discloses a resolution upscaling stage reference picture list enhancement method, device, equipment and medium for video coding, relates to the technical field of video coding, and comprises the following steps: determining whether a current picture is a resolution upscaling picture during coding; if yes, counting the number of reference pictures with a resolution not lower than that of the current picture in an original active reference picture list, and comparing the number with a preset number; if the counted number is less than the preset number, selecting a candidate picture from a decoded picture buffer, replacing a target low-resolution reference picture with the largest time distance from the current picture, updating the counted number and repeating the process until the counted number reaches a standard or no candidate picture is available, and obtaining a replaced reference picture list for inter-frame prediction coding of the current picture. The application can dynamically adjust the reference picture list in the resolution upscaling stage, take into account the time correlation and the spatial detail reservation, and improve the prediction efficiency.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and in particular to a method, apparatus, device, and medium for enhancing reference charts during the resolution upscaling stage of video coding. Background Technology

[0002] In multi-resolution reference map coding, the encoder often dynamically adjusts the coding resolution based on bandwidth, scene complexity, or bit rate control needs. This means encoding at a lower resolution for a period of time, and then reverting to a higher resolution once conditions improve. This technique is an important means of improving coding efficiency.

[0003] However, existing technologies have significant shortcomings in the resolution recovery stage. With increased resolution, the current image requires richer spatial details to support prediction, but the reference map list generated according to a predetermined temporal structure often retains many reference maps from previous low-resolution encoding stages. While these reference maps can be matched to the current image in size using existing resampling mechanisms, resampling is merely pixel interpolation and cannot recover the texture, edge, and chromaticity details lost in the low-resolution stage. This leads to increased prediction residuals and causes bitrate fluctuations or even sudden spikes near the resolution switch location. Furthermore, existing processing methods are rather simplistic and extreme: directly removing all low-resolution reference maps results in the loss of strong temporal correlation between images with closer temporal distances; while completely using the original list ignores existing images of equal or higher resolution in the decoded image cache, leading to insufficient utilization of reference resources and difficulty in balancing temporal correlation and spatial detail prediction capabilities.

[0004] Therefore, how to effectively and dynamically adjust the reference map list during the resolution increase stage, and achieve a balance between temporal correlation and spatial detail preservation to improve image prediction efficiency, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for enhancing the reference map list during the resolution upscaling stage of video coding. This method enables effective dynamic adjustment of the reference map list during the resolution upscaling stage, achieving a balance between temporal relevance and spatial detail preservation to improve image prediction efficiency. The specific solution is as follows: In a first aspect, this application provides a reference map list enhancement method for the resolution upscaling stage of video encoding, applied to a video encoder, comprising: During the encoding of AVS3 video, based on the resolution relationship between the current image to be encoded and the previous encoded image, it is determined whether the current image is an image with increased resolution; the image with increased resolution is the current image with a resolution higher than that of the previous encoded image. When the current image is an image with increased resolution, count the number of reference images in the original active reference image list with a resolution not lower than that of the current image, obtain the count, and compare the count with a preset number. When the statistical count is less than the preset count, a candidate image with a resolution not lower than the current image resolution and not included in the original active reference image list is selected from the video encoder's decoded image buffer. From the target low-resolution reference images included in the original active reference image list, the target low-resolution reference image with the largest time distance from the current image is selected as the reference image to be replaced. The candidate image is used to replace the reference image to be replaced, thereby completing one reference image replacement operation of the original active reference image list. The target low-resolution reference image is a reference image in the original active reference image list with a resolution lower than the current image resolution. The time distance is the absolute value of the difference between the image sequence count value of the reference image and the current image. Update the statistical count and repeat the reference image replacement operation until the statistical count is not less than the preset count, or the candidate image is not present in the decoded image buffer, and obtain the corresponding replaced active reference image list, so as to perform inter-frame predictive coding operation on the current image based on the replaced active reference image list.

[0006] Optionally, determining whether the current image is an upscaled image based on the resolution relationship between the current image to be encoded and the previous encoded image includes: Obtain the resolution level corresponding to the current image from the pre-set multi-resolution levels, as well as the resolution level corresponding to the previous encoded image; If the value of the resolution level corresponding to the current image is less than the value of the resolution level corresponding to the previous encoded image, then the current image is determined to be an image with increased resolution. Without setting a multi-resolution layer, obtain the width and height of the current image, as well as the width and height of the previous encoded image; If the width of the current image is not less than the width of the previous encoded image, and the height of the current image is not less than the height of the previous encoded image, and the width of the current image is greater than the width of the previous encoded image or the height of the current image is greater than the height of the previous encoded image, then the current image is determined to be an image with increased resolution. If the total number of luminance sampling points in the current image is greater than the total number of luminance sampling points in the previous encoded image, then the current image is determined to be an image with increased resolution, provided that all resolutions supported by the video encoder have the same aspect ratio.

[0007] Optionally, the method further includes: Determine the processing window following the upscaled image; the processing window is a window composed of target images located after the upscaled image in the encoding order; If the current image is not an image with increased resolution and the current image is within the processing window, perform the step of counting the number of reference images with a resolution not lower than the current image resolution in the original active reference image list of the current image, and perform the reference image replacement operation; When the number of target images is zero, the reference image replacement operation only applies to the image with increased resolution.

[0008] Optionally, selecting candidate images from the video encoder's decoded image buffer that have a resolution not lower than the current image's resolution and are not included in the original active reference image list includes: The reference images in the decoded image buffer of the video encoder are traversed, and reference images that meet preset conditions are identified as candidate images. A candidate set is constructed based on the candidate images. The preset conditions include that the reference image is in a referable state, the reference image meets the preset time direction constraint and preset image type constraint corresponding to the original active reference image list, the resolution of the reference image is not lower than the resolution of the current image, and the reference image is not included in the original active reference image list. A candidate image is selected from the candidate set based on a preset candidate selection strategy, and used as a candidate image to replace the reference image to be replaced.

[0009] Optionally, selecting a candidate image from the candidate set based on a preset candidate selection strategy includes: When the preset candidate selection strategy is a resolution-first strategy, the candidate image with the highest resolution is selected from the candidate set; If there are multiple candidate images with the highest resolution in the candidate set, then the candidate image with the smallest time distance from the current image is selected from the candidate images with the highest resolution. If there are multiple candidate images among the candidate images with the highest resolution that are closest to the current image in time, then select a candidate image from the candidate images that are closest to the current image in time that has the same time direction, the same image type, or the same time level as the reference image to be replaced. If there are multiple candidate images that have the same time direction, the same image type, or the same time level as the reference image to be replaced, then one candidate image is selected from the decoded image cache based on the storage order of the candidate images in the decoded image cache or the order of the image sequence count values. When the preset candidate selection strategy is a distance-first strategy, the candidate image with the smallest time distance from the current image is selected from the candidate set. If there are multiple candidate images that are closest to the current image in time, then select the candidate image with the highest resolution from the candidate images that are closest to the current image in time. If there are multiple candidate images with the highest resolution among the candidate images that are closest to the current image in time, then a candidate image with the same time direction, the same image type, or the same time level as the reference image to be replaced is selected from the candidate images with the highest resolution.

[0010] Optionally, selecting the target low-resolution reference image with the largest temporal distance from the current image from the target low-resolution reference images included in the original active reference image list as the reference image to be replaced includes: If there are multiple target low-resolution reference images that are the longest time apart from the current image, then the reference image with the lowest resolution is selected from the list of original active reference images as the reference image to be replaced. If there are multiple reference images with the lowest resolution, then the reference image that is ranked last or has the lowest priority in the original active reference image list is selected as the reference image to be replaced.

[0011] Optionally, performing inter-frame predictive coding of the current image based on the replaced active reference map list includes: The reference maps in the replaced active reference map list with a resolution lower than that of the current image are resampled to the resolution of the current image. Based on the resampled reference maps and the reference maps in the replaced active reference map list with a resolution not lower than that of the current image, motion search, motion compensation, and rate distortion optimization are performed on the current image to complete the inter-frame predictive coding operation of the current image.

[0012] Secondly, this application provides a reference map list enhancement device for the resolution upscaling stage of video encoding, applied to a video encoder, comprising: The resolution determination module is used to determine whether the current image is an image with increased resolution based on the resolution relationship between the current image to be encoded and the previous encoded image during the encoding process of AVS3 video; the image with increased resolution is the current image with a resolution higher than that of the previous encoded image. The quantity comparison module is used to count the number of reference images with a resolution not lower than the current image resolution in the original active reference image list of the current image when the current image is an image with increased resolution, obtain the statistical quantity, and compare the statistical quantity with a preset quantity. The reference image replacement module is used to, when the statistical count is less than the preset count, select candidate images from the video encoder's decoded image buffer that have a resolution not lower than the current image's resolution and are not included in the original active reference image list; select the target low-resolution reference image with the largest time distance from the current image from the target low-resolution reference images included in the original active reference image list as the reference image to be replaced; and replace the target low-resolution reference image with the candidate image to complete one reference image replacement operation of the original active reference image list; the target low-resolution reference image is a reference image in the original active reference image list with a resolution lower than the current image's resolution; the time distance is the absolute value of the difference between the image sequence count value of the reference image and the current image; The list update module is used to update the statistical quantity and repeatedly perform the reference image replacement operation until the statistical quantity is not less than the preset quantity, or the candidate image does not exist in the decoded image cache, and obtain the corresponding replaced active reference image list, so as to perform the inter-frame predictive coding operation of the current image based on the replaced active reference image list.

[0013] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the aforementioned method for enhancing the reference map list during the resolution upscaling stage of video encoding.

[0014] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for enhancing the reference map list during the resolution upscaling stage of video encoding.

[0015] In this application, during the encoding of AVS3 video, based on the resolution relationship between the current image to be encoded and the previous encoded image, it is determined whether the current image is an upscaled image; the upscaled image is the current image with a resolution higher than that of the previous encoded image; if the current image is an upscaled image, the number of reference images with a resolution not lower than that of the current image in the original active reference image list is counted, and the count is compared with a preset number; when the count is less than the preset number, candidate images with a resolution not lower than that of the current image and not included in the original active reference image list are selected from the decoding image buffer of the video encoder, and the target low-resolution image included in the original active reference image list is selected from the target low-resolution image. In the resolution reference map, the target low-resolution reference map with the largest temporal distance from the current image is selected as the reference map to be replaced. The candidate image is used to replace the reference map to be replaced, thereby completing one reference map replacement operation of the original active reference map list. The target low-resolution reference map is a reference map in the original active reference map list whose resolution is lower than that of the current image. The temporal distance is the absolute value of the difference between the image sequence count value of the reference map and the current image. The statistical count is updated and the reference map replacement operation is repeated until the statistical count is not less than the preset count, or the candidate image does not exist in the decoded image buffer, and a corresponding replaced active reference map list is obtained. The inter-frame predictive coding operation of the current image is then performed based on the replaced active reference map list. As can be seen from the above, in this application, by dynamically detecting the number of high-resolution reference maps in the original active reference map list during the resolution increase stage, and replacing the low-resolution reference map with the one with the largest temporal distance with a high-resolution candidate image in the decoded image buffer that meets the conditions, and iteratively updating the reference map list until the preset conditions are met, a higher quality replaced active reference map list is obtained, thereby improving the accuracy and efficiency of inter-frame prediction. In this way, the technical solution provided by this application can effectively and dynamically adjust the reference image list during the resolution increase stage, achieve a balance between temporal correlation and spatial detail preservation, significantly improve image prediction efficiency, and effectively suppress bitrate fluctuations near the resolution switching position. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1This application discloses a flowchart of a reference diagram list enhancement method for the resolution upscaling stage of video coding. Figure 2 This application discloses a flowchart of a specific reference diagram list enhancement method for the resolution upscaling stage of video coding. Figure 3 This is a schematic diagram of a reference diagram list enhancement device structure for a resolution upscaling stage of video encoding disclosed in this application; Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Currently, existing technologies have significant shortcomings in the resolution restoration stage. After resolution is increased, the current image requires richer spatial details to support prediction. However, the reference map list generated according to a predetermined temporal structure often retains many reference maps from the previous low-resolution coding stage. Although these reference maps can be matched to the current image in size using existing resampling mechanisms, resampling is merely pixel interpolation and cannot recover the texture, edge, and chromaticity details lost in the low-resolution stage. This leads to increased prediction residuals and causes bitrate fluctuations or even sudden increases near the resolution switch location. Furthermore, existing processing methods are rather simplistic and extreme: directly removing all low-resolution reference maps results in the loss of strong temporal correlation between images with a recent temporal distance; completely using the original list ignores existing images of equal or higher resolution in the decoded image buffer, resulting in insufficient utilization of reference resources and difficulty in balancing temporal correlation and spatial detail prediction capabilities. Therefore, this application provides a method, apparatus, device, and medium for enhancing the reference map list during the resolution increase stage of video coding. This method can effectively and dynamically adjust the reference map list during the resolution increase stage, achieving a balance between temporal correlation and spatial detail preservation capabilities to improve image prediction efficiency.

[0020] See Figure 1 As shown, this embodiment of the invention discloses a reference map list enhancement method for the resolution upscaling stage of video encoding, applied to a video encoder, including: Step S11: During the encoding of AVS3 (Audio Video Coding Standard 3) video, based on the resolution relationship between the current image to be encoded and the previous encoded image, it is determined whether the current image is an image with increased resolution; the image with increased resolution is the current image with a resolution higher than that of the previous encoded image.

[0021] In this embodiment, when encoding a video sequence, the AVS3 encoder reads the image to be encoded frame by frame. Before encoding each frame, it first determines whether the resolution of the current image has changed relative to the previous encoded frame. It should be noted that the previous encoded image refers to the image immediately preceding the current image in the encoding sequence, and its encoding resolution may differ from that of the current image. To accurately determine whether the current image is an upgraded resolution image, this embodiment provides multiple determination methods, allowing the encoder to flexibly select the appropriate determination strategy based on the actual encoding configuration.

[0022] In one specific implementation, the resolution level corresponding to the current image and the resolution level corresponding to the previous encoded image are obtained from a pre-set multi-resolution level; if the value of the resolution level corresponding to the current image is less than the value of the resolution level corresponding to the previous encoded image, the current image is determined to be an image with increased resolution.

[0023] In another specific implementation, without setting a multi-resolution layer, the width and height of the current image, as well as the width and height of the previous encoded image, are obtained; if the width of the current image is not less than the width of the previous encoded image, and the height of the current image is not less than the height of the previous encoded image, and the width of the current image is greater than the width of the previous encoded image or the height of the current image is greater than the height of the previous encoded image, then the current image is determined to be an upscaled image.

[0024] In a third specific implementation, if the resolutions supported by the video encoder have the same aspect ratio, and the total number of luminance sampling points of the current image is greater than the total number of luminance sampling points of the previous encoded image, then the current image is determined to be an image with increased resolution.

[0025] After determining that the current image is an upscaled image, the encoder also needs to determine the processing window following the upscaled image; the processing window is a window composed of target images located after the upscaled image in the encoding order. If the current image is not an upscaled image but is located within the processing window, the step of counting the number of reference images in the original active reference image list with a resolution not lower than the current image resolution, and the reference image replacement operation, are performed. It should be noted that when the number of target images is zero, the reference image replacement operation only applies to the upscaled image, so that in subsequent encoding processes, the reference image list counting and reference image replacement operations of this embodiment can also be triggered for non-upscaled images located within the processing window.

[0026] Step S12: When the current image is an image with increased resolution, count the number of reference images in the original active reference image list of the current image whose resolution is not lower than that of the current image, obtain the count, and compare the count with a preset number.

[0027] In this embodiment, the original active reference map list can be a set of reference images used by the encoder for inter-frame prediction of the current image before any enhancement processing. After determining that the current image is an upscaled image, the encoder traverses each reference map in the list, obtains the actual encoded resolution of each reference map, compares this resolution with the resolution of the current image, and accumulates the number of reference maps with a resolution not lower than the resolution of the current image, thus obtaining a statistical count. The encoder compares the above statistical count with a preset number to determine the current supply status of high-resolution reference maps. The comparison result will serve as the basis for subsequent reference map list adjustment operations.

[0028] Step S13: When the statistical count is less than the preset count, select a candidate image from the video encoder's decoded image buffer that has a resolution not lower than the current image's resolution and is not included in the original active reference image list. From the target low-resolution reference images included in the original active reference image list, select the target low-resolution reference image with the largest time distance from the current image as the reference image to be replaced. Use the candidate image to replace the reference image to be replaced, thereby completing one reference image replacement operation of the original active reference image list. The target low-resolution reference image is a reference image in the original active reference image list with a resolution lower than the current image's resolution. The time distance is the absolute value of the difference between the image sequence count value of the reference image and the current image.

[0029] In this embodiment, when the number of statistics is less than the preset number, it indicates that there are not enough high-resolution reference maps in the current original active reference map list that can be used for inter-frame prediction of the current image, which cannot meet the needs of spatial detail prediction in the resolution enhancement stage. Therefore, the enhancement replacement operation of the reference map list is triggered.

[0030] Specifically, the reference images in the decoded image buffer of the video encoder are traversed, and reference images that meet preset conditions are identified as candidate images. A candidate set is then constructed based on these candidate images. The preset conditions include: the reference image is in a referable state; the reference image satisfies preset time direction constraints and preset image type constraints corresponding to the original active reference image list; the resolution of the reference image is not lower than the resolution of the current image; and the reference image is not included in the original active reference image list. After the candidate set is constructed, a candidate image is selected from the candidate set based on a preset candidate selection strategy to replace the reference image to be replaced.

[0031] This embodiment provides two candidate selection strategies, which the encoder can choose according to the actual encoding configuration.

[0032] When the preset candidate selection strategy is a resolution-first strategy, the candidate image with the highest resolution is selected from the candidate set. If there are multiple candidate images with the highest resolution in the candidate set, the candidate image with the smallest time distance from the current image is selected from the candidate images with the highest resolution. If there are multiple candidate images with the smallest time distance from the current image in the candidate images with the highest resolution, the candidate image with the same time direction, the same image type, or the same time level as the reference image to be replaced is selected from the candidate image with the smallest time distance from the current image. If there are multiple candidate images with the same time direction, the same image type, or the same time level as the reference image to be replaced, one candidate image is selected from the decoded image cache based on the storage order of the candidate images in the decoded image cache or the order of the image sequence count value.

[0033] When the preset candidate selection strategy is a distance-first strategy, the candidate image with the smallest time distance from the current image is selected from the candidate set; if there are multiple candidate images with the smallest time distance from the current image, the candidate image with the highest resolution is selected from among them; if there are multiple candidate images with the highest resolution among the candidate images with the smallest time distance from the current image, the candidate image with the same time direction, the same image type, or the same time level as the reference image to be replaced is selected from among the candidate images with the highest resolution.

[0034] Furthermore, when determining the reference image to be replaced, the encoder filters all reference images with a resolution lower than the current image resolution from the original active reference image list as target low-resolution reference images. Then, it selects the target low-resolution reference image with the largest time distance from the current image as the reference image to be replaced, so as to preserve the temporal correlation advantage of low-resolution reference images with a closer time distance as much as possible.

[0035] If there are multiple target low-resolution reference images that are the longest in time from the current image, then the reference image with the lowest resolution is selected from the list of original active reference images as the reference image to be replaced; if there are multiple reference images with the lowest resolution, then the reference image with the lowest resolution is selected from the list of original active reference images as the reference image to be replaced.

[0036] After selecting candidate images and the reference image to be replaced, the encoder uses the candidate images to replace the position of the reference image to be replaced in the original active reference image list, thus completing one reference image replacement operation. If the number of selected images is still less than the preset number, the encoder can repeat the replacement operation until the number of selected images reaches the preset number or the candidate set is empty.

[0037] Step S14: Update the statistical quantity and repeat the reference image replacement operation until the statistical quantity is not less than the preset quantity, or the candidate image does not exist in the decoded image buffer, and obtain the corresponding replaced active reference image list, so as to perform the inter-frame predictive coding operation of the current image based on the replaced active reference image list.

[0038] In this embodiment, after completing a reference image replacement operation, the encoder updates the statistical count by incrementing it by one to reflect the new number of reference images in the currently active reference image list with a resolution no lower than the current image resolution after the replacement operation. The encoder then compares the updated statistical count with a preset number again. If the updated statistical count is still less than the preset number, the above-described candidate image selection, reference image selection to be replaced, and replacement operations are repeated until the statistical count is no less than the preset number, or there are no candidate images in the decoded image cache that meet the preset conditions.

[0039] After the loop terminates, the encoder obtains a list of active reference maps after one or more replacement operations, i.e., the replaced active reference map list. Based on this replaced active reference map list, the encoder performs inter-frame predictive coding operations on the current image. Specifically, reference maps in the replaced active reference map list with a resolution lower than that of the current image are resampled to the resolution of the current image. Based on the resampled reference maps and reference maps in the replaced active reference map list with a resolution not lower than that of the current image, motion search, motion compensation, and rate-distortion optimization are performed on the current image to complete the inter-frame predictive coding operation for the current image.

[0040] As can be seen from the above, in this application, by dynamically detecting the number of high-resolution reference images in the original active reference image list during the resolution upscaling stage, and replacing the low-resolution reference image with the one with the largest temporal distance with a high-resolution candidate image that meets the conditions in the decoded image buffer, the reference image list is iteratively updated until the preset conditions are met, thereby obtaining a higher-quality replaced active reference image list, thus improving the accuracy and efficiency of inter-frame prediction. In this way, the technical solution provided by this application can effectively and dynamically adjust the reference image list during the resolution upscaling stage, achieving a balance between temporal correlation and spatial detail preservation, significantly improving image prediction efficiency, and effectively suppressing bitrate fluctuations near the resolution switching position.

[0041] The following is combined with Figure 2 The schematic diagram shown illustrates the technical solution of the embodiments of this application in detail.

[0042] In general, in this scheme, the encoder first obtains the actual encoding resolution of the current image, the list of original active reference images, and the reference image information in the decoded image cache. Based on the resolution relationship between the current image and the previous encoded image or the previous reference image, it determines whether the current image is an image with increased resolution and whether the current image is located within the preset processing window after the resolution is increased.

[0043] When the activation conditions are met, the encoder counts the number of equivalent or higher resolution reference images in the current active reference image list. If this number has reached the target number T, the original list remains unchanged; if the number is insufficient, a candidate reference image that meets the conditions is selected from the DPB (DecodedPicture Buffer), and this candidate reference image replaces the lowest-resolution active reference image in the list that is furthest in time. This candidate selection and replacement process is repeated until the number of equivalent or higher resolution active reference images reaches the target number T, or no available candidate images remain.

[0044] The activation conditions include: the multi-resolution reference map function is enabled; the current image is an inter-frame coded image; the current image is an upscaled image, or is located within the first W upscaled images; the number of reference maps of equal or higher resolution in the current active reference map list is less than the target number T; and at least one low-resolution reference map exists in the current active reference map list.

[0045] After the replacement is complete, the encoder will proceed to the subsequent encoding process.

[0046] For each image to be encoded, the following information needs to be obtained: the image sequence count and actual encoding resolution of the current image; the original reference map list L0 (i.e., the forward reference map list) and / or L1 (i.e., the backward reference map list) and the number of active reference maps; the image sequence count, resolution level, and availability status of each candidate reference map in the decoded image buffer; the processing window length W after resolution is increased; the number of targets T of active reference maps of equal or higher resolution; and the nearest neighbor search range N.

[0047] W, T, and N can be fixed by the encoding configuration, or they can be adaptively determined based on the image type, time level, resolution switching magnitude, or encoding statistics. When W is 0, only the first image after the resolution is increased is processed; T can be 1, 2, or other integers not exceeding the number of active reference images; N indicates that at most N images before the current image are checked during nearest neighbor search.

[0048] Specifically, for determining the resolution increase stage, the encoder can determine the resolution relationship between the reference image and the current image based on pre-set multi-resolution levels. A smaller resolution level value indicates a higher resolution. When the resolution level of the reference image is lower than that of the current image, the reference image is determined to be a higher resolution reference image; when the resolution levels are the same, the reference image is determined to be a reference image of equal resolution.

[0049] If the resolution level of the current image is higher than that of the previous encoded image, the current image is classified as an upscaled image. If no resolution level is set, the current image is classified as an upscaled image if its width and height are both not less than those of the previous encoded image, and at least one dimension is greater than the corresponding dimension of the previous encoded image. For cases where all resolutions have the same aspect ratio, the classification can also be made by comparing the total number of luminance sampling points.

[0050] In practical encoding, this method can be applied only to the image with increased resolution, or it can be applied to the preceding W images after that image. The latter approach can cover the transitional period where the reference image list is still affected by the low-resolution stage after resolution restoration.

[0051] For RPL (i.e., Reference Picture List) statistics, specifically, the encoder traverses the active reference entries in RPL L0 and / or L1 respectively, and counts the number C of reference pictures with equal or higher resolution therein. If C≥T, this list does not need to be modified; if C<T, the process proceeds to candidate selection and replacement.

[0052] When the coding structure specifies different temporal directions for L0 and L1, this method processes the two lists separately, and ensures that the candidate pictures satisfy the reference direction of the corresponding list, picture availability, and existing coding specification constraints.

[0053] For reference picture selection in the DPB, specifically, the temporal distance is defined as: ; where, denotes the picture order count value of the current picture, denotes the candidate reference picture i 's picture order count value, denotes the temporal distance between the candidate reference picture and the current picture.

[0054] The general candidate selection manner traverses the available reference pictures in the DPB. A candidate picture shall simultaneously meet the following conditions: being in a referenceable state; satisfying the temporal direction and picture type constraints of the current list; having a resolution not lower than that of the current picture; and not yet appearing in the active reference picture list to be enhanced.

[0055] The encoder may adopt any one of the following candidate picture selection strategies.

[0056] The first one is the resolution-first strategy. In the DPB, the candidate picture with the highest resolution is selected first. If there are multiple candidate pictures with the highest resolution, the candidate picture with the smallest temporal distance to the current picture is selected therefrom. If there are still multiple candidate reference pictures meeting the above conditions, a candidate reference picture that has the same temporal direction, same picture type or same temporal layer as the replaced reference picture is preferentially selected; if they still cannot be distinguished, the determination is made according to the storage order of the candidate reference pictures in the decoded picture buffer or the preset picture order count value order.

[0057] The second one is the distance-first strategy. In the DPB, the candidate picture with the smallest temporal distance to the current picture is selected first. If there are multiple candidate pictures with the smallest temporal distance, the candidate picture with the highest resolution is selected therefrom. If there are still multiple candidate pictures meeting the above conditions, a candidate reference picture that has the same temporal direction, picture type or temporal layer as the replaced reference picture is preferentially selected.

[0058] For the selection of the replaced reference picture, specifically, in the active reference picture list to be enhanced, the replacement object is selected only from the low-resolution active reference pictures, and the low-resolution active reference picture with the largest temporal distance from the current picture is preferably used as the replacement object.

[0059] When multiple low-resolution reference pictures have the same temporal distance, the reference picture with lower resolution can be preferentially replaced; if they are still the same, the reference picture located at the end of the list or at a lower priority position can be replaced. This rule enables recent low-resolution reference pictures to be retained, thereby reducing disturbance to the original reference structure.

[0060] For the general candidate replacement procedure, specifically, for each RPL that needs to be enhanced, the encoder performs the following steps: 1. Traverse the RPL to obtain the number C of reference pictures with equal or higher resolution; 2. If C≥T, keep the list unchanged and end the processing; 3. If C<T, filter candidate pictures with equal or higher resolution that meet the conditions from the DPB; 4. Select a candidate picture from the candidate set according to a predetermined candidate picture selection strategy; when the resolution-first strategy is adopted, the candidate picture with the highest resolution is preferentially selected; when the distance-first strategy is adopted, the candidate picture with the smallest temporal distance from the current picture is preferentially selected; 5. From the low-resolution reference pictures in the current active reference picture list, select the reference picture with the largest temporal distance from the current picture as the replacement object; 6. Replace the selected low-resolution reference picture with the candidate picture, and the total number of active reference pictures remains unchanged; 7. Update C and repeat steps 3 to 6 until C≥T or there is no available candidate picture. A replacement example can be shown in Table 1 below: Table 1 Replacement Example

[0061] In the above table, values similar to 1280×720 are resolutions, and in values similar to 79 (960×544), 79 denotes POC, and 960×544 denotes the resolution.

[0062] For the optional adaptive extension, specifically, the proportion R=Nuse / Nall of the number of coding blocks that finally use reference pictures with equal or higher resolution for prediction in the current picture can be counted, where Nuse denotes the number of coding blocks using reference pictures with equal or higher resolution, and Nall denotes the total number of coding blocks involved in the statistics. If R is higher than a preset threshold, the target number T can be increased or the neighbor search range N can be expanded in the subsequent processing window; if R is relatively low, T can be decreased or the processing window W can be shortened. This adaptive extension is not essential for implementing the basic solution.

[0063] Correspondingly, with reference to Figure 3 as shown, an embodiment of the present application provides an apparatus for enhancing a reference picture list in a resolution upscaling stage for video coding, which is applied to a video encoder, comprising: The resolution determination module 11 is used to determine whether the current image is an image with increased resolution based on the resolution relationship between the current image to be encoded and the previous encoded image during the encoding process of AVS3 video; the image with increased resolution is the current image with a resolution higher than that of the previous encoded image. The quantity comparison module 12 is used to count the number of reference images with a resolution not lower than the current image resolution in the original active reference image list of the current image when the current image is an image with increased resolution, obtain the count, and compare the count with a preset number. The reference image replacement module 13 is used to select, when the statistical count is less than the preset count, a candidate image with a resolution not lower than the current image and not included in the original active reference image list from the decoded image buffer of the video encoder; select the target low-resolution reference image with the largest time distance from the current image from the target low-resolution reference images included in the original active reference image list as the reference image to be replaced; and replace the reference image to be replaced with the candidate image to complete one reference image replacement operation of the original active reference image list; the target low-resolution reference image is a reference image in the original active reference image list with a resolution lower than the current image; the time distance is the absolute value of the difference between the image sequence count value of the reference image and the current image; The list update module 14 is used to update the statistical quantity and repeatedly perform the reference image replacement operation until the statistical quantity is not less than the preset quantity, or the candidate image does not exist in the decoded image cache, and obtain the corresponding replaced active reference image list, so as to perform the inter-frame predictive coding operation of the current image based on the replaced active reference image list.

[0064] In some specific embodiments, the resolution determination module 11 specifically includes: The layer acquisition unit is used to acquire the resolution layer corresponding to the current image and the resolution layer corresponding to the previous encoded image from a pre-set multi-resolution layer. The first resolution determination unit is used to determine that the current image is an image with increased resolution if the value of the resolution level corresponding to the current image is less than the value of the resolution level corresponding to the previous encoded image. The information acquisition unit is used to acquire the width and height of the current image, as well as the width and height of the previous encoded image, without setting a multi-resolution layer. The second resolution determination unit is used to determine that the current image is an image with increased resolution if the width of the current image is not less than the width of the previous encoded image, the height of the current image is not less than the height of the previous encoded image, and the width of the current image is greater than the width of the previous encoded image or the height of the current image is greater than the height of the previous encoded image. The third resolution determination unit is used to determine that the current image is an image with increased resolution if the total number of luminance sampling points of the current image is greater than the total number of luminance sampling points of the previous encoded image, provided that the resolutions supported by the video encoder have the same aspect ratio.

[0065] In some specific embodiments, the device further includes: A window determination unit is used to determine the processing window after the upscaled image; the processing window is a window composed of target images located after the upscaled image in the encoding order; An operation execution unit is configured to perform the following steps when the current image is not an image with increased resolution and the current image is located within the processing window: counting the number of reference images in the original active reference image list of the current image with a resolution not lower than that of the current image, and performing the reference image replacement operation. When the number of target images is zero, the reference image replacement operation only applies to the image with increased resolution.

[0066] In some specific embodiments, the reference diagram replacement module 13 specifically includes: A set construction unit is used to traverse the reference images in the decoded image buffer of the video encoder, determine the reference images that meet preset conditions as candidate images, and construct a candidate set based on the candidate images; the preset conditions include that the reference image is in a referable state, the reference image meets the preset time direction constraints and preset image type constraints corresponding to the original active reference image list, the resolution of the reference image is not lower than the resolution of the current image, and the reference image is not included in the original active reference image list. The image selection unit is used to select a candidate image from the candidate set based on a preset candidate selection strategy, so as to replace the reference image to be replaced.

[0067] In some specific embodiments, the image selection unit specifically includes: The first image selection subunit is used to select the candidate image with the highest resolution from the candidate set when the preset candidate selection strategy is a resolution priority strategy. The second image selection subunit is used to select the candidate image with the smallest time distance from the current image from the candidate images if there are multiple candidate images with the highest resolution in the candidate set. The third image selection subunit is used to select, if there are multiple candidate images with the smallest time distance from the current image among the candidate images with the highest resolution, a candidate image with the same time direction, the same image type, or the same time level as the reference image to be replaced from the candidate images with the smallest time distance from the current image. The fourth image selection subunit is used to select one candidate image from the decoded image cache if there are multiple candidate images with the same time direction, the same image type, or the same time level as the reference image to be replaced; based on the storage order of the candidate images in the decoded image cache or the order of the image order count value. The fifth image selection subunit is used to select the candidate image with the smallest time distance from the current image from the candidate set when the preset candidate selection strategy is a distance priority strategy. The sixth image selection subunit is used to select the candidate image with the highest resolution from the candidate images with the smallest time distance from the current image if there are multiple candidate images with the smallest time distance from the current image. The seventh image selection subunit is used to select, if there are multiple candidate images with the highest resolution among the candidate images with the smallest time distance from the current image, a candidate image with the same time direction, the same image type, or the same time level as the reference image to be replaced from the candidate images with the highest resolution.

[0068] In some specific embodiments, the reference diagram replacement module 13 specifically includes: The first reference image determination unit is used to select the reference image with the lowest resolution from the original active reference image list as the reference image to be replaced if there are multiple target low-resolution reference images that are the largest time distance from the current image. The second reference image determination unit is used to select, if there are multiple reference images with the lowest resolution, the reference image that is arranged last or has the lowest priority in the original active reference image list as the reference image to be replaced.

[0069] In some specific embodiments, the list update module 14 specifically includes: The predictive coding unit is used to resample reference maps with a resolution lower than that of the current image in the replaced active reference map list to the resolution of the current image, and perform motion search, motion compensation, and rate distortion optimization on the current image based on the resampled reference maps and reference maps with a resolution not lower than that of the current image in the replaced active reference map list, so as to complete the inter-frame predictive coding operation of the current image.

[0070] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the resolution upscaling reference list enhancement method for video encoding disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be a computer.

[0071] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0072] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0073] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the resolution upscaling reference chart enhancement method for video encoding executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0074] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for enhancing a reference map list during the resolution upscaling stage of video encoding. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0075] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0076] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0077] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0078] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for enhancing a reference map list during the resolution upscaling stage of video coding, characterized in that, Applied to video encoders, including: During the encoding of AVS3 video, based on the resolution relationship between the current image to be encoded and the previous encoded image, it is determined whether the current image is an image with increased resolution; the image with increased resolution is the current image with a resolution higher than that of the previous encoded image. When the current image is an image with increased resolution, count the number of reference images in the original active reference image list with a resolution not lower than that of the current image, obtain the count, and compare the count with a preset number. When the statistical count is less than the preset count, a candidate image with a resolution not lower than the current image resolution and not included in the original active reference image list is selected from the video encoder's decoded image buffer. From the target low-resolution reference images included in the original active reference image list, the target low-resolution reference image with the largest time distance from the current image is selected as the reference image to be replaced. The candidate image is used to replace the reference image to be replaced, thereby completing one reference image replacement operation of the original active reference image list. The target low-resolution reference image is a reference image in the original active reference image list with a resolution lower than the current image resolution. The time distance is the absolute value of the difference between the image sequence count value of the reference image and the current image. Update the statistical count and repeat the reference image replacement operation until the statistical count is not less than the preset count, or the candidate image is not present in the decoded image buffer, and obtain the corresponding replaced active reference image list, so as to perform inter-frame predictive coding operation on the current image based on the replaced active reference image list.

2. The reference map list enhancement method for resolution upscaling stage of video encoding according to claim 1, characterized in that, The determination of whether the current image is an upscaled image based on the resolution relationship between the current image to be encoded and the previous encoded image includes: Obtain the resolution level corresponding to the current image from the pre-set multi-resolution levels, as well as the resolution level corresponding to the previous encoded image; If the value of the resolution level corresponding to the current image is less than the value of the resolution level corresponding to the previous encoded image, then the current image is determined to be an image with increased resolution. Without setting a multi-resolution layer, obtain the width and height of the current image, as well as the width and height of the previous encoded image; If the width of the current image is not less than the width of the previous encoded image, and the height of the current image is not less than the height of the previous encoded image, and the width of the current image is greater than the width of the previous encoded image or the height of the current image is greater than the height of the previous encoded image, then the current image is determined to be an image with increased resolution. If the total number of luminance sampling points in the current image is greater than the total number of luminance sampling points in the previous encoded image, then the current image is determined to be an image with increased resolution, provided that all resolutions supported by the video encoder have the same aspect ratio.

3. The reference map list enhancement method for resolution upscaling stage of video encoding according to claim 1, characterized in that, Also includes: Determine the processing window after the image resolution is increased; The processing window is a window composed of target images located after the upscaled image in the encoding order; If the current image is not an image with increased resolution and the current image is within the processing window, perform the step of counting the number of reference images with a resolution not lower than the current image resolution in the original active reference image list of the current image, and perform the reference image replacement operation; When the number of target images is zero, the reference image replacement operation only applies to the image with increased resolution.

4. The reference map list enhancement method for resolution upscaling stage of video encoding according to claim 1, characterized in that, The step of selecting candidate images from the video encoder's decoded image buffer that have a resolution not lower than the current image's resolution and are not included in the original active reference image list includes: The reference images in the decoded image buffer of the video encoder are traversed, and reference images that meet preset conditions are identified as candidate images. A candidate set is constructed based on the candidate images. The preset conditions include that the reference image is in a referable state, the reference image meets the preset time direction constraint and preset image type constraint corresponding to the original active reference image list, the resolution of the reference image is not lower than the resolution of the current image, and the reference image is not included in the original active reference image list. A candidate image is selected from the candidate set based on a preset candidate selection strategy, and used as a candidate image to replace the reference image to be replaced.

5. The reference map list enhancement method for resolution upscaling stage of video encoding according to claim 4, characterized in that, The step of selecting a candidate image from the candidate set based on a preset candidate selection strategy includes: When the preset candidate selection strategy is a resolution-first strategy, the candidate image with the highest resolution is selected from the candidate set; If there are multiple candidate images with the highest resolution in the candidate set, then the candidate image with the smallest time distance from the current image is selected from the candidate images with the highest resolution. If there are multiple candidate images among the candidate images with the highest resolution that are closest to the current image in time, then select a candidate image from the candidate images that are closest to the current image in time that has the same time direction, the same image type, or the same time level as the reference image to be replaced. If there are multiple candidate images that have the same time direction, the same image type, or the same time level as the reference image to be replaced, then one candidate image is selected from the decoded image cache based on the storage order of the candidate images in the decoded image cache or the order of the image sequence count values. When the preset candidate selection strategy is a distance-first strategy, the candidate image with the smallest time distance from the current image is selected from the candidate set. If there are multiple candidate images that are closest to the current image in time, then select the candidate image with the highest resolution from the candidate images that are closest to the current image in time. If there are multiple candidate images with the highest resolution among the candidate images that are closest to the current image in time, then a candidate image with the same time direction, the same image type, or the same time level as the reference image to be replaced is selected from the candidate images with the highest resolution.

6. The reference map list enhancement method for resolution upscaling stage of video encoding according to claim 1, characterized in that, The step of selecting the target low-resolution reference image with the largest temporal distance from the current image from the target low-resolution reference images included in the original active reference image list as the reference image to be replaced includes: If there are multiple target low-resolution reference images that are the longest time apart from the current image, then the reference image with the lowest resolution is selected from the list of original active reference images as the reference image to be replaced. If there are multiple reference images with the lowest resolution, then the reference image that is ranked last or has the lowest priority in the original active reference image list is selected as the reference image to be replaced.

7. The method for enhancing a reference map list in the resolution upscaling stage of video coding according to any one of claims 1 to 6, characterized in that, The inter-frame predictive coding operation for the current image based on the replaced active reference map list includes: The reference maps in the replaced active reference map list with a resolution lower than that of the current image are resampled to the resolution of the current image. Based on the resampled reference maps and the reference maps in the replaced active reference map list with a resolution not lower than that of the current image, motion search, motion compensation, and rate distortion optimization are performed on the current image to complete the inter-frame predictive coding operation of the current image.

8. A reference map list enhancement device for a resolution upscaling stage in video encoding, characterized in that, Applied to video encoders, including: The resolution determination module is used to determine whether the current image is an image with increased resolution based on the resolution relationship between the current image to be encoded and the previous encoded image during the encoding process of AVS3 video; the image with increased resolution is the current image with a resolution higher than that of the previous encoded image. The quantity comparison module is used to count the number of reference images with a resolution not lower than the current image resolution in the original active reference image list of the current image when the current image is an image with increased resolution, obtain the statistical quantity, and compare the statistical quantity with a preset quantity. The reference image replacement module is used to, when the statistical count is less than the preset count, select candidate images from the decoded image buffer of the video encoder that have a resolution not lower than the resolution of the current image and are not included in the original active reference image list; select the target low-resolution reference image with the largest time distance from the current image from the target low-resolution reference images included in the original active reference image list as the reference image to be replaced; and replace the reference image to be replaced with the candidate image to complete one reference image replacement operation of the original active reference image list; the target low-resolution reference image is a reference image in the original active reference image list with a resolution lower than the resolution of the current image; the time distance is the absolute value of the difference between the image sequence count value of the reference image and the current image; The list update module is used to update the statistical quantity and repeatedly perform the reference image replacement operation until the statistical quantity is not less than the preset quantity, or the candidate image does not exist in the decoded image cache, and obtain the corresponding replaced active reference image list, so as to perform the inter-frame predictive coding operation of the current image based on the replaced active reference image list.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the resolution upscaling reference list enhancement method for video encoding as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the resolution upscaling reference list enhancement method for video encoding as described in any one of claims 1 to 7.

Images