Reference frame processing method, device and server

Abstract

The present disclosure provides a reference frame-based processing method, device and server, which relates to the field of electronic information technology and can solve the problem of low image coding processing efficiency. The specific technical solution is: obtain the current reference frame used for current frame image coding, and then determine how to update the reference frame sequence through the current reference frame based on the temporal correlation and spatial correlation between image frames, so as to reduce coding redundancy and improve coding efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of electronic information technology, and in particular to a reference frame processing method, device and server. Background Art

[0002] With the rapid development of cloud desktop applications and other remote technologies, the requirements for video coding efficiency and quality are increasing. When processing screen content, traditional coding methods use short-term reference frames for coding in most cases because the images closer to the frame to be encoded have stronger reference. However, in the desktop content coding scenario, the screen content may contain a large amount of text, graphics, images, videos and other elements. When the user frequently drags the window out of the screen and then drags it back in, the image content on the screen will change significantly, especially the window edges and covered areas. This change may cause the direct similarity between adjacent frames to decrease, thereby increasing the difficulty of coding and the size of the bitstream. The coding method based on short-term reference frames may not be able to effectively capture these changes, resulting in low coding efficiency. Summary of the invention

[0003] The embodiments of the present disclosure provide a reference frame-based processing method, device, and server, which can solve the problem of low coding efficiency in the process of existing coding to process remote data. The technical solution is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a method for processing a reference frame is provided, the method comprising:

[0005] Get the current frame image;

[0006] Processing a current frame image based on an initial reference frame in a reference frame sequence, and obtaining a current reference frame of the current frame image, wherein the length of the reference frame sequence is N, N is an integer greater than or equal to 2, and the reference frame sequence can store at most M short-term reference frames and NM long-term reference frames, and M is an integer greater than or equal to 1;

[0007] According to the current reference frame, updating and processing the reference frame sequence;

[0008] Wherein, the updating and processing of the reference frame sequence based on the current reference frame includes:

[0009] If the reference frame sequence is full after encoding the current frame image, a target reference frame to be removed is determined in the reference frame sequence, and after removing the target reference frame, the current reference frame is stored in the reference frame sequence.

[0010] According to a second aspect of an embodiment of the present disclosure, a reference frame processing device is provided, the device comprising:

[0011] Image acquisition module, image encoding module and reference frame updating module;

[0012] An image acquisition module is used to acquire the current frame image;

[0013] an image encoding module, configured to encode and process a current frame image based on an initial reference frame in a reference frame sequence, and obtain a current reference frame of the current frame image, wherein the length of the reference frame sequence is N, and N is an integer greater than or equal to 2, and the reference frame sequence can store at most M short-term reference frames and NM long-term reference frames, and M is an integer greater than or equal to 1;

[0014] A reference frame updating module, used for updating and processing the reference frame sequence according to the current reference frame;

[0015] Wherein, updating and processing the reference frame sequence based on the current reference frame includes:

[0016] If the reference frame sequence is full after encoding the current frame image, a target reference frame to be removed is determined in the reference frame sequence, and after removing the target reference frame, the current reference frame is stored in the reference frame sequence.

[0017] According to a third aspect of an embodiment of the present disclosure, a server is provided, comprising: a processor and a memory storing a computer program executable on the processor, wherein when the processor executes the computer program, the steps of the reference frame processing method as described in the above example are implemented.

[0018] The reference frame processing method provided in the embodiment of the present application uses a long-term reference frame as a reference for encoding a current image frame to capture the long-term dependency between image frames, and then determines how to update the reference frame sequence through the current reference frame based on the temporal correlation and spatial correlation between the image frames, thereby reducing coding redundancy and improving coding efficiency. Specifically, the method includes:

[0019] 1. Improve coding efficiency: By introducing long-term reference frame technology, unnecessary calculations and processing processes can be reduced, thereby significantly improving coding efficiency.

[0020] 2. Reduce image distortion: The long-term reference frame encoding method can better capture the characteristics of the screen content, reduce information loss during the encoding process, and thus reduce the occurrence of image distortion.

[0021] 3. Enhanced adaptability: Optimization and innovation have been made based on the characteristics of screen content encoding, making the encoding method more suitable for screen content encoding scenarios and better meeting the needs of emerging applications.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] Figure 1 is a schematic diagram of a scene in a reference frame processing method provided by an embodiment of the present disclosure;

[0025] Figure 2 is a flowchart of a reference frame processing method provided by an embodiment of the present disclosure;

[0026] Figure 3 An example of a reference frame sequence in a reference frame processing method provided in an embodiment of the present disclosure Figure 1 ;

[0027] Figure 4 An example of a reference frame sequence in a reference frame processing method provided in an embodiment of the present disclosure Figure 2 ;

[0028] Figure 5 An example of a reference frame sequence in a reference frame processing method provided in an embodiment of the present disclosure Figure 3 . DETAILED DESCRIPTION

[0029] The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0030] At present, various zero-terminal solutions have been proposed. All programs of the zero terminal run in the cloud server. The zero terminal uses the corresponding application by accessing the cloud server and controls the corresponding application through reverse control. The server will send the real-time processing screen to the zero terminal. In this way, for the zero terminal, it is just like operating locally. Users can remotely control multiple applications by manipulating the desktop of the zero terminal.

[0031] Based on the above scenario of user operation of remote cloud desktop, the screen content may contain a large amount of text, graphics, images, videos and other elements. When the user frequently drags the window out of the screen and then drags it back in, the image content on the screen will change significantly, especially the window edge and coverage area. This change may reduce the direct similarity between adjacent frames, thereby increasing the difficulty of encoding and the bitstream size. Traditional encoding methods may not be able to effectively capture these changes, resulting in a sudden increase in the bitstream.

[0032] The above-mentioned process of encoding screen data includes long-term encoding reference frames and short-term encoding reference frames. Long-term reference frames usually contain stable or relatively stable elements on the screen, such as background, desktop icons, fixed text, etc. These elements may remain unchanged for a long time in the video sequence, so they provide important background information for the current frame. Short-term reference frames refer to those frames that are close to the current frame and are mainly used to eliminate short-term temporal correlations in the video.

[0033] In an embodiment of the present application, a method based on a data processing algorithm and long-term reference frame coding technology is adopted for the image coding processing of video data. It can effectively deal with the problem of sudden increase in code rate caused by frequent dragging of windows, and reduce the bandwidth burden by identifying the long-term dependencies between image frames. It not only improves the coding efficiency, but also optimizes the bandwidth usage.

[0034] Please see attached Figure 1 , which shows the data processing flow when the user drags the window, and specifically describes the data processing steps based on the reference frame from the i-th frame to the j-th frame and then to the k-th frame. Figure 1 In the figure, the outer rectangle represents the screen boundary, while the inner small rectangle represents the dragged window. In the process of encoding the kth frame, if only the short-term reference frame is relied on, the reference frame may be the jth frame. However, since the window in the jth frame has completely moved out of the screen boundary, the moving window in the kth frame cannot find the corresponding reference area in the jth frame, so the window area of ​​the kth frame is likely to be encoded intra-frame. This phenomenon will lead to a larger bitrate, and the problem will become more significant as the range of the moving window increases.

[0035] In the process of solving the above problems, if the method provided by the present application is adopted, when encoding the kth frame, the long-term reference frame can be used to trace back to the i-th frame, when the window has not yet moved out of the screen, so as to confirm the reference area that matches the k-th frame window. By matching the historical position of the window on the screen, redundant information can be reduced, the encoding quality can be improved, and the use of intra-frame encoding can be reduced, the code stream size can be optimized, and the encoding efficiency can be improved; especially in application scenarios where the window movement range is large, the role of the long-term reference frame is more significant.

[0036] For desktop office scenarios, dragging windows occurs frequently. If bandwidth resources can be effectively saved under bandwidth-limited network conditions, it will bring users a better user experience.

[0037] In the SCC application scenario, the long-term reference frame can record the historical state of the window on the screen, even if the window is not visible in the short-term reference frame.

[0038] Please see attached Figure 2 The following embodiments take a server or a data encoding device as the execution subject, and apply the method provided in the embodiment of the present application to reference frame processing as an example for specific description. The reference frame processing method provided in the embodiment of the present application includes the following steps 201 to 203:

[0039] Step 201: Get the current frame image.

[0040] In this study, the current frame image refers to the image at a specific moment in the video sequence, or a frame in the image sequence. The current frame image can include static image information or dynamic image information that integrates multiple dynamic and interactive elements; the current frame image can be a local image or a cloud image, which provides convenience for users to process data in the cloud and effectively supports remote access and real-time processing of large-scale data sets.

[0041] Step 202: Encode and process the current frame image based on the initial reference frame in the reference frame sequence, and obtain the current reference frame of the current frame image.

[0042] According to the preset coding rules and the preset reference frame sequence, each image frame in the target video is encoded in sequence.

[0043] For example, when encoding and processing the second image frame, the second image frame matches 2 reference frames, and the preset frame sequence stores 2 reference frames; when encoding and processing the fifth image frame, the second image frame matches 5 reference frames, and the preset frame sequence stores 5 reference frames, and the 5 reference frames may include 4 long-term reference frames and 1 short-term reference frame, or 3 long-term reference frames and 2 short-term reference frames.

[0044] The length of the reference frame sequence is N, where N is an integer greater than or equal to 2. The reference frame sequence can store at most M short-term reference frames and NM long-term reference frames, where M is an integer greater than or equal to 1.

[0045] For example, when encoding the first frame image, the preset reference frame can be used as the preset reference frame sequence for encoding processing, and when encoding the second frame image, the reference frame used when encoding the first frame image can be used as the preset reference frame sequence for encoding processing.

[0046] The method provided in the embodiment of the present application stores multiple reference frames by presetting a frame sequence, thereby improving encoding efficiency.

[0047] Before encoding the frame image, appropriate image frames can be selected as long-term reference frames based on the attribute information of the image frames (such as content changes, motion information, etc.). These long-term reference frames usually contain static or relatively stable elements on the screen, such as backgrounds, fixed icons, etc.

[0048] Step 203: Update and process the reference frame sequence according to the current reference frame.

[0049] The next frame image of the current image is encoded according to the updated reference frame sequence.

[0050] If the number of the N initial reference frames is less than the maximum storage capacity of the preset reference frame sequence, the N initial reference frames are directly stored in the preset reference frame sequence.

[0051] If the number of the N initial reference frames is equal to the maximum storage capacity of the preset reference frame sequence, the target reference frame among the N initial reference frames is removed and the removed initial reference frame is stored in the preset reference frame sequence so that the number of reference frames matched by the next frame image matches the maximum capacity of the preset reference frame sequence.

[0052] For example, when encoding the sixth image frame, the second image frame matches 6 reference frames, and the preset reference frame sequence stores 6 reference frames. If the storage capacity of the preset reference frame sequence is set to 6, the preset reference frame list will be full after the 6th frame is encoded. Then, when the encoding is processed to the 7th image frame, there will be one more reference frame. Thereafter, the reference frame list needs to be updated after each encoding.

[0053] The number of target reference frames to be removed is based on the capacity of the preset reference frame sequence, and one frame or multiple frames may be removed.

[0054] Accordingly, it can be seen that after determining the number of reference frames of each frame image frame by frame and removing them, the problem of reference frame sequence overflow will not occur.

[0055] Based on the short-term reference frame and the long-term reference frame, inter-frame prediction is performed on the current image frame to obtain the target matching block, and then encoding is performed according to the target matching block to reduce coding redundancy and improve coding efficiency.

[0056] The reference frame processing method provided in this embodiment obtains historical reference frame data before encoding and processing frame images, and performs an evaluation strategy for the historical reference frames that comprehensively considers encoding quality and reference frame similarity, so as to more accurately measure the reference value of the historical reference frames, and decide whether to remove or replace a reference frame accordingly, thereby dynamically updating the long-term reference frame list according to changes in video content to adapt to different encoding requirements and application scenarios.

[0057] The method provided in the present application also includes: encoding and processing the N+1th frame image according to the processed preset reference frame sequence.

[0058] The encoding process of the N+1th frame image may be performed by using a preset reference frame that directly stores the reference frame used by the Nth frame image, or by using a preset reference frame sequence that has been removed.

[0059] In the implementation process, combined with the Figure 1 In the scenario shown, when encoding the kth frame, the long-term reference frame can be used to trace back to the ith frame, when the window has not yet moved out of the screen, so as to find the reference area corresponding to the kth frame window. In this way, the use of intra-frame coding can be reduced, the bitstream size can be optimized, and the coding efficiency can be improved. Especially when the window movement range is large, the role of the long-term reference frame is more significant. By matching the historical position of the window on the screen, redundant information can be reduced and the coding quality can be improved.

[0060] In an optional embodiment, in step 203 of the encoding method provided by the present application, the target reference frame to be removed is determined in the reference frame sequence, thereby updating the reference frame sequence, including steps 301 to 302:

[0061] Step 301: Calculate the matching value of each initial reference frame in the reference frame sequence according to a preset matching factor.

[0062] The matching factors at least include the similarity between the reference frame and the frame image to be encoded and / or the encoding quality of the reference frame;

[0063] The matching factors at least include the similarity between the reference frame and the frame image to be encoded and / or the encoding quality of the reference frame. When calculating the matching value of each initial reference frame, it can be based on a single preset matching factor or a combination of multiple preset matching factors, so as to balance the encoding quality and encoding efficiency.

[0064] Step 302: Sort the matching values ​​of each of the initial reference frames, and remove the target reference frame with the lowest matching value among the N initial reference frames.

[0065] The long-term reference frame is far away from the current frame. Theoretically, in the same GOP, all frames before the current frame except the short-term reference frame can be used as long-term reference frames, so they need to be screened.

[0066] For example, when processing screen content containing a large number of static or stable elements, long-term reference frames can significantly reduce the bitstream size; however, the storage and management of long-term reference frames require additional memory resources and may increase the complexity of encoding; therefore, in the process of image encoding based on long-term reference frames, a trade-off needs to be made between encoding efficiency and quality. Choosing a suitable long-term reference frame helps improve encoding quality, but if the long-term reference frame is not selected properly, it may lead to a decrease in encoding quality and waste of storage space.

[0067] In an optional embodiment, in the encoding method provided by the present application, respectively calculating the matching value of each initial reference frame in the reference frame sequence in step 301 includes:

[0068] Step 401: Calculate a first matching value of each initial reference frame according to a first matching factor.

[0069] The first matching factor includes the similarity between the reference frame and the frame image to be encoded, thereby reflecting the predictability between the current frame and the reference frame.

[0070] The first matching value may be calculated by the number of motion matching macroblocks, residual energy or distortion metric, transform domain analysis, or rate-distortion optimization. The following examples are described using the calculation of the number of motion matching macroblocks as an example, but are not limited thereto.

[0071] Step 402: Calculate a second matching value of each initial reference frame according to the second matching factor.

[0072] The second matching factor includes the encoding quality of the frame image to be encoded, thereby evaluating the image's ability to maintain its original information during transmission, processing or storage, which may include image clarity, contrast, color restoration, detail retention and other aspects.

[0073] The second matching value may be calculated by means of a structural similarity index, a mean square error, visual information fidelity, a peak signal-to-noise ratio, etc. The following examples are described using the calculation of a peak signal-to-noise ratio PSNR value as an example, but are not limited thereto.

[0074] In an optional embodiment, in the encoding method provided by the present application, in step 203, the target reference frame in the N initial reference frames is removed, and a balance can be achieved between the first matching factor and the second matching factor, so as to determine which reference frame to remove. In specific implementation, the following strategies can be adopted:

[0075] First, a preset first matching threshold and a preset second matching threshold are set. If the first matching value is lower than the first matching threshold or the second matching threshold is higher than the second matching threshold, it is considered as a candidate for removal or replacement.

[0076] For example, when the number of motion matching macroblocks of a certain reference frame is lower than a threshold or the PSNR value is higher than a threshold, it is considered as a candidate for removal or replacement.

[0077] Secondly, the first matching factor and the second matching factor are evaluated comprehensively.

[0078] For example, a comprehensive evaluation is performed by combining the number of motion matching macroblocks and the PSNR value. If a reference frame has a large number of motion matching macroblocks but a low PSNR value, or a small number of motion matching macroblocks but a high PSNR value, then the weights of the two need to be comprehensively considered to decide whether to remove or replace the reference frame based on the actual application scenario and requirements.

[0079] Finally, the historical usage of the reference frame, such as the frequency of use, the time of the most recent use, etc., can also be considered to further assist in decision making.

[0080] In an optional embodiment, the step of removing the target reference frame from the N initial reference frames in step 203 of the encoding method provided by the present application includes:

[0081] If the first matching value and the second matching value of the current reference frame are both smaller than the first matching value and the second matching value of the remaining reference frames of the N initial reference frames, the current reference frame is determined to be the target reference frame to be removed.

[0082] For example, if the similarity and encoding quality between the current reference frame and the Nth frame are both smaller than those of the remaining reference frames of the N initial reference frames, the current reference frame is determined to be the target reference frame to be removed.

[0083] In an optional embodiment, the step of removing the target reference frame from the N initial reference frames in step 203 of the encoding method provided by the present application includes:

[0084] Step 501: If the first matching value and the second matching value of any reference frame among the N initial reference frames cannot simultaneously satisfy the requirement of being smaller than the first matching value and the second matching value of the remaining reference frames, then calculate the removal cost value of each initial reference frame;

[0085] Step 502: If the removal cost of the current reference frame is greater than all other initial reference frames, the current reference frame is determined to be a target reference frame to be removed.

[0086] In an optional embodiment, the calculation of the removal cost value of each initial reference frame in step 502 of the encoding method provided by the present application includes:

[0087] Calculate the PSNR value between each initial reference frame and the Nth frame, and the number of matching motion matching macroblocks;

[0088] Calculate the removal cost of each initial reference frame according to the number of motion matching macroblocks and PSNR value of each initial reference frame and the target calculation formula;

[0089] Determine the target reference frame to be removed according to the removal cost value of each initial reference frame,

[0090] Wherein, the target calculation formula includes:

[0091]

[0092] Among them, cost is used to indicate the removal cost of the current reference frame, N is used to indicate the number of motion matching macroblocks in the current reference frame, and PSNR average Used to indicate the average PSNR value of the current reference frame.

[0093] In an optional embodiment, the step of removing the target reference frame from the N initial reference frames in step 203 of the encoding method provided by the present application includes:

[0094] If the target reference frame to be removed is the initial short-term reference frame, the current short-term reference frame is deleted, and the initial long-term reference frame is used as the target long-term reference frame;

[0095] If the target reference frame to be removed is an initial long-term reference frame, the target reference frame in the NM initial long-term reference frames is removed, and the target reference frame is replaced by the initial short-term reference frame.

[0096] In an optional embodiment, in the encoding method provided by the present application, if the M is greater than 1, replacing the target reference frame with an initial short-term reference frame includes:

[0097] Obtain the distance between each short-term reference frame and the Nth frame image;

[0098] According to the distance between each short-term reference frame and the Nth frame image, the short-term reference frame with the farthest distance from the current frame among the short-term reference frames is selected to replace the target reference frame.

[0099] Through the above replacement process, the short-term reference frame can always be kept as the frame closest to the current position among all reference frames, thereby balancing the encoding quality and encoding efficiency.

[0100] In an optional embodiment, the encoding method provided by the present application further includes:

[0101] Encoding the N+1th frame of image according to the target long-term reference frame matched by the N+1th frame of image and the target short-term reference frame matched by the N+1th frame of image;

[0102] The target short-term reference frame is generated at least according to the similarity between the N+1th frame image and the Nth frame image.

[0103] In order to better explain the above data processing process, please refer to the attached manual. Figure 3 The data processing logic diagram of the image sequence shown in the figure, in the data processing process, each frame of video image may include: a long-term reference frame and / or a short-term reference frame during the encoding process, wherein:

[0104] Different reference frames are displayed by filling different patterns in the box, where a white box represents the current frame, a diagonal box represents a long-term reference frame, and a gray box represents a short-term reference frame. The short-term reference frame is the previous adjacent frame of the current frame. In this example, the short-term reference frame is the previous frame of the current frame. When encoding the first 6 frames, all the previous frames are used as short-term reference frames or long-term reference frames. For example, when encoding the 6th frame, frames 0-4 are long-term reference frames, and frame 5 is a short-term reference frame, so before the 6th frame (including the 6th frame), all reconstructed frames will be used as reference frames.

[0105] Long-term reference frame (hereinafter identified as LTR): used to store key frames frequently referenced in the entire GOP, the number of which is 5 (hereinafter identified as LTR_NUM=5).

[0106] Short-term reference frame (hereinafter identified as STR): used to store reference frames in a short period of time (such as near the current frame), the number is 1 (hereinafter identified as STR_NUM=1).

[0107] Based on the above reference frame, the two-stage processing within the GOP (Group of Pictures) includes:

[0108] The first stage: the first LTR_NUM+STR_NUM frames (i.e. the first 6 frames), mainly for the selection and setting of the initial reference frame, including steps 1.1 to 1.3:

[0109] Step 1.1: Read the first LTR_NUM+STR_NUM frames (i.e., the first 6 frames) in the GOP.

[0110] Step 1.2: Select the previous LTR_NUM frames as the initial long-term reference frames based on content importance (such as inter-frame difference, motion vector, etc.).

[0111] Step 1.3: Select the STR_NUM frame immediately before the current frame (ie, the 6th frame) as the initial short-term reference frame.

[0112] The second stage: from the LTR_NUM+STR_NUM+1 frame (ie, the 7th frame) to the end of the GOP, the main task is to update and select reference frames to adapt to changes in video content.

[0113] Processing stage (Phase II), Phase II includes steps 2.1 to 2.4:

[0114] Step 2.1: Starting from the LTR_NUM+STR_NUM+1 frame (ie, the 7th frame), read the remaining frames in the GOP frame by frame.

[0115] Step 2.2: For each frame, first determine whether the long-term reference frame needs to be updated. If the current frame meets the update conditions (such as large content changes, long time intervals, etc.), replace the least important long-term reference frame.

[0116] Step 2.3: Select the STR_NUM frame immediately before the current frame (ie, the frame before the current frame) as the short-term reference frame.

[0117] Step 2.4: When performing encoding or other processing, appropriate long-term and short-term reference frames are selected according to the requirements of the current frame.

[0118] Based on the attached Figure 3 The coding scenario of the reference frames shown in further detail explains how to select appropriate long-term and short-term reference frames according to the needs of the current frame:

[0119] Combined with Figure 1 The application scenario and the processing steps in the above method, for each GOP, initialize the long-term reference frame list (LTR list) to be empty and set its size to 5; the short-term reference frame (STR) is initialized to the previous frame of the current frame, then in the process of encoding the first 6 frames: for the first 6 frames in the GOP, since there are not enough frames to fill the LTR list, all encoded frames are regarded as potential LTR candidate frames (if they meet the LTR selection criteria).

[0120] However, at the 6th frame, the long-term reference frame list is full, so before the encoding of the 6th frame is completed, the frame image referenced by the image encoding of the 7th frame must be selected:

[0121] That is, either the short-term reference frame 5 is discarded, the long-term reference frame of the next frame is 0-4, and the short-term reference frame is 6;

[0122] Either replace a frame in the long-term reference frame list with the fifth frame (assuming it is the second frame);

[0123] The decision process starting from the seventh frame in the method provided in the embodiment of the present application may be based on the target index of the candidate frame, wherein the calculation process of the target index may include:

[0124] For each candidate frame (including the current STR, i.e., the STR of the 6th frame is the 5th frame, and the frames in the LTR list), the number of motion matching macroblocks (representing the difference between frames or dynamic content) and the PSNR value (peak signal-to-noise ratio, representing the quality of the reconstructed frame) are calculated.

[0125] The candidate frames are sorted according to the number of motion matching macroblocks and the PSNR value, so as to determine the candidate frames that have more motion matching macroblocks (indicating that they are frequently referenced during the encoding process) and higher PSNR values ​​(indicating higher quality).

[0126] Based on the above decision-making ideas, the following situations will occur:

[0127] Case 1: If the comprehensive evaluation of the number of motion matching macroblocks and PSNR value of STR (i.e., the previous frame of the current frame, for example, the STR of the 7th frame is the 6th frame) is better than the weakest frame in the LTR list (i.e., the frame with the least number of motion matching macroblocks and the lowest PSNR value), then the replacement operation is performed. For example, if the 5th frame (as a candidate for the STR of the 7th frame) is better than the 2nd frame in the LTR list, then the 2nd frame is replaced with the 5th frame.

[0128] Case 2: If the comprehensive evaluation of STR is not better than any frame in the LTR list, the LTR list is kept unchanged, and only the previous frame of the current frame is used as the new STR.

[0129] To better explain how to select the long-term reference frame, please refer to the attached Figure 4 The data processing diagram, the calculation process may include:

[0130] like Figure 4 As shown, a macroblock in the motion area of ​​the current frame may match multiple reference frames. The small black square of reference 2 in the second macroblock in the figure is the reference block with the largest PSNR. This application adds 1 to the number of motion macroblock matches for each reference frame and records the sum of the PSNR values ​​of all motion matching macroblocks.

[0131] This operation is performed for each potential motion macroblock in the current frame. When a decision needs to be made, the average PSNR value of the motion matching macroblocks of each reference frame is calculated first, and then the cost value is calculated using formula (1).

[0132]

[0133] By calculating the cost value of each initial reference frame, the reference frame with the largest cost value needs to be removed.

[0134] Please refer to the attached Figure 5 The long-term reference frame sequence after removal includes two groups of long-term reference frame sequences in the figure. The upper group in the figure represents the sequence after removing the 5th frame reference frame in the initial reference frame; the lower group in the figure represents that the 2nd frame in the initial reference frame needs to be removed, and the 5th frame in the initial reference frame replaces the 2nd frame.

[0135] In the reference frame processing method provided in the present application, the updating strategy of the long-term reference frame is completed before the current frame is encoded, so as to encode the current frame using the updated reference frame list.

[0136] The reference frame processing method provided in the present application is aimed at the short-term reference frame management strategy, including: when the number of short-term reference frames is greater than 1, if it is necessary to remove the long-term reference frame to add a new short-term reference frame, the frame farthest from the current frame in the STR list is preferentially selected for replacement, thereby ensuring that the frame in the STR list is always the closest to the current frame, thereby effectively utilizing the inter-frame correlation in the short term.

[0137] The reference frame processing method provided in this application realizes the refined processing of image content by dynamically adjusting the long-term reference frame and macroblock coding strategy. This method can significantly improve the coding efficiency while ensuring the visual quality of the reconstructed image, and is particularly suitable for application scenarios with limited bandwidth resources but requiring high-quality video transmission.

[0138] The present disclosure provides a reference frame processing device, the device comprising:

[0139] Image acquisition module, image encoding module and reference frame updating module;

[0140] An image acquisition module is used to acquire the current frame image;

[0141] an image encoding module, configured to encode and process a current frame image based on an initial reference frame in a reference frame sequence, and obtain a current reference frame of the current frame image, wherein the length of the reference frame sequence is N, and N is an integer greater than or equal to 2, and the reference frame sequence can store at most M short-term reference frames and NM long-term reference frames, and M is an integer greater than or equal to 1;

[0142] A reference frame updating module, used for updating and processing the reference frame sequence according to the current reference frame;

[0143] Wherein, updating and processing the reference frame sequence based on the current reference frame includes:

[0144] If the reference frame sequence is full after encoding the current frame image, a target reference frame to be removed is determined in the reference frame sequence, and after removing the target reference frame, the current reference frame is stored in the reference frame sequence.

[0145] An embodiment of the present disclosure provides a server, comprising: a processor and a memory storing a computer program executable on the processor, wherein the processor implements the steps of the reference frame processing method in the above embodiment when executing the computer program.

[0146] For the specific definition of the above-mentioned server, please refer to the definition of the processing method of the reference frame in the above text, which will not be repeated here. Each module in the above-mentioned data processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0147] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0148] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0150] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for processing a reference frame, characterized in that: The method comprises: Get the current frame image; Processing a current frame image based on an initial reference frame in a reference frame sequence, and obtaining a current reference frame of the current frame image, wherein the length of the reference frame sequence is N, N is an integer greater than or equal to 2, and the reference frame sequence can store at most M short-term reference frames and NM long-term reference frames, and M is an integer greater than or equal to 1; According to the current reference frame, updating and processing the reference frame sequence; Wherein, the updating and processing of the reference frame sequence based on the current reference frame includes: If the reference frame sequence is full after encoding the current frame image, a target reference frame to be removed is determined in the reference frame sequence, and after removing the target reference frame, the current reference frame is stored in the reference frame sequence.

2. The method for processing a reference frame according to claim 1, characterized in that: The step of determining a target reference frame to be removed from the reference frame sequence includes: Calculating the matching value of each initial reference frame in the reference frame sequence according to preset matching factors, wherein the matching factors at least include the similarity between the reference frame and the frame image to be encoded and / or the encoding quality of the reference frame; The matching value of each initial reference frame is sorted and the target reference frame with the lowest matching value among the N initial reference frames is removed.

3. The method for processing a reference frame according to claim 2, characterized in that: The calculating the matching value of each initial reference frame in the reference frame sequence comprises: Calculate the number of motion matching macroblocks between the current reference frame and the Nth frame image, as well as the peak signal-to-noise ratio; According to the similarity of the current reference frame and the peak signal-to-noise ratio of each macroblock, the matching value of the current reference frame is obtained.

4. The method for processing a reference frame according to claim 2, characterized in that: The first matching factor includes the similarity between the reference frame and the frame image to be encoded, and the second matching factor includes the encoding quality of the reference frame and the frame image to be encoded, and determining the target reference frame to be removed in the reference frame sequence includes: Calculating a first matching value for each initial reference frame according to the first matching factor; Calculating a second matching value for each initial reference frame according to the second matching factor; If the first matching value and the second matching value of the current reference frame are both smaller than the first matching value and the second matching value of the remaining reference frames in the reference frame sequence, the current reference frame is determined to be a target reference frame to be removed.

5. The method for processing a reference frame according to claim 4, characterized in that: Determining a target reference frame to be removed from the reference frame sequence includes: If the first matching value and the second matching value of any reference frame in the N initial reference frames cannot simultaneously satisfy the requirement of being smaller than the first matching value and the second matching value of the remaining reference frames in the reference frame sequence, then calculating the removal cost value of each initial reference frame; If the removal cost of the current reference frame is greater than all other initial reference frames in the reference frame sequence, the current reference frame is determined to be a target reference frame to be removed.

6. The method for processing a reference frame according to claim 5, characterized in that: The calculating of the removal cost value of each initial reference frame includes: Calculate the PSNR value of the motion matching macroblock of each macroblock in each initial reference frame image in turn; Calculate the removal cost of each initial reference frame according to the PSNR value and the target calculation formula; Determine the target reference frame to be removed according to the removal cost value of each initial reference frame, Wherein, the target calculation formula includes: The cost is used to indicate the removal cost of the current reference frame, N is used to indicate the number of motion matching macroblocks in the current reference frame, and PSNR average Used to indicate the average PSNR value of the current reference frame.

7. The method for processing a reference frame according to claim 1, characterized in that: The removing the target reference frame from the N initial reference frames comprises: If the target reference frame to be removed is the initial short-term reference frame, the current short-term reference frame is deleted, and the initial long-term reference frame is used as the target long-term reference frame; If the target reference frame to be removed is an initial long-term reference frame, the target reference frame in the NM initial long-term reference frames is removed, and the target reference frame is replaced by the initial short-term reference frame.

8. The method for processing a reference frame according to claim 7, characterized in that: If the M is greater than 1, replacing the target reference frame with an initial short-term reference frame includes: Obtain the distance between each short-term reference frame and the Nth frame image; According to the distance between each short-term reference frame and the Nth frame image, the short-term reference frame with the farthest distance from the current frame among the short-term reference frames is selected to replace the target reference frame.

9. A reference frame processing device, characterized in that: The device comprises: Image acquisition module, image encoding module and reference frame updating module; An image acquisition module is used to acquire the current frame image; an image encoding module, configured to encode and process a current frame image based on an initial reference frame in a reference frame sequence, and obtain a current reference frame of the current frame image, wherein the length of the reference frame sequence is N, and N is an integer greater than or equal to 2, and the reference frame sequence can store at most M short-term reference frames and NM long-term reference frames, and M is an integer greater than or equal to 1; A reference frame updating module, used for updating and processing the reference frame sequence according to the current reference frame; Wherein, updating and processing the reference frame sequence based on the current reference frame includes: If the reference frame sequence is full after encoding the current frame image, a target reference frame to be removed is determined in the reference frame sequence, and after removing the target reference frame, the current reference frame is stored in the reference frame sequence.

10. A server, characterized in that: The server stores computer executable instructions, and the computer executable instructions are used to execute the reference frame processing method described in any one of claims 1 to 8.

Images