Reference frame management method and system based on screen content coding

Abstract

The invention discloses a reference frame management method and system based on screen content coding. The method comprises the following steps: maintaining attribute information of a reference frame screen window in a reference frame queue in an operating system; obtaining attribute information of a screen window of a current to-be-coded frame, and selecting a reference frame for the current to-be-coded frame according to the attribute information of the screen window; and autonomously updating the reference frame queue according to a preset updating mechanism. According to the technical scheme provided by the invention, the reference frame queue is maintained by utilizing the characteristics of screen content coding, and the optimal reference frame is quickly selected from the reference frame queue by virtue of the attribute information stored in the screen window of the operating system, so that the screening efficiency of the optimal reference frame is greatly improved. In addition, through dynamic management of the reference frame queue, old reference frames are updated in time, and redundancy of the reference frames in the reference frame queue is avoided while effective prediction is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of image coding, and in particular to a reference frame management method and system based on screen content coding. Background Art

[0002] With the development of Internet video communication technology, desktop remote technology, cloud desktop technology, etc. have been widely used. In these applications, it is necessary to capture the computer display screen, and then use video coding technology to encode and transmit the computer display screen. Screen Content Coding (SCC) is a video coding technology developed for application scenarios such as desktop sharing and cloud desktop. Screen content has characteristics and features different from camera video images: 1. Screen content mainly refers to text, graphics, images and other content on the screen, which are usually generated by computer screens, mobile device screens or application interfaces. These contents usually have high details and sharpness, containing a large number of edges, details and color changes. Therefore, when encoding screen content, high-quality reference frames are crucial. High-quality reference frames are required for accurate motion estimation and prediction. Low-quality reference frames may lead to prediction errors and distortion. 2. Screen content may involve frequent page switching, resulting in frequent changes in screen content, so the update frequency of reference frames is also important. Timely updating of reference frames can ensure that the encoder can accurately capture changes in screen content and make effective predictions. 3. In the use scenarios of screen content encoding, desktop sharing, video conferencing and other scenarios have relatively high requirements for encoding efficiency and require low-latency encoding. However, due to the relatively large frame resolution in screen content encoding, searching for the optimal reference frame that matches the current frame among multiple reference frames and updating the reference frame in a timely manner face problems such as high complexity and time consumption. Summary of the invention

[0003] Since the prior art has the above-mentioned problems, an embodiment of the present invention proposes a reference frame management method and system based on screen content coding.

[0004] Specifically, the embodiment of the present invention provides the following technical solutions:

[0005] In a first aspect, an embodiment of the present invention provides a reference frame management method based on screen content coding, comprising:

[0006] A reference frame queue and attribute information of the reference frame screen window in the reference frame queue are maintained.

[0007] Acquire the attribute information of the screen window of the current frame to be encoded, and select a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window.

[0008] The reference frame queue is updated according to a preset update mechanism.

[0009] In a second aspect, an embodiment of the present invention provides a reference frame management system based on screen content coding, including:

[0010] The information maintenance module is used to maintain a reference frame queue and the attribute information of the reference frame screen window in the reference frame queue.

[0011] The reference frame selection module is used to obtain the attribute information of the screen window of the current frame to be encoded, and select a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window.

[0012] The reference frame queue updating module is used to update the reference frame queue according to a preset updating mechanism.

[0013] It can be seen from the above technical solutions that the reference frame management method and system based on screen content coding provided by the embodiments of the present invention have the following benefits: Compared with the traditional selection of reference frames by analyzing the content characteristics of video frames, the technical solution provided by the present invention utilizes the characteristics of screen content coding to maintain a reference frame queue, and quickly selects the optimal reference frame from the reference frame queue with the help of the attribute information of the screen window stored in the operating system, which greatly improves the efficiency of optimal reference frame screening. In addition, through the dynamic management of the reference frame queue, the outdated reference frame is updated in time, and the redundancy of the reference frames in the reference frame queue is avoided while ensuring effective prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 It is a flowchart of a reference frame management method based on screen content coding provided by an embodiment of the present invention.

[0016] Figure 2 It is a structural diagram of a reference frame management system based on screen content coding provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0018] Figure 1 FIG. 1 is a flowchart of a reference frame management method based on screen content coding provided by an embodiment of the present invention. Figure 1 As shown, the reference frame management method based on screen content coding provided by an embodiment of the present invention includes:

[0019] A reference frame queue and attribute information of the reference frame screen window in the reference frame queue are maintained.

[0020] The attribute information of the screen window includes but is not limited to: window unique identifier, window size, window position, and window stacking order.

[0021] It is understandable that, for the screen content encoding scenario, the screen content includes one or more windows, and different windows display different contents. The attribute information of the screen window is stored in the operating system, and the attribute information of the corresponding screen window can be obtained by calling the relevant API.

[0022] Acquire the attribute information of the screen window of the current frame to be encoded, and select a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window.

[0023] In one embodiment, selecting a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window includes:

[0024] The window unique identifier of the current frame to be encoded is compared with the window unique identifier of the reference frame screen window in the reference frame queue, and one or more reference frames containing the same window unique identifier as the current frame to be encoded are selected from the reference frame queue as reference frames of the current frame to be encoded.

[0025] If the window unique identifier of the current to-be-encoded frame screen window does not exist in the attribute information of the reference frame screen window in the reference frame queue, a reference frame is selected from the reference frame queue according to the content feature of the current to-be-encoded frame.

[0026] The content features can be represented by the following parameters: pixel average, color histogram, gradient histogram, texture features, etc.

[0027] For example, as an implementation method, selecting a reference frame in a reference frame queue according to content features of the current frame to be encoded includes: calculating a frame-level pixel average value of the current frame to be encoded and a frame-level pixel average value of reference frames in the reference frame queue, and selecting a reference frame in the reference frame queue whose frame-level pixel average value is closest to the frame-level pixel average value of the current frame to be encoded as the reference frame of the current frame to be encoded.

[0028] For example, as another implementation method, the root selects a reference frame from a reference frame queue based on content features of the current frame to be encoded, including: calculating a color histogram of the current frame to be encoded and a color histogram of reference frames in the reference frame queue, and then evaluating the similarity between the current frame to be encoded and the reference frames in the reference frame queue through a similarity measurement method, such as a Bhattacharyya distance method, a histogram intersection method, etc., and selecting the reference frame with the highest similarity as the reference frame of the current frame to be encoded.

[0029] It is understandable that compared to natural images captured by the acquisition device of the camera device, the screen content image often contains a large number of repetitive patterns and regular structures. The regularity of the screen content can be used for efficient encoding during encoding. For example, under normal circumstances, if the window unique identifier of the screen window is the same in different video frames, the screen content presented between the different video frames is likely to be highly similar or even the same. Therefore, this feature can be used to quickly filter reference frames for the current frame to be encoded in order to reduce latency and save encoding overhead.

[0030] In another embodiment, selecting a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window includes:

[0031] The window unique identifiers of the current frame to be encoded and the reference frame screen windows in the reference frame queue are compared, and multiple reference frames containing the same window unique identifier as the current frame to be encoded are selected from the reference frame queue as candidate reference frames of the current frame to be encoded.

[0032] The similarity between the current frame to be encoded and the candidate reference frame is evaluated according to the attribute information of the screen window other than the window unique identifier, and then a reference frame is selected for the current frame to be encoded. The higher the similarity, the higher the reference priority of the candidate reference frame.

[0033] The step of evaluating the similarity between the current frame to be encoded and the candidate reference frame according to the attribute information of the screen window other than the window unique identifier, thereby selecting a reference frame for the current frame to be encoded, comprises:

[0034] As an implementation method, the overlapping area of ​​the windows in the current frame to be encoded and the candidate reference frame is calculated. The larger the overlapping area, the higher the similarity, and the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0035] The window overlapping area can be calculated by obtaining the window size and window position information stored in the system.

[0036] As another implementation, the relative positions of the windows in the current frame to be encoded and the candidate reference frames are calculated. The closer the relative positions are, the higher the similarity is, and the higher the reference priority of the candidate reference frame is. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0037] The relative position of the window can be calculated by acquiring the window position information stored in the system.

[0038] As another implementation, the stacking order of windows in the current frame to be encoded and the candidate reference frame is compared. The closer the stacking order is, the higher the similarity is, and the higher the reference priority of the candidate reference frame is. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0039] The stacking order of the windows can be determined by obtaining the stacking order of the windows stored in the system.

[0040] As another implementation method, the window overlap area, the window relative position, and the weight of the window stacking order are preset to comprehensively evaluate the similarity between the current frame to be encoded and the candidate reference frame. The higher the similarity, the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0041] Exemplarily, the weight of the window overlap area is set to t1, the weight of the window relative position is set to t2, and the weight of the window stacking order is set to t3, and the reference priority is quantified into a grade score, and the higher the similarity, the higher the grade score. The grade scores are calculated as a, b, and c respectively according to the window overlap area, the window relative position, and the window stacking order. At this time, the reference priority of the candidate reference frame obtained by comprehensive evaluation is (t1*a+t2*b+t3*c).

[0042] The reference priorities of all the candidate reference frames are calculated in the same way, and the candidate reference frame with the highest reference priority is selected as the reference frame of the current frame to be encoded.

[0043] It can be understood that, in this step, the specific method for evaluating the similarity between the current frame to be encoded and the candidate reference frame is not limited in the embodiment of the present invention, and different evaluation methods can be adopted according to the content scenario of actual screen encoding.

[0044] The reference frame queue is updated according to a preset update mechanism.

[0045] The preset update mechanism includes:

[0046] Set the frame number threshold T of the reference frame queue reference frame.

[0047] When the number of reference frames in the reference frame queue reaches the threshold T, the reference frame to be added to the reference frame queue replaces the reference frame with the least number of references in the reference frame queue.

[0048] It can be understood that this replacement strategy can ensure that new reference frames can be updated to the reference frame queue in a timely manner, and helps to avoid storing too many outdated reference frames in the reference frame queue.

[0049] Figure 2 FIG. 1 is a schematic diagram of a reference frame management system based on screen content coding provided by an embodiment of the present invention. Figure 2 As shown, the reference frame management system based on screen content coding provided by an embodiment of the present invention includes:

[0050] The information maintenance module is used to maintain a reference frame queue and the attribute information of the reference frame screen window in the reference frame queue.

[0051] The attribute information of the screen window includes but is not limited to: window unique identifier, window size, window position, and window stacking order.

[0052] The reference frame selection module is used to obtain the attribute information of the screen window of the current frame to be encoded, and select a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window.

[0053] In one embodiment, selecting a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window includes:

[0054] The window unique identifier of the current frame to be encoded is compared with the window unique identifier of the reference frame screen window in the reference frame queue, and one or more reference frames containing the same window unique identifier as the current frame to be encoded are selected from the reference frame queue as reference frames of the current frame to be encoded.

[0055] If the window unique identifier of the current to-be-encoded frame screen window does not exist in the attribute information of the reference frame screen window in the reference frame queue, a reference frame is selected from the reference frame queue according to the content feature of the current to-be-encoded frame.

[0056] The content features can be represented by the following parameters: pixel average, color histogram, gradient histogram, texture features, etc.

[0057] For example, as an implementation method, selecting a reference frame in a reference frame queue according to content features of the current frame to be encoded includes: calculating a frame-level pixel average value of the current frame to be encoded and a frame-level pixel average value of reference frames in the reference frame queue, and selecting a reference frame in the reference frame queue whose frame-level pixel average value is closest to the frame-level pixel average value of the current frame to be encoded as the reference frame of the current frame to be encoded.

[0058] For example, as another implementation method, the root selects a reference frame from a reference frame queue based on content features of the current frame to be encoded, including: calculating a color histogram of the current frame to be encoded and a color histogram of reference frames in the reference frame queue, and then evaluating the similarity between the current frame to be encoded and the reference frames in the reference frame queue through a similarity measurement method, such as a Bhattacharyya distance method, a histogram intersection method, etc., and selecting the reference frame with the highest similarity as the reference frame of the current frame to be encoded.

[0059] In another embodiment, selecting a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window includes:

[0060] The window unique identifiers of the current frame to be encoded and the reference frame screen windows in the reference frame queue are compared, and multiple reference frames containing the same window unique identifier as the current frame to be encoded are selected from the reference frame queue as candidate reference frames of the current frame to be encoded.

[0061] The similarity between the current frame to be encoded and the candidate reference frame is evaluated according to the attribute information of the screen window other than the window unique identifier, and then a reference frame is selected for the current frame to be encoded. The higher the similarity, the higher the reference priority of the candidate reference frame.

[0062] The step of evaluating the similarity between the current frame to be encoded and the candidate reference frame according to the attribute information of the screen window other than the window unique identifier, thereby selecting a reference frame for the current frame to be encoded, comprises:

[0063] As an implementation method, the overlapping area of ​​the windows in the current frame to be encoded and the candidate reference frame is calculated. The larger the overlapping area, the higher the similarity, and the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0064] The window overlapping area can be calculated by obtaining the window size and window position information stored in the system.

[0065] As another implementation, the relative positions of the windows in the current frame to be encoded and the candidate reference frames are calculated. The closer the relative positions are, the higher the similarity is, and the higher the reference priority of the candidate reference frame is. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0066] The relative position of the window can be calculated by acquiring the window position information stored in the system.

[0067] As another implementation, the stacking order of windows in the current frame to be encoded and the candidate reference frame is compared. The closer the stacking order is, the higher the similarity is, and the higher the reference priority of the candidate reference frame is. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0068] The stacking order of the windows can be determined by obtaining the stacking order of the windows stored in the system.

[0069] As another implementation method, the window overlap area, the window relative position, and the weight of the window stacking order are preset to comprehensively evaluate the similarity between the current frame to be encoded and the candidate reference frame. The higher the similarity, the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0070] Exemplarily, the weight of the window overlap area is set to t1, the weight of the window relative position is set to t2, and the weight of the window stacking order is set to t3, and the reference priority is quantified into a grade score, and the higher the similarity, the higher the grade score. The grade scores are calculated as a, b, and c respectively according to the window overlap area, the window relative position, and the window stacking order. At this time, the reference priority of the candidate reference frame obtained by comprehensive evaluation is (t1*a+t2*b+t3*c).

[0071] The reference priorities of all the candidate reference frames are calculated in the same way, and the candidate reference frame with the highest reference priority is selected as the reference frame of the current frame to be encoded.

[0072] The reference frame queue updating module is used to update the reference frame queue according to a preset updating mechanism.

[0073] The preset update mechanism includes:

[0074] Set the frame number threshold T of the reference frame queue reference frame.

[0075] When the number of reference frames in the reference frame queue reaches the threshold T, the reference frame to be added to the reference frame queue replaces the reference frame with the least number of references in the reference frame queue.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reference frame management method based on screen content coding, characterized in that: include: Maintaining a reference frame queue and attribute information of the reference frame screen window in the reference frame queue; Acquire the attribute information of the screen window of the current frame to be encoded, select a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window; finally, update the reference frame queue according to a preset update mechanism.

2. The reference frame management method based on screen content coding according to claim 1, characterized in that: The attribute information of the screen window includes: window unique identifier, window size, window position, and window stacking order.

3. The reference frame management method based on screen content coding according to claim 1, characterized in that: The selecting a reference frame for the current frame to be encoded according to the attribute information of the screen window includes: Comparing the window unique identifiers of the current frame to be encoded and the reference frame screen windows in the reference frame queue, and selecting one or more reference frames containing the same window unique identifier as the current frame to be encoded from the reference frame queue as reference frames of the current frame to be encoded; If the window unique identifier of the current to-be-encoded frame screen window does not exist in the attribute information of the reference frame screen window in the reference frame queue, selecting a reference frame in the reference frame queue according to the content feature of the current to-be-encoded frame; The content features can be represented by the following parameters: pixel average, color histogram, gradient histogram, texture features.

4. The reference frame management method based on screen content coding according to claim 3, characterized in that: After comparing the window unique identifiers of the current frame to be encoded with the reference frame screen windows in the reference frame queue, and selecting a plurality of reference frames having the same window unique identifier as the current frame to be encoded from the reference frame queue, the method further includes: Filtering out from the reference frame queue a plurality of reference frames having the same window unique identifier as the current frame to be encoded as candidate reference frames; According to the attribute information of the screen window other than the window unique identifier, the similarity between the current frame to be encoded and the candidate reference frame is evaluated, thereby selecting a reference frame for the current frame to be encoded, wherein the higher the similarity, the higher the reference priority of the candidate reference frame; The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

5. The reference frame management method based on screen content coding according to claim 1, characterized in that: The preset update mechanism includes: Set the frame number threshold T of the reference frame queue reference frame; When the number of reference frames in the reference frame queue reaches the threshold T, the reference frame to be added to the reference frame queue replaces the reference frame with the least number of references in the reference frame queue.

6. A reference frame management system based on screen content coding, characterized in that: include: An information maintenance module, used for maintaining a reference frame queue and attribute information of a reference frame screen window in the reference frame queue; A reference frame selection module, used for obtaining attribute information of a screen window of a current frame to be encoded, and selecting a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window; The reference queue updating module is used to update the reference frame queue according to a preset updating mechanism.

7. The reference frame management system based on screen content coding according to claim 6, characterized in that: The attribute information of the screen window includes: window unique identifier, window size, window position, and window stacking order.

8. The reference frame management system based on screen content coding according to claim 6, characterized in that: The selecting a reference frame for the current frame to be encoded according to the attribute information of the screen window includes: Comparing the window unique identifiers of the current frame to be encoded and the reference frame screen windows in the reference frame queue, and selecting one or more reference frames containing the same window unique identifier as the current frame to be encoded from the reference frame queue as reference frames of the current frame to be encoded; If the window unique identifier of the current to-be-encoded frame screen window does not exist in the attribute information of the reference frame screen window in the reference frame queue, selecting a reference frame in the reference frame queue according to the content feature of the current to-be-encoded frame; The content features can be represented by the following parameters: pixel average, color histogram, gradient histogram, texture features.

9. The reference frame management system based on screen content coding according to claim 8, characterized in that: After comparing the window unique identifiers of the current frame to be encoded with the reference frame screen windows in the reference frame queue, and selecting a plurality of reference frames having the same window unique identifier as the current frame to be encoded from the reference frame queue, the method further includes: Filtering out from the reference frame queue a plurality of reference frames having the same window unique identifier as the current frame to be encoded as candidate reference frames; According to the attribute information of the screen window other than the window unique identifier, the similarity between the current frame to be encoded and the candidate reference frame is evaluated, thereby selecting a reference frame for the current frame to be encoded, wherein the higher the similarity, the higher the reference priority of the candidate reference frame; The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

10. The reference frame management system based on screen content coding according to claim 6, characterized in that: The preset update mechanism includes: Set the frame number threshold T of the reference frame queue reference frame; When the number of reference frames in the reference frame queue reaches the threshold T, the reference frame to be added to the reference frame queue replaces the reference frame with the least number of references in the reference frame queue.

Images