A reference frame management method and system based on screen content coding

Abstract

The present invention discloses a reference frame management method and system based on screen content coding, the method comprising: maintaining attribute information of a reference frame screen window in a reference frame queue in an operating system; obtaining attribute information of a screen window of a current frame to be coded, selecting a reference frame for the current frame to be coded based on the attribute information of the screen window; and autonomously updating the reference frame queue according to a preset update mechanism. The technical solution provided by the present invention utilizes the characteristics of screen content coding to maintain a reference frame queue, and uses the attribute information of the screen window stored in the operating system to quickly select the optimal reference frame from the reference frame queue, thereby greatly improving the efficiency of optimal reference frame screening. In addition, through dynamic management of the reference frame queue, outdated reference frames are updated in a timely manner, ensuring effective prediction while avoiding redundancy of reference frames in the reference frame queue.

Description

Technical Field

[0001] The present invention relates to the technical field of image coding, and in particular to a reference frame management method and system based on screen content coding. Background Art

[0002] With the development of internet-based video communication technologies, desktop remote access and cloud desktop technologies have gained widespread application. These applications require capturing computer display images, encoding them using video coding techniques, and transmitting them. Screen content coding (SCC) is a video coding technology developed for scenarios like desktop sharing and cloud desktops. Screen content has characteristics and properties that differ from camera video images: 1. Screen content primarily refers to on-screen content such as text, graphics, and images, typically generated by a computer screen, mobile device screen, or application interface. This content typically exhibits high detail and sharpness, containing numerous edges, details, and color variations. Therefore, high-quality reference frames are crucial for encoding screen content. They are required for accurate motion estimation and prediction; low-quality reference frames can lead to prediction errors and distortion. 2. Screen content may involve frequent page switching, resulting in frequent changes in screen content. Therefore, the frequency of reference frame updates is also crucial. Timely reference frame updates ensure that the encoder can accurately capture changes in screen content and perform effective predictions. 3. Screen content encoding scenarios, such as desktop sharing and video conferencing, require relatively high encoding efficiency and low-latency encoding. However, due to the relatively high frame resolution in screen content encoding, searching for the optimal reference frame among multiple reference frames to match the current frame and updating the reference frames in a timely manner are complex and time-consuming. Summary of the Invention

[0003] Since the prior art has the above-mentioned problems, an embodiment of the present invention proposes a reference frame management method and system based on screen content coding.

[0004] Specifically, the embodiments of the present invention provide the following technical solutions:

[0005] In a first aspect, an embodiment of the present invention provides a reference frame management method based on screen content coding, comprising:

[0006] A reference frame queue and attribute information of the reference frame screen window in the reference frame queue are maintained.

[0007] Attribute information of a screen window of a current frame to be encoded is obtained, and a reference frame is selected for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window.

[0008] The reference frame queue is updated according to a preset update mechanism.

[0009] In a second aspect, an embodiment of the present invention provides a reference frame management system based on screen content coding, comprising:

[0010] The information maintenance module is used to maintain a reference frame queue and attribute information of the reference frame screen window in the reference frame queue.

[0011] The reference frame selection module is used to obtain attribute information of the screen window of the current frame to be encoded, and select a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window.

[0012] The reference frame queue updating module is used to update the reference frame queue according to a preset updating mechanism.

[0013] As can be seen from the above technical solutions, the reference frame management method and system based on screen content coding provided by the embodiments of the present invention have the following benefits: Compared to the traditional method of selecting reference frames by analyzing the content characteristics of video frames, the technical solution provided by the present invention utilizes the characteristics of screen content coding to maintain a reference frame queue. Using the attribute information of the screen window stored in the operating system, the optimal reference frame is quickly selected from the reference frame queue, greatly improving the efficiency of optimal reference frame screening. In addition, through dynamic management of the reference frame queue, outdated reference frames are updated in a timely manner, ensuring effective prediction while avoiding redundancy of reference frames in the reference frame queue. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 This is a flowchart of a reference frame management method based on screen content coding provided by one embodiment of the present invention.

[0016] Figure 2 It is a structural diagram of a reference frame management system based on screen content coding provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following embodiments of the present invention are further described in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0018] Figure 1 FIG. 1 shows a flow chart of a reference frame management method based on screen content coding provided by an embodiment of the present invention. Figure 1 As shown, the reference frame management method based on screen content coding provided by an embodiment of the present invention includes:

[0019] A reference frame queue and attribute information of the reference frame screen window in the reference frame queue are maintained.

[0020] The attribute information of the screen window includes but is not limited to: window unique identifier, window size, window position, and window stacking order.

[0021] It is understandable that, for the screen content encoding scenario, the screen content includes one or more windows, and different windows display different content. The attribute information of the screen window is stored in the operating system, and the attribute information of the corresponding screen window can be obtained by calling the relevant API.

[0022] Attribute information of a screen window of a current frame to be encoded is obtained, and a reference frame is selected for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window.

[0023] In one embodiment, selecting a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window includes:

[0024] Compare the window unique identifiers of the current frame to be encoded and the reference frame screen windows in the reference frame queue, and select one or more reference frames from the reference frame queue that contain the same window unique identifier as the current frame to be encoded as reference frames of the current frame to be encoded.

[0025] If the window unique identifier of the current to-be-encoded frame screen window does not exist in the attribute information of the reference frame screen window in the reference frame queue, a reference frame is selected from the reference frame queue according to the content characteristics of the current to-be-encoded frame.

[0026] The content features can be represented by the following parameters: pixel average, color histogram, gradient histogram, texture features, etc.

[0027] For example, as an implementation method, selecting a reference frame in a reference frame queue based on content features of the current frame to be encoded includes: calculating the frame-level pixel average of the current frame to be encoded and the frame-level pixel average of the reference frames in the reference frame queue, and selecting a reference frame in the reference frame queue whose frame-level pixel average is closest to the frame-level pixel average of the current frame to be encoded as the reference frame of the current frame to be encoded.

[0028] For example, as another implementation method, the root selects a reference frame from a reference frame queue based on content features of the current frame to be encoded, including: calculating a color histogram of the current frame to be encoded and a color histogram of the reference frames in the reference frame queue, and then evaluating the similarity between the current frame to be encoded and the reference frames in the reference frame queue through a similarity measurement method, such as a Bhattacharyya distance method, a histogram intersection method, etc., and selecting the reference frame with the highest similarity as the reference frame of the current frame to be encoded.

[0029] It is understandable that compared to natural images captured by the acquisition device of the camera device, the screen content image often contains a large number of repetitive patterns and regular structures. The regularity of the screen content can be used for efficient encoding during encoding. For example, under normal circumstances, if the window unique identifier of the screen window is the same in different video frames, the screen content presented between the different video frames is likely to be highly similar or even the same. Therefore, this feature can be used to quickly filter the reference frame for the current frame to be encoded in order to reduce latency and save encoding overhead.

[0030] In another embodiment, selecting a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window includes:

[0031] The window unique identifiers of the current frame to be encoded and the reference frame screen windows in the reference frame queue are compared, and multiple reference frames containing the same window unique identifier as the current frame to be encoded are screened out from the reference frame queue as candidate reference frames of the current frame to be encoded.

[0032] The similarity between the current frame to be encoded and the candidate reference frame is evaluated based on the attribute information of the screen window other than the window unique identifier, and then a reference frame is selected for the current frame to be encoded. The higher the similarity, the higher the reference priority of the candidate reference frame.

[0033] The evaluating the similarity between the current frame to be encoded and the candidate reference frames based on the attribute information of the screen window other than the window unique identifier, thereby selecting a reference frame for the current frame to be encoded, includes:

[0034] As an implementation method, the overlapping area of ​​the windows between the current frame to be encoded and the candidate reference frames is calculated. The larger the overlapping area, the higher the similarity, and the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0035] The window overlapping area can be calculated by obtaining the window size and window position information stored in the system.

[0036] As another implementation, the relative positions of the windows in the current frame to be encoded and the candidate reference frames are calculated. The closer the relative positions are, the higher the degree of similarity is, and the higher the reference priority of the candidate reference frame is. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0037] The relative position of the window can be calculated by obtaining the window position information stored in the system.

[0038] As another implementation, the stacking order of the windows in the current frame to be encoded and the candidate reference frames is compared. The closer the stacking order, the higher the similarity, and the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0039] The stacking order of the windows can be determined by obtaining the window stacking order stored in the system.

[0040] As another implementation, weights are preset for window overlap area, window relative position, and window stacking order, and the degree of similarity between the current frame to be encoded and the candidate reference frames is comprehensively evaluated. The higher the degree of similarity, the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority becomes the reference frame for the current frame to be encoded.

[0041] For example, the weight of the window overlap area is set to t1, the weight of the window relative position is set to t2, and the weight of the window stacking order is set to t3. The reference priority is quantified into a grade score, with higher similarity indicating a higher grade score. Grade scores a, b, and c are calculated based on the window overlap area, window relative position, and window stacking order, respectively. In this case, the reference priority of the candidate reference frame obtained through comprehensive evaluation is (t1*a+t2*b+t3*c).

[0042] The reference priorities of all the candidate reference frames are calculated in the same way, and the candidate reference frame with the highest reference priority is selected as the reference frame of the current frame to be encoded.

[0043] It is understandable that, in this step, the specific method for evaluating the similarity between the current frame to be encoded and the candidate reference frame is not limited in the embodiment of the present invention, and different evaluation methods can be adopted according to the actual screen encoding content scenario.

[0044] The reference frame queue is updated according to a preset update mechanism.

[0045] The preset update mechanism includes:

[0046] Set the frame number threshold T of the reference frames in the reference frame queue.

[0047] When the number of reference frames in the reference frame queue reaches the threshold T, the reference frame to be added to the reference frame queue replaces the reference frame with the least number of references in the reference frame queue.

[0048] It can be understood that this replacement strategy can ensure that new reference frames can be updated to the reference frame queue in a timely manner, and helps to avoid storing too many outdated reference frames in the reference frame queue.

[0049] Figure 2 FIG. 1 shows a schematic diagram of a reference frame management system based on screen content coding provided by an embodiment of the present invention. Figure 2 As shown, the reference frame management system based on screen content coding provided by an embodiment of the present invention includes:

[0050] The information maintenance module is used to maintain a reference frame queue and attribute information of the reference frame screen window in the reference frame queue.

[0051] The attribute information of the screen window includes but is not limited to: window unique identifier, window size, window position, and window stacking order.

[0052] The reference frame selection module is used to obtain attribute information of the screen window of the current frame to be encoded, and select a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window.

[0053] In one embodiment, selecting a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window includes:

[0054] Compare the window unique identifiers of the current frame to be encoded and the reference frame screen windows in the reference frame queue, and select one or more reference frames from the reference frame queue that contain the same window unique identifier as the current frame to be encoded as reference frames of the current frame to be encoded.

[0055] If the window unique identifier of the current to-be-encoded frame screen window does not exist in the attribute information of the reference frame screen window in the reference frame queue, a reference frame is selected from the reference frame queue according to the content characteristics of the current to-be-encoded frame.

[0056] The content features can be represented by the following parameters: pixel average, color histogram, gradient histogram, texture features, etc.

[0057] For example, as an implementation method, selecting a reference frame in a reference frame queue based on content features of the current frame to be encoded includes: calculating the frame-level pixel average of the current frame to be encoded and the frame-level pixel average of the reference frames in the reference frame queue, and selecting a reference frame in the reference frame queue whose frame-level pixel average is closest to the frame-level pixel average of the current frame to be encoded as the reference frame of the current frame to be encoded.

[0058] For example, as another implementation method, the root selects a reference frame from a reference frame queue based on content features of the current frame to be encoded, including: calculating a color histogram of the current frame to be encoded and a color histogram of the reference frames in the reference frame queue, and then evaluating the similarity between the current frame to be encoded and the reference frames in the reference frame queue through a similarity measurement method, such as a Bhattacharyya distance method, a histogram intersection method, etc., and selecting the reference frame with the highest similarity as the reference frame of the current frame to be encoded.

[0059] In another embodiment, selecting a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window includes:

[0060] The window unique identifiers of the current frame to be encoded and the reference frame screen windows in the reference frame queue are compared, and multiple reference frames containing the same window unique identifier as the current frame to be encoded are screened out from the reference frame queue as candidate reference frames of the current frame to be encoded.

[0061] The similarity between the current frame to be encoded and the candidate reference frame is evaluated based on the attribute information of the screen window other than the window unique identifier, and then a reference frame is selected for the current frame to be encoded. The higher the similarity, the higher the reference priority of the candidate reference frame.

[0062] The evaluating the similarity between the current frame to be encoded and the candidate reference frames based on the attribute information of the screen window other than the window unique identifier, thereby selecting a reference frame for the current frame to be encoded, includes:

[0063] As an implementation method, the overlapping area of ​​the windows between the current frame to be encoded and the candidate reference frames is calculated. The larger the overlapping area, the higher the similarity, and the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0064] The window overlapping area can be calculated by obtaining the window size and window position information stored in the system.

[0065] As another implementation, the relative positions of the windows in the current frame to be encoded and the candidate reference frames are calculated. The closer the relative positions are, the higher the degree of similarity is, and the higher the reference priority of the candidate reference frame is. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0066] The relative position of the window can be calculated by obtaining the window position information stored in the system.

[0067] As another implementation, the stacking order of the windows in the current frame to be encoded and the candidate reference frames is compared. The closer the stacking order, the higher the similarity, and the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

[0068] The stacking order of the windows can be determined by obtaining the window stacking order stored in the system.

[0069] As another implementation, weights are preset for window overlap area, window relative position, and window stacking order, and the degree of similarity between the current frame to be encoded and the candidate reference frames is comprehensively evaluated. The higher the degree of similarity, the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority becomes the reference frame for the current frame to be encoded.

[0070] For example, the weight of the window overlap area is set to t1, the weight of the window relative position is set to t2, and the weight of the window stacking order is set to t3. The reference priority is quantified into a grade score, with higher similarity indicating a higher grade score. Grade scores a, b, and c are calculated based on the window overlap area, window relative position, and window stacking order, respectively. In this case, the reference priority of the candidate reference frame obtained through comprehensive evaluation is (t1*a+t2*b+t3*c).

[0071] The reference priorities of all the candidate reference frames are calculated in the same way, and the candidate reference frame with the highest reference priority is selected as the reference frame of the current frame to be encoded.

[0072] The reference frame queue updating module is used to update the reference frame queue according to a preset updating mechanism.

[0073] The preset update mechanism includes:

[0074] Set the frame number threshold T of the reference frames in the reference frame queue.

[0075] When the number of reference frames in the reference frame queue reaches the threshold T, the reference frame to be added to the reference frame queue replaces the reference frame with the least number of references in the reference frame queue.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A reference frame management method based on screen content coding, characterized in that: include: Maintaining a reference frame queue and attribute information of the reference frame screen window in the reference frame queue; Acquiring attribute information of a screen window of a current frame to be encoded, and selecting a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window; Finally, the reference frame queue is updated according to a preset update mechanism; The attribute information of the screen window includes: window unique identifier, window size, window position, and window stacking order; The selecting a reference frame for the current frame to be encoded according to the attribute information of the screen window includes: Comparing the window unique identifiers of the current frame to be encoded with those of the reference frame screen windows in the reference frame queue, and selecting from the reference frame queue one or more reference frames containing the same window unique identifier as the current frame to be encoded as reference frames of the current frame to be encoded; If the window unique identifier of the current to-be-encoded frame screen window does not exist in the attribute information of the reference frame screen window in the reference frame queue, selecting a reference frame in the reference frame queue according to the content feature of the current to-be-encoded frame; After comparing the window unique identifiers of the current frame to be encoded with the reference frame screen windows in the reference frame queue, and selecting a plurality of reference frames containing the same window unique identifier as the current frame to be encoded from the reference frame queue, the method further includes: Filtering a plurality of reference frames containing the same window unique identifier as the current frame to be encoded from the reference frame queue as candidate reference frames; evaluating the similarity between the current frame to be encoded and the candidate reference frames according to the attribute information of the screen window other than the window unique identifier, thereby selecting a reference frame for the current frame to be encoded, wherein the higher the similarity, the higher the reference priority of the candidate reference frame; The evaluating the similarity between the current frame to be encoded and the candidate reference frames based on the attribute information of the screen window other than the window unique identifier, thereby selecting a reference frame for the current frame to be encoded, includes: Calculating the overlapping area of ​​the windows in the current frame to be encoded and the candidate reference frame, wherein the larger the overlapping area, the higher the similarity, and the higher the reference priority of the candidate reference frame; Alternatively, the relative position of the window in the current frame to be encoded and the window in the candidate reference frame is calculated, and the closer the relative position is, the higher the similarity is, and the higher the reference priority of the candidate reference frame is; or, comparing the window stacking order of the current frame to be encoded and the window stacking order of the candidate reference frame; the closer the stacking order is, the higher the similarity is, and the higher the reference priority of the candidate reference frame; Alternatively, the window overlap area, the relative position of the windows, and the weight of the window stacking order are preset to comprehensively evaluate the similarity between the current frame to be encoded and the candidate reference frame. The higher the similarity, the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded.

2. The reference frame management method based on screen content coding according to claim 1, characterized in that: The content features can be represented by the following parameters: pixel average, color histogram, gradient histogram, and texture features.

3. The reference frame management method based on screen content coding according to claim 1, characterized in that: The preset update mechanism includes: Set the frame number threshold T of the reference frame queue; When the number of reference frames in the reference frame queue reaches the threshold T, the reference frame to be added to the reference frame queue replaces the reference frame with the least number of references in the reference frame queue.

4. A reference frame management system based on screen content coding, characterized in that: include: An information maintenance module, configured to maintain a reference frame queue and attribute information of the reference frame screen windows in the reference frame queue; The attribute information of the screen window includes: window unique identifier, window size, window position, and window stacking order; A reference frame selection module, configured to obtain attribute information of a screen window of a current frame to be encoded, and select a reference frame for the current frame to be encoded in the reference frame queue according to the attribute information of the screen window; The selecting a reference frame for the current frame to be encoded according to the attribute information of the screen window includes: Comparing the window unique identifiers of the current frame to be encoded with those of the reference frame screen windows in the reference frame queue, and selecting from the reference frame queue one or more reference frames containing the same window unique identifier as the current frame to be encoded as reference frames of the current frame to be encoded; If the window unique identifier of the current to-be-encoded frame screen window does not exist in the attribute information of the reference frame screen window in the reference frame queue, selecting a reference frame in the reference frame queue according to the content feature of the current to-be-encoded frame; After comparing the window unique identifiers of the current frame to be encoded with the reference frame screen windows in the reference frame queue, and selecting a plurality of reference frames containing the same window unique identifier as the current frame to be encoded from the reference frame queue, the method further includes: Filtering a plurality of reference frames containing the same window unique identifier as the current frame to be encoded from the reference frame queue as candidate reference frames; evaluating the similarity between the current frame to be encoded and the candidate reference frames according to the attribute information of the screen window other than the window unique identifier, thereby selecting a reference frame for the current frame to be encoded, wherein the higher the similarity, the higher the reference priority of the candidate reference frame; The evaluating the similarity between the current frame to be encoded and the candidate reference frames based on the attribute information of the screen window other than the window unique identifier, thereby selecting a reference frame for the current frame to be encoded, includes: Calculating the overlapping area of ​​the windows in the current frame to be encoded and the candidate reference frame, wherein the larger the overlapping area, the higher the similarity, and the higher the reference priority of the candidate reference frame; Alternatively, the relative position of the window in the current frame to be encoded and the window in the candidate reference frame is calculated, and the closer the relative position is, the higher the similarity is, and the higher the reference priority of the candidate reference frame is; or, comparing the window stacking order of the current frame to be encoded and the window stacking order of the candidate reference frame; the closer the stacking order is, the higher the similarity is, and the higher the reference priority of the candidate reference frame; Alternatively, the window overlap area, the relative position of the windows, and the weight of the window stacking order are preset to comprehensively evaluate the similarity between the current frame to be encoded and the candidate reference frame. The higher the similarity, the higher the reference priority of the candidate reference frame. The candidate reference frame with the highest reference priority is the reference frame of the current frame to be encoded; The reference queue updating module is used to update the reference frame queue according to a preset updating mechanism.

5. The reference frame management system based on screen content coding according to claim 4, characterized in that: The content features can be represented by the following parameters: pixel average, color histogram, gradient histogram, and texture features.

6. The reference frame management system based on screen content coding according to claim 4, characterized in that: The preset update mechanism includes: Set the frame number threshold T of the reference frame queue; When the number of reference frames in the reference frame queue reaches the threshold T, the reference frame to be added to the reference frame queue replaces the reference frame with the least number of references in the reference frame queue.

Images