A method and system for encoding salient regions in real-time rendered scenes

Abstract

A method and system for encoding salient regions in real-time rendering scenarios includes: configuring the importance of various objects on the application screen; generating a video stream in real time based on user-input control information; acquiring and rendering the current frame from the video stream; then, based on the configured object importance, re-rendering the corresponding importance texture map of the current frame in the GPU buffer through the rendering context; finally, sending the current frame and its importance texture map into the CPU; calculating the QP increment that needs to be adjusted for each macroblock in the current frame based on the importance weight of macroblocks in the importance texture map; then adjusting the QP value of each macroblock based on the QP increment; and encoding the current frame based on the QP value. This invention relates to the field of information and communication networks, and can acquire and encode accurate object-level salient regions from the rendering pipeline in real time, enhancing the video quality of user-interested regions while ensuring real-time performance. It is suitable for real-time rendering scenarios such as video games.

Description

Technical Field

[0001] This invention relates to a salient region encoding method and system for real-time rendering scenarios, and relates to the field of information and communication networks. Background Technology

[0002] The human visual system is less sensitive to images outside the visual center. Region of Interest (ROI) based coding is a method to improve video quality, typically used for offline video transmission. The basic unit of coding is a macroblock, a 16x16 pixel block. Its core idea is to improve the subjective quality of experience (QOE) of the video by allocating more coding resources to macroblocks that users are more likely to be interested in, given a fixed network bandwidth.

[0003] Real-time rendering scenarios, such as video games and virtual reality applications, differ from offline video transmission. In real-time rendering, the system generates a video stream based on real-time user input control information, rather than transmitting pre-rendered offline video. This necessitates real-time synchronous salient region coding, presenting additional challenges. In this scenario, salient region coding can be broken down into two steps: acquiring salient regions that closely approximate the user's actual area of ​​interest and allocating more coding resources to these salient regions within a given bandwidth. Furthermore, minimizing the impact of these steps on real-time performance is a key requirement that distinguishes this scenario from offline video transmission. Typically, video games require a frame rate of 60 FPS or higher on the user's device to ensure a smooth experience, while virtual reality scenarios have even more stringent frame rate requirements. Offline video salient region coding methods often predict salient regions as rectangular bounding boxes, which is insufficiently accurate. In real-time rendering scenarios, because the rendering pipeline itself uses object-level rendering target information, there is the potential to divide salient regions into macroblocks.

[0004] The relevant technical solutions currently include the following:

[0005] 1. Paper 1: Content-aware video encoding for cloud gaming [C] (Authors: Hegazy M, Diab K, Saeedi M, et al., Journal: Proceedings of the 10th ACM multimediasystems conference. 2019: 60-73.). This technical solution applies salient region encoding to video games in real-time rendering scenarios and proposes a salient region encoding algorithm. However, it only describes a method for obtaining salient regions in real time. It is powerless for objects of interest with occlusion relationships because the occlusion relationship between objects cannot be inferred without combining the depth information in the rendering pipeline. Therefore, the salient region in this technical solution is a rectangular region, just like in offline video methods, and cannot be accurate to the outline of the object.

[0006] 2. Paper 2: Enhancing video encoding for cloud gaming using rendering information [J] (Authors: Liu Y, Dey S, Lu Y, Journal: IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(12): 1960-1974.) Similar to Paper 1, it introduces the idea of ​​obtaining object-level ROI using rendering pipeline information, but it does not provide an effective method for determining salient regions. This technical solution proposes to obtain accurate object-level salient regions by obtaining information from the rendering pipeline, but the pixel-by-pixel weighted algorithm does not take real-time performance into account.

[0007] 3. Paper 3: Deepgame: Efficient video encoding for cloud gaming [C] (Authors: Mossad O, Diab K, Amer I, et al., Journal: Proceedings of the 29th ACM International Conference on Multimedia. 2021: 1387-1395.). This technical solution introduces deep learning methods to obtain salient regions. Using a pre-trained neural network model, all targets in the scene are first identified using YOLOv3, and then salient regions are inferred using LSTM. However, this type of solution requires eye tracking to collect a large amount of information as a training set. Considering the diversity of video game screen types and the continuous updates of game content, this method of collection and training would be extremely cumbersome and lacks universality. The addition of deep learning algorithms also has a significant impact on real-time performance, and the learned salient regions are neither refined enough to pinpoint the contours of objects of interest, nor can they provide high accuracy in some game applications with special visual styles.

[0008] 4. Chinese Patent CN 108063946 A relates to an image encoding method and apparatus, storage medium, and electronic device. The method includes: acquiring an image frame matching the current playback viewpoint from panoramic video image frames in a video stream to be played; determining a first region and a second region contained in the image frame, wherein the first region is a region of interest (ROI) in the image frame, and the second region is a region in the image frame excluding the ROI; acquiring the encoding weight of a first macroblock contained in the first region and the encoding weight of a second macroblock contained in the second region, wherein the encoding weight of the first macroblock is greater than the encoding weight of the second macroblock; encoding the first region according to the encoding weight of the first macroblock, and encoding the second region according to the encoding weight of the second macroblock. This technical solution processes a panoramic video that has already been rendered offline and does not involve processing video frames generated in real-time.

[0009] Therefore, how to obtain accurate object-level salient regions from the rendering pipeline in real time and encode these salient regions to enhance the video quality of areas of interest to users while ensuring real-time performance, and to make them suitable for real-time rendering scenarios such as video games and virtual reality applications, has become a key technical issue of concern for engineers. Summary of the Invention

[0010] In view of this, the purpose of this invention is to provide a salient region encoding method and system for real-time rendering scenarios, which can obtain accurate object-level salient regions from the rendering pipeline in real time and encode the salient regions, thereby enhancing the video quality of the user's area of ​​interest while ensuring real-time performance. This method is applicable to real-time rendering scenarios such as video games and virtual reality applications.

[0011] To achieve the above objectives, the present invention provides a salient region encoding method for real-time rendering scenes, comprising:

[0012] Step 1: Configure the importance of various objects on the application screen;

[0013] Step 2: Generate a video stream in real time based on the control information input by the user. Obtain and render the current frame from the real-time video stream. Then, based on the configured object importance, render the importance texture map corresponding to the current frame in the GPU buffer through the rendering context. Finally, send the current frame and its importance texture map into the CPU. The importance texture map uses different RGB texture values ​​to identify the importance weight of macroblocks corresponding to different objects in the current frame based on the configured object importance.

[0014] Step 3: Based on the importance weight of macroblocks in the importance texture map, calculate the QP increment that each macroblock in the current frame needs to be adjusted according to its importance weight. Then, adjust the QP value of each macroblock according to the QP increment, and encode the current frame according to the adjusted QP value.

[0015] To achieve the above objectives, the present invention also provides a salient region encoding system for real-time rendering scenes, comprising:

[0016] The salient region extraction device configures the importance of various objects on the application screen, generates a video stream in real time based on the control information input by the user, obtains and renders the current frame from the real-time video stream, and then, based on the configured object importance, renders and generates the importance texture map corresponding to the current frame in the GPU buffer through the rendering context. Finally, the current frame and its importance texture map are sent to the CPU. The importance texture map uses different RGB texture values ​​to identify the importance weight of macroblocks corresponding to different objects in the current frame based on the configured object importance.

[0017] The salient region encoding device calculates the QP increment that each macroblock in the current frame needs to be adjusted according to its importance weight in the importance texture map, then adjusts the QP value of each macroblock according to the QP increment, and encodes the current frame according to the adjusted QP value.

[0018] To achieve the above objectives, the present invention also provides a computing device, comprising:

[0019] Memory and processor;

[0020] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the salient region encoding method in the real-time rendering scene.

[0021] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the salient region encoding method in the real-time rendering scene.

[0022] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a method for obtaining accurate object-level salient region weight information in real time from the rendering pipeline. The salient regions can be accurate to the object outline, and it is suitable for application scenarios with complex occlusion relationships. It also has almost no negative impact on the frame rate of game applications. Users (application developers) can configure the importance of objects of interest based on their prior knowledge of the game, and then encode the salient regions of the final output image. No additional processing is required for each object of interest in the scene, and it has good scalability. This invention also achieves strong real-time performance by utilizing the GPU parallel computing capabilities of a custom post-processing rendering channel and compute shaders. The post-processing rendering channel renders the importance texture map for each frame in the GPU, and then uses G... The PU multi-threading technology smooths, filters, and compresses the importance texture map, ultimately sending it back to the CPU and delivering it to the encoder. The overall overhead is less than 1ms on a typical device, demonstrating strong real-time performance. Existing solutions often neglect real-time performance, and overly refined weight allocation algorithms lose accuracy when adjusting quantization parameters during encoding. This invention also provides a method for real-time salient region encoding using importance weights, significantly improving video quality under the same bandwidth or saving bandwidth requirements for the same video quality. A first-order RQ model is used to calculate the average quality of the overall image, and then iterative numerical calculations are performed based on the importance weights of salient regions and the encoding context of the current frame to determine the optimal encoding resource allocation scheme, thereby achieving the effect of not changing the total amount of encoding resources occupied by the frame. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a salient region encoding method in a real-time rendering scene, as shown in an exemplary embodiment of the present invention.

[0024] Figure 2 This is an exemplary embodiment of the present invention. Figure 1 The detailed process flowchart for step three.

[0025] Figure 3 This is a schematic diagram illustrating the structure of a salient region encoding system in a real-time rendering scene, as shown in an exemplary embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0028] This invention proposes a real-time salient region encoding method based on the rendering pipeline and user prior information, which can be divided into two parts: salient region extraction and salient region encoding. The former generates an importance texture map based on the importance configuration given by the user, and further processes and compresses the information using GPU multi-threading technology before delivering it to the encoder. The latter reallocates encoding resources to each macroblock on the screen using this importance weight information to enhance the image quality of the user's regions of interest.

[0029] like Figure 1 As shown, the present invention provides a salient region encoding method for real-time rendering scenes, comprising:

[0030] Step 1: Configure the importance of various objects on the application screen;

[0031] Step 2: Generate a video stream in real time based on the control information input by the user. Obtain and render the current frame from the real-time video stream. Then, based on the configured object importance, render the importance texture map corresponding to the current frame in the GPU buffer through the rendering context. Finally, send the current frame and its importance texture map into the CPU. The importance texture map uses different RGB texture values ​​to identify the importance weight of macroblocks corresponding to different objects in the current frame based on the configured object importance.

[0032] Step 3: Based on the importance weight of macroblocks in the importance texture map, calculate the QP (Quantization Parameter) increment that each macroblock in the current frame needs to be adjusted according to its importance weight. Then, adjust the QP value of each macroblock according to the QP increment, and encode the current frame according to the adjusted QP value.

[0033] In step one, specifically, the user (application developer) can configure the importance of various objects that ultimately appear on the application screen based on their prior knowledge of the application. Here, "objects" is used in a broad sense and can also be called "display objects." For example, for most games, the user interface (UI) usually needs to be emphasized, so the UI's importance can be configured higher. For most third-person games, the player's attention is often focused on the controlled character, so the character's importance can also be configured higher.

[0034] In step two, the invention may further include a post-processing rendering channel. This channel, combining the importance configuration and the rendering context, renders an importance texture map for the current frame. The rendering context contains information such as the objects to be rendered and depth, allowing for the sequential rendering of various types of objects of interest into a GPU buffer while preserving the correct depth relationships. For example, for a resolution of 1296*1440 (typically seen in VR), since the basic unit of encoding, the macroblock, is 16*16 in size, the rendered importance texture map will have a resolution of 81*90. Step two, based on the configured object importance, further includes rendering the corresponding importance texture map for the current frame in the GPU buffer using the rendering context.

[0035] The post-processing rendering pass creates an importance texture map in the GPU buffer. The importance texture map is used to store the importance weights of all macroblocks in the current frame. First, the rendering target is set to the importance texture map. For each configured object class, the list of objects of that class that need to be rendered to the screen in the current frame is obtained from the culling results of the rendering context. Then, rendering is performed, and the importance weights are mapped to RGB values. All objects in the object list are rendered to the rendering target (i.e., the importance texture map) using RGB values ​​representing different importance weights. At the same time, depth read and write are enabled.

[0036] Encoding and transmission by the encoder need to be performed in the CPU. The importance texture map is actually an RGB texture stored in the GPU. Directly reading it into the CPU would significantly impact real-time performance because data transfer between the GPU and CPU is typically done via the PCI Express (PCIe) bus, and the transmission speed is limited by the PCIe version and the number of channels. In real-time salient region encoding, the salient region acquisition process must be performed synchronously every frame, which would cause a significant drop in frame rate. On the other hand, the importance texture map currently stored in the GPU is not smooth. For example, when two objects with large differences in importance weights are very close to each other on the screen, it may cause a sudden change in the image quality of adjacent macroblocks, resulting in discomfort. Therefore, this invention also introduces a computation shader, which is a technique that utilizes GPU multi-threading to efficiently process large-scale parallel computing. This can effectively reduce the amount of data sent back from the GPU to the CPU, smoothing the weights while significantly reducing the impact on real-time performance. In step two, sending the current frame and its importance texture map into the CPU can further include:

[0037] The computation shader first performs an average filter on the importance texture map, that is, for each pixel in the importance texture map, its importance weight is corrected to the average weight within a certain area (e.g., 5*5). Then, the RGB texture is compressed into an integer array, and finally the importance texture map compressed into an integer array is sent to the CPU.

[0038] This invention can be based on the open-source OpenH264 encoder. According to the importance texture map corresponding to the current frame in the video stream, the quantization parameters (QP) of each macroblock in the frame are adjusted to achieve a reallocation of encoding resources. For example... Figure 2 As shown, Figure 1 In step three, based on the importance weights of macroblocks in the importance texture map, the QP increment that each macroblock in the current frame needs to be adjusted according to its importance weight is calculated. Then, the QP value of each macroblock is adjusted according to the QP increment. This may further include:

[0039] Step 31: Calculate the positive and negative contribution values ​​of the importance of each macroblock in the current frame: SP i =max(S) avg -S i ,0), SN i =min(S) i -S avg ,0), where SP i SN i These are the positive and negative contribution values ​​of the importance of the i-th macroblock, respectively, S. avg S is the average importance weight of all macroblocks in the current frame. iIt is the importance weight of the i-th macroblock, and max(·) and min(·) represent the calculation of the maximum and minimum values, respectively;

[0040] The adjustment objective of this invention is to modify the bit rate of macroblocks while minimizing the change in average QP, reducing the QP in significant regions, and increasing the QP in non-significant regions (the lower the QP, the higher the quality).

[0041] Step 32: Calculate the QP increment that needs to be adjusted for each macroblock in the current frame according to its importance weight: ΔQP i =α*SP i -β*SN i Where α and β are adjustment coefficients, ΔQP i It is the QP increment that the i-th macroblock needs to be adjusted according to its importance weight;

[0042] Step 33: Calculate the adjusted QP value for each macroblock in the current frame: QP i ′ =QP i +ΔQP i QP i QP i ′ These are the QP values ​​before and after the adjustment of the i-th macroblock, respectively.

[0043] It is worth mentioning that in step 32, the values ​​of α and β can also be calculated using linear programming, as follows:

[0044] Step 321: Set the upper and lower bounds of the two linear programming parameters α and β: Set the lower bound of α and β to 0. The formula for calculating the upper bound is as follows: Where, α s β s These are the upper bounds of α and β, respectively, QP lb QP ub These are the upper and lower bounds of QP for the current frame, given by the encoder, and min(SP). i ) represents the minimum positive contribution value of the importance of all macroblocks in the current frame, max(SN) i This represents the maximum negative contribution value of all macroblocks in the current frame.

[0045] Step 322: Using the Gauss-Seidel method, calculate α and β when the sum of the QP increments that all macroblocks in the current frame need to adjust according to their importance weights is at its minimum. The minimum α and β are the calculated α and β.

[0046] Experiments have shown that the method of this invention can save 7%-19% of bandwidth when applied to the same video quality.

[0047] like Figure 3 As shown, the present invention provides a salient region encoding system for real-time rendering scenes, comprising:

[0048] The salient region extraction device configures the importance of various objects on the application screen, generates a video stream in real time based on the control information input by the user, obtains and renders the current frame from the real-time video stream, and then, based on the configured object importance, renders and generates the importance texture map corresponding to the current frame in the GPU buffer through the rendering context. Finally, the current frame and its importance texture map are sent to the CPU. The importance texture map uses different RGB texture values ​​to identify the importance weight of macroblocks corresponding to different objects in the current frame based on the configured object importance.

[0049] The salient region encoding device calculates the QP increment that each macroblock in the current frame needs to be adjusted according to its importance weight in the importance texture map, then adjusts the QP value of each macroblock according to the QP increment, and encodes the current frame according to the adjusted QP value.

[0050] See Figure 4 , Figure 4 This is a structural block diagram of a computing device 400 illustrated in an exemplary embodiment of this specification. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0051] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0052] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0053] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 400 can also be a mobile or stationary server or cloud server, etc.

[0054] The processor 420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the salient region encoding method in the above-described real-time rendering scene.

[0055] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the salient region encoding method in the real-time rendering scene described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the salient region encoding method in the real-time rendering scene described above.

[0056] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the salient region encoding method described above for real-time rendering scenarios.

[0057] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the salient region encoding method in the real-time rendering scene described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the salient region encoding method or system in the real-time rendering scene described above.

[0058] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described method for encoding salient regions in a real-time rendering scene.

[0059] The above is an illustrative scheme of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solution of the salient region encoding method in the real-time rendering scene described above. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the salient region encoding method or system in the real-time rendering scene described above.

[0060] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0061] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0062] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for encoding salient regions in real-time rendered scenes, the method comprising: Including: Step 1: Configure the importance of various objects on the application screen; Step 2: Generate a video stream in real time based on the control information input by the user. Obtain and render the current frame from the real-time video stream. Then, based on the configured object importance, render the importance texture map corresponding to the current frame in the GPU buffer through the rendering context. Finally, send the current frame and its importance texture map into the CPU. The importance texture map uses different RGB texture values ​​to identify the importance weight of macroblocks corresponding to different objects in the current frame based on the configured object importance. Step 3: Based on the importance weights of macroblocks in the importance texture map, calculate the QP increment that each macroblock in the current frame needs to adjust according to its importance weight. Then, adjust the QP value of each macroblock according to the QP increments, and encode the current frame based on the adjusted QP values. In step two, based on the configured object importance, the importance texture map corresponding to the current frame is generated in the GPU buffer through the rendering context, including: The post-processing rendering pass creates an importance texture map in the GPU buffer. This importance texture map stores the importance weights of all macroblocks in the current frame. First, the rendering target is set to the importance texture map. For each configured object class, the list of objects of that class that need to be rendered to the screen in the current frame is obtained from the culling results of the rendering context. Then, rendering is performed, mapping the importance weights to RGB values. All objects in the object list are rendered to the rendering target using RGB values ​​representing different importance weights, while simultaneously enabling depth read / write. In step two, the current frame and its importance texture map are sent to the CPU, including: The computation shader first performs an average filter on the importance texture map, that is, for each pixel in the importance texture map, its importance weight is corrected to the average weight within a certain area. Then, the RGB texture is compressed into an integer array, and finally the importance texture map compressed into an integer array is sent to the CPU.

2. The method according to claim 1, characterized in that, In step three, based on the importance weights of macroblocks in the importance texture map, the QP increment that each macroblock in the current frame needs to be adjusted according to its importance weight is calculated. Then, the QP value of each macroblock is adjusted according to the QP increment, including: Step 31: Calculate the positive and negative contribution values ​​of the importance of each macroblock in the current frame: , ,in, , These are the positive and negative contribution values ​​of the importance of the i-th macroblock, respectively. It is the average importance weight of all macroblocks in the current frame. It is the importance weight of the i-th macroblock. , These represent calculating the maximum and minimum values, respectively. Step 32: Calculate the QP increment that needs to be adjusted for each macroblock in the current frame according to its importance weight: ,in, , It is an adjustment factor. It is the QP increment that the i-th macroblock needs to be adjusted according to its importance weight; Step 33: Calculate the adjusted QP value for each macroblock in the current frame: ,in, , These are the QP values ​​before and after the adjustment of the i-th macroblock, respectively.

3. The method according to claim 2, characterized in that, In step 32, , The value of is obtained by linear programming, and the specific process is as follows: Step 321, Settings and The upper and lower bounds of these two linear programming parameters: , The lower bound is set to 0, and the upper bound is calculated using the following formula: , ,in, , They are , The upper bound value, , These are the upper and lower bounds of the QP for the current frame, respectively, given by the encoder. This represents the minimum positive contribution value of all macroblocks in the current frame. This represents the maximum negative contribution value of all macroblocks in the current frame. Step 322: Using the Gauss-Seidel method, find the minimum value when the sum of the QP increments that all macroblocks in the current frame need to be adjusted according to their importance weights. and The minimum value and That is, obtained through calculation. and .

4. A salient region encoding system for real-time rendering scenes, characterized in that, The system is used to execute the salient region encoding method for real-time rendering scenes as described in claim 1, and the system includes: The salient region extraction device configures the importance of various objects on the application screen, generates a video stream in real time based on the control information input by the user, obtains and renders the current frame from the real-time video stream, and then, based on the configured object importance, renders and generates the importance texture map corresponding to the current frame in the GPU buffer through the rendering context. Finally, the current frame and its importance texture map are sent to the CPU. The importance texture map uses different RGB texture values ​​to identify the importance weight of macroblocks corresponding to different objects in the current frame based on the configured object importance. The salient region encoding device calculates the QP increment that each macroblock in the current frame needs to be adjusted according to its importance weight in the importance texture map, then adjusts the QP value of each macroblock according to the QP increment, and encodes the current frame according to the adjusted QP value.

5. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the salient region encoding method in any one of claims 1-3 for real-time rendering scenes.

6. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the salient region encoding method for real-time rendering scenes as described in any one of claims 1-3.

Images