Salient region coding method and system in real-time rendering scene

Abstract

The invention discloses a method and a system for coding a salient region in a real-time rendering scene. The method comprises the following steps: configuring importance of various objects on an application picture; a video stream is generated in real time according to control information input by a user, a current frame picture is obtained from the video stream and rendered and generated, then an importance texture map corresponding to the current frame picture is rendered and generated in a GPU buffer area through a rendering context according to the importance of a configured object, and finally the current frame picture and the importance texture map of the current frame picture are sent into a CPU; and according to the importance weight of the macro block in the importance texture map, calculating the QP increment required to be adjusted of each macro block in the current frame picture, then adjusting the QP value of each macro block according to the QP increment, and coding the current frame picture according to the QP value. The invention relates to the field of information communication networks, can obtain an accurate object-level salient region from a rendering pipeline in real time and complete coding, enhances the video quality of a region of interest of a user on the premise of ensuring the real-time performance, and is suitable for real-time rendering scenes such as electronic games and the like.

Description

Technical Field

[0001] The present invention relates to a significant area coding method and system in a real-time rendering scene, and relates to the field of information communication networks. Background Art

[0002] The human visual system is less sensitive to images outside the central area of ​​vision. Region of Interest (ROI)-based coding is a method to improve video image quality, which is usually used for offline video transmission. The basic unit of coding is macroblock, which is a 16*16 pixel block. The core idea is to improve the subjective quality of experience (QOE) of the video by allocating more coding resources to macroblocks that users are more likely to be interested in when the network bandwidth is fixed.

[0003] Real-time rendering scenarios, such as video games and virtual reality applications, are different from offline video transmission. In the case of real-time rendering, the system generates a video stream based on the control information input by the user in real time, rather than transmitting pre-rendered offline videos. This requires that salient area encoding also be performed synchronously in real time, which brings additional challenges. In this scenario, the salient area encoding technology can be divided into two steps: obtaining salient areas close to the user's actual area of ​​interest and allocating more encoding resources to the salient areas under certain bandwidth conditions. In addition, how to minimize the impact of the above steps on real-time performance is a requirement that distinguishes this scenario from offline video transmission scenarios. Generally speaking, video games need to reach a frame rate of more than 60FPS on one user device to ensure a smooth user experience, and virtual reality scenarios have even more stringent frame rate requirements. The salient area predicted by the salient area encoding method of offline videos is usually a rectangular box, which is not accurate enough. In real-time rendering scenarios, since the rendering pipeline itself uses object-level rendering target information, it has the potential to divide the salient area into macroblocks.

[0004] At present, there are several related technical solutions:

[0005] 1. Paper 1: Content-aware video encoding for cloud gaming [C] (Author: Hegazy M, Diab K, Saeedi M, et al., Published in: Proceedings of the 10th ACM multimedia systems conference. 2019: 60-73.), this technical solution applies salient area coding to electronic games in real-time rendering scenarios, and proposes a salient area coding algorithm. However, it only briefly describes the method of obtaining salient areas in real time, and is powerless for objects of interest with occlusion relationships, because the occlusion relationship between objects cannot be inferred without combining the depth information in the rendering pipeline. Therefore, the salient area in this technical solution is a rectangular area like the offline video method, and cannot be accurately determined to the outline of the object.

[0006] 2. Paper 2: Enhancing video encoding for cloud gaming using rendering information [J] (Author: Liu Y, Dey S, Lu Y, Published in: IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(12): 1960-1974.) is similar to Paper 1 and introduces the idea of ​​using rendering pipeline information to obtain object-level ROI, but it also does not provide an effective method for determining salient areas. This technical solution proposes to obtain accurate object-level salient areas by obtaining information from the rendering pipeline, but the pixel-by-pixel weighting algorithm does not take real-time performance into consideration.

[0007] 3. Paper 3: Deepgame: Efficient video encoding for cloud gaming [C] (Author: Mossad O, Diab K, Amer I, et al, Published in: Proceedings of the 29th ACM International Conference on Multimedia. 2021: 1387-1395.), this technical solution introduces a deep learning method to obtain salient areas. Using a pre-trained neural network model, all targets in the picture are first identified through a YOLOv3, and then the salient areas are inferred through an LSTM. However, this type of technical solution requires the use of eye tracking to collect a large amount of information as a training set. Considering the diversity of electronic game screens and the continuous updating of game content, this collection and training method will be extremely cumbersome and not universal. The addition of deep learning algorithms also has a significant impact on real-time performance, and the salient areas learned are not fine enough to accurately identify the outline of the object of interest, and cannot provide high accuracy in some game applications with special screen styles.

[0008] 4. Chinese patent CN 108063946 A relates to an image encoding method and device, a storage medium and an electronic device, the method comprising: obtaining an image screen that matches the current playback perspective from a panoramic video image frame in the video stream data to be played; determining a first area and a second area contained in the image screen, wherein the first area is an area of ​​interest in the image screen, and the second area is an area other than the area of ​​interest in the image screen; obtaining the encoding weight of a first macroblock contained in the first area, and the encoding weight of a second macroblock contained in the second area, wherein the encoding weight of the first macroblock is greater than the encoding weight of the second macroblock; encoding the first area according to the encoding weight of the first macroblock, and encoding the second area according to the encoding weight of the second macroblock. This technical solution processes a panoramic video that has been rendered offline, and does not involve the processing of video screens generated by real-time rendering.

[0009] Therefore, how to obtain accurate object-level salient areas from the rendering pipeline in real time and encode the salient areas, so as to enhance the video quality of the user's area of ​​interest while ensuring real-time performance, which is suitable for real-time rendering scenarios such as electronic games and virtual reality applications, has become a technical issue that technicians have focused on. Summary of the invention

[0010] In view of this, the purpose of the present invention is to provide a salient area encoding method and system in a real-time rendering scenario, which can obtain accurate object-level salient areas from the rendering pipeline in real time and encode the salient areas, thereby enhancing the video quality of the user's area of ​​interest while ensuring real-time performance, and is suitable for real-time rendering scenarios such as electronic games and virtual reality applications.

[0011] In order to achieve the above object, the present invention provides a salient area encoding method in a real-time rendering scene, comprising:

[0012] Step 1: Configure the importance of various objects on the application screen;

[0013] Step 2: Generate a video stream in real time according to the control information input by the user, obtain and render the current frame from the real-time video stream, and then render the importance texture map corresponding to the current frame in the GPU buffer through the rendering context according to the importance of the configured object, and finally send the current frame and its importance texture map to the CPU, where the importance texture map uses different RGB texture values ​​to identify the importance weights of macroblocks corresponding to different objects in the current frame according to the importance of the configured object;

[0014] Step 3: According to the importance weights of the macroblocks in the importance texture map, calculate the QP increment that needs to be adjusted for each macroblock in the current frame according to the importance weight, then adjust the QP value of each macroblock according to the QP increment, and encode the current frame according to the adjusted QP value.

[0015] In order to achieve the above object, the present invention also provides a salient area coding system in a real-time rendering scene, comprising:

[0016] A salient region extraction device is configured to configure the importance of various objects on the application screen, generate a video stream in real time according to the control information input by the user, obtain and render the current frame from the real-time video stream, and then render the importance texture map corresponding to the current frame in the GPU buffer through the rendering context according to the importance of the configured object, and finally send the current frame and its importance texture map to the CPU, wherein the importance texture map uses different RGB texture values ​​to identify the importance weights of macroblocks corresponding to different objects in the current frame according to the importance of the configured object;

[0017] The salient area encoding device calculates the QP increment that needs to be adjusted for each macroblock in the current frame according to the importance weight of the macroblock in the importance texture map, then adjusts the QP value of each macroblock according to the QP increment, and encodes the current frame according to the adjusted QP value.

[0018] In order to achieve the above object, the present invention further provides a computing device, comprising:

[0019] Memory and processor;

[0020] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the salient area encoding method in the real-time rendering scene are implemented.

[0021] In order to achieve the above object, the present invention further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the salient area encoding method in the real-time rendering scene.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a method for obtaining accurate object-level salient area weight information from a rendering pipeline in real time, the salient area can be accurate to the object outline, and is suitable for application scenarios with complex occlusion relationships, and has almost no negative impact on the frame rate of game applications. Users (application developers) can configure the importance of objects of interest based on their prior knowledge of the game, and can encode the salient areas of the final output image, without the need for additional processing of each object of interest in the scene, and has good scalability; the present invention also achieves strong real-time performance by using the GPU parallel computing capability of a custom post-processing rendering channel and a compute shader, the post-processing rendering channel renders the importance texture map of each frame in the GPU, and then passes through the GPU PU multi-threading technology performs smoothing filtering and compression on the importance texture map, and finally transmits it back to the CPU and delivers it to the encoder. The overall overhead is less than 1ms on general equipment, and it has strong real-time performance. However, existing solutions usually ignore real-time performance, and the overly sophisticated weight allocation algorithm loses its accuracy when entering the encoding quantization parameter adjustment step. The present invention also provides a method for real-time salient area encoding using importance weights, which greatly improves the video quality under the same bandwidth or saves the bandwidth requirement under the same video quality. The first-order RQ model is used to calculate the average quality of the entire picture, and then iterative numerical calculations are performed based on the importance weights of the salient areas and the encoding context of the current frame to determine the optimal encoding resource allocation scheme, thereby achieving the effect of not changing the total amount of encoding resources occupied by the frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The flowchart of a salient area encoding method in a real-time rendering scene is shown in an exemplary embodiment of the present invention.

[0024] Figure 2 An exemplary embodiment of the present invention is shown Figure 1 Detailed process flow chart of step three.

[0025] Figure 3 It is a schematic diagram of the structure of a salient area encoding system in a real-time rendering scenario shown in an exemplary embodiment of the present invention.

[0026] Figure 4 It is a schematic diagram of the structure of a computer device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0028] The present invention proposes a real-time salient area coding method based on rendering pipeline and user prior information, which can be divided into two parts: salient area extraction and salient area coding. The former generates an importance texture map according to the importance configuration given by the user, and further processes and compresses the information to deliver it to the encoder through GPU multi-threading technology. The latter reallocates coding resources for each macroblock on the screen through the importance weight information to achieve the effect of enhancing the picture quality of the user's area of ​​interest.

[0029] like Figure 1 As shown, the present invention provides a significant area encoding method in a real-time rendering scene, comprising:

[0030] Step 1: Configure the importance of various objects on the application screen;

[0031] Step 2: Generate a video stream in real time according to the control information input by the user, obtain and render the current frame from the real-time video stream, and then render the importance texture map corresponding to the current frame in the GPU buffer through the rendering context according to the importance of the configured object, and finally send the current frame and its importance texture map to the CPU, where the importance texture map uses different RGB texture values ​​to identify the importance weights of macroblocks corresponding to different objects in the current frame according to the importance of the configured object;

[0032] Step 3: According to the importance weights of the macroblocks in the importance texture map, the QP (Quantization Parameter) increment that needs to be adjusted for each macroblock in the current frame is calculated, and then the QP value of each macroblock is adjusted according to the QP increment, and the current frame is encoded according to the adjusted QP value.

[0033] In step one, specifically, the user (application developer) can configure the importance of various objects that eventually appear on the application screen based on their prior knowledge of the application. The "object" here is in a broad sense and can also be called an "object". For example, for most games, the user interface (UI) part usually needs to be focused on, so the importance of the UI can be configured to be higher. For most third-person games, the player's attention is often focused on the controlled character, so the importance of the character can also be configured to be higher.

[0034] In step two, the present invention may also include a post-processing rendering channel, which combines the importance configuration with the rendering context to render the current frame to generate an importance texture map. The rendering context contains information such as the object to be rendered and the depth, so according to the configuration, each type of object of interest can be rendered into a GPU buffer in turn, and the correct depth relationship is retained. For example, for a resolution of 1296*1440 (usually seen in VR), since the size of the basic unit macroblock of encoding is 16*16, the resolution of the rendered importance texture map is 81*90. In step two, according to the configured object importance, the importance texture map corresponding to the current frame is generated by re-rendering in the GPU buffer through the rendering context, which may further include:

[0035] The post-processing rendering channel creates an importance texture map in the GPU buffer. The importance texture map is used to store the importance weights of all macroblocks for the current frame. First, set the rendering target to the importance texture map. For each type of object configured, obtain the object list of this type that needs to be rendered to the screen in the current frame from the culling result of the rendering context, and then perform rendering. Map the importance weights to RGB values, and use RGB values ​​representing different importance weights to render all objects in the object list to the rendering target (i.e., the importance texture map), and turn on depth reading and writing at the same time.

[0036] The encoding and transmission of the encoder need to be performed in the CPU. The importance texture map is actually an RGB texture stored in the GPU. If it is directly read into the CPU, it will have a greater impact on real-time performance. This is because the data transmission between the GPU and the CPU is usually completed through the PCI Express (PCIe) bus, and the transmission speed is limited by the PCIe version and the number of channels. In real-time salient area encoding, the salient area acquisition process must be performed synchronously for each frame, which will cause a large drop in frame rate. On the other hand, the importance texture map currently stored in the GPU is not smooth. For example, when two objects with a large difference in importance weights are very close to each other on the screen, visually, the picture quality of adjacent macroblocks may suddenly change, causing discomfort. Therefore, the present invention also introduces a computational shader, that is, a technology that uses GPU multi-threading to efficiently process large-scale parallel calculations, which can effectively reduce the amount of data transmitted back from the GPU to the CPU, smooth the weights, and greatly reduce the impact on real-time performance. In step two, the current frame picture and its importance texture map are sent to the CPU, and it can further include:

[0037] The compute shader first performs average filtering on the importance texture map, that is, for each pixel in the importance texture map, its importance weight is corrected to the weight average within a certain area range (for example, 5*5), and then the RGB texture is compressed into an integer array, and finally the importance texture map compressed into an integer array is sent to the CPU.

[0038] The present invention can be based on the open source openh264 encoder, and adjust the quantization parameter (QP) of each macroblock in the picture according to the importance texture map corresponding to the current frame picture in the video stream to achieve the reallocation of coding resources. Figure 2 As shown, Figure 1 In step 3, according to the importance weight of the macroblock in the importance texture map, the QP increment that needs to be adjusted for each macroblock in the current frame according to the importance weight is calculated, and then the QP value of each macroblock is adjusted according to the QP increment, which may further include:

[0039] Step 31: Calculate the positive and negative contribution values ​​of each macroblock in the current frame: SP i =max(S avg -S i ,0),SN i =min(S i -S avg ,0), where SP i , SN i are the positive and negative contributions of the importance of the i-th macroblock, S avg is the average importance weight of all macroblocks in the current frame, S iis the importance weight of the i-th macroblock, max(·) and min(·) represent the calculated maximum and minimum values, respectively;

[0040] The adjustment goal of the present invention is to modify the bit rate of the macroblock while minimizing the change in the average QP, reducing the QP of the salient area, and increasing the QP of the non-salient area (the lower the QP, the higher the quality);

[0041] Step 32: Calculate the QP increment that needs to be adjusted for each macroblock in the current frame according to the importance weight: ΔQP i =α*SP i -β*SN i , where α, β are adjustment coefficients, ΔQP i is the QP increment that needs to be adjusted for the i-th macroblock according to the importance weight;

[0042] Step 33, calculate the adjusted QP value of each macroblock in the current frame: QP i ′ =QP i +ΔQP i , where QP i , QP i ′ are the QP values ​​before and after adjustment of the i-th macroblock.

[0043] It is worth mentioning that in step 32, the values ​​of α and β can also be calculated by linear programming method, and the specific process is as follows:

[0044] Step 321, set the upper and lower bounds of the two linear programming parameters α and β: set the lower bounds of α and β to 0, and the upper bounds are calculated as follows: Among them, α s , β s are the upper bounds of α and β, respectively, and QP lb , QP ub They are the upper and lower QP limits of the current frame given by the encoder, min(SP i ) represents the minimum positive contribution value of all macroblocks in the current frame, max(SN i ) represents the maximum negative contribution value of the importance of all macroblocks in the current frame;

[0045] Step 322: Use the Gauss-Seidel method to solve α and β when the sum of the QP increments that need to be adjusted for all macroblocks in the current frame according to the importance weights is the minimum, and the minimum α and β are the calculated α and β.

[0046] Experimental verification shows that the method of the present invention can save 7%-19% of bandwidth under the same video quality.

[0047] like Figure 3 As shown, the present invention provides a salient area coding system in a real-time rendering scene, comprising:

[0048] A salient region extraction device is configured to configure the importance of various objects on the application screen, generate a video stream in real time according to the control information input by the user, obtain and render the current frame from the real-time video stream, and then render the importance texture map corresponding to the current frame in the GPU buffer through the rendering context according to the importance of the configured object, and finally send the current frame and its importance texture map to the CPU, wherein the importance texture map uses different RGB texture values ​​to identify the importance weights of macroblocks corresponding to different objects in the current frame according to the importance of the configured object;

[0049] The salient area encoding device calculates the QP increment that needs to be adjusted for each macroblock in the current frame according to the importance weight of the macroblock in the importance texture map, then adjusts the QP value of each macroblock according to the QP increment, and encodes the current frame according to the adjusted QP value.

[0050] See also Figure 4 , Figure 4 4 is a block diagram of a computing device 400 shown in an exemplary embodiment of this specification. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.

[0051] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0052] In one embodiment of the present specification, the above components of the computing device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 4 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0053] The computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 400 may also be a mobile or stationary server or a cloud server, etc.

[0054] The processor 420 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned salient area encoding method in the real-time rendering scenario.

[0055] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the salient area encoding method in the above-mentioned real-time rendering scenario belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the salient area encoding method in the above-mentioned real-time rendering scenario.

[0056] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned method for encoding a salient area in a real-time rendering scenario.

[0057] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the salient area encoding method in the above-mentioned real-time rendering scenario belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the salient area encoding method or system in the above-mentioned real-time rendering scenario.

[0058] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned method for encoding a salient area in a real-time rendering scenario.

[0059] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the salient area coding method in the above-mentioned real-time rendering scenario belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the salient area coding method or system in the above-mentioned real-time rendering scenario.

[0060] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0061] The computer instructions include computer program codes, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0062] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for encoding a salient area in a real-time rendering scene, characterized in that: Included are: Step 1: Configure the importance of various objects on the application screen; Step 2: Generate a video stream in real time according to the control information input by the user, obtain and render the current frame from the real-time video stream, and then render the importance texture map corresponding to the current frame in the GPU buffer through the rendering context according to the importance of the configured object, and finally send the current frame and its importance texture map to the CPU, where the importance texture map uses different RGB texture values ​​to identify the importance weights of macroblocks corresponding to different objects in the current frame according to the importance of the configured object; Step 3: According to the importance weights of the macroblocks in the importance texture map, calculate the QP increment that needs to be adjusted for each macroblock in the current frame according to the importance weight, then adjust the QP value of each macroblock according to the QP increment, and encode the current frame according to the adjusted QP value.

2. The method according to claim 1, characterized in that In step 2, according to the configured object importance, the importance texture map corresponding to the current frame is generated by re-rendering in the GPU buffer through the rendering context, including: The post-processing rendering channel creates an importance texture map in the GPU buffer. The importance texture map is used to store the importance weights of all macroblocks for the current frame. First, set the rendering target to the importance texture map. For each type of object configured, obtain the object list of this type that needs to be rendered to the screen in the current frame from the culling result of the rendering context, and then perform rendering. Map the importance weights to RGB values, and use RGB values ​​representing different importance weights to render all objects in the object list to the rendering target, while turning on depth reading and writing.

3. The method according to claim 1, characterized in that In step 2, the current frame and its importance texture map are sent to the CPU, including: The compute shader first performs average filtering on the importance texture map, that is, for each pixel in the importance texture map, its importance weight is corrected to the weight average within a certain area, and then the RGB texture is compressed into an integer array, and finally the importance texture map compressed into an integer array is sent to the CPU.

4. The method according to claim 1, characterized in that: In step 3, according to the importance weight of the macroblock in the importance texture map, the QP increment that needs to be adjusted for each macroblock in the current frame according to the importance weight is calculated, and then the QP value of each macroblock is adjusted according to the QP increment, including: Step 31: Calculate the positive and negative contribution values ​​of each macroblock in the current frame: SP i =max(S avg -S i ,0),SN i =min(S i -S avg ,0), Among them, SP i , SN i are the positive and negative contributions of the importance of the i-th macroblock, S avg is the average importance weight of all macroblocks in the current frame, S i is the importance weight of the i-th macroblock, max(·) and min(·) represent the calculated maximum and minimum values, respectively; Step 32: Calculate the QP increment that needs to be adjusted for each macroblock in the current frame according to the importance weight: ΔQP i =α*SP i -β*SN i , where α, β are adjustment coefficients, ΔQP i is the QP increment that needs to be adjusted for the i-th macroblock according to the importance weight; Step 33, calculate the adjusted QP value of each macroblock in the current frame: QP i ′ =QP i +ΔQP i , where QP i , QP i ′ are the QP values ​​before and after adjustment of the i-th macroblock.

5. The method according to claim 4, characterized in that In step 32, the values ​​of α and β are calculated by a linear programming method, and the specific process is as follows: Step 321, set the upper and lower bounds of the two linear programming parameters α and β: set the lower bounds of α and β to 0, and the upper bounds are calculated as follows: Among them, α s , β s are the upper bounds of α and β, respectively, and QP lb , QP ub They are the upper and lower QP limits of the current frame given by the encoder, min(SP i ) represents the minimum positive contribution value of all macroblocks in the current frame, max(SN i ) represents the maximum negative contribution value of the importance of all macroblocks in the current frame; Step 322: Use the Gauss-Seidel method to solve α and β when the sum of the QP increments that need to be adjusted for all macroblocks in the current frame according to the importance weights is the minimum, and the minimum α and β are the calculated α and β.

6. A salient area coding system in a real-time rendering scene, characterized in that: Included are: A salient region extraction device is configured to configure the importance of various objects on the application screen, generate a video stream in real time according to the control information input by the user, obtain and render the current frame from the real-time video stream, and then render the importance texture map corresponding to the current frame in the GPU buffer through the rendering context according to the importance of the configured object, and finally send the current frame and its importance texture map to the CPU, wherein the importance texture map uses different RGB texture values ​​to identify the importance weights of macroblocks corresponding to different objects in the current frame according to the importance of the configured object; The salient area encoding device calculates the QP increment that needs to be adjusted for each macroblock in the current frame according to the importance weight of the macroblock in the importance texture map, then adjusts the QP value of each macroblock according to the QP increment, and encodes the current frame according to the adjusted QP value.

7. A computing device, characterized in that include: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the salient area encoding method in a real-time rendering scene according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, can implement the steps of the significant area encoding method in a real-time rendering scene as described in any one of claims 1 to 5.

Images