A method for video encoding in a game engine and related equipment

Abstract

The present invention discloses a method for video encoding in a game engine and related equipment, comprising: sequentially obtaining video frames rendered and output by a graphics processing unit (GPU), and obtaining video encoding parameters from the game engine; and encoding the video frames rendered and output by the GPU based on the encoding behavior determined by the video encoding parameters. The video encoding method disclosed in the present invention can obtain additional video encoding parameters through the game engine, and can effectively improve the encoding efficiency of the video encoding engine.

Description

Technical Field

[0001] The present invention relates to the field of graphics processing technology, and in particular to a method for video encoding in a game engine and related equipment. Background Art

[0002] Typically, during the hardware video encoding process, the hardware video encoder sequentially receives images from each frame in the video stream for encoding, attempting to achieve the best video quality for the target bitrate. Since hardware encoding usually requires consistent encoding time (regardless of the complexity of each frame), and hardware design is cost-constrained by factors such as chip size, the hardware video encoder has limited ability to understand each picture frame it is encoding. For example, motion vector searches, which are commonly used in video encoding, have a limited range of motion vector searches because a larger search range implemented in hardware usually costs more memory storage, memory access, performance impact, and higher operating power consumption. For another example, for a video stream that contains long-term continuous static content, if Lookahead encoding is not implemented, the hardware video encoder will not be able to discover the statistical information patterns of content behavior over multiple frames.

[0003] But lookahead encoding is usually implemented in two steps to achieve video encoding. The first pass of encoding performs up to 40 frames to record frame encoding statistics, and the second pass of encoding as the actual encoding can use this statistics to correctly adjust the encoding parameters to achieve better performance in terms of quality and bitrate distribution. However, lookahead encoding will have a serious impact on the delay of frame input and encoded bitstream output.

[0004] Figure 1 A simplified system block diagram of a cloud gaming service is shown. Users typically interact with games or desktop applications (local or cloud) through physical components such as keyboards, mice, and tablets. The game engine is capable of processing data and providing rendering instructions to the graphics processing unit (GPU) to perform rendering on the frame buffer. Typically, the frames after GPU rendering are processed by the display component, so no special instructions are required to avoid image tearing except for frame synchronization instructions. As Internet video streaming and cloud gaming become more and more mainstream, these frames are not sent to the display component, but are processed by the video encoder engine because there is no physical display component. Each rendered frame provided by the GPU will perform video encoding in sequence. The video encoder engine will also need to store a copy of the converted frame in the reference frame buffer for future frame prediction encoding. Finally, a compressed encoded bitstream output is generated by the video encoder engine for transmission to the receiving end for reverse operation to reproduce the frame rendered by the GPU to display the frame information on the client.

[0005] A game or desktop application engine is usually software that runs on an operating system using a general-purpose CPU. For games, it usually processes and executes game commands from different sources including physical game input sources, audio and sound, network scripts, game object management, and most importantly, rendering to the GPU engine through a direct interface.

[0006] After the GPU renders, the final objects are rendered into a two-dimensional viewpoint, called a display frame buffer. Because there may be multiple frame buffers, some of them are used for consumption (displayed on the screen or encoded by the video encoder), while others are used for rendering operations.

[0007] The video encoder engine is the user of the 2D frame obtained after GPU rendering. It needs to encode the current frame, that is, the newly available frame after GPU rendering, through the video encoder. Summary of the invention

[0008] The embodiment of the present invention provides a method and related equipment for video encoding in a game engine, so as to improve the efficiency and speed of encoding.

[0009] In a first aspect, an embodiment of the present invention provides a method for video encoding in a game engine, comprising:

[0010] Sequentially obtain video frames rendered and output by the image processing unit GPU, and obtain video encoding parameters from the game engine;

[0011] Based on the encoding behavior determined by the video encoding parameters, the video frames rendered and output by the GPU are encoded.

[0012] In some embodiments, the acquired video encoding parameters include at least one of the following parameters: a global motion offset position parameter, a forced scene change parameter, a time interval for the next frame, a timestamp of a GPU rendered frame, and an expected parameter for game frame stream bitrate control.

[0013] In some embodiments, when the video encoding parameters include global motion offset position parameters, encoding the video frame output by the GPU rendering based on the encoding behavior determined by the video encoding parameters includes:

[0014] Outside the search range of the video encoder, a search is performed based on the global motion offset position parameter to implement motion estimation calculation between frames.

[0015] In some embodiments, searching outside a search range of the video encoder based on the global motion offset position parameter comprises:

[0016] A target pixel block is configured based on the global motion offset position parameter, and block matching is performed with the current position of the target pixel block as a starting point to perform a global offset search.

[0017] In some embodiments, performing a global offset search further includes:

[0018] A global offset region is configured based on the target image in the window, and no global offset search is performed outside the global offset region.

[0019] In some embodiments, performing a global offset search further includes:

[0020] In the case where there are multiple overlapping windows, an XOR logic operation is used to selectively enable the global offset for different windows.

[0021] In some embodiments, where the forced scene change parameters include global motion offset position parameters, the forced scene change parameters are configured as:

[0022] When a scene change is determined based on the logic of the game engine as an external parameter input, the subsequent scene is forced to be encoded using I frames to skip motion estimation and compensation.

[0023] In a second aspect, an embodiment of the present invention provides a device for video encoding in a game engine, including a video encoding engine, configured to:

[0024] Sequentially obtain video frames rendered and output by the image processing unit GPU, and obtain video encoding parameters from the game engine;

[0025] Based on the encoding behavior determined by the video encoding parameters, the video frames rendered and output by the GPU are encoded.

[0026] In a third aspect, an embodiment of the present invention provides a computer device, including a processor, a memory, and a communication bus;

[0027] The communication bus is used to realize the connection and communication between the processor and the memory;

[0028] The processor is used to execute one or more computer programs stored in the memory to implement the steps of the aforementioned video encoding method.

[0029] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned video encoding method are implemented.

[0030] The embodiment of the present invention obtains video encoding parameters from the game engine, and encodes the video frames rendered and output by the GPU based on the encoding behavior determined by the video encoding parameters. Thus, additional video encoding parameters can be obtained through the game engine, which can effectively improve the encoding efficiency of the video encoding engine.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0033] Figure 1 A system block diagram of an existing game service is shown;

[0034] Figure 2 A basic flow chart of an embodiment of the present disclosure is shown;

[0035] Figure 3 A system block diagram of a game service according to an embodiment of the present disclosure is shown.

[0036] Figure 4 A schematic diagram of using global offset to perform motion estimation according to an embodiment of the present disclosure is shown.

[0037] Figure 5 A schematic diagram of inter-frame motion estimation of pixel blocks in the prior art is shown.

[0038] Figure 6 A schematic diagram of inter-frame motion estimation of a pixel block according to an embodiment of the present disclosure is shown.

[0039] Figure 7 Another schematic diagram of inter-frame motion estimation of pixel blocks according to an embodiment of the present disclosure is shown.

[0040] Figure 8 A schematic diagram of player perspective definition according to an embodiment of the present disclosure is shown.

[0041] Fig. 9 The flowchart of the operation of the forced scene change parameters according to the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] The first embodiment of the present invention provides a method for video encoding in a game engine, such as Figure 2 As shown, including:

[0044] S201, sequentially obtain the video frames rendered and output by the image processing unit GPU, and obtain video encoding parameters from the game engine. In some embodiments, the obtained video encoding parameters include at least one of the following parameters: global motion offset position parameters, forced scene change parameters, time interval of the next frame, timestamp of GPU rendered frames, and expected parameters for game frame stream rate control. As an optional method of obtaining video encoding parameters from the game engine, the game engine can send video encoding parameters in the form of an SDK interface. The video rendering engine can obtain the required parameters from the SDK interface of the game engine. The specific SDK may include a thin drawing submission interface provided by Vulkan, directx12, and Metal, and can also submit the rendering of the game engine to the GPU. Modern game engines are becoming more and more complex, and can directly process GPU command buffers, memory allocation, and synchronization decisions, thereby mapping well to the multi-core / multi-threaded functions of modern CPUs, allowing parallel operations. In some embodiments, the game engine can provide parameters to the video encoder in the form of an SDK, and the specific parameters that can be provided may include various game behaviors, such as the conversion of a world view to a rendered 2D view, physical input information, static object positions, frame rendering behaviors, etc. The role of the SDK is to convert game-related information into encoding parameters used by the video encoder engine.

[0045] The GPU rendering engine is a data-centric, ultra-high-performance parallel pipeline engine. It requires commands from the game engine to perform real-time computer graphics rendering to a displayable frame buffer. The main GPU hardware engines currently come from NVIDIA and AMD. The process of performing rendering actually requires multiple stages of data processing.

[0046] S202: Encode the video frames rendered and output by the GPU based on the encoding behavior determined by the video encoding parameters.

[0047] After the GPU renders, the final objects are rendered to a 2D viewpoint, called the display frame buffer. Because there may be multiple frame buffers, some of them are used for consumption (displayed on the screen or encoded by the video encoder), while others are used for rendering operations. The video encoder engine is the user of the 2D frame obtained after the GPU renders. It needs to encode the current frame, that is, the newly available frame after the GPU renders, through the video encoder.

[0048] The embodiment of the present invention obtains video encoding parameters from the game engine, and encodes the video frames rendered and output by the GPU based on the encoding behavior determined by the video encoding parameters. Thus, additional video encoding parameters can be obtained through the game engine, which can effectively improve the encoding efficiency of the video encoding engine.

[0049] In some embodiments, when the video encoding parameters include global motion offset position parameters, encoding the video frame output by the GPU rendering based on the encoding behavior determined by the video encoding parameters includes:

[0050] Outside the search range of the video encoder, a search is performed based on the global motion offset position parameter to implement motion estimation calculation between frames.

[0051] like Figure 4 As shown, in the video encoding method of this embodiment, the global motion offset position parameter can be applied to the motion estimation hardware, wherein the global motion offset position parameter can provide the amount that the entire frame has moved in the 2D position, thereby simplifying the motion estimation calculation for inter-frame prediction. In this example, the global motion offset position parameter is applied to the search range outside the video encoder for targeted search, thereby achieving the effect of simplifying the operation. Existing hardware video encoders can perform searches of up to 128 pixels (vertical / horizontal) in motion evaluation. It is mainly used for inter-frame prediction, including the full frame range of the previous frame (including future frames for B frames). For motion scenes in games, especially at high resolutions, fast motion can easily exceed the 128-pixel capability of the motion estimator. The method disclosed in the present invention uses global motion offset, and the encoder of the video encoding engine can perform a 128-pixel search from the corresponding offset position according to the global motion offset position parameter, thereby greatly simplifying the computational complexity of motion estimation.

[0052] In some embodiments, searching outside a search range of the video encoder based on the global motion offset position parameter comprises:

[0053] A target pixel block is configured based on the global motion offset position parameter, and block matching is performed with the current position of the target pixel block as a starting point to perform a global offset search.

[0054] like Figure 5 As shown in FIG. 1 , in a conventional implementation of a search range for inter-frame prediction, a pixel block will be used to perform inter-frame prediction, but since the search range of a video encoder is limited, a matching block cannot be found, and thus the existing search scheme cannot achieve an ideal search effect. The video encoding method disclosed in the present invention, such as Figure 6As shown, the search range can be offset according to the global motion offset position parameter. The inter-frame search is still performed through pixel blocks, for example, pixel blocks are used to search and match the previous frame with the current frame. The pixel block used for the search can be searched from the starting position of its current position - the global offset. The global offset may not be 100% accurate, but as long as it is within the maximum search range of the video encoder, a good match can be found, thereby simplifying the calculation amount of motion estimation. In this embodiment, the global offset can be optionally configured using a specific window. Only in the game, the GUI panel can also not translate at all and remain static.

[0055] In some embodiments, performing a global offset search further includes:

[0056] A global offset region is configured based on the target image in the window, and no global offset search is performed outside the global offset region.

[0057] Figure 7 An example of a window global offset application is shown, in which the image portion in the window can be configured to use the global offset, and the global offset will be disabled outside the area, that is, the global offset can be disabled in the area outside the image portion.

[0058] In some embodiments, performing a global offset search further includes:

[0059] In the case of multiple window overlaps, an XOR logic operation is used to selectively enable global offset for different windows. That is, in practical applications, for complex scenes with multiple window overlaps, XOR logic can be used for multiple window overlaps. Thus, global offset is selectively enabled for different windows according to the global motion offset position parameters.

[0060] Game engines usually focus on a part of the game engine world space for rendering based on the player's perspective. To simplify the description, take a two-dimensional game engine as an example. Figure 8 As shown, the player's visual angle defines relative to the camera distance and angle. For some 2D games, its relative distance is often maintained, but allows to scroll in world space to obtain different information of the game world. Game engines like GODOT provide screen scrolling application programming interface (API) scrolling, while locking the object of interest. However, in order to support the resolution of different terminal screens, the game engine can normalize the value of the visual angle to between 0.0 and 1.0. Most game engines have been able to convert the standardized value to the point on the screen defined in pixels. In some embodiments, these two APIs can be combined together, and the scrolling produced can be converted into pixels, and then can be passed to the video encoder.

[0061] In some embodiments, when the forced scene change parameters include global motion offset position parameters, the forced scene change parameters are configured to: when a scene change is determined based on the logic of the game engine as an external parameter input, force the subsequent scene to be encoded using I frames to skip motion estimation and compensation.

[0062] Specifically, rate control usually controls the encoding bit rate so that the result meets the required target bit rate. It controls the frame type, quantization and entropy coding of the hybrid encoder. Fig. 9 As shown, by providing additional support for forced scene change, during the encoding process, a new scene can be forced to be encoded using I frames, thereby skipping the motion estimation and compensation steps.

[0063] Through the logic of the game engine, as an external parameter input, the encoder can be notified of scene changes so that the current frame can be encoded. No special hardware is needed to detect scene changes. Fig. 9 As shown, the motion estimation and compensation logic hardware is not used at this time and can be put into power saving mode.

[0064] The video encoding method disclosed herein uses additional parameters to allow motion estimation to be performed within the offset search range, effectively expanding the search range of the hardware video encoder. By using the parameters corresponding to the game engine's scene changes, the motion estimation logic can be completely bypassed to save power and improve encoding efficiency and speed.

[0065] In a second aspect, an embodiment of the present invention provides a device for video encoding in a game engine, including a video encoding engine, configured to:

[0066] Sequentially obtain video frames rendered and output by the image processing unit GPU, and obtain video encoding parameters from the game engine;

[0067] Based on the encoding behavior determined by the video encoding parameters, the video frames rendered and output by the GPU are encoded.

[0068] In a third aspect, an embodiment of the present invention provides a computer device, including a processor, a memory, and a communication bus;

[0069] The communication bus is used to realize the connection and communication between the processor and the memory;

[0070] The processor is used to execute one or more computer programs stored in the memory to implement the steps of the aforementioned video encoding method.

[0071] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned video encoding method are implemented.

[0072] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0073] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for a terminal (which can be a mobile phone, computer, server or network equipment, etc.) to execute the methods described in each embodiment of the present invention.

[0075] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for video encoding in a game engine, characterized in that: include: Sequentially obtain video frames rendered and output by the image processing unit GPU, and obtain video encoding parameters from the game engine; Encoding the video frame rendered and output by the GPU based on the encoding behavior determined by the video encoding parameters; In a case where the video encoding parameters include global motion offset position parameters, encoding the video frame output by the GPU rendering based on the encoding behavior determined by the video encoding parameters includes: Searching outside the search range of the video encoder based on the global motion offset position parameter to implement motion estimation calculation between frames; Searching outside the search range of the video encoder based on the global motion offset position parameter comprises: Configuring a target pixel block based on the global motion offset position parameter, and performing block matching with the current position of the target pixel block as a starting point to perform a global offset search; Perform a global offset search, also including: A global offset region is configured based on the target image in the window, and no global offset search is performed outside the global offset region.

2. The method for video encoding in a game engine as claimed in claim 1, characterized in that: The obtained video encoding parameters include at least one of the following parameters: a global motion offset position parameter, a forced scene change parameter, a time interval for the next frame, a timestamp of a GPU rendered frame, and an expected parameter for game frame stream bitrate control.

3. The method for video encoding in a game engine as claimed in claim 1, characterized in that: Perform a global offset search, also including: In the case where there are multiple overlapping windows, an XOR logic operation is used to selectively enable the global offset for different windows.

4. The method for video encoding in a game engine as claimed in claim 2, characterized in that: In case the forced scene change parameters include global motion offset position parameters, the forced scene change parameters are configured as: When a scene change is determined based on the logic of the game engine as an external parameter input, the subsequent scene is forced to be encoded using I frames to skip motion estimation and compensation.

5. A device for video encoding in a game engine, characterized in that: Includes a video encoding engine, configured to: Sequentially obtain video frames rendered and output by the image processing unit GPU, and obtain video encoding parameters from the game engine; Encoding the video frame rendered and output by the GPU based on the encoding behavior determined by the video encoding parameters; In a case where the video encoding parameters include global motion offset position parameters, encoding the video frame output by the GPU rendering based on the encoding behavior determined by the video encoding parameters includes: Searching outside the search range of the video encoder based on the global motion offset position parameter to implement motion estimation calculation between frames; Searching outside the search range of the video encoder based on the global motion offset position parameter comprises: Configuring a target pixel block based on the global motion offset position parameter, and performing block matching with the current position of the target pixel block as a starting point to perform a global offset search; Perform a global offset search, also including: A global offset region is configured based on the target image in the window, and no global offset search is performed outside the global offset region.

6. A computer device, characterized in that: including a processor, a memory and a communication bus; The communication bus is used to realize the connection and communication between the processor and the memory; The processor is configured to execute one or more computer programs stored in the memory to implement the steps of the video encoding method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video encoding method according to any one of claims 1 to 4 are implemented.

Images