Image processing method, apparatus, medium, device, and program product

Abstract

This application provides an image processing method, apparatus, medium, device, and program product, which can be applied to various scenarios such as cloud technology, cloud gaming, and game display. The method acquires a first rendered image corresponding to the game process of a target cloud game in the cloud, and second rendered images corresponding to other processes; determines a target rendered image based on the current running state of the target cloud game and at least one of the first and second rendered images; encodes the target rendered image and transmits it to the target client, allowing the target client to display the game screen corresponding to the target cloud game based on the encoded target rendered image, thereby significantly improving the latency of client-side screen rendering and enhancing the user's immersive gaming experience.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to an image processing method, apparatus, medium, device, and program product. Background Technology

[0002] With the continuous development of cloud technology, cloud gaming is becoming increasingly popular in the gaming industry. Cloud gaming performs the game's graphics rendering on the graphics processing unit (GPU) on the cloud server and transmits the rendering results to the user's client via the network.

[0003] In related technologies, the cloud server displays the process image on the screen, and then the cloud game image capture manager captures the image displayed on the monitor. The captured image is then formatted, encoded, and transmitted to the client. However, the screen display process increases the latency of the client's screen rendering, reducing the user's immersive gaming experience. Summary of the Invention

[0004] This application provides an image processing method, apparatus, medium, device, and program product that can significantly improve the latency of cloud gaming client screen rendering and enhance the user's immersive gaming experience.

[0005] In a first aspect, embodiments of this application provide an image processing method, including:

[0006] Obtain the first rendered image corresponding to the game process of the target cloud game in the cloud, and the second rendered images corresponding to other processes;

[0007] The target rendering image is determined based on the current running status of the target cloud game and at least one of the first rendering image and the second rendering image;

[0008] The target rendered image is encoded and then transmitted to the target client so that the target client can display the game screen corresponding to the target cloud game based on the encoded target rendered image.

[0009] Secondly, embodiments of this application provide an image processing apparatus, including:

[0010] The acquisition module is used to acquire the first rendered image corresponding to the game process of the target cloud game in the cloud, and the second rendered image corresponding to other processes;

[0011] The determining module is used to determine the target rendering image based on the current running state of the target cloud game and at least one of the first rendering image and the second rendering image;

[0012] The image processing module is used to encode the target rendered image and transmit it to the target client so that the target client can display the game screen corresponding to the target cloud game based on the encoded target rendered image.

[0013] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer device, cause the computer device to perform the image processing method as described in the first aspect above.

[0014] Fourthly, embodiments of this application provide a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the image processing method as described in the first aspect by calling the computer program stored in the memory.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the image processing method described in the first aspect above.

[0016] This application provides an image processing method, apparatus, medium, device, and program product. It acquires a first rendered image corresponding to a game process of a target cloud game in the cloud, and second rendered images corresponding to other processes. Then, based on the current running state of the target cloud game and at least one of the first and second rendered images, a target rendered image is determined. The target rendered image is then encoded and transmitted to a target client, allowing the target client to display the game screen corresponding to the target cloud game based on the encoded target rendered image. This application directly transmits the target rendered image to the client after determination, eliminating the need to capture the rendered image on a cloud display. This improves the speed of transmitting the target rendered image to the client, significantly reduces client-side rendering latency, and enhances the user's immersive gaming experience. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the first process of the image processing method in the related cloud gaming technology provided in the embodiments of this application.

[0018] Figure 2 This is a schematic diagram of the second process of the image processing method in the related cloud gaming technology provided in the embodiments of this application.

[0019] Figure 3 This is a schematic diagram of the structure of the image processing system provided in the embodiments of this application.

[0020] Figure 4 This is a schematic flowchart of the image processing method provided in the embodiments of this application.

[0021] Figure 5 This is a schematic diagram illustrating an application scenario of the image processing method provided in the embodiments of this application.

[0022] Figure 6 This is a schematic diagram of the image processing method provided in the embodiments of this application.

[0023] Figure 7 This is a schematic diagram of another application scenario of the image processing method provided in the embodiments of this application.

[0024] Figure 8 This is a schematic diagram of the structure of a binary tree provided in an embodiment of this application.

[0025] Figure 9 This is a schematic block diagram of an image processing apparatus provided in an embodiment of this application.

[0026] Figure 10 This is a schematic block diagram of another image processing apparatus provided in the embodiments of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art without creative effort regarding the embodiments of this application are within the scope of protection of this application.

[0028] Before describing the embodiments of this application, the relevant terms involved in this application will be explained.

[0029] Sandboxing is a component relationship design pattern used in browsers and other applications to protect security.

[0030] Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0031] Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied to the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will all require robust system support, which can only be achieved through cloud computing.

[0032] Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing. It enables thin clients with relatively limited graphics processing and data processing capabilities to run high-quality games. In cloud gaming, the game does not reside on the player's terminal but runs on a cloud server. The cloud server renders the game scene as a video and audio stream, which is then transmitted to the player's terminal via the network. The player's terminal does not need powerful graphics processing and data processing capabilities; it only needs basic streaming media playback capabilities and the ability to receive player input commands and send them to the cloud server.

[0033] A graphics processing unit (GPU), also known as a display core, visual processor, or display chip, is a processor specifically designed to perform image and graphics-related calculations on personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones).

[0034] DirectX (Direct eXtension, DX) is a multimedia programming interface and an application programming interface (API).

[0035] The Open Graphics Library (OpenGL) is a cross-language, cross-platform application programming interface for rendering 2D and 3D vector graphics.

[0036] The Graphics Device Interface (GDI) is a subsystem in Windows XP that is primarily responsible for outputting information to display screens and printing devices. It is a set of application programming interfaces implemented using C++ classes.

[0037] Hook functions are part of the message handling mechanism. By setting hooks, applications can filter all messages and events at the system level and access messages that are normally inaccessible. Essentially, a hook is a piece of code used to process system messages; it is attached to the system via a system call.

[0038] A window is the basic unit in the Windows operating system used to display information and receive user input. The operating system component responsible for managing window-related functions is called the Window Manager. When the Windows operating system initializes, it generates a window called the desktop window. The desktop window covers the entire screen, and all other windows are displayed on top of it.

[0039] This application provides an image processing method, apparatus, medium, device, and program product. Specifically, the image processing method of this application can be executed by a computer device.

[0040] Currently, in related cloud gaming technologies, virtual machine technology is used in the cloud. When a cloud game runs, the game screen is usually displayed on the virtual machine's monitor. Then, the cloud game image capture manager captures the image from the monitor, processes the image, and transmits it to the client. The following examples illustrate cloud gaming in windowed and full-screen modes.

[0041] Please see Figure 1 In the current windowed mode of cloud gaming, the system's window management compositor acquires images of all visible windows, such as process windows, desktop windows, and other process windows. Then, the compositor calculates the window overlap and renders all visible images onto a single texture using the graphics card. It then waits for the virtual machine's vertical sync signal to display the texture on the monitor. Afterward, the cloud gaming image capture manager captures a frame from the monitor, converts it to an encodeable image format using an image format converter, encodes it, and transmits it to the client. Therefore, in windowed mode, the system window management compositor's acquisition of window images introduces a rendering and compositing delay; the graphics card's display of the image on the monitor adds a vertical sync signal delay; and the cloud gaming image capture manager's capture of the monitor adds another image copying delay, resulting in a delay in the cloud gaming client's display. Furthermore, in windowed mode, cloud gaming client users may see desktop or other process windows appearing in the cloud gaming client's screen, affecting the user's immersive experience.

[0042] Please see Figure 2In the current full-screen mode of cloud gaming, the game window fills the entire screen, and the operating system enters screen-exclusive mode. In this mode, the textures rendered by the cloud game process can directly wait for the monitor's vertical sync signal to display them. Then, the cloud gaming image capture manager captures a frame of the image displayed on the monitor, converts this frame to an encodeable image format using an image format converter, encodes it, and transmits it to the cloud gaming client. It is evident that in full-screen mode, the graphics card adds a delay to displaying the textures on the monitor due to the vertical sync signal, and the cloud gaming image capture manager adds another delay to capturing the image from the monitor, resulting in a delay in the cloud gaming client's display.

[0043] Therefore, embodiments of this application provide an image processing method, apparatus, medium, device, and program product that can be applied to various scenarios such as cloud technology, cloud gaming, and game display. It can also be applied to various head-mounted display devices such as VR (Virtual Reality) and AR (Augmented Reality) and various scenarios. By directly transmitting the target rendered image to the cloud gaming client after determining the target rendered image, without needing to display the rendered image on a cloud display for capture, the speed of transmitting the target rendered image to the cloud gaming client is improved, the latency of the cloud gaming client's screen rendering is significantly reduced, and the user's immersive gaming experience is enhanced.

[0044] Please see Figure 3 , Figure 3 This is a schematic diagram of the image processing system provided in an embodiment of this application. The system includes at least one terminal 1, which can connect to different servers 2 via a network 3. When the system includes multiple terminals 1, multiple servers 2, and multiple networks 3, different terminals 1 can connect to each other via different networks 3 and different servers 2. The network 3 can be a wireless network or a wired network. Additionally, different terminals 1 can also connect to other terminals 1 or to servers 2 using their own Bluetooth network or hotspot network. For example, multiple users can connect online via different terminals 1 and synchronize with each other through appropriate networks to support multiplayer games.

[0045] Server 2 can be a cloud gaming server. Cloud gaming server 2 can be used to: obtain a first rendered image corresponding to the game process of the target cloud game, and a second rendered image corresponding to other processes; determine a target rendered image based on the current running state of the target cloud game and at least one of the first and second rendered images; encode the target rendered image and transmit it to terminal 10, so that the cloud gaming client running on terminal 10 can display the game screen corresponding to the target cloud game based on the encoded target rendered image.

[0046] Terminal 10 can be used to display the game screen transmitted from cloud gaming server 2 on the cloud gaming client. Client users can perform game operations using input devices on the cloud gaming client, such as game controllers, touchscreens, and keyboards. The terminal then transmits the game operation data to the cloud server via control flow.

[0047] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0048] Please see Figure 4 , Figure 4 A flowchart illustrating the image processing method described in an embodiment of this application is shown. The method mainly includes steps 101 to 103, as explained below:

[0049] Step 101: Obtain the first rendered image corresponding to the game process of the target cloud game in the cloud, and the second rendered image corresponding to other processes.

[0050] The cloud component includes a target cloud game. When the game program runs, the cloud receives operation data from the cloud game client regarding the target cloud game, generates the target cloud game's visual data, and renders the corresponding first rendered image based on this game operation data. Other processes can primarily include any processes running in the cloud other than the game process, such as the desktop process.

[0051] Specifically, the cloud includes an image capture manager and multiple image capturers, each corresponding to a different process. Step 101 mainly includes: obtaining the first rendered image corresponding to the game process of the target cloud game through the image capturer corresponding to the game process among the multiple image capturers; obtaining the second rendered image corresponding to other processes through the image capturers corresponding to other processes among the multiple image capturers; and sharing the obtained first rendered image and second rendered image to the image capture manager.

[0052] Specifically, the cloud includes a GPU, which can render the first and second rendered images mentioned above. Then, an image capture device corresponding to each process captures the rendered image of its respective process from the GPU. For example, ... Figure 5As shown, processes developed using DirectX, OpenGL, and GDI are described as examples, not as limitations. These three processes produce rendered images after GPU rendering, and the corresponding image capture devices for these three game processes can capture these rendered images. These images are then shared to the cloud gaming image capture manager. The cloud gaming image capture manager is primarily used to determine the target rendered image based on the current running state of the target game and to determine the data flow of the target rendered image.

[0053] In some embodiments, the method further includes: after acquiring a first rendered image corresponding to the game process of the target cloud game through an image capturer corresponding to the game process among multiple image capturers, and acquiring a second rendered image corresponding to other processes through an image capturer corresponding to other processes among multiple image capturers, the first rendered image and the second rendered image are prevented from being displayed on the cloud display.

[0054] Specifically, each image capturer primarily captures the corresponding rendered image through the dynamic link library and related key functions provided by the corresponding process. For example, DX captures the corresponding rendered image through the rendering and display function `present` provided by the key dynamic link library `DXGI.dll`, OpenGL captures the corresponding rendered image through the rendering and display function `glSwapBuffer` provided by the key dynamic link library `OpenGL.dll`, and GDI captures the corresponding rendered image through the rendering and display function `NtGdiBitBlt` provided by the key dynamic link library `Win32u.dll`. For instance, the DX capturer can capture the rendered image to be displayed within the context of the rendering and display function `present`, and then share the rendered image with the cloud gaming image capture manager. Furthermore, it prevents the execution of subsequent image display instructions by this rendering and display function, thereby preventing the rendered image from being displayed.

[0055] As mentioned above, it is understandable that each image capturer can mainly include hook functions to "hook" the corresponding rendering on-screen function and capture the rendered image from it. For example, the DX capturer includes a hook function that "hook" the rendering on-screen function present and captures the rendered image that is about to be displayed.

[0056] Specifically, in relevant cloud gaming technologies, in cloud gaming windowed mode, rendered images from all processes are typically synchronized to the system window management compositor first. However, in this embodiment, the rendered image can be directly obtained through an image capture device. This eliminates the latency caused by synchronizing rendered images from all processes to the system window management compositor. Furthermore, in relevant cloud gaming technologies, regardless of whether in cloud gaming windowed or full-screen mode, the rendered image is usually displayed on a cloud display, and then the rendered image is captured by the image capture device. However, in this embodiment, there is no need to display the image on the screen; the rendered image is directly captured from the context of the rendering and display function by the image capture device, and the rendering and display of the rendered image is prohibited. This not only eliminates the latency caused by the vertical synchronization signal during display but also reduces a significant amount of PCIe (Peripheral Component Interconnect Express) transmission, thereby improving rendering performance.

[0057] Step 102: Determine the target rendering image based on the current running status of the target cloud game and at least one of the first rendering image and the second rendering image.

[0058] The current running state primarily includes whether the target cloud game is in a game or not. Specifically, the current running state of the target cloud game can be determined by whether a game process is detected and whether a specific game screen is generated. For example, the specific game screen can include the screen generated after the game starts, or the game screen corresponding to any scene in the game. For instance, if the creation of the game process corresponding to the target cloud game is detected and a specific game screen is generated, the current running state of the target cloud game is determined to be in a game. Otherwise, the current running state of the target cloud game is determined to be not in a game.

[0059] Specifically, in relevant cloud gaming technologies, regardless of the current running state of the target cloud game, all rendered images corresponding to all processes are identified as the target rendered image and then composited, encoded, and transmitted to the target client in subsequent steps. However, different display requirements may exist in different running states of the target cloud game. For example, when the target cloud game is in progress, the client user's attention is usually focused on the game. In this case, only the rendered image corresponding to the game process is identified as the target rendered image and transmitted to the target client in subsequent steps, eliminating the need for image compositing and thus removing the latency caused by image compositing. In this embodiment, identifying the target rendered image separately for the states of being in progress and not being in progress allows for better allocation of processor resources and improves the gaming experience for the client user.

[0060] Step 103: After encoding the target rendered image, transmit it to the target client so that the target client can display the game screen corresponding to the target cloud game based on the encoded target rendered image.

[0061] The target client can be a browser client or an application client; this embodiment does not limit this. The target client supports multiple operating system platforms, such as Windows and macOS. After the cloud transmits the encoded target rendered image to the target client, the target client can decode the target rendered image and then display the game screen corresponding to the target cloud game based on the target rendered image.

[0062] In some embodiments, step 102 may mainly include: when the current running state is in a game, determining the target rendering image from the first rendering image.

[0063] It's easy to understand that when the target game is in progress, the client user's attention is usually focused on the game. At this time, the first rendered image corresponding to the game process can be directly designated as the target rendered image for transmission to the cloud gaming client for display in subsequent steps. This eliminates the need to designate the second rendered images corresponding to other processes, such as the desktop process, as the target rendered image. Thus, when the target game is in progress, the cloud gaming client displays a screen that fills the entire client, without showing the screens of other processes, enhancing the client user's immersive gaming experience. Furthermore, the elimination of image compositing eliminates the latency associated with image compositing.

[0064] In this embodiment, step 103 may mainly include: converting the first rendered image corresponding to the game process of the target cloud game into a preset target format; encoding the converted first rendered image; and transmitting the encoded first rendered image to the target client.

[0065] The preset target format can be an encoding format, such as YUV format.

[0066] Specifically, this can be achieved by converting the first rendered image corresponding to the game process of the target cloud game into an encodeable format, encoding the converted first rendered image, and then transmitting the encoded first rendered image to the target client. For example, the rendered image is usually in RGB format. The cloud can include an image format converter and an encoder. The image format converter can be used to convert the image format, and the encoder can encode the converted image. For details, please refer to [link to relevant documentation]. Figure 5The first rendered image can be transmitted to an image format converter, which converts the RGB image to YUV format. Then, the YUV image is encoded using an encoder. In subsequent steps, the encoded stream can be transmitted to the target client.

[0067] Specifically, please refer to Figure 7 The following example illustrates cloud gaming in windowed mode. After the cloud transmits the target rendered image stream of the target cloud game to the target client 1000, the target client 1000 first decodes the stream and then displays the decoded target rendered image on its display screen 1100. Since the current running state is in a game, the cloud directly identifies the first rendered image corresponding to the game process as the target rendered image, and does not identify the second rendered image corresponding to processes such as the desktop process. Therefore, the image displayed on the target client 1000's display screen 1100 will fill the entire screen, meaning the target rendered image displayed on the target client 1000's display screen 1100 is full-screen. For example, ... Figure 7 As shown, after the window of the cloud game "XX Alliance" is transmitted to the target client 1000, the window will fill the display screen 1100 of the target client 1000, which can enhance the immersive experience of the client user in playing the game.

[0068] In some embodiments, step 102 may mainly include: when the current running state is not in a game, determining both the first rendered image and the second rendered image as the target rendered image.

[0069] Specifically, when the target game is not currently in a game, all rendered images corresponding to processes running in the cloud can be identified as the target rendered image. After subsequent compositing and encoding, these images are sent to the target client for display. It's easy to understand that when the target game is not currently in a game, the client user is typically performing graphical interface operations or game matchmaking actions. In this case, the slight delay in image compositing will not affect the client user's gaming experience.

[0070] In this embodiment, step 103 may mainly include: combining the first rendered image and the second rendered image to obtain the target image; converting the target image into a new format; encoding the converted target image; and transmitting the encoded target image to the target client.

[0071] The preset target format can be an encoding format, such as YUV format.

[0072] Specifically, the first and second rendered images can be combined into a single target image, which is then converted to an encodeable format. The converted target image is then encoded and transmitted to the target client. For example, the target image is typically in RGB format. The cloud can include an image compositer, an image format converter, and an encoder. The image compositer is primarily used to combine multiple images into a single image. For more details, please refer to [link to relevant documentation]. Figure 6 The first and second rendered images can be transmitted to an image compositor, which combines them into a single target image. Next, the target image is transmitted to an image format converter, which converts it from RGB to YUV format. Then, the YUV image is encoded using an encoder. In subsequent steps, the encoded bitstream can be transmitted to the target client.

[0073] Specifically, when the target game is not currently in a game state, the first and second rendered images can be combined into a single image. Both the first and second rendered images can include corresponding Z-order information and coordinate information. Based on the Z-order information and coordinate information, the corresponding rendered images can be combined into a single image from the bottom layer to the top layer according to the Z-order information and coordinate information. Specifically, the Z-order of a window indicates its position in the stack of overlapping windows. This stack of windows extends vertically outwards from the screen along a virtual axis, the "Z-axis." Windows at the top of the Z-order overlap other windows in the Z-order, and windows at the bottom of the Z-order are covered by other windows in the Z-order.

[0074] Specifically, the step "combining the first rendered image and the second rendered image to obtain the target image" may include: combining the first rendered image and the second rendered image according to the coverage state of the windows corresponding to the processes corresponding to the first rendered image and the second rendered image to obtain the target image.

[0075] The overlap between the windows corresponding to the first and second rendered images can be determined based on the Z-order information contained in the first rendered image. Specifically, the rendered image corresponding to the top window in the Z-order can cover all other non-top windows. Generally, the newly created window is usually placed at the top of the Z-order, which is the currently used window.

[0076] In this embodiment, the method may further include: creating an initial binary tree; in response to a window change signal, determining a target binary tree based on the window change signal and the initial binary tree, wherein the target binary tree is used to characterize the window coverage state of each process, including the game process and other processes; and determining the window coverage state of each process based on the target binary tree.

[0077] A binary tree can be used to represent the window coverage state. Specifically, each node in the binary tree corresponds to a window of a process. The window corresponding to the left node of a target node in the binary tree can be understood as a child window of that target node's corresponding window, and the window corresponding to the right node of a target node can be understood as a sibling window of that target node's corresponding window. Child windows can only be displayed within the client area of ​​their corresponding parent window; any portion exceeding the parent window's client area will be clipped. Furthermore, when the parent window is hidden, all its child windows are also hidden. When the parent window is minimized, the child windows are minimized along with the parent window. When the parent window is destroyed, all its corresponding child windows are destroyed. All direct child windows of the same parent window are siblings, meaning they are equal and there is no master-slave relationship.

[0078] For example, please refer to Figure 8 In this binary tree, the left node of top-level window 1 is child window 1 of top-level window 1, and the right node of child window 1 of top-level window 1 is child window 2 of top-level window 1. That is, child window 1 and child window 2 of top-level window 1 are child windows of top-level window 1, and child window 2 of top-level window 1 is a sibling window of child window 1 of top-level window 1. Specifically, when the window state changes, the binary tree can be updated in real time to update the window coverage state.

[0079] Specifically, the cloud may include a cloud gaming image compositor, used to composite rendered images from various processes based on the window's coverage state. Each rendered image can include corresponding Z-order information, and the cloud gaming image compositor can create an initial binary tree based on this Z-order information. For example, the initial binary tree can at least include the node corresponding to the window of the desktop process. Subsequently, when a window change signal is detected, such as a window creation signal, a window destruction signal, or a change in window stacking order, the binary tree can be updated to represent the coverage state of the window corresponding to each process.

[0080] Specifically, the window change signal includes a window creation signal, which is used to indicate the creation of a first target window. The window creation signal includes the parent node information of the first target window. The step "determine the target binary tree based on the window change signal and the initial binary tree" may specifically include: determining the first parent node of the first target window in the initial binary tree based on the parent node information of the first target window; updating the left child node of the first parent node to the first target window; determining the left child node of the first parent node before the update to the right child node of the first target window, thus obtaining the target binary tree.

[0081] For example, when a window creation signal is received, indicating the creation of window A, the parent node corresponding to window A in the initial binary tree can be found based on the parent node information of window A. Then, the left child node of the parent node corresponding to window A is updated to window A, and the parent node of window A before the update is determined as the right child node of the node corresponding to window A.

[0082] Specifically, the window change signal includes a window destruction signal, which is used to indicate the destruction of the second target window. The second target window includes the parent node information of the second target window. The step "determine the target binary tree based on the window change signal and the initial binary tree" may specifically include: determining the second parent node of the second target window in the initial binary tree based on the parent node information of the second target window; traversing the right child nodes of all left child nodes of the second parent node to determine the previous and next sibling nodes of the corresponding node of the second target window; updating the right child node of the previous sibling node of the corresponding node of the second target window to the next sibling node of the corresponding node of the second target window, thereby obtaining the target binary tree.

[0083] For example, when a window destruction signal is received, this signal indicates that window B should be destroyed. At this time, the parent node of the corresponding node of window B in the initial binary tree can be found based on the parent node information of window B. Then, the right child of all the left child nodes of the parent node of the corresponding node of window B is traversed to determine the previous and next sibling nodes of the corresponding node of window B. The right child of the previous sibling node of the corresponding node of window B is updated to the next sibling node of the corresponding node of window B, thus obtaining the target binary tree.

[0084] Specifically, the window change signal includes a window stacking order change signal, which is used to change the stacking order of the third target window. The window stacking order change signal includes the current parent node information and target sibling node information of the third target window. The step "determine the target binary tree based on the window change signal and the initial binary tree" may specifically include: determining the current third parent node of the third target window in the initial binary tree based on the current parent node information of the third target window; traversing the right child nodes of all left child nodes of the current third parent node to determine the previous and next sibling nodes of the corresponding node of the third target window; updating the right child node of the previous sibling node of the corresponding node of the third target window to the next sibling node of the corresponding node of the third target window; determining the target previous and next sibling nodes of the third target window based on the target sibling node information of the third target window; updating the right child node of the target previous sibling node of the third target window to the third target window; and determining the right child node of the third target window as the target next sibling node.

[0085] For example, when a window stacking order change signal is received, this signal is used to change the Z-order of window C. First, the node corresponding to window C can be deleted from the initial binary tree. Then, according to the desired Z-order position, the node corresponding to window C is inserted back into the initial binary tree. Specifically, the current parent node of the node corresponding to window C can be determined first based on the current parent node information of window C. Then, the right child nodes of all left child nodes of the current parent node are traversed to determine the previous and next sibling nodes of the node corresponding to window C. Next, the right child node of the previous sibling node of the node corresponding to window C is updated to the next sibling node of the node corresponding to window C. Then, according to the target sibling node information of window C, the target previous and next sibling nodes of the node corresponding to window C are determined. Finally, the right child node of the target previous sibling node of the node corresponding to window C is updated to window C, and the right child node of the node corresponding to window C is determined to be the target next sibling node.

[0086] For a better illustration of the image processing method provided in the embodiments of this application, please refer to... Figure 6 The image processing method provided in this application can be summarized as follows: First, the cloud game process and other processes can render corresponding images using the graphics card. Then, they share their respective rendered images with the cloud game image capture manager. Each process can have a different image capturer to capture its own rendered image. For example, the image capturer may have a hook function to "hook" the rendering and display function corresponding to its process, capture the rendered image from the rendering and display function, and prevent subsequent rendering and display steps from being executed. This eliminates the delay caused by synchronizing the rendered images of all processes to the system window management compositor and reduces a significant amount of PCIe transmission, thereby improving rendering performance. Then, the cloud game image capture manager can determine whether the target cloud game is currently running in the game, and determine the target rendered image and data flow based on the current running state. For example, ... Figure 6As shown, when the target cloud game is not currently in a game, the target rendered image stream is directed to the image compositer for image compositing. The composited image is then sent to the image format converter for format conversion, transforming it into an encodeable format. Next, the image is sent to the encoder for encoding, and finally, the bitstream is transmitted to the target client. Alternatively, when the target cloud game is not currently in a game, the target rendered image stream is directly directed to the image format converter for format conversion, transforming the composited image into an encodeable format. The image is then sent to the encoder for encoding, and finally, the bitstream is transmitted to the target client. When the target cloud game is in a game, only the rendered image corresponding to the game process can be identified as the target rendered image, thus eliminating the need for image compositing and removing the latency associated with it.

[0087] As described above, in this embodiment, regardless of whether the user is in a game or what game mode they are in, it is not necessary to send the rendered image to the system window management compositor, thus eliminating the delay in this step. Furthermore, in this embodiment, it is not necessary to capture the rendered image after it is displayed on screen, thus eliminating the delay caused by vertical signal synchronization during display. Moreover, when in a game, there is no need for image compositing operations, eliminating the delay caused by image compositing operations. Therefore, this significantly improves the rendering latency in cloud gaming and enhances the immersive experience for client users.

[0088] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0089] The image processing method provided in this application involves acquiring a first rendered image corresponding to the game process of a target cloud game in the cloud, and second rendered images corresponding to other processes. Then, based on the current running state of the target cloud game and at least one of the first and second rendered images, a target rendered image is determined. The target rendered image is then encoded and transmitted to the target client, allowing the target client to display the game screen corresponding to the target cloud game based on the encoded target rendered image. This application embodiment directly transmits the target rendered image to the client after determination, eliminating the need to capture the rendered image on a cloud display. This improves the speed of transmitting the target rendered image to the client, significantly reduces client-side screen rendering latency, and enhances the user's immersive gaming experience.

[0090] The method embodiments of this application have been described in detail above. The following description, in conjunction with... Figure 9 The present application describes the device embodiments in detail. It should be understood that the device embodiments correspond to the method embodiments, and similar descriptions can be referred to the method embodiments.

[0091] Figure 9 This is a schematic structural diagram of an image processing apparatus 10 according to an embodiment of this application, such as... Figure 9 As shown, the image processing apparatus 10 may include:

[0092] The acquisition module 11 is used to acquire the first rendered image corresponding to the game process of the target cloud game in the cloud, and the second rendered image corresponding to other processes;

[0093] The determining module 12 is used to determine the target rendering image based on the current running status of the target cloud game and at least one of the first rendering image and the second rendering image;

[0094] The image processing module 13 is used to encode the target rendered image and transmit it to the target client so that the target client can display the game screen corresponding to the target cloud game based on the encoded target rendered image.

[0095] Optionally, module 12 can be used to determine the target rendering image from the first rendering image when the current running state is in the game.

[0096] Optionally, the image processing module 13 can be used to: convert the format of the first rendered image corresponding to the game process of the target cloud game; encode the converted first rendered image; and transmit the encoded first rendered image to the target client.

[0097] Optionally, module 12 can be used to determine both the first and second rendered images as the target rendered image when the current running state is not in the game.

[0098] Optionally, the image processing module 13 can be used to: combine the first rendered image and the second rendered image to obtain a target image; convert the format of the target image; encode the converted target image; and transmit the encoded target image to the target client.

[0099] Optionally, the image processing module 13 can be used to: synthesize the first rendered image and the second rendered image according to the coverage state of the windows corresponding to the processes of the first rendered image and the second rendered image to obtain the target image.

[0100] Optionally, the image processing device 10 may also include a processing module for: creating an initial binary tree; determining a target binary tree in response to a window change signal, based on the window change signal and the initial binary tree, the target binary tree being used to characterize the current window coverage state of each process, including the game process and other processes; and determining the window coverage state of each process based on the target binary tree.

[0101] Optionally, the window change signal includes a window creation signal, which is used to indicate the creation of a first target window. The window creation signal includes the parent node information of the first target window. The processing module can specifically be used to: determine the first parent node of the first target window in the initial binary tree based on the parent node information of the first target window; update the left child node of the first parent node to the first target window; determine the left child node of the first parent node before the update as the right child node of the first target window, and obtain the target binary tree.

[0102] Optionally, the window change signal includes a window destruction signal, which is used to indicate the destruction of the second target window. The second target window includes the parent node information of the second target window. The processing module can specifically be used to: determine the second parent node of the second target window in the initial binary tree based on the parent node information of the second target window; traverse the right child nodes of all left child nodes of the second parent node to determine the previous and next sibling nodes of the corresponding node of the second target window; update the right child node of the previous sibling node of the corresponding node of the second target window to the next sibling node of the corresponding node of the second target window, and obtain the target binary tree.

[0103] Optionally, the window change signal includes a window stacking order change signal, which is used to change the stacking order of the third target window. The window stacking order change signal includes the current parent node information and target sibling node information of the third target window. The processing module can be specifically used to: determine the current third parent node of the third target window in the initial binary tree based on the current parent node information of the third target window; traverse the right child nodes of all left child nodes of the current third parent node to determine the previous and next sibling nodes of the corresponding node of the third target window; update the right child node of the previous sibling node of the corresponding node of the third target window to the next sibling node of the corresponding node of the third target window; determine the target previous and target next sibling nodes of the third target window based on the target sibling node information of the third target window; update the right child node of the target previous sibling node of the third target window to the third target window; and determine the right child node of the third target window as the target next sibling node.

[0104] Optionally, the cloud includes an image capture manager and multiple image capturers, each corresponding to a different process. The acquisition module 11 can be used to: acquire the first rendered image corresponding to the game process of the target cloud game through the image capturer corresponding to the game process among the multiple image capturers; acquire the second rendered image corresponding to other processes through the image capturers corresponding to other processes among the multiple image capturers; and share the acquired first and second rendered images to the image capture manager.

[0105] It should be noted that the functions of each module in the image processing device 10 in this application embodiment can be referred to the specific implementation methods in the above method embodiments, and will not be repeated here.

[0106] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0107] The image processing apparatus 10 provided in this application embodiment acquires a first rendered image corresponding to the game process of the target cloud game in the cloud and a second rendered image corresponding to other processes through an acquisition module 11. Then, a determination module 12 determines the target rendered image based on the current running state of the target cloud game and at least one of the first and second rendered images. Next, the image processing module 13 encodes the target rendered image and transmits it to the target client so that the target client can display the game screen corresponding to the target cloud game based on the encoded target rendered image, thereby significantly improving the latency of client screen rendering and enhancing the user's immersive gaming experience.

[0108] Figure 10 This is another schematic structural diagram of the image processing apparatus provided in the embodiments of this application, such as... Figure 10 As shown, the image processing device 20 may include: a communication interface 21, a memory 22, a processor 23, and a communication bus 24. The communication interface 21, memory 22, and processor 23 communicate with each other via the communication bus 24. The communication interface 21 is used for data communication between the image processing device 20 and external devices. The memory 22 can be used to store software programs and modules, and the processor 23 executes the software programs and modules stored in the memory 22, such as the software programs for the corresponding operations in the aforementioned method embodiments.

[0109] In some embodiments, the processor 23 may invoke software programs and modules stored in the memory 22 to perform the following operations: obtain a first rendered image corresponding to the game process of the target cloud game in the cloud, and a second rendered image corresponding to other processes; determine a target rendered image based on the current running state of the target cloud game and at least one of the first and second rendered images; encode the target rendered image and transmit it to the target client so that the target client can display the game screen corresponding to the target cloud game based on the encoded target rendered image.

[0110] In some embodiments, this application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0111] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the image processing method of this application embodiment; for brevity, further details are omitted here.

[0112] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the corresponding flow in the image processing method of this application embodiment. For simplicity, further details are omitted here.

[0113] This application also provides a computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding process in the image processing method of this application embodiment. For simplicity, further details are omitted here.

[0114] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0115] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0116] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0118] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0119] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0121] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0122] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the first rendered image corresponding to the game process of the target cloud game in the cloud, and the second rendered images corresponding to other processes; Determining a target rendering image based on the current running state of the target cloud game and at least one of the first rendering image and the second rendering image includes: when the current running state is not in a game, determining both the first rendering image and the second rendering image as the target rendering image; The target rendered image is encoded and then transmitted to the target client so that the target client can display the game screen corresponding to the target cloud game based on the encoded target rendered image. The step of encoding the target rendered image and transmitting it to the target client includes: The method of combining the first rendered image and the second rendered image to obtain a target image includes: combining the first rendered image and the second rendered image according to the coverage state of the window corresponding to the process of the first rendered image and the second rendered image to obtain a target image; The target image is converted to a different format. The converted target image is then encoded. The encoded target image is transmitted to the target client; The method further includes: Create the initial binary tree; In response to a window change signal, a target binary tree is determined based on the window change signal and the initial binary tree. The target binary tree is used to characterize the window coverage state of each process, including the game process and the other processes. The window coverage state of each process is determined based on the target binary tree.

2. The image processing method according to claim 1, characterized in that, Determining the target rendering image based on the current running state of the target cloud game and at least one of the first rendering image and the second rendering image includes: When the current running state is in a game, the first rendered image is used to determine the target rendered image.

3. The image processing method according to claim 2, characterized in that, The step of encoding the target rendered image and transmitting it to the target client includes: Convert the format of the first rendered image corresponding to the game process of the target cloud game; The converted first rendered image is then encoded. The encoded first rendered image is transmitted to the target client.

4. The image processing method according to claim 1, characterized in that, The window change signal includes a window creation signal, which indicates the creation of a first target window. The window creation signal includes parent node information of the first target window. Determining the target binary tree based on the window change signal and the initial binary tree includes: Based on the parent node information of the first target window, determine the first parent node of the first target window in the initial binary tree; Update the left child node of the first parent node to the first target window; The left child node of the first parent node before the update is determined as the right child node of the first target window, thus obtaining the target binary tree.

5. The image processing method according to claim 1, characterized in that, The window change signal includes a window destruction signal, which indicates the destruction of a second target window. The second target window includes its parent node information. Determining the target binary tree based on the window change signal and the initial binary tree includes: Based on the parent node information of the second target window, determine the second parent node of the second target window in the initial binary tree; Iterate through the right child nodes of all left child nodes of the second parent node to determine the previous and next sibling nodes of the node corresponding to the second target window. The right child node of the preceding sibling node of the node corresponding to the second target window is updated to the following sibling node of the node corresponding to the second target window to obtain the target binary tree.

6. The image processing method according to claim 1, characterized in that, The window change signal includes a window stacking order change signal, which is used to change the stacking order of the third target window. The window stacking order change signal includes the current parent node information and target sibling node information of the third target window. The step of determining the target binary tree based on the window change signal and the initial binary tree includes: Based on the current parent node information of the third target window, determine the current third parent node of the third target window in the initial binary tree; Traverse the right child nodes of all left child nodes of the current third parent node to determine the previous and next sibling nodes of the node corresponding to the third target window. Update the right child node of the preceding sibling node of the node corresponding to the third target window to the following sibling node of the node corresponding to the third target window. Based on the target sibling node information of the third target window, determine the target's previous sibling node and target's next sibling node in the third target window; Update the right child node of the first sibling node of the target in the third target window to the third target window, and determine the right child node of the third target window to be the next sibling node of the target.

7. The image processing method according to any one of claims 1-6, characterized in that, The cloud includes an image capture manager and multiple image capturers, each corresponding to a different process. The process of acquiring the first rendered image corresponding to the game process of the target cloud game in the cloud, and the second rendered images corresponding to other processes, includes: The first rendered image corresponding to the game process of the target cloud game is obtained by using the image capture device corresponding to the game process among the multiple image capture devices; The second rendered image corresponding to the other processes is obtained through the image capture device corresponding to the other processes among the plurality of image capture devices; The first and second rendered images are shared to the image capture manager.

8. The image processing method according to claim 7, characterized in that, The method further includes: After obtaining the first rendered image corresponding to the game process of the target cloud game through the image capturer corresponding to the game process among the plurality of image capturers, and obtaining the second rendered image corresponding to the other process through the image capturer corresponding to the other process among the plurality of image capturers, the first rendered image and the second rendered image are prohibited from being displayed on the cloud display.

9. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the first rendered image corresponding to the game process of the target cloud game in the cloud, and the second rendered image corresponding to other processes; The determining module is configured to determine a target rendering image based on the current running state of the target cloud game and at least one of the first rendering image and the second rendering image, including: when the current running state is not in a game, determining both the first rendering image and the second rendering image as the target rendering image; An image processing module is used to encode the target rendered image and transmit it to the target client so that the target client can display the game screen corresponding to the target cloud game based on the encoded target rendered image; The image processing module is configured to: combine the first rendered image and the second rendered image to obtain a target image, including: combining the first rendered image and the second rendered image according to the coverage state of the windows corresponding to the processes corresponding to the first rendered image and the second rendered image to obtain a target image; converting the format of the target image; encoding the converted target image; and transmitting the encoded target image to a target client. The processing module is configured to: create an initial binary tree; in response to a window change signal, determine a target binary tree based on the window change signal and the initial binary tree, wherein the target binary tree is used to characterize the window coverage state of each process, including the game process and the other processes; and determine the window coverage state of each process based on the target binary tree.

10. The image processing apparatus according to claim 9, characterized in that, The window change signal includes a window creation signal, which indicates the creation of a first target window. The window creation signal includes parent node information of the first target window. The processing module is used to: Based on the parent node information of the first target window, determine the first parent node of the first target window in the initial binary tree; Update the left child node of the first parent node to the first target window; The left child node of the first parent node before the update is determined as the right child node of the first target window, thus obtaining the target binary tree.

11. The image processing apparatus according to claim 9, characterized in that, The window change signal includes a window destruction signal, which indicates the destruction of a second target window. The second target window includes its parent node information. The processing module is used to: Based on the parent node information of the second target window, determine the second parent node of the second target window in the initial binary tree; Iterate through the right child nodes of all left child nodes of the second parent node to determine the previous and next sibling nodes of the node corresponding to the second target window. The right child node of the preceding sibling node of the node corresponding to the second target window is updated to the following sibling node of the node corresponding to the second target window to obtain the target binary tree.

12. The image processing apparatus according to claim 9, characterized in that, The window change signal includes a window stacking order change signal, which is used to change the stacking order of the third target window. The window stacking order change signal includes the current parent node information and target sibling node information of the third target window. The processing module is used for: Based on the current parent node information of the third target window, determine the current third parent node of the third target window in the initial binary tree; Traverse the right child nodes of all left child nodes of the current third parent node to determine the previous and next sibling nodes of the node corresponding to the third target window. Update the right child node of the preceding sibling node of the node corresponding to the third target window to the following sibling node of the node corresponding to the third target window. Based on the target sibling node information of the third target window, determine the target's previous sibling node and target's next sibling node in the third target window; Update the right child node of the first sibling node of the target in the third target window to the third target window, and determine the right child node of the third target window to be the next sibling node of the target.

13. The image processing apparatus according to any one of claims 9-12, characterized in that, The cloud includes an image capture manager and multiple image capturers, each corresponding to a different process; the acquisition module is used for: The first rendered image corresponding to the game process of the target cloud game is obtained by using the image capture device corresponding to the game process among the multiple image capture devices; The second rendered image corresponding to the other processes is obtained through the image capture device corresponding to the other processes among the plurality of image capture devices; The first and second rendered images are shared to the image capture manager.

14. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on a computer device, the computer device performs the image processing method as described in any one of claims 1 to 8.

15. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the image processing method according to any one of claims 1 to 8 by calling the computer program stored in the memory.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image processing method according to claims 1 to 8.

Images