Immersive cloud game image end-cloud collaborative rendering method and related equipment

Abstract

The invention provides an immersive cloud game image end-cloud collaborative rendering method and related equipment. The method comprises the steps of determining a to-be-rendered image and image rendering data; performing image analysis on the to-be-rendered image, performing foreground and background division according to an analysis result, and determining a to-be-rendered foreground image and a to-be-rendered background image; according to the image rendering data, rendering the to-be-rendered foreground image and the to-be-rendered background image through a device end and a cloud server end, and determining the foreground image and the background image; and merging the foreground image and the background image through the device end, completing rendering of the cloud game image, and displaying the rendered image.

Description

Technical Field

[0001] The present application relates to the technical field, and in particular to an immersive cloud game image end-cloud collaborative rendering method and related equipment. Background Art

[0002] With the rapid development of science and technology, cloud gaming has gradually become an important part of modern digital entertainment. Cloud gaming allows players to run high-quality games on low-configuration devices. In this process, the operation and rendering of games rely on the powerful data processing capabilities of cloud computing. The operation and rendering are mainly completed on cloud servers, and then the rendering results are transmitted to the player's device in real time. The player's device receives the rendered video stream through the network. This model effectively reduces the player's demand for high-performance hardware and expands the audience base of high-end games.

[0003] The rendering quality of cloud games is limited by the player's network bandwidth and stability. An unstable or low-bandwidth network environment may cause the image quality to deteriorate and affect the overall gaming experience. In real-time interactive gaming scenarios, even a slight delay may seriously affect the player's gaming experience. Summary of the invention

[0004] In view of this, the purpose of this application is to propose an immersive cloud game image end-cloud collaborative rendering method and related equipment.

[0005] Based on the above objectives, the present application provides an immersive cloud game image end-cloud collaborative rendering method, including:

[0006] Determine the image to be rendered and the image rendering data;

[0007] Performing image analysis on the image to be rendered, and dividing the foreground and background according to the analysis result to determine the foreground image to be rendered and the background image to be rendered;

[0008] According to the image rendering data, the foreground image to be rendered and the background image to be rendered are rendered by a device end and a cloud server end to determine a foreground image and a background image;

[0009] The foreground image and the background image are merged by the device end to complete image rendering, and the rendered image is displayed.

[0010] Optionally, performing image analysis on the image to be rendered, dividing the foreground and the background according to the analysis result, and determining the foreground image to be rendered and the background image to be rendered includes:

[0011] Performing image analysis on the image to be rendered to determine layer information or image information of the image to be rendered;

[0012] The image to be rendered is divided into a foreground and a background according to the layer information or the image information, and the foreground image to be rendered and the background image to be rendered are determined.

[0013] Optionally, rendering the foreground image to be rendered and the background image to be rendered by a device end and a cloud server end according to the image rendering data to determine the foreground image and the background image includes:

[0014] Determine foreground image rendering data and background image rendering data according to the image rendering data, the foreground image to be rendered, and the background image to be rendered;

[0015] According to the foreground image rendering data and the background image rendering data, the foreground image to be rendered and the background image to be rendered are rendered by the device end and the cloud server end to determine the foreground image and the background image.

[0016] Optionally, rendering the foreground image to be rendered and the background image to be rendered by a device end and a cloud server end according to the foreground image rendering data and the background image rendering data to determine the foreground image and the background image includes:

[0017] Determine the computing power of the device end, the computing power of the cloud server end, the network bandwidth, the first image rendering computing power and the second image rendering computing power; wherein the first image rendering computing power is the computing power required for image rendering according to the foreground image rendering data, and the second image rendering computing power is the computing power required for image rendering according to the background image rendering data;

[0018] Determining an image rendering delay according to the computing power of the device end, the computing power of the cloud server end, the network bandwidth, the first image rendering computing power, and the second image rendering computing power;

[0019] Determining an image rendering mode according to the image rendering delay and a preset image rendering delay;

[0020] According to the image rendering mode, the foreground image to be rendered and the background image to be rendered are rendered by the device side and the cloud server side to determine the foreground image and the background image.

[0021] Optionally, determining the image rendering delay according to the computing power of the device side, the computing power of the cloud server side, the network bandwidth, the first image rendering computing power, and the second image rendering computing power includes:

[0022] Determining a device-side rendering computing amount and a cloud server-side rendering computing amount according to the device-side computing power, the cloud server-side computing power, the first image rendering computing power, and the second image rendering computing power;

[0023] Determining the rendering delay according to the computing power of the device side, the computing power of the cloud server side, the rendering calculation amount of the device side, and the rendering calculation amount of the cloud server side;

[0024] Determining a transmission delay according to the image rendering data and the network bandwidth;

[0025] The image rendering delay is determined according to the rendering delay and the transmission delay.

[0026] Optionally, the background image to be rendered includes: a first rendering task, a second rendering task, a third rendering task, a fourth rendering task and a fifth rendering task, wherein the first rendering task is a background rendering task at the current position, the second rendering task is a background rendering task at the next coordinate in the positive direction of the x-axis, the third rendering task is a background rendering task at the next coordinate in the negative direction of the x-axis, the fourth rendering task is a background rendering task at the next coordinate in the positive direction of the y-axis, and the fifth rendering task is a background rendering task at the next coordinate in the negative direction of the y-axis;

[0027] The method further comprises:

[0028] In response to determining that the image rendering delay is less than the preset image rendering delay, the device end and the cloud server end render the foreground image to be rendered and the background image to be rendered respectively according to the foreground image rendering data and the background image rendering data to determine the foreground image and the background image;

[0029] In response to determining that the image rendering delay is not less than the preset image rendering delay, the first image rendering computing power, the second image rendering computing power, the network transmission requirements and a pre-built rendering gear table are determined; the foreground image to be rendered and the background image to be rendered are rendered according to the first image rendering computing power, the second image rendering computing power, the network transmission requirements and the rendering gear table, and the foreground image and the background image are determined.

[0030] Based on the same inventive concept, the embodiment of the present application also provides an immersive cloud game image end-cloud collaborative rendering device, including:

[0031] A data determination module is configured to determine an image to be rendered and image rendering data;

[0032] An image division module is configured to perform image analysis on the image to be rendered, and divide the image into foreground and background according to the analysis result to determine a foreground image to be rendered and a background image to be rendered;

[0033] An image rendering module is configured to render the foreground image to be rendered and the background image to be rendered through a device end and a cloud server end according to the image rendering data, and determine a foreground image and a background image;

[0034] The rendering image display module is configured to merge the foreground image and the background image through the device end, complete the rendering of the cloud game image, and display the rendered image.

[0035] Based on the same inventive concept, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an immersive cloud gaming image end-cloud collaborative rendering method as described in any one of the above items is implemented.

[0036] Based on the same inventive concept, an embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned immersive cloud game image end-cloud collaborative rendering methods.

[0037] Based on the same inventive concept, an embodiment of the present application also provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer executes any of the above-mentioned immersive cloud game image end-cloud collaborative rendering methods.

[0038] From the above, it can be seen that the immersive cloud game image end-cloud collaborative rendering method and related equipment provided by the present application include: determining the image to be rendered and image rendering data; performing image analysis on the image to be rendered, and dividing the foreground and background according to the analysis results to determine the foreground image to be rendered and the background image to be rendered; according to the image rendering data, the foreground image to be rendered and the background image to be rendered are rendered by the device end and the cloud server end to determine the foreground image and the background image; the foreground image and the background image are merged by the device end to complete the rendering of the cloud game image, and the rendered image is displayed. The present application renders in parts, that is, the foreground image to be rendered and the background image to be rendered are rendered by the device end and the cloud server end respectively, the foreground image and the background image are determined, and the background image that has been rendered by the cloud server end is received by the device end, and the foreground image and the background image are merged. Thereby reducing the time from requesting to obtaining the picture, effectively improving the performance and user experience of cloud games, so that cloud games can meet the needs of high-speed response while ensuring high-quality rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A schematic diagram of the process of the end-cloud collaborative rendering method of immersive cloud game images according to an embodiment of the present application;

[0041] Figure 2 A schematic diagram of an embodiment of a method for collaborative rendering of images of an immersive cloud game on a terminal and a cloud according to an embodiment of the present application;

[0042] Figure 3 A schematic diagram of an embodiment of a method for collaborative rendering of images of an immersive cloud game on a terminal and a cloud according to an embodiment of the present application;

[0043] Figure 4 A schematic diagram of an immersive cloud game image end-cloud collaborative rendering device according to an embodiment of the present application;

[0044] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0046] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0047] In order to facilitate understanding of the technical solutions of the present disclosure, some technical terms involved in the present disclosure are introduced below.

[0048] Cloud gaming is an online gaming method based on cloud computing technology. It transfers the game computing process to the cloud server and transmits the processed game screen to the user's client device through the Internet for operation. This model allows players to play large-scale games without having high-end hardware devices.

[0049] In order to make the technical solution of the present disclosure clearer and easier to understand, the immersive cloud game image end-cloud collaborative rendering method provided by the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0050] As mentioned in the background technology section, with the rapid development of science and technology, cloud gaming has gradually become an important part of modern digital entertainment. Cloud gaming allows players to run high-quality games on low-configuration devices. In this process, the operation and rendering of the game rely on the powerful data processing capabilities of cloud computing. The operation and rendering are mainly completed on the cloud server, and then the rendering results are transmitted to the player's device in real time. The player's device receives the rendered video stream through the network. This model effectively reduces the player's demand for high-performance hardware and expands the audience base of high-end games.

[0051] The rendering quality of cloud games is limited by the player's network bandwidth and stability. An unstable or low-bandwidth network environment may cause the image quality to deteriorate and affect the overall gaming experience. In real-time interactive gaming scenarios, even a slight delay may seriously affect the player's gaming experience.

[0052] In addition, the applicant has discovered through research that the computing power of mobile devices, especially smartphones, has been significantly improved in recent years. With the improvement of the performance of end-side devices, if we continue to rely solely on the cloud for all game rendering, we will not be able to fully utilize the computing resources of local devices. The non-use of such resources not only means potential waste of computing power, but may also cause unnecessary network transmission burden, increasing game response time and possible delays.

[0053] Furthermore, in normal rendering logic, there is no distinction between foreground and background objects. After all objects and textures in the engine are rendered, the corresponding images are transmitted and displayed on the user side according to the position of the player's main perspective camera. However, the advantage of separate foreground and background rendering is that the rendering cost of the background is high, but the dynamics are low. It can be rendered in the cloud and transmitted back in the form of video streams, which saves terminal computing power and ensures the integrity of the scene rendering. The foreground objects are highly dynamic and have lower requirements for rendering capabilities, so they are suitable for rendering on the terminal side to ensure low latency. It should be noted that the concepts of "background" and "background" are the same.

[0054] In view of this, an embodiment of the present application provides an immersive cloud game image end-cloud collaborative rendering method and related equipment, including: determining an image to be rendered and image rendering data; performing image analysis on the image to be rendered, and dividing the foreground and background according to the analysis results to determine the foreground image to be rendered and the background image to be rendered; according to the image rendering data, rendering the foreground image to be rendered and the background image to be rendered by the device end and the cloud server end to determine the foreground image and the background image; merging the foreground image and the background image by the device end to complete the rendering of the cloud game image, and displaying the rendered image. The present application uses partial rendering, that is, rendering the foreground image to be rendered and the background image to be rendered by the device end and the cloud server end respectively, determining the foreground image and the background image, and receiving the background image that has been rendered by the cloud server end by the device end, and merging the foreground image and the background image. Thereby reducing the time from requesting to obtaining the picture, effectively improving the performance and user experience of cloud games, so that cloud games can meet the needs of high-speed response while ensuring high-quality rendering.

[0055] like Figure 1 As shown, the immersive cloud game image end-cloud collaborative rendering method includes:

[0056] Step S102, determining an image to be rendered and image rendering data;

[0057] In specific implementation, the "image to be rendered" is an image whose framework has been built but without character, game map and color rendering. It needs to be rendered through image rendering data to present a normal game screen, and the "image rendering data" is the data used to render the image to be rendered.

[0058] In some implementations, the image to be rendered should be determined first, and image rendering data corresponding to the image to be rendered in the database is obtained according to the image to be rendered.

[0059] Step S104, performing image analysis on the image to be rendered, and dividing the foreground and background according to the analysis result to determine the foreground image to be rendered and the background image to be rendered;

[0060] In a specific implementation, image analysis is performed on the image to be rendered, and foreground and background are divided according to the analysis result to determine the foreground image to be rendered and the background image to be rendered, including: performing image analysis on the image to be rendered to determine the layer information of the image to be rendered; dividing the foreground and background of the image to be rendered according to the layer information to determine the foreground image to be rendered and the background image to be rendered. And performing image analysis on the image to be rendered to determine the image information of the image to be rendered, dividing the foreground and background of the image to be rendered according to the image information to determine the foreground image to be rendered and the background image to be rendered.

[0061] In some embodiments, the game scene can be a game scene of a shooting game. For example, in a first-person shooter game, the "image to be rendered" may include a player's hand model, a gun model, an enemy, smoke, bullet special effects, and a distant building background or a grass background, etc. Among them, the distant building background or grass background has less changes and lower dynamics, and can be considered as the background part, that is, as the "background image to be rendered" in this application, while the player's hand model, gun model, enemy, smoke and bullet special effects are models that are slightly far away but highly dynamic and close to the user's perspective, and can be used as the foreground part of the image, that is, as the "foreground image to be rendered" in this application.

[0062] In some optional implementations, the foreground and background divisions of the rendered image can be performed by means of depth information utilization, rendering order control, image processing technology, scene management and LOD technology, masking and real-time rendering technology, so as to determine the foreground image to be rendered and the background image to be rendered in the present application.

[0063] Step S106: Rendering the foreground image to be rendered and the background image to be rendered through the device end and the cloud server end according to the image rendering data to determine the foreground image and the background image;

[0064] In a specific implementation, according to the image rendering data, the foreground image to be rendered and the background image to be rendered are rendered by the device end and the cloud server end to determine the foreground image and the background image, including:

[0065] Determine foreground image rendering data and background image rendering data according to the image rendering data, the foreground image to be rendered, and the background image to be rendered;

[0066] According to the foreground image rendering data and the background image rendering data, the foreground image to be rendered and the background image to be rendered are rendered by the device end and the cloud server end to determine the foreground image and the background image.

[0067] In some optional implementations, the background image to be rendered includes: a first rendering task, a second rendering task, a third rendering task, a fourth rendering task, and a fifth rendering task, wherein the first rendering task is a background rendering task at the current position, the second rendering task is a background rendering task at the next coordinate in the positive direction of the x-axis, the third rendering task is a background rendering task at the next coordinate in the negative direction of the x-axis, the fourth rendering task is a background rendering task at the next coordinate in the positive direction of the y-axis, and the fifth rendering task is a background rendering task at the next coordinate in the negative direction of the y-axis. Figure 2As shown, if the current position is the position of number 0, the first rendering task is the background rendering task of the position of number 0, the second rendering task is the background rendering task of the position of number 3, that is, the background rendering task on the right side of 0, the third rendering task is the background rendering task of the position of number 2, that is, the background rendering task on the left side of 0, the fourth rendering task is the background rendering task of the position of number 1, that is, the background rendering task on the front side of 0, and the fifth rendering task is the background rendering task of the position of number 4, that is, the background rendering task on the back side of 0. Among them, the first, second, third, fourth, and fifth are all for illustration and are not intended to be limiting, and can be adjusted according to specific circumstances.

[0068] In some embodiments, Figure 2 As shown in the figure, when the character is at a certain coordinate in the scene (at the beginning, it is at number 0), the terminal not only obtains the panoramic frame at number 0, but also extracts the next coordinate point in the four directions of front, back, left, and right as the center in the cloud, constructs a panoramic frame of the background, and pre-fetches it to the terminal, while the terminal is only responsible for rendering the foreground model and merging the foreground and background. When the user controls the character to move to the coordinate of number 3 in the next step, the pre-fetched panoramic frame at coordinate 3 is directly used, and the panoramic frames of the three coordinates of numbers 5, 6, and 7 need to be pre-fetched, because the panoramic frame at number 0 has been obtained and saved in the previous step. Similarly, when the user continues to move to coordinate 2, the pre-fetched panoramic frame at coordinate 2 is used, and the corresponding panoramic frames at coordinates 5, 6, 7 and are pre-fetched at the same time. The panoramic frames here all refer to the 360° image of the background screen. The division of coordinates is based on the step length of the character's movement, and the step length of the movement can be determined according to the speed of the character's movement in the engine. For example, if the player's reaction time is 1s, then the background screen needs to be switched after moving 1s.

[0069] In some optional implementations, according to the foreground image rendering data and the background image rendering data, rendering the foreground image to be rendered and the background image to be rendered by the device end and the cloud server end to determine the foreground image and the background image specifically includes:

[0070] Determine the computing power of the device end, the computing power of the cloud server end, the network bandwidth, the first image rendering computing power and the second image rendering computing power; wherein the first image rendering computing power is the computing power required for image rendering according to the foreground image rendering data, and the second image rendering computing power is the computing power required for image rendering according to the background image rendering data;

[0071] Determining an image rendering delay according to the computing power of the device end, the computing power of the cloud server end, the network bandwidth, the first image rendering computing power, and the second image rendering computing power;

[0072] Determining an image rendering mode according to the image rendering delay and a preset image rendering delay;

[0073] According to the image rendering mode, the foreground image to be rendered and the background image to be rendered are rendered by the device side and the cloud server side to determine the foreground image and the background image.

[0074] In some optional implementations, determining the image rendering delay according to the computing power of the device end, the computing power of the cloud server end, the network bandwidth, the first image rendering computing power, and the second image rendering computing power includes:

[0075] Determining a device-side rendering computing amount and a cloud server-side rendering computing amount according to the device-side computing power, the cloud server-side computing power, the first image rendering computing power, and the second image rendering computing power;

[0076] Determining the rendering delay according to the computing power of the device side, the computing power of the cloud server side, the rendering calculation amount of the device side, and the rendering calculation amount of the cloud server side;

[0077] Determining a transmission delay according to the image rendering data and the network bandwidth;

[0078] The image rendering delay is determined according to the rendering delay and the transmission delay.

[0079] In some optional embodiments, since the background image to be rendered is divided into a first task to be rendered, a second task to be rendered, a third task to be rendered, a fourth task to be rendered and a fifth task to be rendered, the background image rendering data is also correspondingly divided into a first rendering data, a second rendering data, a third rendering data, a fourth rendering data and a fifth rendering data, which are used to render the corresponding rendering tasks.

[0080] Specifically, Figure 3 In the DAG (directed acyclic graph) shown in the figure, subtasks are represented by circular nodes, and the order and dependency between tasks are represented by straight lines with arrows. Solid arrows represent tasks that must be executed, and dashed arrows represent tasks that can be executed or not. Among all subtasks, the rendering of the current background (subtask) and the rendering of the foreground (subtask) must be executed. These two subtasks are basic tasks to ensure the normal display of the screen. The color in the circular node represents the location where the subtask can be run. Pink represents tasks that can only be run on the client side (foreground rendering and merging of foreground and background). These tasks will destroy the task framework of foreground and background collaboration if they are placed in the cloud; sky blue represents tasks that can be calculated on the client side or cloud server. These tasks are subtasks that need to be assigned in the subsequent tier scheduling (blue) algorithm; gray represents process tasks. These tasks are only used as business logic to connect various parts, and the computing requirements and network requirements can be ignored.

[0081] Further, such as Figure 3 As shown in the left part, for the rendering of the background image to be rendered, by default, the device side renders the current foreground image to be rendered, and the cloud server side renders the first task to be rendered, the second task to be rendered, the third task to be rendered, the fourth task to be rendered, and the fifth task to be rendered, that is, all 5 backgrounds are rendered by the cloud side, and only the foreground is rendered on the terminal side. However, as the number of users increases, the worsening network conditions will lead to increased transmission delay. When the total delay exceeds the preset image rendering delay of 50ms, it is necessary to obtain the gear table shown in Table 1 below, and adjust the rendering tasks responsible for the device side and the cloud server side according to the gear table (adjust the user gear to a higher gear) to reduce the burden on the network side.

[0082]

[0083] Specifically, when adjusting the gear, it is necessary to know the computing power on both the end and cloud sides and the current network loan bandwidth, and then combine the known typical values ​​of computing volume and data volume in the gear table to calculate the maximum gear that can maximize the use of the end-side computing power and meet the delay requirement within 50ms.

[0084] It should be noted that for the rendering of the current background, if it is executed on the cloud, it needs to be further converted into a panoramic video frame and returned to the end side for merging the foreground and background; if it is executed on the end side, then there is no need for additional conversion into the form of video frames, because at this time the rendering of the background is completed directly on the end side, and the end side can directly execute the foreground and background as a complete rendering process.

[0085] In some optional implementations, performing image rendering according to the gear table and the preset image rendering delay specifically includes: in response to determining that the image rendering delay is less than the preset image rendering delay, the device end and the cloud server end respectively render the foreground image to be rendered and the background image to be rendered according to the foreground image rendering data and the background image rendering data, and determine the foreground image and the background image;

[0086] In response to determining that the image rendering delay is not less than the preset image rendering delay, the first image rendering computing power, the second image rendering computing power, the network transmission requirements and a pre-built rendering gear table are determined; the foreground image to be rendered and the background image to be rendered are rendered according to the first image rendering computing power, the second image rendering computing power, the network transmission requirements and the rendering gear table, and the foreground image and the background image are determined.

[0087] Step S108: Merge the foreground image and the background image through the device end to complete the rendering of the cloud game image, and display the rendered image.

[0088] It should be noted that in Figure 3 In the figure, the straight lines with arrows not only indicate the dependencies between the subtasks, but also the QoS requirements of each task. In our solution, QoS requirements are mainly divided into computing power requirements and network transmission requirements. The computing power requirements come from the computing power overhead of the rendering engine when rendering the foreground or background, and the encoding and decoding overhead when converting the background rendering results into video frames. It is worth noting that the encoding and decoding overhead only occurs when the background rendering is performed in the cloud, because if the result of the classification is the background rendering on the end side, it does not need to be converted into video frames for transmission, and can be directly displayed locally. The source of the network transmission demand is that when the background rendering task is executed in the cloud, the rendering result is transmitted back in the form of a panoramic video frame. The specified amount of data is transmitted through the network, and the corresponding demand for network bandwidth is generated according to the latency requirement.

[0089] As can be seen from the above, the immersive cloud game image end-cloud collaborative rendering method, device, electronic device, storage medium and program product provided by the present application include: determining the image to be rendered and image rendering data; performing image analysis on the image to be rendered, and dividing the foreground and background according to the analysis results to determine the foreground image to be rendered and the background image to be rendered; according to the image rendering data, the foreground image to be rendered and the background image to be rendered are rendered by the device end and the cloud server end to determine the foreground image and the background image; the foreground image and the background image are merged by the device end to complete the rendering of the cloud game image, and the rendered image is displayed. The present application renders in parts, that is, the foreground image to be rendered and the background image to be rendered are rendered by the device end and the cloud server end respectively, the foreground image and the background image are determined, and the background image that has been rendered by the cloud server end is received by the device end, and the foreground image and the background image are merged. Thereby reducing the time from requesting to obtaining the picture, effectively improving the performance and user experience of cloud games, so that cloud games can meet the needs of high-speed response while ensuring high-quality rendering.

[0090] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0091] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0092] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an immersive cloud game image end-cloud collaborative rendering device.

[0093] refer to Figure 4 , the immersive cloud game image end-cloud collaborative rendering device comprises:

[0094] A data determination module 402 is configured to determine an image to be rendered and image rendering data;

[0095] The image division module 404 is configured to perform image analysis on the image to be rendered, and divide the image into foreground and background according to the analysis result to determine the foreground image to be rendered and the background image to be rendered;

[0096] The image rendering module 406 is configured to render the foreground image to be rendered and the background image to be rendered through the device end and the cloud server end according to the image rendering data, and determine the foreground image and the background image;

[0097] The rendering image display module 408 is configured to merge the foreground image and the background image through the device end, complete the rendering of the cloud game image, and display the rendered image.

[0098] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0099] The device of the above-mentioned embodiment is used to implement the corresponding immersive cloud game image end-cloud collaborative rendering method in any of the aforementioned embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0100] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the immersive cloud game image end-cloud collaborative rendering method described in any of the above embodiments is implemented.

[0101] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0102] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0103] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0104] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0105] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0106] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0107] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0108] The electronic device of the above-mentioned embodiment is used to implement the corresponding immersive cloud game image end-cloud collaborative rendering method in any of the aforementioned embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0109] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the immersive cloud game image end-cloud collaborative rendering method as described in any of the above embodiments.

[0110] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0111] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the immersive cloud game image end-cloud collaborative rendering method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0112] Based on the same inventive concept, corresponding to the immersive cloud game image end-cloud collaborative rendering method described in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor executes the immersive cloud game image end-cloud collaborative rendering method. Corresponding to the execution subject corresponding to each step in each embodiment of the immersive cloud game image end-cloud collaborative rendering method, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0113] The computer program product of the above-mentioned embodiment is used to enable the computer and / or the processor to execute the immersive cloud game image end-cloud collaborative rendering method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0114] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0115] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0116] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0117] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for collaborative rendering of immersive cloud game images between end and cloud, characterized in that: include: Determine the image to be rendered and the image rendering data; Performing image analysis on the image to be rendered, and dividing the foreground and background according to the analysis result to determine the foreground image to be rendered and the background image to be rendered; According to the image rendering data, the foreground image to be rendered and the background image to be rendered are rendered by a device end and a cloud server end to determine a foreground image and a background image; The foreground image and the background image are merged by the device to complete the rendering of the cloud game image, and the rendered image is displayed.

2. The method according to claim 1, characterized in that The performing image analysis on the image to be rendered, dividing the foreground and the background according to the analysis result, and determining the foreground image to be rendered and the background image to be rendered includes: Performing image analysis on the image to be rendered to determine layer information or image information of the image to be rendered; The image to be rendered is divided into a foreground and a background according to the layer information or the image information, and the foreground image to be rendered and the background image to be rendered are determined.

3. The method according to claim 1, characterized in that The method of rendering the foreground image to be rendered and the background image to be rendered by a device end and a cloud server end according to the image rendering data to determine the foreground image and the background image includes: Determine foreground image rendering data and background image rendering data according to the image rendering data, the foreground image to be rendered, and the background image to be rendered; According to the foreground image rendering data and the background image rendering data, the foreground image to be rendered and the background image to be rendered are rendered by the device end and the cloud server end to determine the foreground image and the background image.

4. The method according to claim 3, characterized in that The method of rendering the foreground image to be rendered and the background image to be rendered by a device end and a cloud server end according to the foreground image rendering data and the background image rendering data to determine the foreground image and the background image includes: Determine the computing power of the device end, the computing power of the cloud server end, the network bandwidth, the first image rendering computing power and the second image rendering computing power; wherein the first image rendering computing power is the computing power required for image rendering according to the foreground image rendering data, and the second image rendering computing power is the computing power required for image rendering according to the background image rendering data; Determining an image rendering delay according to the computing power of the device end, the computing power of the cloud server end, the network bandwidth, the first image rendering computing power, and the second image rendering computing power; Determining an image rendering mode according to the image rendering delay and a preset image rendering delay; According to the image rendering mode, the foreground image to be rendered and the background image to be rendered are rendered by the device side and the cloud server side to determine the foreground image and the background image.

5. The method according to claim 4, characterized in that The determining the image rendering delay according to the computing power of the device end, the computing power of the cloud server end, the network bandwidth, the first image rendering computing power, and the second image rendering computing power includes: Determining a device-side rendering computing amount and a cloud server-side rendering computing amount according to the device-side computing power, the cloud server-side computing power, the first image rendering computing power, and the second image rendering computing power; Determining the rendering delay according to the computing power of the device side, the computing power of the cloud server side, the rendering calculation amount of the device side, and the rendering calculation amount of the cloud server side; Determining a transmission delay according to the image rendering data and the network bandwidth; The image rendering delay is determined according to the rendering delay and the transmission delay.

6. The method according to claim 5, characterized in that The background image to be rendered includes: a first rendering task, a second rendering task, a third rendering task, a fourth rendering task and a fifth rendering task, wherein the first rendering task is a background rendering task at the current position, the second rendering task is a background rendering task for the next coordinate in the positive direction of the x-axis, the third rendering task is a background rendering task for the next coordinate in the negative direction of the x-axis, the fourth rendering task is a background rendering task for the next coordinate in the positive direction of the y-axis, and the fifth rendering task is a background rendering task for the next coordinate in the negative direction of the y-axis; The method further comprises: In response to determining that the image rendering delay is less than the preset image rendering delay, the device end and the cloud server end render the foreground image to be rendered and the background image to be rendered respectively according to the foreground image rendering data and the background image rendering data, and determine the foreground image and the background image; background image rendering data: In response to determining that the image rendering delay is not less than the preset image rendering delay, the first image rendering computing power, the second image rendering computing power, the network transmission requirements and a pre-built rendering gear table are determined; the foreground image to be rendered and the background image to be rendered are rendered according to the first image rendering computing power, the second image rendering computing power, the network transmission requirements and the rendering gear table, and the foreground image and the background image are determined.

7. An immersive cloud game image end-cloud collaborative rendering device, characterized in that: include: A data determination module is configured to determine an image to be rendered and image rendering data; An image division module is configured to perform image analysis on the image to be rendered, and divide the image into foreground and background according to the analysis result to determine a foreground image to be rendered and a background image to be rendered; An image rendering module is configured to render the foreground image to be rendered and the background image to be rendered through a device end and a cloud server end according to the image rendering data, and determine a foreground image and a background image; The rendering image display module is configured to merge the foreground image and the background image through the device end, complete the rendering of the cloud game image, and display the rendered image.

8. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 6.

Images