Distributed real-time rendering method and system based on edge calculation and medium

Abstract

The invention provides a distributed real-time rendering method, system and device based on edge computing. The method comprises the steps that scene complexity of a to-be-rendered task is evaluated and matched with preset rendering capacity of an edge node, if the rendering capacity of the edge node is insufficient, the to-be-rendered task is sent to a cloud end for processing, if the rendering capacity of the edge node is sufficient, the to-be-rendered task is subjected to task splitting according to rendering scene category feature data, and the to-be-rendered task is obtained; acquiring a to-be-rendered sub-task, distributing the to-be-rendered sub-task to an edge node for real-time rendering to obtain a sub-task rendering picture, evaluating the rendering consistency of each node, performing synthesis processing to obtain a real-time rendering picture, and transmitting the real-time rendering picture to a user side for display; according to the method, the rendering capability of the edge node is verified through intelligent matching, the to-be-rendered task is split and allocated according to the rendering scene, the rendering difficulty and the load state of the edge node, and node rendering consistency evaluation is performed in real time, so that real-time rendering based on edge calculation is realized.

Description

Technical Field

[0001] This application relates to the technical fields of computer graphics and edge computing. Specifically, it relates to a distributed real-time rendering method, system, and device based on edge computing. Background Art

[0002] Traditional rendering technologies relying on a single local graphics processor or a cloud computing center are limited by hardware performance and transmission latency, which easily lead to problems such as rendering frame drops and latency, and cannot meet application scenarios with high real-time requirements. Currently, although there have been studies on applying edge computing technology to real-time rendering, the rationality and adaptability of rendering task splitting and allocation are still insufficient, resulting in unbalanced loads on edge computing nodes, inconsistent quality of real-time rendering images, and lack of intelligence in real-time rendering image evaluation and synthesis, making it difficult to guarantee the quality of synthesized images. Therefore, there is an urgent need for a highly adaptable distributed real-time rendering method.

[0003] In response to the above problems, there is an urgent need for effective technical solutions. Summary of the Invention

[0004] The purpose of this application is to provide a distributed real-time rendering method, system, and device based on edge computing, which can verify the rendering capabilities of edge nodes through intelligent matching, split and allocate rendering tasks to be processed according to the rendering scenario, rendering difficulty, and load status of edge nodes, and perform real-time evaluation of node rendering consistency, thereby achieving real-time rendering based on edge computing.

[0005] This application also provides a distributed real-time rendering method based on edge computing, including the following steps: Obtain rendering requirement information according to a rendering instruction, and extract rendering scenario category feature data and rendering complexity evaluation data; Process the rendering complexity evaluation data to obtain the scene complexity of the rendering task to be processed; Match the scene complexity with the preset rendering capabilities of edge nodes. If the rendering capabilities of an edge node are insufficient, send the rendering task to the cloud for processing; If the rendering capabilities of an edge node are sufficient, split the rendering task according to the rendering scenario category feature data to obtain sub-rendering tasks to be processed; Allocate the sub-rendering tasks to be processed to edge nodes through a preset method to obtain node rendering sub-tasks, and perform real-time rendering to obtain sub-task rendering images; Synthesize the sub-task rendering images through a preset synthesis method to obtain a real-time rendering image, and transmit it to the user side for display.

[0006] Optionally, in the distributed real-time rendering method based on edge computing described in this application, the step of obtaining rendering requirement information according to the rendering instruction and extracting rendering scene category feature data and rendering complexity evaluation data includes: Obtain rendering requirement information according to the rendering instruction, and extract rendering scene category feature data and rendering complexity evaluation data; The rendering scene category feature data includes static scene feature data, dynamic scene feature data, or multi-dimensional independent scene feature data; The rendering complexity evaluation data includes the dynamic object magnitude, geometric feature data, texture feature data, and light source feature data.

[0007] Optionally, in the distributed real-time rendering method based on edge computing described in this application, the step of processing the rendering complexity evaluation data to obtain the scene complexity of the to-be-rendered task includes: Query the preset mapping table of dynamic object magnitude and weight value according to the dynamic object magnitude to obtain the geometric feature weight value, texture feature weight value, and light source feature weight value; Perform weighted summation processing on the geometric feature data, texture feature data, and light source feature data in combination with the geometric feature weight value, texture feature weight value, and light source feature weight value to obtain the scene complexity of the to-be-rendered task.

[0008] Optionally, in the distributed real-time rendering method based on edge computing described in this application, if the rendering ability of the edge node is sufficient, the step of splitting the to-be-rendered task according to the rendering scene category feature data to obtain to-be-rendered subtasks includes: Split the to-be-rendered task to obtain to-be-rendered subtasks, including to-be-rendered block subtasks, to-be-rendered frame subtasks, and to-be-rendered scene subtasks; If it is static scene feature data, split the to-be-rendered task by space to obtain to-be-rendered block subtasks; If it is dynamic scene feature data, split the to-be-rendered task by time to obtain to-be-rendered frame subtasks; If it is multi-dimensional independent scene feature data, split the to-be-rendered task by scene to obtain to-be-rendered scene subtasks.

[0009] Optionally, in the distributed real-time rendering method based on edge computing described in this application, it further includes: Obtain the corresponding rendering requirement information according to the to-be-rendered block subtasks, to-be-rendered frame subtasks, or to-be-rendered scene subtasks; The rendering requirement information includes rendering accuracy, light requirement data, and picture display requirement data; Input the rendering accuracy, lighting requirement data, and screen display requirement data into a preset rendering difficulty prediction model for processing to obtain a rendering difficulty category label, including high difficulty or low difficulty; Classify the to-be-rendered block subtasks, to-be-rendered frame subtasks, or to-be-rendered scene subtasks according to the high difficulty or low difficulty to obtain a to-be-rendered block subtask set, a to-be-rendered frame subtask set, or a to-be-rendered scene subtask set.

[0010] Optionally, in the distributed real-time rendering method based on edge computing described in this application, it further includes: Obtain the CPU utilization rate, GPU utilization rate, memory occupancy rate, and network bandwidth usage data of the edge node; Perform weighted summation calculation on the CPU utilization rate, GPU utilization rate, memory occupancy rate, and network bandwidth usage data to obtain the real-time load data of the edge node; Compare the real-time load data of the edge node with a preset load warning threshold to obtain the real-time load status of the edge node; If it is less than or equal to the preset load warning threshold, determine that the real-time load status is low load; If it is greater than the preset load warning threshold, determine that the real-time load status is high load.

[0011] Optionally, in the distributed real-time rendering method based on edge computing described in this application, the step of allocating the to-be-rendered subtasks to the edge nodes through a preset method to obtain node rendering subtasks and performing real-time rendering to obtain subtask rendering images includes: Allocate the to-be-rendered block subtask set, to-be-rendered frame subtask set, or to-be-rendered scene subtask set in combination with the real-time load status to obtain node rendering subtasks; Extract a test set according to the node rendering subtasks to obtain rendering test subtasks; The edge node renders the rendering test subtasks to obtain rendering test time-consuming data; Process the rendering test time-consuming data to obtain node rendering time-consuming prediction data; Compare the node rendering time-consuming prediction data with a preset rendering time limit threshold; If it is less than the preset rendering time limit threshold, perform real-time rendering to obtain subtask rendering images; If it is greater than or equal to the preset rendering time limit threshold, output a warning response.

[0012] Optionally, in the distributed real-time rendering method based on edge computing described in this application, it further includes: Allocate the rendering test subtasks to each edge node for real-time rendering simultaneously to obtain rendering test images corresponding to the edge nodes; Perform consistency evaluation on the rendered test screen; If the consistency evaluation passes, perform real-time rendering; If the consistency evaluation fails, send the corresponding edge node to the client for display.

[0013] In a second aspect, the present application provides a distributed real-time rendering system based on edge computing. The system includes: a memory and a processor. The memory includes a program of a distributed real-time rendering method based on edge computing. When the program of the distributed real-time rendering method based on edge computing is executed by the processor, the following steps are implemented: Obtain rendering requirement information according to a rendering instruction, and extract rendering scene category feature data and rendering complexity evaluation data; Process according to the rendering complexity evaluation data to obtain the scene complexity of the task to be rendered; Match the scene complexity with the preset rendering capabilities of the edge nodes. If the rendering capabilities of the edge nodes are insufficient, send the task to be rendered to the cloud for processing; If the rendering capabilities of the edge nodes are sufficient, split the task to be rendered according to the rendering scene category feature data to obtain subtasks to be rendered; Allocate the subtasks to be rendered to the edge nodes through a preset method to obtain node rendering subtasks, and perform real-time rendering to obtain subtask rendering screens; Perform synthesis processing on the subtask rendering screens through a preset synthesis method to obtain a real-time rendering screen and transmit it to the client for display.

[0014] In a third aspect, the present application further provides a distributed real-time rendering device based on edge computing, including: An edge node and cloud collaboration module, including an edge computing node unit and a cloud server unit, for constructing a rendering edge computing node and cloud collaborative rendering system; A rendering acquisition module, for real-time acquisition of data and information of rendering tasks; A rendering evaluation module, for evaluating rendering tasks and edge computing nodes; A rendering task handling module, for splitting and allocating tasks to be rendered; A rendering task synthesis module, for synthesizing subtask rendering screens.

[0015] As can be seen from the above, a distributed real-time rendering method, system, and device based on edge computing provided by this application intelligently match and verify the rendering capabilities of edge nodes, split and allocate rendering tasks to be processed according to the rendering scene, rendering difficulty, and the load status of edge nodes, and perform real-time evaluation of node rendering consistency, thereby achieving real-time rendering based on edge computing.

[0016] Other features and advantages of this application will be described in the following specification. Moreover, some of them will become apparent from the specification or be understood by implementing the embodiments of this application. The objectives and other advantages of this application can be achieved and obtained through the structures specifically pointed out in the written specification and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To clearly illustrate the technical solutions of the embodiments of this application, the following briefly introduces the accompanying drawings required for the embodiments of this application. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a flowchart of a distributed real-time rendering method based on edge computing provided for an embodiment of this application; Figure 2 It is a flowchart of obtaining the scene complexity of a rendering task to be processed for a distributed real-time rendering method based on edge computing provided for an embodiment of this application; Figure 3 It is a flowchart of obtaining a sub-task to be rendered for a distributed real-time rendering method based on edge computing provided for an embodiment of this application; Figure 4 It is a flowchart of obtaining a rendered image of a sub-task for a distributed real-time rendering method based on edge computing provided for an embodiment of this application; Figure 5 It is a device diagram of a distributed real-time rendering device based on edge computing provided for an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Components of the embodiments of the present application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application claimed, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0020] It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first" and "second" are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0021] Please refer to Figure 1 , Figure 1 which is a flowchart of a distributed real-time rendering method based on edge computing in some embodiments of the present application. This distributed real-time rendering method based on edge computing is used in terminal devices such as computers and mobile phone terminals. This distributed real-time rendering method based on edge computing includes the following steps: S11. Obtain rendering requirement information according to a rendering instruction, and extract rendering scene category feature data and rendering complexity evaluation data; S12. Process according to the rendering complexity evaluation data to obtain the scene complexity of the task to be rendered; S13. Match the scene complexity with the preset rendering ability of the edge node. If the rendering ability of the edge node is insufficient, send the task to be rendered to the cloud for processing; S14. If the rendering ability of the edge node is sufficient, split the task to be rendered according to the rendering scene category feature data to obtain subtasks to be rendered; S15. Allocate the subtasks to be rendered to the edge nodes through a preset method to obtain node rendering subtasks, and perform real-time rendering to obtain subtask rendering images; S16. Synthesize and process the subtask rendering images through a preset synthesis method to obtain a real-time rendering image and transmit it to the user side for display.

[0022] It should be noted that after receiving the rendering instruction, in order to improve the real-time performance of rendering, edge computing technology is adopted. To verify the rendering ability of edge nodes, first, the scene complexity of the rendering task is evaluated to determine whether the rendering ability of the edge nodes is sufficient. If it is insufficient, the rendering task is sent to the cloud computing center for rendering processing to achieve collaborative processing between the edge nodes and the cloud. If the ability is sufficient, the task is split according to the rendering scene category feature data representing the scene type to obtain the sub-tasks to be rendered. Then, the rendering difficulty of the obtained sub-tasks to be rendered and the real-time load of the edge nodes are evaluated, and the sub-tasks to be rendered are allocated to each edge node for real-time rendering according to the evaluation results. At the same time, the consistency and rendering quality of the rendering of each node are monitored in real time. Finally, the rendered pictures of the sub-tasks completed by each edge node are synthesized through a preset synthesis method to obtain a real-time rendering picture, which is transmitted to the user side for display, thus realizing the rationality and adaptability of the splitting and allocation of the rendering task.

[0023] According to an embodiment of the present invention, obtaining the rendering requirement information according to the rendering instruction and extracting the rendering scene category feature data and the rendering complexity evaluation data includes: Obtaining the rendering requirement information according to the rendering instruction and extracting the rendering scene category feature data and the rendering complexity evaluation data; The rendering scene category feature data includes static scene feature data, dynamic scene feature data, or multi-dimensional independent scene feature data; The rendering complexity evaluation data includes the dynamic object magnitude, geometric feature data, texture feature data, and light source feature data.

[0024] It should be noted that in order to improve the rationality of task splitting and allocation and improve the rendering quality, the corresponding rendering complexity is evaluated according to different rendering scenes to evaluate whether the rendering ability of the edge nodes matches. Among them, the static scene feature data is such as a factory layout diagram, the dynamic scene feature data is such as an animation video, the multi-dimensional independent scene feature data is such as multiple player characters in a 3D game, the dynamic object magnitude is defined by those skilled in the art according to the number of dynamic objects in the scene and can be dynamically adjusted, from 0 to 10, a total of 10 levels, and the number of dynamic objects at level 10 is the largest, which is used to query the scene complexity evaluation weight value.

[0025] Please refer to Figure 2 , Figure 2 is a flowchart of obtaining the scene complexity of the task to be rendered in a distributed real-time rendering method based on edge computing in some embodiments of the present application. According to an embodiment of the present invention, processing according to the rendering complexity evaluation data to obtain the scene complexity of the task to be rendered includes: S21. Query the preset mapping table of dynamic object magnitudes and weight values according to the dynamic object magnitude, and obtain the geometric feature weight value, the texture feature weight value, and the light source feature weight value; S22. Perform weighted summation processing on the geometric feature data, the texture feature data, and the light source feature data in combination with the geometric feature weight value, the texture feature weight value, and the light source feature weight value to obtain the scene complexity of the to-be-rendered task.

[0026] It should be noted that the weight values corresponding to the respective evaluation data are obtained by querying the preset mapping table of dynamic object magnitudes and weight values according to the evaluated dynamic object magnitude, and then weighted summation processing is performed to obtain the scene complexity corresponding to the to-be-rendered task, which is used for subsequent matching with the rendering capabilities of the edge nodes. Among them, the geometric feature data refers to the sum of the number of polygons and the number of curved surfaces, the texture feature data is evaluated by those skilled in the art based on different texture types and the corresponding texture quantity and texture resolution, and the light source feature data is evaluated by those skilled in the art based on the number of different types of light sources.

[0027] Please refer to Figure 3 , Figure 3 is a flowchart of obtaining the to-be-rendered subtask of a distributed real-time rendering method based on edge computing in some embodiments of the present application. According to an embodiment of the present invention, if the rendering capabilities of the edge nodes are sufficient, the to-be-rendered task is split according to the rendering scene category feature data to obtain the to-be-rendered subtasks, including: S31. Split the to-be-rendered task to obtain to-be-rendered subtasks, including to-be-rendered block subtasks, to-be-rendered frame subtasks, and to-be-rendered scene subtasks; S32. If it is static scene feature data, split the to-be-rendered task into spatial tasks to obtain to-be-rendered block subtasks; S33. If it is dynamic scene feature data, split the to-be-rendered task into temporal tasks to obtain to-be-rendered frame subtasks; S34. If it is multi-dimensional independent scene feature data, split the to-be-rendered task into scene tasks to obtain to-be-rendered scene subtasks.

[0028] It should be noted that to improve the adaptability of task splitting and allocation, static scenes are split into spatial tasks, and each block is assigned to an edge node. Dynamic scenes are split into temporal tasks, and consecutive video frames are assigned to an edge node. Multi-dimensional independent scenes are split into scene tasks, and each independent scene is assigned to an edge node. Among them, the to-be-rendered block subtasks can also be divided into multiple small blocks, the to-be-rendered frame subtasks can also be divided into multiple video frames, and the to-be-rendered scene subtasks can also be divided into multiple small scenes, which are used for subsequent rendering time-consuming evaluation and rendering consistency evaluation of each node.

[0029] According to an embodiment of the present invention, it further includes: Obtain corresponding rendering requirement information according to the to-be-rendered block subtask, to-be-rendered frame subtask or to-be-rendered scene subtask; The rendering requirement information includes rendering precision, lighting requirement data, and picture display requirement data; Input the rendering precision, lighting requirement data, and picture display requirement data into a preset rendering difficulty prediction model for processing to obtain a rendering difficulty category label, including high difficulty or low difficulty; Classify the to-be-rendered block subtask, to-be-rendered frame subtask or to-be-rendered scene subtask according to the high difficulty or low difficulty to obtain a to-be-rendered block subtask set, a to-be-rendered frame subtask set or a to-be-rendered scene subtask set.

[0030] It should be noted that since the rendering difficulties of the divided to-be-rendered subtasks are different, in order to classify and assign tasks, the corresponding rendering precision, lighting requirement data, and picture display requirement data are processed through a preset rendering difficulty prediction model to obtain a rendering difficulty category label; among them, the preset rendering difficulty prediction model is trained by obtaining the rendering precision, lighting requirement data, and picture display requirement data of a large number of historical rendering tasks and the corresponding rendering difficulty category labels. The rendering precision is obtained by weighted summation of geometric precision, texture precision, and color precision. The lighting requirement data is obtained by weighted summation of static lighting, dynamic lighting, and lighting animation. The picture display requirement data is obtained by weighted summation of the picture resolution and frame rate. The specific weight values are preset by those skilled in the art according to specific rendering tasks and can be dynamically adjusted. Then, the to-be-rendered block subtask, to-be-rendered frame subtask or to-be-rendered scene subtask is classified according to high difficulty or low difficulty to obtain a to-be-rendered block subtask set including high difficulty and / or low difficulty, a to-be-rendered frame subtask set including high difficulty and / or low difficulty, and a to-be-rendered scene subtask set including high difficulty and / or low difficulty.

[0031] According to an embodiment of the present invention, it further includes: Obtain the CPU utilization rate, GPU utilization rate, memory occupancy rate, and network bandwidth usage data of the edge node; Perform weighted summation calculation on the CPU utilization rate, GPU utilization rate, memory occupancy rate, and network bandwidth usage data to obtain the real-time load data of the edge node; Compare the real-time load data of the edge node with a preset load warning threshold to obtain the real-time load status of the edge node; If it is less than or equal to the preset load warning threshold, determine that the real-time load status is low load; If it is greater than the preset load warning threshold, determine that the real-time load status is high load.

[0032] It should be noted that, in order to achieve the adaptation of the sub-tasks to be rendered to the edge nodes, the real-time load of the edge nodes is evaluated to obtain the real-time load data of the edge nodes, and a threshold comparison is performed. The real-time load status is divided into low load or high load. The low load is responsible for rendering tasks with high difficulty, and the high load is responsible for rendering tasks with low difficulty.

[0033] Please refer to Figure 4 , Figure 4 FIG. is a flowchart of obtaining the rendering screen of sub-tasks in a distributed real-time rendering method based on edge computing in some embodiments of the present application. According to an embodiment of the present invention, the step of allocating the sub-tasks to be rendered to edge nodes through a preset method to obtain node rendering sub-tasks and performing real-time rendering to obtain the rendering screen of sub-tasks includes: S41. Combining the set of sub-tasks of the block to be rendered, the set of sub-tasks of the frame to be rendered, or the set of sub-tasks of the scene to be rendered with the real-time load status for task allocation to obtain node rendering sub-tasks; S42. Extracting a test set according to the node rendering sub-tasks to obtain rendering test sub-tasks; S43. The edge node renders the rendering test sub-tasks to obtain rendering test time-consuming data; S44. Processing the rendering test time-consuming data to obtain predicted node rendering time-consuming data; S45. Comparing the predicted node rendering time-consuming data with a preset rendering time limit threshold; S46. If it is less than the preset rendering time limit threshold, perform real-time rendering to obtain the rendering screen of sub-tasks; S47. If it is greater than or equal to the preset rendering time limit threshold, output a warning response.

[0034] It should be noted that after allocating the set of sub-tasks with high difficulty to the edge nodes with low load and the set of sub-tasks with low difficulty to the edge nodes with high load, and before performing real-time rendering, in order to determine whether the time-consuming of each node for task rendering can meet the rendering timeliness, first, each edge node extracts the rendering test sub-tasks for rendering processing and counts the rendering test time-consuming data, then, edge node time-consuming prediction is performed to obtain the predicted node rendering time-consuming data, and finally, it is determined whether the time limit requirement is met through threshold comparison.

[0035] According to an embodiment of the present invention, it further includes: Allocating the rendering test sub-tasks to each edge node for real-time rendering simultaneously to obtain the rendering test screens corresponding to the edge nodes; Performing consistency evaluation on the rendering test screens; If the consistency evaluation is passed, perform real-time rendering; If the consistency evaluation fails, the corresponding edge node is sent to the user side for display.

[0036] It should be noted that, in order to evaluate the consistency of each edge node in task rendering and improve the picture synthesis effect, each edge node performs real-time rendering on the same test sub-task to obtain the rendering test pictures corresponding to each edge node, and conducts a consistency evaluation on them. Only when the evaluation passes is real-time rendering allowed to avoid wasting computing resources.

[0037] It is worth mentioning that, according to the embodiments of the present invention, it further includes: Obtain the synthesis test sub-task of the first edge node; Allocate the synthesis test sub-task to the first edge node and the adjacent second edge node respectively for real-time rendering to obtain the corresponding synthesis test pictures; Conduct a quality evaluation on the synthesis test pictures; If the quality evaluation passes, synthesize the sub-task rendering pictures of the first edge node and the adjacent second edge node; If the quality evaluation fails, send the corresponding edge node to the user side for display.

[0038] It should be noted that since the sub-task rendering pictures after rendering by each edge node need to be synthesized, to avoid unqualified picture quality after synthesis, after the real-time rendering consistency evaluation of each edge node passes, the rendering quality of adjacent edge nodes to be synthesized is further evaluated before synthesis. First, extract the synthesis test sub-task of the first edge node, and then simultaneously allocate it to two adjacent edge nodes to be synthesized for real-time rendering respectively to obtain the synthesis test pictures corresponding to the first edge node and the adjacent second edge node, and conduct quality evaluations respectively. If both adjacent edge nodes pass, synthesis processing is allowed; otherwise, the edge node with a failed quality evaluation is sent to the user side for display.

[0039] It is worth mentioning that, according to the embodiments of the present invention, the consistency evaluation of the rendering test pictures includes: Conduct spatial consistency, temporal consistency, visual consistency, and interaction consistency evaluations on the rendering test pictures to obtain spatial consistency evaluation results, temporal consistency evaluation results, visual consistency evaluation results, and interaction consistency evaluation results; Conduct an AND operation on the spatial consistency evaluation result, temporal consistency evaluation result, visual consistency evaluation result, and interaction consistency evaluation result; If the evaluation passes, it is determined that the consistency evaluation passes.

[0040] It should be noted that the evaluation of the rendering consistency of each edge node includes the evaluation of spatial consistency, temporal consistency, visual consistency, and interaction consistency. Among them, the spatial consistency is evaluated by the deviation of the geometric position, size, and orientation of an object or scene from the corresponding preset geometric position, preset size, and preset orientation respectively; the temporal consistency is evaluated by the timing sequence of a dynamic scene to determine whether there are frame drops or latency exceeding the tolerance; the visual consistency is evaluated by the deviation of brightness, contrast, and color from the corresponding preset brightness, preset contrast, and preset color respectively; the interaction consistency is evaluated by the feedback consistency of different edge nodes to state changes; the evaluation results of spatial consistency, temporal consistency, visual consistency, and interaction consistency respectively include passing the evaluation or failing the evaluation.

[0041] The present invention also discloses a distributed real-time rendering system based on edge computing, including a memory and a processor. The memory includes a program of a distributed real-time rendering method based on edge computing. When the program of the distributed real-time rendering method based on edge computing is executed by the processor, the following steps are implemented: Obtain rendering requirement information according to a rendering instruction, and extract rendering scene category feature data and rendering complexity evaluation data; Process the rendering complexity evaluation data to obtain the scene complexity of the task to be rendered; Match the scene complexity with the preset rendering capabilities of edge nodes. If the rendering capabilities of the edge nodes are insufficient, send the task to be rendered to the cloud for processing; If the rendering capabilities of the edge nodes are sufficient, split the task to be rendered according to the rendering scene category feature data to obtain subtasks to be rendered; Allocate the subtasks to be rendered to edge nodes through a preset method to obtain node rendering subtasks, and perform real-time rendering to obtain subtask rendering images; Perform synthesis processing on the subtask rendering images through a preset synthesis method to obtain a real-time rendering image, and transmit it to the user terminal for display.

[0042] It should be noted that after receiving the rendering instruction, in order to improve the real-time performance of rendering, edge computing technology is adopted. To verify the rendering ability of edge nodes, first, the scene complexity of the rendering task is evaluated to determine whether the rendering ability of the edge nodes is sufficient. If it is insufficient, the rendering task is sent to the cloud computing center for rendering processing to achieve collaborative processing between the edge nodes and the cloud. If the ability is sufficient, the task is split according to the rendering scene category feature data representing the scene type to obtain the sub-tasks to be rendered. Then, the rendering difficulty of the obtained sub-tasks to be rendered and the real-time load of the edge nodes are evaluated, and the sub-tasks to be rendered are assigned to each edge node for real-time rendering according to the evaluation results. At the same time, the consistency and rendering quality of the rendering of each node are monitored in real time. Finally, the rendered images of the sub-tasks completed by each edge node are synthesized through a preset synthesis method to obtain a real-time rendering image, which is then transmitted to the user side for display, thus realizing the rationality and adaptability of the splitting and assignment of the rendering task.

[0043] According to an embodiment of the present invention, the obtaining of the rendering requirement information according to the rendering instruction, and the extraction of the rendering scene category feature data and the rendering complexity evaluation data include: Obtaining the rendering requirement information according to the rendering instruction, and extracting the rendering scene category feature data and the rendering complexity evaluation data; The rendering scene category feature data includes static scene feature data, dynamic scene feature data, or multi-dimensional independent scene feature data; The rendering complexity evaluation data includes the dynamic object magnitude, geometric feature data, texture feature data, and light source feature data.

[0044] It should be noted that in order to improve the rationality of task splitting and assignment and improve the rendering quality, the corresponding rendering complexity is evaluated for different rendering scenes to evaluate whether the rendering ability of the edge nodes matches. Among them, the static scene feature data is such as a factory layout diagram, the dynamic scene feature data is such as an animation video, the multi-dimensional independent scene feature data is such as multiple player characters in a 3D game, the dynamic object magnitude is defined by those skilled in the art according to the number of dynamic objects in the scene and can be dynamically adjusted, with a total of 10 levels from 0 to 10, and the number of dynamic objects at level 10 is the largest, which is used to query the weight value of the scene complexity evaluation.

[0045] According to an embodiment of the present invention, the processing according to the rendering complexity evaluation data to obtain the scene complexity of the task to be rendered includes: Querying a preset mapping table of dynamic object magnitude and weight value according to the dynamic object magnitude to obtain the geometric feature weight value, the texture feature weight value, and the light source feature weight value; Combining the geometric feature data, texture feature data, and light source feature data with the geometric feature weight value, texture feature weight value, and light source feature weight value for weighted summation processing to obtain the scene complexity of the to-be-rendered task.

[0046] It should be noted that the weight values corresponding to each evaluation data are obtained by querying the preset mapping table of dynamic object magnitudes and weight values according to the evaluated dynamic object magnitudes, and then weighted summation processing is performed to obtain the scene complexity corresponding to the to-be-rendered task, which is used for subsequent matching with the rendering capabilities of edge nodes. Among them, the geometric feature data refers to the sum of the number of polygons and the number of surfaces, the texture feature data is evaluated by those skilled in the art based on different texture types and the corresponding number of textures and texture resolutions, and the light source feature data is evaluated by those skilled in the art based on the number of different types of light sources.

[0047] According to an embodiment of the present invention, if the rendering capabilities of the edge nodes are sufficient, the to-be-rendered task is split into to-be-rendered subtasks according to the rendering scene category feature data, including: Splitting the to-be-rendered task into to-be-rendered subtasks, including to-be-rendered block subtasks, to-be-rendered frame subtasks, and to-be-rendered scene subtasks; If it is static scene feature data, the to-be-rendered task is split into spatial tasks to obtain to-be-rendered block subtasks; If it is dynamic scene feature data, the to-be-rendered task is split into time tasks to obtain to-be-rendered frame subtasks; If it is multi-dimensional independent scene feature data, the to-be-rendered task is split into scene tasks to obtain to-be-rendered scene subtasks.

[0048] It should be noted that to improve the adaptability of task splitting and allocation, static scenes are split into spatial tasks, and each block is assigned to an edge node. Dynamic scenes are split into time tasks, and consecutive video frames are assigned to an edge node. Multi-dimensional independent scenes are split into scene tasks, and each independent scene is assigned to an edge node. Among them, the to-be-rendered block subtasks can also be divided into multiple small blocks, the to-be-rendered frame subtasks can also be divided into multiple video frames, and the to-be-rendered scene subtasks can also be divided into multiple small scenes, which are used for subsequent rendering time-consuming evaluation and rendering consistency evaluation of each node.

[0049] According to an embodiment of the present invention, it further includes: Obtaining corresponding rendering requirement information according to the to-be-rendered block subtask, to-be-rendered frame subtask, or to-be-rendered scene subtask; The rendering requirement information includes rendering accuracy, light demand data, and picture display demand data; Input the rendering accuracy, light requirement data, and screen display requirement data into a preset rendering difficulty prediction model for processing to obtain a rendering difficulty category label, including high difficulty or low difficulty; Classify the to-be-rendered block subtasks, to-be-rendered frame subtasks, or to-be-rendered scene subtasks according to the high difficulty or low difficulty to obtain a to-be-rendered block subtask set, a to-be-rendered frame subtask set, or a to-be-rendered scene subtask set.

[0050] It should be noted that since the rendering difficulties of the divided to-be-rendered subtasks are different, in order to classify and allocate tasks, the corresponding rendering accuracy, light requirement data, and screen display requirement data are processed through a preset rendering difficulty prediction model to obtain a rendering difficulty category label; Among them, the preset rendering difficulty prediction model is obtained by training with the rendering accuracy, light requirement data, and screen display requirement data of a large number of historical rendering tasks and the corresponding rendering difficulty category labels. The rendering accuracy is obtained by weighted summation of geometric accuracy, texture accuracy, and color accuracy. The light requirement data is obtained by weighted summation of static light, dynamic light, and light animation. The screen display requirement data is obtained by weighted summation of screen resolution and frame rate. The specific weight values are preset by those skilled in the art according to specific rendering tasks and can be dynamically adjusted. Then, the to-be-rendered block subtasks, to-be-rendered frame subtasks, or to-be-rendered scene subtasks are classified according to high difficulty or low difficulty to obtain a to-be-rendered block subtask set including high difficulty and / or low difficulty, a to-be-rendered frame subtask set including high difficulty and / or low difficulty, and a to-be-rendered scene subtask set including high difficulty and / or low difficulty.

[0051] According to an embodiment of the present invention, it further includes: Obtain the CPU utilization rate, GPU utilization rate, memory occupancy rate, and network bandwidth usage data of the edge node; Perform weighted summation calculation on the CPU utilization rate, GPU utilization rate, memory occupancy rate, and network bandwidth usage data to obtain the real-time load data of the edge node; Compare the real-time load data of the edge node with a preset load warning threshold to obtain the real-time load status of the edge node; If it is less than or equal to the preset load warning threshold, determine that the real-time load status is low load; If it is greater than the preset load warning threshold, determine that the real-time load status is high load.

[0052] It should be noted that in order to achieve the adaptation of the to-be-rendered subtasks to the edge node, the real-time load of the edge node is evaluated to obtain the real-time load data of the edge node, and a threshold comparison is performed. The real-time load status is divided into low load or high load. The low load is responsible for rendering high-difficulty tasks, and the high load is responsible for rendering low-difficulty tasks.

[0053] According to an embodiment of the present invention, the steps of allocating the sub-tasks to be rendered to edge nodes through a preset method to obtain node rendering sub-tasks and performing real-time rendering to obtain sub-task rendering images include: Combining the sub-task sets of blocks to be rendered, frames to be rendered, or scenes to be rendered with the real-time load status to perform task allocation and obtain node rendering sub-tasks; Extracting a test set according to the node rendering sub-tasks to obtain rendering test sub-tasks; Edge nodes render the rendering test sub-tasks to obtain rendering test time-consuming data; Processing the rendering test time-consuming data to obtain node rendering time-consuming prediction data; Comparing the node rendering time-consuming prediction data with a preset rendering time limit threshold; If it is less than the preset rendering time limit threshold, perform real-time rendering to obtain sub-task rendering images; If it is greater than or equal to the preset rendering time limit threshold, output a warning response.

[0054] It should be noted that after allocating high-difficulty sub-task sets to low-load edge nodes and low-difficulty sub-task sets to high-load edge nodes, before performing real-time rendering, in order to determine whether the time-consuming of each node for task rendering can meet the rendering timeliness, first, each edge node extracts rendering test sub-tasks for rendering processing and counts the rendering test time-consuming data, then, performs edge node time-consuming prediction to obtain node rendering time-consuming prediction data, and finally, determines whether the time limit requirement is met through threshold comparison.

[0055] According to an embodiment of the present invention, it further includes: Allocating the rendering test sub-tasks to each edge node simultaneously for real-time rendering to obtain rendering test images corresponding to the edge nodes; Performing consistency evaluation on the rendering test images; If the consistency evaluation passes, perform real-time rendering; If the consistency evaluation fails, send the corresponding edge node to the user side for display.

[0056] It should be noted that in order to evaluate the consistency of each edge node for task rendering and improve the image synthesis effect, each edge node performs real-time rendering on the same test sub-task to obtain rendering test images corresponding to each edge node, and performs consistency evaluation on them. Only when the evaluation passes is real-time rendering allowed to avoid waste of computing resources.

[0057] It is worth mentioning that according to an embodiment of the present invention, it further includes: Obtain the synthetic test subtask of the first edge node; Allocate the synthetic test subtask to the first edge node and the adjacent second edge node respectively for real-time rendering to obtain corresponding synthetic test images; Evaluate the quality of the synthetic test images; If the quality evaluation passes, synthesize the subtask rendering images of the first edge node and the adjacent second edge node; If the quality evaluation fails, send the corresponding edge node to the user side for display.

[0058] It should be noted that since the subtask rendering images after rendering by each edge node need to be synthesized, to avoid unqualified image quality after synthesis, after the real-time rendering consistency evaluation of each edge node passes, the rendering quality of adjacent edge nodes to be synthesized is further evaluated before synthesis. First, extract the synthetic test subtask of the first edge node, and then allocate it to two adjacent edge nodes to be synthesized for real-time rendering respectively to obtain the corresponding synthetic test images of the first edge node and the adjacent second edge node, and perform quality evaluation respectively. If both adjacent edge nodes pass, synthesis processing is allowed; otherwise, the edge node with failed quality evaluation is sent to the user side for display.

[0059] It is worth mentioning that according to the embodiments of the present invention, the evaluation of the consistency of the rendering test images includes: Evaluate the spatial consistency, temporal consistency, visual consistency, and interaction consistency of the rendering test images to obtain spatial consistency evaluation results, temporal consistency evaluation results, visual consistency evaluation results, and interaction consistency evaluation results; Perform an AND operation on the spatial consistency evaluation results, temporal consistency evaluation results, visual consistency evaluation results, and interaction consistency evaluation results; If the evaluation passes, it is determined that the consistency evaluation passes.

[0060] It should be noted that the evaluation of the rendering consistency of each edge node includes spatial consistency, temporal consistency, visual consistency, and interaction consistency evaluations. Among them, spatial consistency is evaluated by the deviations of the geometric position, size, and orientation of an object or scene from the corresponding preset geometric position, preset size, and preset orientation respectively; temporal consistency is evaluated by the timing sequence of a dynamic scene to determine whether there are frame drops or latency exceedances; visual consistency is evaluated by the deviations of brightness, contrast, and color from the corresponding preset brightness, preset contrast, and preset color respectively; interaction consistency is evaluated by the feedback consistency of different edge nodes to state changes. The evaluation results of spatial consistency, temporal consistency, visual consistency, and interaction consistency respectively include passing the evaluation or failing the evaluation.

[0061] Please refer to Figure 5 , Figure 5 which is a device diagram of a distributed real-time rendering device based on edge computing in some embodiments of the present application.

[0062] The third aspect of the present invention provides a distributed real-time rendering device 5 based on edge computing, including: An edge node and cloud collaboration module 51, including an edge computing node unit 511 and a cloud server unit 512, for constructing a collaborative rendering system of rendering edge computing nodes and the cloud; A rendering acquisition module 52, for real-time acquisition of data and information of rendering tasks; A rendering evaluation module 53, for evaluating rendering tasks and edge computing nodes; A rendering task handling module 54, for splitting and distributing rendering tasks to be processed; A rendering task synthesis module 55, for synthesizing sub-task rendering images.

[0063] A distributed real-time rendering method, system, and device disclosed by the present invention verify the rendering capabilities of edge nodes through intelligent matching, split and distribute rendering tasks to be processed according to the rendering scene, rendering difficulty, and the load status of edge nodes, and perform real-time evaluation of node rendering consistency, thereby realizing real-time rendering based on edge computing.

[0064] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed with each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0065] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units. They can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0066] In addition, each functional unit in the embodiments of the present invention can be all integrated in a processing unit, or each unit can be separately used as a unit, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0067] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a readable storage medium. When the program is executed, it executes the steps including the above method embodiments. The foregoing storage medium includes various media that can store program codes, such as removable storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0068] Alternatively, if the above-mentioned integrated units of the present invention are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present invention. The foregoing storage medium includes various media that can store program codes, such as removable storage devices, ROM, RAM, magnetic disks, or optical discs.

Claims

1. A distributed real-time rendering method based on edge computing, characterized in that, It includes the following steps: Obtain rendering requirement information according to the rendering instruction, and extract rendering scene category feature data and rendering complexity evaluation data; Process according to the rendering complexity evaluation data to obtain the scene complexity of the to-be-rendered task; Match the scene complexity with the preset rendering capabilities of the edge nodes. If the rendering capabilities of the edge nodes are insufficient, send the to-be-rendered task to the cloud for processing; If the rendering capabilities of the edge nodes are sufficient, split the to-be-rendered task according to the rendering scene category feature data to obtain to-be-rendered subtasks; Allocate the to-be-rendered subtasks to the edge nodes through a preset method to obtain node rendering subtasks, and perform real-time rendering to obtain subtask rendering pictures; Synthesize and process the subtask rendering pictures through a preset synthesis method to obtain a real-time rendering picture, and transmit it to the user side for display.

2. The distributed real-time rendering method based on edge computing according to claim 1, wherein The obtaining of rendering requirement information according to the rendering instruction and the extraction of rendering scene category feature data and rendering complexity evaluation data include: Obtain rendering requirement information according to the rendering instruction, and extract rendering scene category feature data and rendering complexity evaluation data; The rendering scene category feature data includes static scene feature data, dynamic scene feature data, or multi-dimensional independent scene feature data; The rendering complexity evaluation data includes the dynamic object magnitude, geometric feature data, texture feature data, and light source feature data.

3. The distributed real-time rendering method based on edge computing according to claim 2, wherein The processing according to the rendering complexity evaluation data to obtain the scene complexity of the to-be-rendered task includes: Query a preset mapping table of dynamic object magnitude and weight values according to the dynamic object magnitude to obtain geometric feature weight values, texture feature weight values, and light source feature weight values; Perform weighted summation processing on the geometric feature data, texture feature data, and light source feature data in combination with the geometric feature weight values, texture feature weight values, and light source feature weight values to obtain the scene complexity of the to-be-rendered task.

4. The distributed real-time rendering method based on edge computing according to claim 3, characterized in that The splitting of the to-be-rendered task according to the rendering scene category feature data to obtain to-be-rendered subtasks when the rendering capabilities of the edge nodes are sufficient includes: Split the to-be-rendered task to obtain to-be-rendered subtasks, including to-be-rendered block subtasks, to-be-rendered frame subtasks, and to-be-rendered scene subtasks; If it is static scene feature data, perform spatial task splitting on the to-be-rendered task to obtain to-be-rendered block subtasks; If it is dynamic scene feature data, perform time task splitting on the to-be-rendered task to obtain to-be-rendered frame subtasks; If it is multi-dimensional independent scene feature data, perform scene task splitting on the to-be-rendered task to obtain to-be-rendered scene subtasks.

5. The distributed real-time rendering method based on edge computing according to claim 4, wherein It also includes: Obtain corresponding rendering requirement information according to the to-be-rendered block subtasks, to-be-rendered frame subtasks, or to-be-rendered scene subtasks; The rendering requirement information includes rendering precision, light requirement data, and picture display requirement data; Input the rendering precision, light requirement data, and picture display requirement data into a preset rendering difficulty prediction model for processing to obtain a rendering difficulty category label, including high difficulty or low difficulty; Classify the to-be-rendered block subtasks, to-be-rendered frame subtasks, or to-be-rendered scene subtasks according to the high or low difficulty to obtain a to-be-rendered block subtask set, a to-be-rendered frame subtask set, or a to-be-rendered scene subtask set.

6. The distributed real-time rendering method based on edge computing according to claim 5, wherein, It further includes: Obtain the CPU utilization rate, GPU utilization rate, memory occupancy rate, and network bandwidth usage data of the edge node; Perform a weighted summation calculation on the CPU utilization rate, GPU utilization rate, memory occupancy rate, and network bandwidth usage data to obtain the real-time load data of the edge node; Compare the real-time load data of the edge node with a preset load warning threshold to obtain the real-time load status of the edge node; If it is less than or equal to the preset load warning threshold, determine that the real-time load status is low load; If it is greater than the preset load warning threshold, determine that the real-time load status is high load.

7. The distributed real-time rendering method based on edge computing according to claim 6, wherein The step of allocating the to-be-rendered subtasks to the edge nodes through a preset method to obtain node rendering subtasks and performing real-time rendering to obtain subtask rendering images includes: Allocate the to-be-rendered block subtask set, to-be-rendered frame subtask set, or to-be-rendered scene subtask set in combination with the real-time load status to obtain node rendering subtasks; Extract a test set according to the node rendering subtasks to obtain rendering test subtasks; The edge node renders the rendering test subtasks to obtain rendering test time-consuming data; Process the rendering test time-consuming data to obtain node rendering time-consuming prediction data; Compare the node rendering time-consuming prediction data with a preset rendering time limit threshold; If it is less than the preset rendering time limit threshold, perform real-time rendering to obtain subtask rendering images; If it is greater than or equal to the preset rendering time limit threshold, output a warning response.

8. The distributed real-time rendering method based on edge computing according to claim 7, characterized in that It further includes: Allocate the rendering test subtasks to each edge node for real-time rendering simultaneously to obtain rendering test images corresponding to the edge nodes; Perform a consistency evaluation on the rendering test images; If the consistency evaluation passes, perform real-time rendering; If the consistency evaluation fails, send the corresponding edge node to the user side for display.

9. A distributed real-time rendering system based on edge computing, characterized in that, It includes a memory and a processor. The memory includes a program for the distributed real-time rendering method based on edge computing. When the program for the distributed real-time rendering method based on edge computing is executed by the processor, the following steps are implemented: Obtain rendering requirement information according to a rendering instruction, and extract rendering scene category feature data and rendering complexity evaluation data; Process the rendering complexity evaluation data to obtain the scene complexity of the to-be-rendered task; Match the scene complexity with the preset rendering capabilities of the edge nodes. If the rendering capabilities of the edge nodes are insufficient, send the to-be-rendered task to the cloud for processing; If the rendering capabilities of the edge nodes are sufficient, split the to-be-rendered task according to the rendering scene category feature data to obtain to-be-rendered subtasks; Allocate the to-be-rendered subtasks to the edge nodes through a preset method to obtain node rendering subtasks and perform real-time rendering to obtain subtask rendering images; The rendered screen of the sub-task is synthesized through a preset synthesis method to obtain a real-time rendered screen, which is then transmitted to the client for display.

10. A distributed real-time rendering device based on edge computing, characterized in that, Including: Edge node and cloud collaboration module, including an edge computing node unit and a cloud server unit, used to build a rendering edge computing node and cloud collaborative rendering system; Rendering acquisition module, used to collect data and information of rendering tasks in real time; Rendering evaluation module, used to evaluate rendering tasks and edge computing nodes; Rendering task handling module, used to split and allocate tasks to be rendered; Rendering task synthesis module, used to synthesize the rendered screen of sub-tasks.

Images