Collaborative Rendering Method, Device, and Storage Medium for City-Level Digital Twins

Abstract

The present application discloses a collaborative rendering method, device, and storage medium for urban-level digital twins, which relates to the field of digital twin technologies and includes: monitoring the input camera control data, and performing view strongly correlated rendering calculations on the target scene based on the camera control data to obtain a strongly correlated rendering result of the target scene; sending the camera control data to the server to perform view weakly correlated rendering calculations on the target scene by the server based on the camera control data and return a weakly correlated rendering result of the target scene; and mixing the strongly correlated rendering result and the weakly correlated rendering result to obtain a collaborative rendering result of the target scene. By transferring the view weakly correlated rendering calculation task to the server for execution, the present application realizes that under the condition of limited client resources, through the collaborative work of the server and the client, both the real-time performance and fluency of user interaction are ensured, and a high-fidelity picture rendering effect is achieved.

Description

Technical Field

[0001] This application relates to the field of digital twin technology, and particularly to a collaborative rendering method, device, and storage medium for urban-level digital twins. Background Art

[0002] At present, the rendering of urban-level digital twin scenes is mainly achieved through cloud rendering or end rendering. Among them, cloud rendering refers to a method of running a rendering program on the server side and pushing the rendering result to the client side for display through a video stream. This method can obtain a highly realistic picture rendering effect on resource-constrained client sides. However, this strong coupling between the server side and the client side and the continuous streaming data transmission method have high requirements for network bandwidth and stability. A low-bandwidth and unstable network environment will affect the transmission rate of the picture, resulting in frame freezing and latency, and affecting the user's interaction experience. End rendering refers to running a rendering program on the client side and displaying the rendering result. Since end rendering does not require external computing resources, it is suitable for high-concurrency business scenarios. However, the picture rendering effect of this method is limited by the client's hardware resources, and it is difficult to support advanced rendering algorithms on low-configuration client sides, and high-fidelity picture rendering cannot be performed.

[0003] Therefore, how to achieve a highly realistic picture rendering effect under the condition of limited client hardware resources is an urgent problem to be solved at present. Summary of the Invention

[0004] The main purpose of this application is to provide a collaborative rendering method, device, and storage medium for urban-level digital twins, aiming to solve the technical problem of how to achieve a highly realistic picture rendering effect under the condition of limited client hardware resources.

[0005] To achieve the above object, this application proposes a collaborative rendering method for urban-level digital twins, which is applied to the client side. The collaborative rendering method for urban-level digital twins includes:

[0006] Monitoring the input camera control data, and performing view strongly related rendering calculation on the target scene based on the camera control data to obtain the strongly related rendering result of the target scene. Among them, the process of the view strongly related rendering calculation is determined by the view strongly related rendering calculation tasks obtained by splitting the original rendering calculation task based on the preset correlation division rule;

[0007] Sending the camera control data to the server side, so that the server side performs view weakly related rendering calculation on the target scene based on the camera control data and returns the weakly related rendering result of the target scene. Among them, the process of the weakly related rendering calculation is determined by the view weakly related rendering calculation tasks obtained by splitting the original rendering calculation task based on the preset correlation division rule;

[0008] Mix the strongly correlated rendering result and the weakly correlated rendering result to obtain the collaborative rendering result of the target scene.

[0009] In one embodiment, the strongly correlated rendering result includes a direct lighting rendering result, and the weakly correlated rendering result includes a light probe index and texture data. The weakly correlated rendering result is returned from the server to the client at a synchronization frequency lower than the rendering frame rate of the direct lighting rendering result.

[0010] The step of mixing the strongly correlated rendering result and the weakly correlated rendering result to obtain the collaborative rendering result of the target scene includes:

[0011] Perform indirect lighting calculation based on the light probe index and the texture data to obtain the indirect lighting rendering result of the target scene.

[0012] Overlay the indirect lighting rendering result and the direct lighting rendering result to obtain the collaborative rendering result of the target scene at the rendering frame rate.

[0013] In one embodiment, the light probe index in the weakly correlated rendering result is compressed data compressed by the server, and the texture data in the weakly correlated rendering result is encoded data encoded by the server. The step of performing indirect lighting calculation based on the light probe index and the texture data includes:

[0014] Decompress the compressed data to obtain the decompressed data of the light probe index, and decode the encoded data to obtain the decoded data of the texture data.

[0015] Perform indirect lighting calculation based on the decompressed data and the decoded data.

[0016] In one embodiment, the indirect lighting rendering result is the target light probe index and target texture data in the current moment's light probe indices and texture data, where the change amount compared with the previous moment's historical light probe indices and historical texture data is greater than a preset threshold. The step of performing indirect lighting calculation based on the light probe index and the texture data includes:

[0017] Update the light probe index and texture data stored in the previous moment based on the target light probe index and target texture data, and perform indirect lighting calculation based on the updated light probe index and texture data.

[0018] In addition, to achieve the above object, the present application also proposes a collaborative rendering method for urban-level digital twins, which is applied to the server. The collaborative rendering method for urban-level digital twins includes:

[0019] Receive the camera control data sent by the client, and perform view weakly related rendering calculation on the target scene based on the camera control data to obtain the weakly related rendering result of the target scene. Wherein, the process of the weakly related rendering calculation is determined by the view weakly related rendering calculation tasks obtained by splitting the original rendering calculation task based on the preset correlation division rule;

[0020] Return the weakly related rendering result to the client, so that the client generates the collaborative rendering result of the target scene based on the weakly related rendering result and the strongly related rendering result obtained by calculation. Wherein, the calculation process of the strongly related rendering result is determined by the view strongly related rendering calculation tasks obtained by splitting the original rendering calculation task based on the preset correlation division rule.

[0021] In one embodiment, the weakly related rendering result includes a light probe index and texture data. The step of performing view weakly related rendering calculation on the target scene based on the camera control data to obtain the weakly related rendering result of the target scene includes:

[0022] Determine the scale information corresponding to the target scene in the camera control data, and determine the detail level of the light probe based on the scale information;

[0023] Activate the target light probes in each light probe in the preset frustum according to the detail level, and calculate the light information and depth information of the target light probe to obtain the light probe index and the texture data.

[0024] In one embodiment, after the step of obtaining the light probe index and the texture data includes:

[0025] Perform delta encoding compression on the light probe index to obtain the compressed data of the light probe index after compression, and perform video encoding on the texture data to obtain the encoded data of the texture data after encoding, so as to return the compressed data and the encoded data as the weakly related rendering result to the client.

[0026] In one embodiment, after the step of obtaining the weakly related rendering result of the target scene includes:

[0027] Save the weakly related rendering result, so that when the expected camera control data sent by any target client is within the preset data range of the camera control data, the weakly related rendering result is returned to the target client as the weakly related rendering result of the expected camera control data.

[0028] In addition, to achieve the above object, the present application further provides an electronic device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the collaborative rendering method for urban-level digital twins as described above.

[0029] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the steps of the collaborative rendering method for urban-level digital twins as described above.

[0030] In addition, to achieve the above object, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the collaborative rendering method for urban-level digital twins as described above.

[0031] One or more technical solutions provided by the present application have at least the following technical effects:

[0032] The present application first monitors the input camera control data, and performs view-strongly correlated rendering calculations on the target scene based on the camera control data to obtain the strongly correlated rendering result of the target scene, so as to ensure the real-time performance and smoothness of user interaction by locally executing the rendering calculation tasks closely related to the view on the client side; the camera control data is sent to the server side, so that the server side performs view-weakly correlated rendering calculations on the target scene based on the camera control data and returns the weakly correlated rendering result of the target scene, so as to assign the rendering calculation tasks with weak view correlation to the server side for execution, thereby using the powerful computing power of the server side to process complex rendering tasks, and at the same time reducing the pressure on the network bandwidth and improving the stability of network transmission by reducing the frequency and amount of transmitted data; the strongly correlated rendering result and the weakly correlated rendering result are mixed to obtain the collaborative rendering result of the target scene, so as to fuse the weakly correlated rendering result returned by the server side with the strongly correlated rendering result calculated locally on the client side, thereby combining the computing advantages of the client side and the server side and achieving a high-fidelity rendering effect.

[0033] In summary, in the present application, the original rendering calculation task is split into a view strongly related rendering calculation task and a view weakly related rendering calculation task according to a preset relevance division rule, and the view weakly related rendering calculation task is transferred to the server for processing, thus avoiding the problem that high-fidelity scene rendering cannot be performed due to limited hardware resources of the client. When the client resources are limited, through the collaborative work of the server and the client, the real-time performance and fluency of user interaction are ensured, and a high-fidelity rendering effect is achieved, thereby improving the performance of the overall rendering system and the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings incorporated herein and constituting a part of this specification illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 FIG.

[0037] Figure 2 is a schematic flowchart provided for Embodiment 1 of the collaborative rendering method for urban-level digital twins according to the present application;

[0038] Figure 3 FIG.

[0039] Figure 4 is a detailed level schematic diagram of the collaborative rendering method for urban-level digital twins provided for Embodiment 2 of the present application;

[0040] Figure 5 FIG.

[0041] Figure 6 is a schematic diagram of the device structure of the hardware operating environment involved in the collaborative rendering method for urban-level digital twins in the embodiments of the present application.

[0042] The implementation, functional features, and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of this application and are not used to limit this application.

[0044] To better understand the technical solutions of this application, the following will be described in detail in combination with the accompanying drawings of the specification and specific implementation manners.

[0045] The main solution of the embodiments of this application is as follows: monitor the input camera control data, and perform view strongly correlated rendering calculation on the target scene based on the camera control data to obtain the strongly correlated rendering result of the target scene. Among them, the process of the view strongly correlated rendering calculation is determined by the view strongly correlated rendering calculation tasks obtained by splitting the original rendering calculation task based on a preset correlation division rule; send the camera control data to the server to perform view weakly correlated rendering calculation on the target scene by the server based on the camera control data and return the weakly correlated rendering result of the target scene. Among them, the process of the weakly correlated rendering calculation is determined by the view weakly correlated rendering calculation tasks obtained by splitting the original rendering calculation task based on the preset correlation division rule; mix the strongly correlated rendering result and the weakly correlated rendering result to obtain the collaborative rendering result of the target scene.

[0046] Since the rendering of urban-level digital twin scenes at the current stage is mainly realized through cloud rendering or end rendering methods. Among them, cloud rendering refers to a method of running a rendering program on the server and pushing the rendering result to the client for display through a video stream. This method can obtain a highly realistic picture rendering effect on resource-constrained clients. However, this method of strong coupling between the server and the client and continuous streaming data transmission has high requirements for network bandwidth and stability. A low-bandwidth and unstable network environment will affect the transmission rate of the picture, resulting in frame freezing and latency, affecting the user's interaction experience; end rendering refers to running a rendering program on the client and displaying the rendering result. Since end rendering does not need to rely on external computing resources, it is suitable for high-concurrency business scenarios. However, the picture rendering effect of this method is limited by the client's hardware resources and it is difficult to support advanced rendering algorithms on low-configuration clients and cannot perform highly realistic picture rendering. Therefore, how to achieve a highly realistic picture rendering effect under the condition of limited client hardware resources is an urgent problem to be solved at present.

[0047] The present application provides a solution. By splitting the original rendering calculation tasks into view-strongly-related and view-weakly-related rendering calculation tasks according to a preset relevance division rule, and transferring the view-weakly-related rendering calculation tasks to the server side for processing, it avoids the problem that high-fidelity scene rendering cannot be performed due to limited hardware resources of the client side. It realizes that under the condition of limited resources of the client side, through the collaborative work of the server side and the client side, it not only ensures the real-time performance and fluency of user interaction, but also achieves a high-fidelity rendering effect, thus improving the performance of the overall rendering system and the user experience.

[0048] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of implementing the above functions. Hereinafter, taking an electronic device as an example, this embodiment and the following embodiments will be described.

[0049] Based on this, the embodiments of the present application provide a collaborative rendering method for urban-level digital twins. Referring to Figure 1 , Figure 1 is a schematic flowchart of the first embodiment of the collaborative rendering method for urban-level digital twins of the present application.

[0050] In this embodiment, the collaborative rendering method for urban-level digital twins is applied to the client side. The collaborative rendering method for urban-level digital twins includes steps S10 to S30:

[0051] Step S10, monitor the input camera control data, and perform view-strongly-related rendering calculation on the target scene based on the camera control data to obtain a strongly-related rendering result of the target scene. Among them, the process of the view-strongly-related rendering calculation is determined by the view-strongly-related rendering calculation tasks obtained by splitting the original rendering calculation tasks according to a preset relevance division rule;

[0052] It should be noted that camera control data refers to the data generated when the user operates the camera, such as panning, rotating, zooming, etc., which is used to determine the perspective and scope of the view; the target scene refers to the virtual environment or model to be rendered, which contains all information such as objects, textures, lighting, etc. in the scene; view strongly related rendering calculations refer to those rendering calculations that are very sensitive to changes in the user's perspective, such as direct lighting, etc., and these calculations need to be updated in real time to maintain the accuracy of the view; strongly related rendering results refer to the stage rendering results obtained after the client performs view strongly related rendering calculations, and this result directly affects the picture quality seen by the user; the preset correlation division rule refers to a set of standards or algorithms used to divide rendering tasks into strongly related and weakly related parts, and the division basis is set according to the degree of correlation between the rendering task and the final view rendering effect; the original rendering calculation task refers to the complete rendering process, including all necessary calculation steps, to generate the final rendered picture; the view strongly related rendering calculation task refers to the calculation task separated from the original rendering calculation task according to the preset correlation division rule and needs to be executed in real time on the client side.

[0053] It can be understood that since it is necessary to ensure the real-time nature of user interaction and the continuity of scene rendering, so step S10 is carried out. By locally executing the rendering calculation task closely related to the view on the client side, it can avoid the problems of rendering delay and degradation of user experience caused by network latency, achieve real-time response to user operations, and thus keep the scene rendering synchronized with the user's perspective.

[0054] Exemplarily, a sensor or input device is installed on the client side to capture the user's operation instructions in real time, such as mouse movement or keyboard input, and these instructions are converted into camera control data. The rendering engine of the client first updates the current scene state according to these data, and then performs view strongly related rendering calculations based on the updated scene state, including but not limited to frustum culling, direct lighting calculation, etc., to ensure that the scene elements under the user's perspective can be rendered in a timely and correct manner. This process screens out the part of the original rendering task that is closely related to the user's perspective according to the preset correlation division rule and executes it locally on the client side as the view strongly related rendering calculation task.

[0055] Step S20, sending the camera control data to the server side to perform view weakly related rendering calculations on the target scene based on the camera control data by the server side and returning the weakly related rendering result of the target scene, where the process of the weakly related rendering calculation is determined by the view weakly related rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule;

[0056] It should be noted that view-weakly related rendering calculations refer to those rendering calculations that are less sensitive to changes in the user's perspective, such as global illumination, ambient occlusion, etc. These calculations can be updated less frequently; the view-weakly related rendering result refers to the rendering result obtained after the server performs view-weakly related rendering calculations, and this result can supplement the high-quality details missing in the client-side rendering; the view-weakly related rendering calculation task refers to the calculation task separated from the original rendering calculation task according to the preset relevance division rules and suitable for execution on the server.

[0057] It can be understood that since the resources of the client are often very limited, it is usually difficult to support advanced rendering algorithms on low-configured clients, and thus high-fidelity scene rendering cannot be performed. Therefore, step S20 is carried out. By allocating the rendering calculation tasks with weak correlation to the view to the server for execution, the powerful computing power of the server can be utilized to process complex rendering tasks, which can avoid the problems of client resource overload and limited rendering quality, thereby improving the overall rendering effect of the system.

[0058] Exemplarily, the client transmits the camera control data to the server through the network. After receiving the data, the server performs view-weakly related rendering calculations according to the same relevance division rules. These calculations include background rendering, distant object detail rendering, light probe calculation, etc. These calculations are less sensitive to the direct feedback of the user's perspective and can therefore be performed on the server. After the calculations are completed, the server returns the rendering result to the client in the form of a data stream.

[0059] Step S30, mix the strongly related rendering result and the weakly related rendering result to obtain the collaborative rendering result of the target scene.

[0060] It should be noted that the collaborative rendering result refers to the final rendered image obtained by combining the strongly related rendering result and the weakly related rendering result, and this rendered image provides a high-quality visual effect.

[0061] It can be understood that in order to combine the computing advantages of the client and the server to provide a complete rendered image, step S30 is carried out. By fusing the weakly related rendering result returned by the server with the strongly related rendering result calculated locally on the client, the problem that a single rendering mode cannot meet the high-quality rendering requirements can be avoided, thereby realizing the combination of the real-time interaction ability of the client and the powerful computing ability of the server, and providing the effect of a high-quality and seamless collaborative rendering experience.

[0062] Exemplarily, after the client receives the weakly correlated rendering result sent back by the server, it synthesizes it with the strongly correlated rendering result executed locally. This synthesis process involves techniques such as image layer overlay, color blending, and resolution matching to ensure that the two rendering results can be seamlessly combined to form a complete, coherent, and highly realistic scene picture. Finally, what the user sees on the client is the result of collaborative rendering, which combines the real-time interactivity of the client and the computing power of the server, providing a high-quality visual experience.

[0063] In a feasible implementation manner, the strongly correlated rendering result includes a direct lighting rendering result, and the weakly correlated rendering result includes a lighting probe index and texture data. The weakly correlated rendering result is returned from the server to the client at a synchronization frequency lower than the rendering frame rate of the direct lighting rendering result.

[0064] It should be noted that the lighting probe index refers to the data index used to locate and reference the pre-calculated lighting information in the scene. This information is usually stored in the lighting probe and is used to simulate the global lighting effect in the scene; texture data refers to the information stored in the image file and is used to add details to the surface of the objects in the scene, such as visual effects like color, pattern, and bump; the rendering frame rate refers to the number of frames that the graphics rendering system can generate per second, and this number directly affects the smoothness and real-time nature of the rendered view; the synchronization frequency refers to the frequency at which the server synchronously returns the rendering result to the client.

[0065] Step S30 may include steps S31 - S32:

[0066] Step S31, perform indirect lighting calculation based on the lighting probe index and the texture data to obtain the indirect lighting rendering result of the target scene.

[0067] It should be noted that indirect lighting calculation refers to calculating the lighting effect after light rays are reflected and refracted multiple times in the scene. This lighting effect does not directly come from the light source but is transmitted through other surfaces in the scene.

[0068] It can be understood that, in order to simulate the global effect of light in the real world, and at the same time, since the rendering result of the view weakly correlated calculation task is less sensitive to real-time performance, performing step S31 can avoid the problems of unnatural lighting effects and lack of realism in scene rendering, thereby achieving synchronization at a frequency lower than the real-time rendering frame rate without affecting the visual experience, while providing a realistic global lighting effect and reducing the data transmission cost.

[0069] Exemplarily, first, a plurality of light probes are pre-placed in the scene, and these probes have calculated and stored the ambient light information during the preprocessing stage. During the actual rendering process, the client queries the corresponding light information according to the light probe index synchronized from the server. Then, using this light information and the texture data synchronized from the server, indirect light calculation is performed through specific light models and algorithms (such as spherical harmonics or environment mapping).

[0070] In a feasible implementation manner, the light probe index in the weakly correlated rendering result is compressed data compressed by the server, and the texture data in the weakly correlated rendering result is encoded data encoded by the server. The step of performing indirect light calculation based on the light probe index and the texture data in step S31 may include steps S311 to S312:

[0071] Step S311, decompress the compressed data to obtain the decompressed data of the light probe index, and decode the encoded data to obtain the decoded data of the texture data;

[0072] It should be noted that the decompressed data refers to the data obtained by decompressing the compressed data, and this data was the original light probe index data before being compressed by the server; the decoded data refers to the data obtained by decoding the encoded data, and this data was the original texture data before being encoded.

[0073] It can be understood that since the original light probe index data and texture data will occupy a large bandwidth during data transmission or storage, resulting in a high transmission cost, performing step S311 can avoid the problems of transmission delay caused by excessive data volume and increased storage cost, thus achieving efficient data transmission and storage and ensuring the rapid availability of data during the client rendering process.

[0074] Exemplarily, after the client receives the compressed data sent by the server, the client uses a dedicated decompression algorithm to decompress the compressed data. This algorithm can be a general compression algorithm such as DEFLATE or a customized algorithm specific to the light probe index to restore the original light probe index data. At the same time, the client uses a corresponding video decoder to decode the received encoded data, and these decoders can be H.265, HEVC, etc. to restore the high-definition image information of the texture data. The decompression and decoding processes are both performed on the hardware resources of the client to ensure the availability and real-time nature of the data.

[0075] Step S312, perform indirect light calculation based on the decompressed data and the decoded data.

[0076] For example, after the client has completed the decompression of the light probe index and the decoding of the texture data, it sends these data to the rendering engine for indirect lighting calculation, that is, according to the decompressed light probe index, the light probe at the corresponding position in the scene is found, and the properties of these light probes (such as position, direction, color, etc.) and the decoded texture data are used to calculate the indirect lighting effect of each pixel in the scene through lighting interpolation and texture mapping technology. This process may involve advanced rendering technologies such as environment mapping, light mapping, and shadow calculation, and finally generate high-quality indirect lighting rendering results, which are superimposed with the direct lighting rendering results to form a complete scene lighting effect.

[0077] In this implementation, by adopting data compression and decompression technology and video encoding and decoding technology in the data transmission process between the server and the client, network bandwidth pressure and delay problems caused by large-scale data transmission are avoided, and the rendering related data transmitted by the server can be efficiently received and utilized when the client hardware resources are limited. This not only reduces the network transmission burden and improves the data transmission efficiency, but also ensures that the client can calculate high-quality indirect lighting effects in real time, thereby achieving the technical effect of improving user experience without sacrificing picture quality.

[0078] Step S32, superimposing the indirect lighting rendering result and the direct lighting rendering result to obtain a collaborative rendering result of the target scene at a rendering frame rate.

[0079] It is understandable that since the client needs to display a relatively high frame rate rendering image to the user, step S32 is performed. By superimposing the low frame rate indirect lighting rendering result to the high frame rate direct lighting rendering result, the influence of the low frame rate indirect lighting rendering result on the final rendering result frame rate can be avoided, thereby achieving detailed and realistic lighting collaborative rendering results at the rendering frame rate, thereby improving the realism of the scene and the rendering visual quality.

[0080] Exemplarily, after the indirect lighting calculation is completed, the obtained indirect lighting rendering result is used as a layer of image data. At the same time, the direct lighting rendering result, that is, the lighting effect directly caused by the light source in the scene, is also used as another layer of image data. In the client rendering engine, these two layers of image data are merged through techniques such as color mixing and transparency overlay. This process occurs at each rendering frame rate, ensuring that the final output rendering picture contains both the bright and shadow effects of direct lighting, and the soft and environmental reflection effects of indirect lighting, thereby generating realistic and dynamic collaborative rendering results.

[0081] In this embodiment, by performing indirect illumination calculation based on illumination probe indices and texture data, and superimposing the indirect illumination rendering result obtained at the synchronous frame rate on the direct illumination rendering result, it avoids the problems of lack of realism and dynamic changes in scene rendering caused by insufficient illumination calculation, as well as high data transmission cost and high latency between the server and the client. Thus, it realizes providing a detailed and realistic global illumination collaborative rendering result at the rendering frame rate, while reducing the data transmission cost and transmission latency, improving the realism and visual quality of scene rendering, and ensuring the smoothness and stability of the rendering process.

[0082] This embodiment provides a collaborative rendering method for urban-level digital twins. By splitting the original rendering calculation task into view-strongly-related and view-weakly-related rendering calculation tasks according to a preset relevance division rule, and transferring the view-weakly-related rendering calculation tasks to the server for processing, it avoids the problem that high-fidelity rendering cannot be performed due to limited hardware resources of the client. It realizes that, under the condition of limited client resources, through the collaborative work of the server and the client, it not only ensures the real-time performance and smoothness of user interaction, but also achieves a high-fidelity rendering effect, thereby improving the performance of the overall rendering system and the user experience.

[0083] In a feasible embodiment, the indirect illumination rendering result is the target illumination probe indices and target texture data in the current moment's illumination probe indices and texture data, where the change amount compared with the previous moment's historical illumination probe indices and historical texture data is greater than a preset threshold. Step S31 may include step S310:

[0084] Step S310, based on the target illumination probe indices and target texture data, updates the illumination probe indices and texture data stored in the previous moment, and performs indirect illumination calculation based on the updated illumination probe indices and texture data.

[0085] It should be noted that the target illumination probe indices refer to a set of illumination probe index data required for the current rendering frame, which represents the illumination information with relatively large changes at specific positions in the scene and is used to update the indirect illumination effect on the object surface during the rendering process; the target texture data refers to the texture image data required for the current rendering frame, which contains the object surface details with relatively large changes, such as color, pattern, and texture information, and is used to update the detail performance of the rendering picture during the rendering process.

[0086] It can be understood that since the rendering results of view weakly related computing tasks have little impact on the display effect of the final rendered image, there is no need to fully synchronize the rendering results of this part from the server to the client for complex calculations. Therefore, step S310 is performed. By only processing the indirect illumination rendering results with relatively large changes transmitted from the server, it is possible to avoid performing computationally inefficient calculations on unnecessary indirect illumination rendering results, thereby achieving the ability to adapt to changes in light and texture in the scene while maintaining the quality and real-time performance of the rendered image, and further achieving a rendering effect that takes into account both low cost and high quality.

[0087] Exemplarily, after the server completes the light probe calculation, it will perform a differential calculation on the light probe calculation results (i.e., each historical light probe index and each historical texture data) activated by the corresponding client at the previous moment and the light probe results calculated at the current moment (i.e., each light probe index and each texture data at the current moment), and based on the comparison of the differential calculation results with a preset change threshold, further determine the light probe index and texture data (i.e., the target light probe index and target texture data) that need to be sent to the corresponding client. The client receives the target light probe index and target texture data from the server, and these data represent the light and texture information with relatively large changes in the current rendering frame. Then, the client accesses the light probe index and texture data at the previous moment in the storage module and compares them with the received target data to replace the corresponding old index in the storage module with the target light probe index to ensure that the rendering process in the client can reference the current light information with relatively large changes; secondly, update the texture data in the storage module, replacing the corresponding old texture image with the new target texture data to ensure that the rendering process in the client can obtain the current detailed information with relatively large changes. After completing the data update, the client uses these updated light probe index and texture data for indirect illumination calculation. This process may include performing light interpolation for each pixel point in the scene, combining ambient light, reflected light, and other lighting effects, calculating the final indirect illumination result, and applying it to the scene rendering, thereby achieving a real-time and high-quality lighting effect in a dynamically changing scene.

[0088] In this embodiment, by only performing data transmission and data processing on the intermediate results of indirect illumination rendering with relatively large changes, the problems of high data transmission cost and high data synchronization delay caused by transmitting based on the full amount of indirect illumination rendering results, and the problem of the client's computing burden caused by performing indirect illumination rendering based on the full amount of intermediate results of indirect illumination rendering are avoided. Thus, when the scene light and texture change, it is possible to update the rendering data in real time and calculate an accurate indirect illumination effect, thereby enhancing the dynamic authenticity of the rendered image and the smoothness of user interaction, achieving a rendering effect with high efficiency and high quality.

[0089] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 2 , in this embodiment, the collaborative rendering method for urban-level digital twins is applied to the client, and the collaborative rendering method for urban-level digital twins includes steps S01 to S02:

[0090] Step S01, receiving the camera control data sent by the client, and performing view weakly related rendering calculation on the target scene based on the camera control data to obtain the weakly related rendering result of the target scene, wherein the process of the weakly related rendering calculation is determined by the view weakly related rendering calculation tasks obtained by splitting the original rendering calculation task based on a preset correlation division rule;

[0091] It can be understood that since the resources of the client are often very limited, it is usually difficult to support advanced rendering algorithms on low-configured clients, and thus high-fidelity image rendering cannot be performed. Therefore, step S01 is performed. By assigning the rendering calculation tasks with weak correlation to the view to the server for execution, the powerful computing power of the server can be used to process complex rendering tasks, which can avoid the problems of client resource overload and limited rendering quality, thereby improving the overall rendering effect of the system.

[0092] Exemplarily, the server first establishes a communication connection with the client and receives the camera control data sent by the client through this connection. These data include information such as the position, direction, and focal length of the camera. The server then updates the scene state according to these camera control data, and then combines the preset correlation division rule to split the original rendering calculation task into multiple view weakly related rendering calculation tasks. These tasks may include indirect illumination rendering, etc., and this part of the tasks has less visual impact on the final view presentation. The server executes these weakly related rendering calculation tasks based on the updated scene state, generates the weakly related rendering result of the target scene, and caches and / or prepares to send these results back to the client.

[0093] In a feasible implementation manner, the weakly related rendering result includes a light probe index and texture data. The step of performing view weakly related rendering calculation on the target scene based on the camera control data in step S01 to obtain the weakly related rendering result of the target scene may include steps S011 to S012:

[0094] Step S011, determining the scale information corresponding to the target scene in the camera control data, and determining the detail level of the light probe based on the scale information;

[0095] It should be noted that the scale information refers to the size range of the target scene described in the camera control data, which can be information such as the size of objects in the scene, the depth range of the scene, or the area covered by the scene; the level of detail (LoD) refers to the fineness of the light probes determined according to the scale information of the scene, and this information determines the distribution density and computational complexity of the light probes in the scene.

[0096] It can be understood that in urban-level scenes, due to the large scale of the wide field of view space, a uniform high-density distribution of light probes will lead to a large amount of data calculation, which not only significantly increases the computational cost, but also reduces the data transmission cost between the server and the client, as well as the real-time rendering efficiency. Therefore, step S011 is performed to organize the light probes through LoD, with light probes uniformly sparsely distributed in the wide field of view and large scale, and densely distributed in the narrow field of view and small scale, so as to load and calculate light probes with different LoD and densities based on different perspectives, ensuring that the total number of light probes is equivalent in different perspective cases and stabilizing the computational load.

[0097] Exemplarily, the server analyzes the received camera control data, including the position, direction, and field of view range of the camera, etc., to calculate the scale information of the target scene. For example, the approximate size of the objects in the scene is determined by the position and field of view angle of the camera. Then, based on this scale information, the server applies a preset rule or algorithm to determine the level of detail of the light probes. For example, please refer to Figure 3 In the figure, the origin in the figure indicates the light probe, and the density of the light probe increases with the increase of LoD. That is, if the objects in the scene are small or the scene is far away, light probes with a lower level of detail are selected; conversely, if the objects are large or the scene is close, light probes with a higher level of detail are selected.

[0098] Step S012, activate the target light probes in each light probe in the preset frustum according to the level of detail, and calculate the light information and depth information of the target light probes to obtain the light probe index and the texture data.

[0099] It should be noted that the preset frustum is a three-dimensional space region centered on the camera and determined according to the field of view range and field of view angle of the camera, which defines which areas of the light probes need to be activated and calculated; the target light probe refers to the specific light probe that needs to be activated for light calculation in the preset frustum according to the current camera view angle and scene scale information; the light information refers to the lighting effects received by each point in the scene, including information such as the intensity, color, and direction of the light; the depth information refers to the distance information from each point in the scene to the camera viewpoint, and this information is used to determine the occlusion relationship and three-dimensional sense between objects.

[0100] It can be understood that in order to accurately simulate the lighting effect during the rendering process, step S012 is performed, which can avoid the problems of wasted computing resources and unnecessary lighting errors caused by uniformly calculating all light probes. By activating and calculating the target light probe, on-demand lighting calculation can be achieved, reducing the computing burden while ensuring the lighting authenticity and depth sense of the rendered scene, thus achieving the effects of improving the rendering efficiency and enhancing the scene realism.

[0101] Exemplarily, after determining the level of detail of the light probe, the server filters and activates the target light probes within the frustum of the current camera (i.e., the spatial area within the camera's field of view) at the selected level of detail. These activated light probes will be calculated on the server using the ray tracing algorithm. Each light probe projects the spherical surface onto a two-dimensional square pixel space (such as 18x18) in an octahedral projection manner to store the spherical distribution. Each pixel stores the lighting and depth information in the corresponding spherical direction. For example, the lighting information includes the lighting intensity, color, direction, etc. of the calculated probe position, which is stored at a resolution of 10x10, and the depth information is stored at a resolution of 18x18. These calculation results are organized into a light probe index and texture data. The light probe index is used to quickly find and reference the lighting information, while the texture data contains the lighting and depth information, which is used to provide a realistic lighting effect for the scene during the rendering process.

[0102] In this embodiment, by adopting the means of dynamically adjusting the level of detail of the light probe and on-demand activating and calculating the light probe, the problems of wasted computing resources and low rendering efficiency caused by uniformly processing all light probes during the rendering process are avoided. It realizes the intelligent allocation of computing resources according to the scene scale and camera perspective, improves the rendering efficiency, and at the same time ensures the authenticity of the lighting effect and the accuracy of the scene depth sense, thus achieving the effects of optimizing the rendering performance and enhancing the rendering quality.

[0103] In a feasible embodiment, after step S012, step S010 may further be included:

[0104] Step S010: Perform incremental coding compression on the light probe index to obtain the compressed data of the light probe index after compression, and perform video coding on the texture data to obtain the encoded data of the texture data after encoding, so as to return the compressed data and the encoded data as the weakly correlated rendering result to the client.

[0105] It can be understood that since it is necessary to perform efficient data compression and encoding on the rendering result on the server side to reduce the amount of data transmitted over the network, performing step S010 can avoid the problems of network bandwidth waste and transmission delay caused by the transmission of uncompressed or unencoded data. By compressing and encoding the light probe index and texture data, the optimization of data transmission can be achieved, the time for the client to receive data can be reduced, and the transmission efficiency of the rendering result can be improved, thereby achieving the effect of enhancing the performance of the overall rendering system.

[0106] Exemplarily, after the server side completes the calculation of the light information and depth information of the light probe, it first performs incremental encoding compression processing on the light probe index. This process includes identifying duplicate patterns and unnecessary information in the index and only recording the different parts from the previous index, thereby reducing the amount of data. Next, the server side uses video encoding technologies such as H.265 or HEVC to encode the texture data, which involves efficiently compressing the color information, light information, and depth information in the texture data to reduce the data size while maintaining the image quality as much as possible. After completing the compression and encoding, the server side packages the compressed light probe index data and the encoded texture data and sends them to the client through network transmission. After receiving these data, the client can decode and decompress them to restore the index and texture data of the light probe, and then combine them with the strongly related rendering result calculated locally to complete the final collaborative rendering. This implementation effectively reduces the amount of data transmitted, improves the transmission efficiency, and at the same time ensures the rendering quality.

[0107] In this implementation, by performing data compression and video encoding on the intermediate rendering result on the server side, the problems of excessive network bandwidth occupation and transmission delay caused by the transmission of a large amount of uncompressed intermediate rendering data are avoided, the amount of data transmitted is reduced, the network transmission efficiency is improved, and the speed of the client receiving and processing the rendering result is accelerated, thereby achieving the effect of optimizing the use of network resources and enhancing the rendering performance.

[0108] Step S02: Return the weakly related rendering result to the client so that the client can generate the collaborative rendering result of the target scene based on the weakly related rendering result and the calculated strongly related rendering result, where the calculation process of the strongly related rendering result is determined by the view strongly related rendering calculation tasks obtained by splitting the original rendering calculation task based on the preset correlation division rule.

[0109] It can be understood that, in order to combine the computing advantages of the client and the server to provide a complete rendering picture, step S02 is performed. By fusing the weakly related rendering results returned by the server with the strongly related rendering results calculated locally on the client, the problem that a single rendering mode cannot meet the high-quality rendering requirements can be avoided, thereby realizing the combination of the real-time interaction ability of the client and the powerful computing ability of the server, and providing the effect of a high-quality and seamless collaborative rendering experience.

[0110] Exemplarily, after the server-side completes the weakly related rendering calculation, it sends the rendering results back to the client through the communication network. Among them, the rendering results can be the final rendering results of indirect lighting or the intermediate rendering results of indirect lighting (i.e., light probe indices and texture data). After the client receives the weakly related rendering results, it synthesizes the weakly related rendering results returned by the server with the strongly related rendering results calculated locally to finally generate the complete collaborative rendering results of the target scene. This implementation allows the client and the server to each undertake a part of the rendering work, thereby realizing efficient distributed rendering.

[0111] In this embodiment, by splitting the original rendering calculation task into a view strongly related and a view weakly related rendering calculation task according to a preset relevance division rule, and transferring the view weakly related rendering calculation task to the server for processing, the problem that high-fidelity scene rendering cannot be performed due to limited hardware resources of the client is avoided. When the client's resources are limited, through the collaborative work of the server and the client, the real-time performance and smoothness of user interaction are ensured, and at the same time, the high-fidelity rendering effect is achieved, thereby improving the performance of the overall rendering system and the user experience.

[0112] In a feasible implementation manner, after step S01, step S100 may further be included:

[0113] Step S100: Save the weakly related rendering results, so that when the desired camera control data sent by any target client is within a preset data range from the camera control data, the weakly related rendering results are returned to the target client as the weakly related rendering results of the desired camera control data.

[0114] It should be noted that the target client refers to the client device that requests the rendering service, which may be a virtual reality user or any user terminal that requires remote rendering services; the expected camera control data refers to the data for controlling the virtual camera view sent by the target client in the future, including parameters such as the position, orientation, and focal length of the camera, which are used to determine the viewing perspective of the user in the virtual environment; the preset data range refers to a set of predefined parameter thresholds used to determine whether two different camera control data sets are similar enough to reuse the same weakly related rendering results; the weakly related rendering result of the expected camera control data refers to the rendering result selected from the saved weakly related rendering results and matching the expected camera control data of the target client.

[0115] It can be understood that in an urban scene, when users access the scene concurrently, there is generally no or little direct interaction with the urban scene that causes changes in scene lighting and models. Therefore, by performing step S100 and synchronously storing the view weakly related rendering results, when another client needs them in the future or the current client needs to render a similar target scene in the future, reusing the rendering results that have been calculated at the current moment can avoid the problem of unnecessary waste of computing resources caused by re-rendering the same or similar perspectives according to each camera control data request. This improves the response speed of the rendering service, reduces the server load, and optimizes the overall system resource utilization. In addition, in a high-concurrency scenario, the server-side view weakly related rendering results can be reused by multiple clients and multiple consecutive frames, thus achieving cross-terminal and cross-frame amortization of rendering tasks, and further reducing the overall computing cost of the rendering system.

[0116] Exemplarily, after the server compresses and encodes the light probe index and texture data, it stores these weakly related rendering results in a cache database and records the corresponding original camera control data at the same time. When receiving the expected camera control data from any target client, the server first compares these data with the camera control data recorded in the cache. If the two are within the preset data range (for example, the position deviation of the camera is less than a certain threshold, the orientation difference is less than a certain angle, etc.), the server will directly extract the corresponding weakly related rendering result from the cache and send it back to the target client as a response. In this way, the target client can reuse the stored rendering results to quickly update its view without having to perform a complete rendering calculation again.

[0117] In this embodiment, by caching and reusing the view weakly related rendering results, the problems of waste of computing resources and rendering delay caused by frequent recalculation of the same or similar scene rendering are avoided, achieving the effects of improving rendering efficiency, reducing server load, and accelerating client response speed.

[0118] Exemplarily, to facilitate understanding of the implementation process of the collaborative rendering method for urban-level digital twins obtained by combining this embodiment with the above-mentioned first embodiment, please refer to Figure 4 , Figure 4 A schematic diagram of a brief process of a collaborative rendering method for urban-level digital twins is provided. Specifically:

[0119] Figure 4 Two main branches of the rendering process are shown. The left branch represents the operations on the server side, and the right branch represents the operations on the client side. The process starts with an initialization phase, where both the server side and the client side need to perform some initial settings. Then both the server side and the client side obtain camera control data from their respective sources, which usually includes the position, orientation, and other relevant parameters of the camera. Among them, the data source of the client is the data input by the user, and the data source of the server is the user input data sent by the client. The server will update the scene state according to the camera control data and determine the level of detail of the light probe, which is to determine the accuracy of the light probe in order to more accurately simulate the performance of light in the scene. Among them, determining the level of detail includes constructing the light probe based on LoD and selecting the LoD light probe based on the view. Then, calculate the light probe based on ray tracing in the server side, update the light probe index and texture data index, compress the light probe index, and encode the light probe texture data to obtain the intermediate rendering result of the indirect light finally calculated by the server side, and then return this result to the client. After the client updates the scene state based on the camera control data input by the user, it performs direct light calculation based on the updated scene state, decompresses and decodes the rendering result returned by the server at the same time, and completes the remaining indirect light calculation. Finally, the direct light rendering result and the indirect light rendering result are superimposed in the client to obtain the collaborative rendering result of the target scene.

[0120] Furthermore, please refer to Figure 5 , which shows an improvement based on the existing urban-level end rendering architecture and cloud rendering architecture to obtain a high-concurrency end-cloud collaborative rendering architecture proposed in this application. In this architecture, direct light rendering calculation, post-processing, and data decoding processing can be performed in any client, while indirect light rendering calculation and data encoding processing can be performed on the server side. Then, the encoded weakly related rendering result is returned to the corresponding client, and post-processing is performed with the strongly related rendering result that has been calculated in the corresponding client to obtain the corresponding collaborative rendering result. Among them, the data service docked with the server provides high-precision model data and light data, and at the same time, the data service synchronizes the lightweight model data to each client.

[0121] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation to the collaborative rendering method for urban-level digital twins of the present application. Based on this technical concept, more forms of simple transformations are within the protection scope of the present application.

[0122] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the collaborative rendering method for urban-level digital twins in the first embodiment above.

[0123] Reference is made below to Figure 6 , which shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0124] As Figure 6As shown, the electronic device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the shown systems. Instead, more or fewer systems may be implemented or had.

[0125] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from the network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are executed.

[0126] The electronic device provided in the present application adopts the collaborative rendering method for urban-level digital twins in the above embodiments, and can solve the technical problem of how to achieve a highly realistic picture rendering effect when the client hardware resources are limited. Compared with the prior art, the beneficial effects of the electronic device provided in the present application are the same as those of the collaborative rendering method for urban-level digital twins provided in the above embodiments, and the other technical features in this electronic device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0127] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0128] As described above, the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all of them should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0129] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the collaborative rendering method for urban-level digital twins in the above embodiments.

[0130] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM: Random Access Memory), read-only memory (ROM: Read Only Memory), erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (RadioFrequency), etc., or any suitable combination of the above.

[0131] The above computer-readable storage medium can be included in an electronic device; or it can exist separately without being assembled into the electronic device.

[0132] The above computer-readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to: monitor input camera control data, and perform view strongly-correlated rendering calculations on a target scene based on the camera control data to obtain a strongly-correlated rendering result of the target scene, wherein the process of the view strongly-correlated rendering calculation is determined by a view strongly-correlated rendering calculation task obtained by splitting an original rendering calculation task based on a preset correlation division rule; send the camera control data to a server to perform view weakly-correlated rendering calculations on the target scene based on the camera control data by the server and return a weakly-correlated rendering result of the target scene, wherein the process of the weakly-correlated rendering calculation is determined by a view weakly-correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule; mix the strongly-correlated rendering result and the weakly-correlated rendering result to obtain a collaborative rendering result of the target scene; and / or receive camera control data sent by a client and perform view weakly-correlated rendering calculations on the target scene based on the camera control data to obtain a weakly-correlated rendering result of the target scene, wherein the process of the weakly-correlated rendering calculation is determined by a view weakly-correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule; return the weakly-correlated rendering result to the client to generate a collaborative rendering result of the target scene by the client based on the weakly-correlated rendering result and a calculated strongly-correlated rendering result, wherein the calculation process of the strongly-correlated rendering result is determined by a view strongly-correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule.

[0133] Computer program code for performing the operations of the present application may be written in one or more programming languages or combinations thereof. The above programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0135] The modules described in the embodiments of the present application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0136] The readable storage medium provided by the present application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned collaborative rendering method for urban-level digital twins, and can solve the technical problem of how to achieve a highly realistic picture rendering effect under the condition of limited client hardware resources. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the collaborative rendering method for urban-level digital twins provided by the above embodiments, and will not be elaborated here.

[0137] The present application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-mentioned collaborative rendering method for urban-level digital twins.

[0138] The computer program product provided by the present application can solve the technical problem of how to achieve a highly realistic picture rendering effect under the condition of limited client hardware resources. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the collaborative rendering method for urban-level digital twins provided by the above embodiments, and will not be elaborated here.

[0139] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made under the technical concept of the present application by using the content of the specification and drawings of the present application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A collaborative rendering method for urban-level digital twins, characterized in that, Applied to the client, the collaborative rendering method for urban-level digital twins includes: Monitoring the input camera control data, and performing view strongly correlated rendering calculations on the target scene based on the camera control data to obtain the strongly correlated rendering result of the target scene. Among them, the process of the view strongly correlated rendering calculation is determined by the view strongly correlated rendering calculation tasks obtained by splitting the original rendering calculation tasks based on a preset correlation division rule. The view strongly correlated rendering calculation tasks at least include frustum culling and direct lighting calculation, and the strongly correlated rendering result at least includes the direct lighting rendering result; Sending the camera control data to the server to perform view weakly correlated rendering calculations on the target scene based on the camera control data through the server and returning the weakly correlated rendering result of the target scene. Among them, the process of the weakly correlated rendering calculation is determined by the view weakly correlated rendering calculation tasks obtained by splitting the original rendering calculation tasks based on the preset correlation division rule. The weakly correlated rendering result includes the light probe index and texture data. The step of performing view weakly correlated rendering calculations on the target scene based on the camera control data includes: determining the scale information corresponding to the target scene in the camera control data, and determining the level of detail of the light probe based on the scale information, where the scale information is the size of the objects in the target scene, the depth range of the target scene, or the coverage range of the target scene; activating the first target light probes in each light probe in the preset frustum according to the level of detail, and calculating the lighting information and depth information of the first target light probes to obtain the light probe index and the texture data; after the server calculates the weakly correlated rendering result of the target scene, saving the weakly correlated rendering result so that when the expected camera control data sent by any target client is within the preset data range of the camera control data, returning the weakly correlated rendering result as the weakly correlated rendering result of the expected camera control data to the target client; Updating the light probe index and texture data stored at the previous moment based on the second target light probe index and target texture data, and performing indirect lighting calculations based on the updated light probe index and texture data to obtain the indirect lighting rendering result of the target scene. Among them, the indirect lighting rendering result is the second target light probe index and target texture data in the light probe indices and texture data at the current moment, whose change amount compared with the historical light probe indices and historical texture data at the previous moment is greater than the preset threshold, and the change amount is determined by performing differential calculations on the light probe indices and texture data at the current moment and the historical light probe indices and historical texture data at the previous moment; Overlaying the indirect lighting rendering result with the strongly correlated rendering result that at least includes the direct lighting rendering result to obtain the collaborative rendering result of the target scene at the rendering frame rate.

2. The collaborative rendering method for urban-level digital twins according to claim 1, wherein The lighting probe index and the texture data are returned from the server to the client at a synchronization frequency lower than the rendering frame rate of the direct lighting rendering result.

3. The collaborative rendering method for urban-level digital twins according to claim 1, wherein The lighting probe index in the weakly correlated rendering result is compressed data compressed by the server, and the texture data in the weakly correlated rendering result is encoded data encoded by the server.

4. A collaborative rendering method for urban-level digital twins, characterized in that, Applied to the server, the collaborative rendering method for urban-level digital twins includes: Receiving camera control data sent by the client, and performing view weakly correlated rendering calculation on the target scene based on the camera control data to obtain a weakly correlated rendering result of the target scene. Among them, the process of the weakly correlated rendering calculation is determined by a view weakly correlated rendering calculation task obtained by splitting the original rendering calculation task based on a preset correlation division rule. The weakly correlated rendering result includes a lighting probe index and texture data. The step of performing view weakly correlated rendering calculation on the target scene based on the camera control data to obtain a weakly correlated rendering result of the target scene includes: determining the scale information corresponding to the target scene in the camera control data, and determining the detail level of the lighting probe based on the scale information. Among them, the scale information is the size of the object in the target scene, the depth range of the target scene, or the coverage range of the target scene; activating the first target lighting probes in each lighting probe in the preset frustum according to the detail level, and calculating the lighting information and depth information of the first target lighting probes to obtain the lighting probe index and the texture data; Saving the weakly correlated rendering result so that when the desired camera control data sent by any target client is within a preset data range of the camera control data, the weakly correlated rendering result is returned to the target client as the weakly correlated rendering result of the desired camera control data; Return the weakly correlated rendering result to the client, so that the client can generate the collaborative rendering result of the target scene based on the weakly correlated rendering result and the strongly correlated rendering result calculated by the client. Among them, the calculation process of the strongly correlated rendering result is determined by the view strongly correlated rendering calculation tasks obtained by splitting the original rendering calculation task based on the preset correlation division rule. The view strongly correlated rendering calculation tasks at least include frustum culling and direct lighting calculation. The strongly correlated rendering result at least includes the direct lighting rendering result. The step of generating the collaborative rendering result of the target scene based on the weakly correlated rendering result and the calculated strongly correlated rendering result includes: updating the lighting probe index and texture data stored at the previous moment based on the second target lighting probe index and target texture data, and performing indirect lighting calculation based on the updated lighting probe index and texture data to obtain the indirect lighting rendering result of the target scene. Among them, the indirect lighting rendering result is the second target lighting probe index and target texture data in the lighting probe indices and texture data at the current moment, the change amount of which compared with the historical lighting probe indices and historical texture data at the previous moment is greater than the preset threshold. The change amount is determined by performing differential calculation on the lighting probe indices and texture data at the current moment and the historical lighting probe indices and historical texture data at the previous moment; superimpose the indirect lighting rendering result and the strongly correlated rendering result including at least the direct lighting rendering result to obtain the collaborative rendering result of the target scene at the rendering frame rate.

5. The collaborative rendering method for city-level digital twins according to claim 4, wherein After the step of obtaining the lighting probe index and the texture data, it includes: Perform incremental coding compression on the lighting probe index to obtain the compressed data of the lighting probe index after compression, and perform video coding on the texture data to obtain the encoded data of the texture data after encoding, so as to return the compressed data and the encoded data as the weakly correlated rendering result to the client.

6. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the collaborative rendering method for urban-level digital twins according to any one of claims 1 to 5.

7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the collaborative rendering method for urban-level digital twins according to any one of claims 1 to 5.

Images