Cooperative rendering method and device for city-level digital twinning and storage medium

Abstract

The invention discloses a collaborative rendering method and device for city-level digital twinning and a storage medium, and relates to the technical field of digital twinning, and the method comprises the steps: monitoring input camera control data, carrying out the view strong correlation rendering calculation of a target scene based on the camera control data, and obtaining a strong correlation rendering result of the target scene; sending the camera control data to a server, performing view weak correlation rendering calculation on the target scene based on the camera control data through the server, and returning a weak correlation rendering result of the target scene; and mixing the strong correlation rendering result and the weak correlation rendering result to obtain a collaborative rendering result of the target scene. According to the method and the device, the view weak correlation rendering calculation task is transferred to the server to be carried out, so that the real-time performance and smoothness of user interaction are ensured and a high-fidelity picture rendering effect is realized through the cooperative work of the server and the client under the condition that the client resources are limited.

Description

Technical Field

[0001] The present application relates to the field of digital twin technology, and in particular to a collaborative rendering method, device and storage medium for city-level digital twins. Background Art

[0002] At present, city-level digital twin scene rendering is mainly achieved through cloud rendering or end rendering. Cloud rendering refers to running the rendering program on the server and pushing the rendering results to the client for display through video streaming. This method can obtain highly realistic picture rendering effects on resource-constrained clients, but this strong coupling between the server and the client and the uninterrupted streaming data transmission method have high requirements on network bandwidth and stability. Low bandwidth and unstable network environment will affect the transmission rate of the picture, resulting in frame jams and delays, affecting the user's interactive experience; end rendering refers to running the rendering program on the client and displaying the rendering results. Since end rendering does not need to rely on external computing resources, it is suitable for high-concurrency business scenarios, but the picture rendering effect of this method is limited by the client hardware resources. It is difficult to support advanced rendering algorithms on low-configuration clients and cannot perform highly realistic picture rendering.

[0003] Therefore, how to achieve highly realistic image rendering effects when client hardware resources are limited is a problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of this application is to provide a collaborative rendering method, device and storage medium for city-level digital twins, aiming to solve the technical problem of how to achieve highly realistic picture rendering effects when client hardware resources are limited.

[0005] To achieve the above objectives, the present application proposes a collaborative rendering method for city-level digital twins, which is applied to a client. The collaborative rendering method for city-level digital twins includes: Monitor input camera control data, and perform view strong correlation rendering calculation on a target scene based on the camera control data to obtain a strong correlation rendering result of the target scene, wherein the process of the view strong correlation rendering calculation is determined by splitting the original rendering calculation task based on a preset correlation partitioning rule to obtain a view strong correlation rendering calculation task; The camera control data is sent to a server, so that the server performs view weak correlation rendering calculation on a target scene based on the camera control data, and returns a weak correlation rendering result of the target scene, wherein the weak correlation rendering calculation process is determined by the view weak correlation rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation partitioning rule; The strongly correlated rendering results and the weakly correlated rendering results are mixed to obtain a collaborative rendering result of the target scene.

[0006] In one embodiment, the strongly correlated rendering result includes a direct lighting rendering result, the weakly correlated rendering result includes a lighting probe index and texture data, and the weakly correlated rendering result is returned from the server to the client at a synchronization frequency lower than a rendering frame rate of the direct lighting rendering result; The step of mixing the strongly correlated rendering results and the weakly correlated rendering results to obtain the collaborative rendering result of the target scene includes: Performing indirect lighting calculation based on the lighting probe index and the texture data to obtain an indirect lighting rendering result of the target scene; The indirect lighting rendering result is superimposed with the direct lighting rendering result to obtain a collaborative rendering result of the target scene at a rendering frame rate.

[0007] In one embodiment, the illumination probe index in the weakly correlated rendering result is compressed data compressed by the server, the texture data in the weakly correlated rendering result is encoded data encoded by the server, and the step of performing indirect illumination calculation based on the illumination probe index and the texture data includes: Decompressing the compressed data to obtain decompressed data of the light probe index, and decoding the encoded data to obtain decoded data of the texture data; Indirect lighting calculation is performed based on the decompressed data and the decoded data.

[0008] In one embodiment, the indirect lighting rendering result is a target lighting probe index and target texture data whose change amount between each lighting probe index and each texture data at the current moment and each historical lighting probe index and each historical texture data at the previous moment is greater than a preset threshold, and the step of performing indirect lighting calculation based on the lighting probe index and the texture data includes: Based on the target light probe index and the target texture data, the light probe index and the texture data stored at the last moment are updated, and indirect lighting calculation is performed based on the updated light probe index and the texture data.

[0009] In addition, to achieve the above purpose, the present application also proposes a collaborative rendering method for city-level digital twins, which is applied to the server. The collaborative rendering method for city-level digital twins includes: Receive camera control data sent by the client, and perform view weak correlation rendering calculation on the target scene based on the camera control data to obtain a weak correlation rendering result of the target scene, wherein the process of the weak correlation rendering calculation is determined by the view weak correlation rendering calculation task obtained by splitting the original rendering calculation task based on a preset correlation partitioning rule; The weakly correlated rendering result is returned to the client, so that the client generates a collaborative rendering result of the target scene based on the weakly correlated rendering result and the calculated strongly correlated rendering result, wherein the calculation process of the strongly correlated rendering result is determined by the view strongly correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation partitioning rule.

[0010] In one embodiment, the weakly correlated rendering result includes a light probe index and texture data, and the step of performing view weakly correlated rendering calculation on the target scene based on the camera control data to obtain the weakly correlated rendering result of the target scene includes: Determining scale information corresponding to the target scene in the camera control data, and determining a level of detail of the light probe based on the scale information; According to the detail level, a target lighting probe in each lighting probe in a preset view frustum is activated, and lighting information and depth information of the target lighting probe are calculated to obtain the lighting probe index and the texture data.

[0011] In one embodiment, the step of obtaining the light probe index and the texture data comprises: The light probe index is incrementally encoded and compressed to obtain compressed data of the light probe index after compression, and the texture data is video encoded to obtain encoded data of the texture data after encoding, so as to return the compressed data and the encoded data as the weakly correlated rendering result to the client.

[0012] In one embodiment, the step of obtaining the weakly correlated rendering result of the target scene includes: The weakly correlated rendering result is saved so that when the expected camera control data sent by any target client is within a preset data range with the camera control data, the weakly correlated rendering result is returned to the target client as the weakly correlated rendering result of the expected camera control data.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the collaborative rendering method for city-level digital twins as described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the collaborative rendering method for city-level digital twins as described above are implemented.

[0015] In addition, to achieve the above-mentioned objectives, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the collaborative rendering method for city-level digital twins as described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: The present application first monitors the input camera control data, and performs view-strong correlation rendering calculation on the target scene based on the camera control data to obtain the strongly correlated rendering result of the target scene, so as to execute the rendering calculation task closely related to the view locally on the client, thereby ensuring the real-time and smoothness of user interaction; the camera control data is sent to the server, so as to perform view-weak correlation rendering calculation on the target scene based on the camera control data by the server, and return the weakly correlated rendering result of the target scene, so as to assign the rendering calculation task with weak view correlation to the server for execution, thereby utilizing the powerful computing power of the server to process complex rendering tasks, and at the same time reducing the frequency and amount of transmitted data, reducing the pressure on network bandwidth and improving the stability of network transmission; the strongly correlated rendering result and the weakly correlated rendering result are mixed to obtain the collaborative rendering result of the target scene, so as to combine the computing advantages of the client and the server by fusing the weakly correlated rendering result returned by the server with the locally calculated strongly correlated rendering result on the client, thereby achieving a high-fidelity rendering effect.

[0017] In summary, the present application avoids the problem of being unable to perform high-fidelity image rendering due to limited client hardware resources by splitting the original rendering calculation task into view-strongly correlated and view-weakly correlated rendering calculation tasks according to preset correlation division rules, and transferring the view-weakly correlated rendering calculation task to the server. This achieves real-time and smooth user interaction while achieving high-fidelity image rendering effects through the collaborative work of the server and the client when client resources are limited, thereby improving the performance of the overall rendering system and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart diagram of Embodiment 1 of the collaborative rendering method for city-level digital twins provided in this application; Figure 2 A flowchart diagram of Embodiment 2 of the collaborative rendering method for city-level digital twins provided in this application; Figure 3 A detailed hierarchical diagram of a collaborative rendering method for city-level digital twins provided in Example 2 of the present application; Figure 4 A brief flowchart of a collaborative rendering method for city-level digital twins provided in Example 2 of the present application; Figure 5 A schematic diagram of the system architecture of a collaborative rendering method for city-level digital twins provided in Example 2 of the present application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the collaborative rendering method for city-level digital twins in an embodiment of the present application.

[0021] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0023] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of the embodiment of the present application is: monitor the input camera control data, and perform view strongly correlated rendering calculation on the target scene based on the camera control data to obtain a strongly correlated rendering result of the target scene, wherein the process of the view strongly correlated rendering calculation is determined by the view strongly correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule; send the camera control data to the server, so that the server performs view weakly correlated rendering calculation on the target scene based on the camera control data, and return the weakly correlated rendering result of the target scene, wherein the process of the weakly correlated rendering calculation is determined by the view weakly correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule; mix the strongly correlated rendering result and the weakly correlated rendering result to obtain a collaborative rendering result of the target scene.

[0025] At present, the rendering of city-level digital twin scenes is mainly achieved through cloud rendering or end rendering. Among them, cloud rendering refers to a method of running a rendering program on the server and pushing the rendering results to the client for display through video streaming. This method can obtain highly realistic picture rendering effects on resource-constrained clients, but this method of strong coupling between the server and the client and uninterrupted streaming data transmission has high requirements for network bandwidth and stability. Low bandwidth and unstable network environment will affect the transmission rate of the picture, resulting in frame jams and delays, affecting the user's interactive experience; end rendering refers to running a rendering program on the client and displaying the rendering results. Since end rendering does not rely on external computing resources, it is suitable for high-concurrency business scenarios, but the picture rendering effect of this method is limited by the client hardware resources. It is difficult to support advanced rendering algorithms on low-configuration clients and cannot perform highly realistic picture rendering. Therefore, how to achieve highly realistic picture rendering effects when the client hardware resources are limited is a problem that needs to be solved urgently.

[0026] The present application provides a solution, which splits the original rendering calculation task into view-strongly correlated and view-weakly correlated rendering calculation tasks according to preset correlation division rules, and transfers the view-weakly correlated rendering calculation tasks to the server, thereby avoiding the problem of being unable to perform high-fidelity image rendering due to limited client hardware resources. When client resources are limited, the server and the client work together to ensure the real-time and smoothness of user interaction and achieve a high-fidelity image rendering effect, thereby improving the performance of the overall rendering system and user experience.

[0027] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions. The following takes an electronic device as an example to illustrate this embodiment and the following embodiments.

[0028] Based on this, the embodiment of the present application provides a collaborative rendering method for city-level digital twins, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the collaborative rendering method for city-level digital twins of the present application.

[0029] In this embodiment, the collaborative rendering method for city-level digital twins is applied to a client, and the collaborative rendering method for city-level digital twins includes steps S10 to S30: Step S10, monitoring the input camera control data, and performing view strong correlation rendering calculation on the target scene based on the camera control data to obtain a strong correlation rendering result of the target scene, wherein the process of the view strong correlation rendering calculation is determined by the view strong correlation rendering calculation task obtained by splitting the original rendering calculation task based on a preset correlation partitioning rule; It should be noted that camera control data refers to the data generated when the user operates the camera, such as translation, rotation, zoom, etc., which is used to determine the viewing angle and range of the view; the target scene refers to the virtual environment or model that needs to be rendered, which contains all objects, textures, lighting and other information in the scene; view-strongly correlated rendering calculations refer to those rendering calculations that are very sensitive to changes in user viewing angles, such as direct lighting, etc. These calculations need to be updated in real time to maintain the accuracy of the view; strongly correlated rendering results refer to the stage rendering results obtained after the client executes the view-strongly correlated rendering calculations, which directly affect the picture quality seen by the user; the preset correlation division rule refers to a set of standards or algorithms used to divide the rendering task into two parts, strongly correlated and weakly correlated, and the division is based on the degree of correlation between the rendering task and the final view presentation effect; the original rendering calculation task refers to the complete rendering process, including all necessary calculation steps to generate the final rendered picture; the view-strongly correlated rendering calculation task refers to the calculation task separated from the original rendering calculation task according to the preset correlation division rule and needs to be executed in real time on the client.

[0030] It is understandable that since it is necessary to ensure the real-time of user interaction and the continuity of scene rendering, step S10 is performed. By locally executing rendering calculation tasks closely related to the view on the client, rendering delays and user experience degradation caused by network delays can be avoided, and real-time response to user operations can be achieved, thereby keeping the scene rendering synchronized with the user's perspective.

[0031] Exemplarily, a sensor or input device is installed on the client to capture the user's operating instructions in real time, such as mouse movement or keyboard input, and these instructions are converted into camera control data. The client's rendering engine first updates the current scene state based on this data, and then performs view-strongly related rendering calculations based on the updated scene state, including but not limited to frustum clipping, direct lighting calculations, etc., to ensure that the scene elements from the user's perspective can be rendered in a timely and correct manner. This process filters out the parts of the original rendering task that are closely related to the user's perspective based on the preset correlation partitioning rules, and executes them locally on the client as view-strongly related rendering calculation tasks.

[0032] Step S20, sending the camera control data to the server, so that the server performs view weak correlation rendering calculation on the target scene based on the camera control data, and returns the weak correlation rendering result of the target scene, wherein the process of the weak correlation rendering calculation is determined by the view weak correlation rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule; It should be noted that view weakly-related rendering calculations refer to those rendering calculations that are not very sensitive to changes in user perspective, such as global illumination, ambient occlusion, etc. These calculations can be updated less frequently; weakly-related rendering results refer to the rendering results obtained after the server performs view weakly-related rendering calculations, which can supplement the high-quality details missing from the client rendering; view weakly-related rendering calculation tasks refer to calculation tasks separated from the original rendering calculation tasks according to preset correlation division rules, which are suitable for execution on the server.

[0033] It is understandable that since the resources of the client are often very limited, it is usually difficult for a low-configuration client to support advanced rendering algorithms, and thus it is impossible to perform highly realistic image rendering. Therefore, step S20 is performed, by assigning rendering calculation tasks that are less relevant to the view to the server for execution, thereby utilizing the powerful computing power of the server to process complex rendering tasks, thereby avoiding problems such as client resource overload and limited rendering quality, thereby improving the overall rendering effect of the system.

[0034] For example, the client transmits the camera control data to the server through the network. After receiving the data, the server performs weak view correlation rendering calculations according to the same correlation partitioning rules. These calculations include background rendering, distant object detail rendering, and light probe calculations. These calculations are not so sensitive to direct feedback from the user's perspective, so they can be performed on the server. After completing the calculation, the server sends the rendering results back to the client in the form of a data stream.

[0035] Step S30: Mix the strongly correlated rendering result and the weakly correlated rendering result to obtain a collaborative rendering result of the target scene.

[0036] It should be noted that the collaborative rendering result refers to the final rendering picture after combining the strongly correlated rendering results and the weakly correlated rendering results, and the rendering picture provides a high-quality visual effect.

[0037] It is understandable that in order to combine the computing advantages of the client and the server to provide a complete rendering picture, step S30 is performed. By fusing the weakly correlated rendering results returned by the server with the strongly correlated rendering results calculated locally on the client, the problem that a single rendering mode cannot meet the high-quality rendering requirements can be avoided, thereby achieving the effect of combining the real-time interactive capabilities of the client and the powerful computing capabilities of the server to provide a high-quality, seamless collaborative rendering experience.

[0038] For example, after receiving the weakly correlated rendering result sent back by the server, the client synthesizes it with the strongly correlated rendering result executed locally. This synthesis process involves techniques such as image layer overlay, color mixing, and resolution matching to ensure that the two rendering results can be seamlessly combined to form a complete, coherent, and highly realistic scene image. Ultimately, what the user sees on the client is the result of collaborative rendering, which combines the real-time interactivity of the client and the computing power of the server to provide a high-quality visual experience.

[0039] In a feasible implementation, the strongly correlated rendering result includes a direct lighting rendering result, the weakly correlated rendering result includes a lighting probe index and texture data, and the weakly correlated rendering result is returned from the server to the client at a synchronization frequency lower than a rendering frame rate of the direct lighting rendering result; It should be noted that the light probe index refers to the data index used to locate and reference pre-calculated lighting information in the scene. This information is usually stored in the light probe and is used to simulate the global lighting effect in the scene; texture data refers to the information stored in the image file, which is used to add details to the surface of objects in the scene, such as color, pattern, bumps and other visual effects; rendering frame rate refers to the number of frames that the graphics rendering system can generate per second, which directly affects the smoothness and real-time performance of the rendered view; synchronization frequency refers to the frequency at which the server synchronizes the rendering results back to the client.

[0040] Step S30 may include steps S31-S32: Step S31, performing indirect lighting calculation based on the lighting probe index and the texture data to obtain an indirect lighting rendering result of the target scene; It should be noted that indirect lighting calculation refers to calculating the lighting effect of light in the scene after multiple reflections and refractions. This lighting effect does not come directly from the light source, but is transmitted through other surfaces in the scene.

[0041] It can be understood that in order to simulate the global effect of light in the real world, and because the rendering results of view weakly related computing tasks are less sensitive to real-time performance, step S31 is performed to avoid the problem of unnatural lighting effects and lack of realism in scene rendering, thereby achieving synchronization at a frequency lower than the real-time rendering frame rate without affecting the visual experience, so as to provide realistic global lighting effects while reducing data transmission costs.

[0042] For example, first, multiple lighting probes are pre-placed in the scene. These probes have calculated and stored the ambient lighting information in the preprocessing stage. In the actual rendering process, the client queries the corresponding lighting information based on the lighting probe index synchronized from the server. Then, using this lighting information and the texture data synchronized from the server, indirect lighting calculation is performed through a specific lighting model and algorithm (such as spherical harmonics or environment map).

[0043] In a feasible implementation, the illumination probe index in the weakly correlated rendering result is compressed data compressed by the server, and the texture data in the weakly correlated rendering result is encoded data encoded by the server. The step of performing indirect illumination calculation based on the illumination probe index and the texture data in step S31 may include steps S311-S312: Step S311, decompressing the compressed data to obtain decompressed data of the illumination probe index, and decoding the encoded data to obtain decoded data of the texture data; It should be noted that decompressed data refers to data obtained by decompressing compressed data, which is the original lighting probe index data before compression on the server; decoded data refers to data obtained by decoding encoded data, which is the original texture data before encoding.

[0044] It is understandable that since the original lighting probe index data and texture data will occupy a large bandwidth during the data transmission or storage process, resulting in higher transmission costs, performing step S311 can avoid transmission delays caused by excessive data volume and increased storage costs, thereby achieving efficient data transmission and storage, and ensuring the rapid availability of data during client rendering.

[0045] Exemplarily, after the client receives the compressed data sent by the server, the client uses a dedicated decompression algorithm to decompress the compressed data, which can be a general compression algorithm such as DEFLATE or a customized algorithm specific to the light probe index, to restore the original light probe index data. At the same time, the client uses the corresponding video decoder to decode the received encoded data, which can be H.265, HEVC, etc., to restore the high-definition image information of the texture data. The decompression and decoding processes are all performed on the client's hardware resources to ensure the availability and real-time nature of the data.

[0046] Step S312: performing indirect lighting calculation based on the decompressed data and the decoded data.

[0047] For example, after the client has completed the decompression of the light probe index and the decoding of the texture data, it sends these data to the rendering engine for indirect lighting calculation, that is, according to the decompressed light probe index, the light probe at the corresponding position in the scene is found, and the properties of these light probes (such as position, direction, color, etc.) and the decoded texture data are used to calculate the indirect lighting effect of each pixel in the scene through lighting interpolation and texture mapping technology. This process may involve advanced rendering technologies such as environment mapping, light mapping, and shadow calculation, and finally generate high-quality indirect lighting rendering results, which are superimposed with the direct lighting rendering results to form a complete scene lighting effect.

[0048] In this implementation, by adopting data compression and decompression technology and video encoding and decoding technology in the data transmission process between the server and the client, network bandwidth pressure and delay problems caused by large-scale data transmission are avoided, and the rendering related data transmitted by the server can be efficiently received and utilized when the client hardware resources are limited. This not only reduces the network transmission burden and improves the data transmission efficiency, but also ensures that the client can calculate high-quality indirect lighting effects in real time, thereby achieving the technical effect of improving user experience without sacrificing picture quality.

[0049] Step S32, superimposing the indirect lighting rendering result and the direct lighting rendering result to obtain a collaborative rendering result of the target scene at a rendering frame rate.

[0050] It is understandable that since the client needs to display a relatively high frame rate rendering image to the user, step S32 is performed. By superimposing the low frame rate indirect lighting rendering result to the high frame rate direct lighting rendering result, the influence of the low frame rate indirect lighting rendering result on the final rendering result frame rate can be avoided, thereby achieving detailed and realistic lighting collaborative rendering results at the rendering frame rate, thereby improving the realism of the scene and the rendering visual quality.

[0051] Exemplarily, after the indirect lighting calculation is completed, the obtained indirect lighting rendering result is used as a layer of image data. At the same time, the direct lighting rendering result, that is, the lighting effect directly caused by the light source in the scene, is also used as another layer of image data. In the client rendering engine, these two layers of image data are merged through techniques such as color mixing and transparency overlay. This process occurs at each rendering frame rate, ensuring that the final output rendering picture contains both the bright and shadow effects of direct lighting, and the soft and environmental reflection effects of indirect lighting, thereby generating realistic and dynamic collaborative rendering results.

[0052] In this implementation, indirect lighting calculation is performed based on the lighting probe index and texture data, and the indirect lighting rendering results obtained at the synchronized frame rate are superimposed with the direct lighting rendering results, thereby avoiding the lack of realism and dynamic changes in scene rendering due to insufficient lighting calculation, as well as high data transmission costs and high delays between the server and the client. This enables detailed and realistic global lighting collaborative rendering results to be provided at the rendering frame rate, while reducing data transmission costs and transmission delays, improving the realism and visual quality of scene rendering, and ensuring the smoothness and stability of the rendering process.

[0053] This embodiment provides a collaborative rendering method for city-level digital twins, which splits the original rendering calculation task into view-strongly correlated and view-weakly correlated rendering calculation tasks according to preset correlation division rules, and transfers the view-weakly correlated rendering calculation task to the server. This avoids the problem of being unable to perform high-fidelity image rendering due to limited client hardware resources, and achieves real-time and smooth user interaction and high-fidelity image rendering effects through the collaborative work of the server and the client when client resources are limited, thereby improving the performance of the overall rendering system and user experience.

[0054] In a feasible implementation, the indirect lighting rendering result is a target lighting probe index and target texture data whose change amount between each lighting probe index and each texture data at the current moment and each historical lighting probe index and each historical texture data at the previous moment is greater than a preset threshold, and step S31 may include step S310: Step S310: Based on the target lighting probe index and target texture data, the lighting probe index and texture data stored at the last moment are updated, and indirect lighting calculation is performed based on the updated lighting probe index and texture data.

[0055] It should be noted that the target light probe index refers to a set of light probe index data required for the current rendering frame. This data represents the lighting information with large changes in a specific position in the scene, and is used to update the indirect lighting effect on the object surface during the rendering process; the target texture data refers to the texture image data required for the current rendering frame. This data contains the surface details of objects with large changes, such as color, pattern, texture and other information, and is used to update the detailed performance of the rendering screen during the rendering process.

[0056] It can be understood that since the rendering results of the view weakly related computing tasks have little effect on the display effect of the final rendered picture, there is no need to synchronize the rendering results of this part from the server to the client for complex calculations, so step S310 is performed. By only processing the indirect lighting rendering results with large changes transmitted from the server, it is possible to avoid unnecessary indirect lighting rendering results. The calculation process with low efficiency and cost can be avoided, thereby achieving the ability to adapt to changes in lighting and texture in the scene while maintaining the quality and real-time performance of the rendered picture, thereby achieving a rendering effect that takes into account both low cost and high quality.

[0057] Exemplarily, after completing the calculation of the light probe, the server will perform a differential calculation on the light probe calculation result activated by the corresponding client at the previous moment (i.e., each historical light probe index and each historical texture data) and the light probe result calculated at the current moment (i.e., each light probe index and each texture data at the current moment), and then determine the light probe index and texture data (i.e., target light probe index and target texture data) that need to be sent to the corresponding client based on the difference calculation result and the preset change threshold. The client receives the target light probe index and target texture data from the server, which represent the light and texture information with large changes in the current rendering frame. Next, the client accesses the light probe index and texture data of the previous moment in the storage module, and compares it with the received target data to replace the corresponding old index in the storage module with the target light probe index, so as to ensure that the rendering process in the client can reference the current light information with large changes; secondly, the texture data in the storage module is updated, and the corresponding old texture image is replaced with the new target texture data, so as to ensure that the rendering process in the client can obtain the current detail information with large changes. After the data is updated, the client uses these updated light probe indices and texture data to perform indirect lighting calculations. This process may include lighting interpolation for each pixel in the scene, combining ambient light, reflected light and other lighting effects to calculate the final indirect lighting result, and apply it to scene rendering, thereby achieving real-time, high-quality lighting effects in dynamically changing scenes.

[0058] In the present implementation, by only transmitting and processing data for the intermediate results of indirect lighting rendering with a large amount of changes, the problems of high data transmission cost and high data synchronization delay caused by transmitting based on the full amount of indirect lighting rendering results, as well as the problem of client computing burden caused by indirect lighting rendering based on the full amount of indirect lighting rendering intermediate results are avoided. As a result, when the scene lighting and texture change, the rendering data can be updated in real time and accurate indirect lighting effects can be calculated, thereby improving the dynamic realism of the rendered image and the smoothness of user interaction, and achieving high-efficiency and high-quality rendering effects.

[0059] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 In this embodiment, the collaborative rendering method for city-level digital twins is applied to a client, and the collaborative rendering method for city-level digital twins includes steps S01-S02: Step S01, receiving camera control data sent by the client, and performing view weak correlation rendering calculation on the target scene based on the camera control data to obtain a weak correlation rendering result of the target scene, wherein the process of the weak correlation rendering calculation is determined by the view weak correlation rendering calculation task obtained by splitting the original rendering calculation task based on a preset correlation partitioning rule; It is understandable that since the resources of the client are often very limited, it is usually difficult for a low-configuration client to support advanced rendering algorithms, and thus it is impossible to perform highly realistic image rendering. Therefore, step S01 is performed to assign rendering calculation tasks that are less relevant to the view to the server for execution, thereby utilizing the powerful computing power of the server to process complex rendering tasks, thereby avoiding problems such as client resource overload and limited rendering quality, thereby improving the overall rendering effect of the system.

[0060] Exemplarily, the server first establishes a communication connection with the client and receives the camera control data sent by the client through the connection, which includes the camera's position, direction, focal length and other information. The server then updates the scene state based on the camera control data, and then, in combination with the preset correlation partitioning rules, splits the original rendering calculation task into multiple view weakly correlated rendering calculation tasks, which may include indirect lighting rendering, etc., which have little visual impact on the final presentation of the view. The server executes these weakly correlated rendering calculation tasks based on the updated scene state, generates weakly correlated rendering results of the target scene, and caches and / or prepares to send these results back to the client.

[0061] In a feasible implementation manner, the weakly correlated rendering result includes a light probe index and texture data. The step of performing view weakly correlated rendering calculation on the target scene based on the camera control data in step S01 to obtain the weakly correlated rendering result of the target scene may include steps S011 to S012: Step S011, determining scale information corresponding to the target scene in the camera control data, and determining a detail level of the light probe based on the scale information; It should be noted that scale information refers to the size range of the target scene described in the camera control data. This information can be the size of objects in the scene, the depth range of the scene, or the area covered by the scene. Level of Detail (LoD) refers to the degree of refinement of the lighting probe determined based on the scale information of the scene. This information determines the distribution density and computational complexity of the lighting probe in the scene.

[0062] It is understandable that in city-level scenes, the wide field of view has a large spatial scale, and the high-density uniform distribution of light probes will lead to a large amount of data calculation, which will significantly increase the computing cost while reducing the cost of data transmission between the server and the client, as well as the real-time efficiency of rendering. Therefore, step S011 is performed to organize the light probes by LoD, and the wide field of view and large scale light probes are evenly and sparsely distributed, while the narrow field of view and small scale light probes are densely distributed, so as to load and calculate light probes of different LoDs and densities based on different viewing angles, thereby ensuring that the total amount of light probes is equivalent under different viewing angles and stabilizing the computing load.

[0063] For example, the server analyzes the received camera control data, including the camera's position, direction, and field of view, to calculate the scale information of the target scene, such as determining the approximate size of objects in the scene by the camera's position and field of view. Then, the server applies a preset rule or algorithm to determine the detail level of the light probe based on this scale information. For example, please refer to Figure 3 The origin in the figure indicates the light probe, and the light probe density increases with the increase of LoD. That is, if the object in the scene is small or the scene is far away, the light probe with a lower detail level is selected; conversely, if the object is large or the scene is close, the light probe with a higher detail level is selected.

[0064] Step S012, activating a target lighting probe in each lighting probe in a preset view frustum according to the detail level, and calculating lighting information and depth information of the target lighting probe to obtain the lighting probe index and the texture data.

[0065] It should be noted that the preset view frustum refers to a three-dimensional space area centered on the camera, determined according to the camera's field of view and field of view angle, which defines which areas the lighting probes need to be activated and calculated; the target light probe refers to a specific light probe that needs to be activated for lighting calculation within the preset view frustum according to the current camera viewing angle and scene scale information; lighting information refers to the lighting effects on each point in the scene, including light intensity, color, direction and other information; depth information refers to the distance information from each point in the scene to the camera's viewpoint, which is used to determine the occlusion relationship between objects and the three-dimensional sense of space.

[0066] It can be understood that in order to accurately simulate the lighting effect during the rendering process, step S012 is performed to avoid the waste of computing resources and unnecessary lighting errors caused by the unified calculation of all lighting probes. By activating and calculating the target lighting probes, lighting can be calculated on demand to reduce the computational burden while ensuring the lighting realism and depth of the rendered scene, thereby achieving the effect of improving rendering efficiency and enhancing the realism of the scene.

[0067] Exemplarily, after determining the detail level of the light probe, the server screens and activates the target light probes in the selected detail level within the view frustum of the current camera (i.e., the spatial area within the camera's field of view). These activated light probes will be calculated on the server using a ray tracing algorithm, and each light probe maps the sphere in a two-dimensional square pixel space (such as 18x18) in an octahedral projection to store the spherical distribution. Each pixel stores the lighting and depth information in the corresponding spherical direction, for example: the lighting information includes information such as the lighting intensity, color, and direction of the calculated probe position, which is stored at a resolution of 10x10, and the depth information is stored at a resolution of 18x18. These calculation results are organized into light probe indexes and texture data. The light probe index is used to quickly find and reference lighting information, while the texture data contains lighting and depth information to provide realistic lighting effects for the scene during rendering.

[0068] In this implementation, by dynamically adjusting the detail level of light probes and activating computational light probes on demand, the waste of computing resources and low rendering efficiency caused by unified processing of all light probes during the rendering process are avoided, and computing resources are intelligently allocated according to the scene scale and camera perspective, thereby improving rendering efficiency and ensuring the authenticity of the lighting effect and the accuracy of the scene's sense of depth, thereby achieving the effect of optimizing rendering performance and improving rendering quality.

[0069] In a feasible implementation manner, after step S012, step S010 may also be included: Step S010, incrementally encode and compress the light probe index to obtain compressed data of the light probe index after compression, and perform video encoding on the texture data to obtain encoded data of the texture data after encoding, so as to return the compressed data and the encoded data as the weakly correlated rendering result to the client.

[0070] It can be understood that since it is necessary to perform efficient data compression and encoding on the server side for the rendering results in order to reduce the amount of data transmitted over the network, performing step S010 can avoid the problems of network bandwidth waste and transmission delay caused by data transmission that has not been efficiently compressed or encoded. By compressing and encoding the light probe index and texture data, data transmission can be optimized, reducing the time it takes for the client to receive data and improving the transmission efficiency of the rendering results, thereby achieving the effect of improving the overall rendering system performance.

[0071] Exemplarily, after the server completes the calculation of the illumination information and depth information of the illumination probe, it first performs incremental encoding compression on the illumination probe index. This process includes identifying repeated patterns and non-essential information in the index, and only recording the difference with the previous index, thereby reducing the amount of data. Next, the server uses video coding technology, such as H.265 or HEVC, to encode the texture data, which involves efficient compression of the color information, illumination information, and depth information in the texture data to reduce the data size while maintaining the image quality as much as possible. After completing the compression and encoding, the server packages the compressed illumination probe index data and the encoded texture data and sends them to the client via network transmission. After receiving these data, the client can decode and decompress them to restore the index and texture data of the illumination probe, and then combine them with the strongly correlated rendering results of the local calculation to complete the final collaborative rendering. This implementation method effectively reduces the amount of data transmission, improves transmission efficiency, and ensures rendering quality.

[0072] In this implementation, by performing data compression and video encoding on the intermediate rendering results on the server side, the problems of excessive network bandwidth occupancy and transmission delay caused by the transmission of a large amount of uncompressed intermediate rendering data are avoided, thereby reducing the amount of data transmitted, improving network transmission efficiency, and accelerating the speed at which the client receives and processes rendering results, thereby achieving the effect of optimizing the use of network resources and improving rendering performance.

[0073] Step S02, returning the weakly correlated rendering result to the client, so that the client generates a collaborative rendering result of the target scene based on the weakly correlated rendering result and the calculated strongly correlated rendering result, wherein the calculation process of the strongly correlated rendering result is determined by the view strongly correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule.

[0074] It is understandable that in order to combine the computing advantages of the client and the server to provide a complete rendering picture, step S02 is performed. By fusing the weakly correlated rendering results returned by the server with the strongly correlated rendering results calculated locally on the client, the problem that a single rendering mode cannot meet the high-quality rendering requirements can be avoided, thereby achieving the effect of combining the real-time interactive capabilities of the client and the powerful computing capabilities of the server to provide a high-quality, seamless collaborative rendering experience.

[0075] Exemplarily, after completing the weakly correlated rendering calculation, the server sends the rendering result back to the client through the communication network, wherein the rendering result can be the final rendering result of indirect lighting, or the intermediate rendering result of indirect lighting (i.e., the lighting probe index and texture data). After receiving the weakly correlated rendering result, the client synthesizes the weakly correlated rendering result returned by the server with the strongly correlated rendering result obtained by local calculation, and finally generates a complete collaborative rendering result of the target scene. This implementation allows the client and server to each undertake part of the rendering work, thereby achieving efficient distributed rendering.

[0076] In this embodiment, the original rendering calculation task is split into view-strongly correlated and view-weakly correlated rendering calculation tasks according to preset correlation division rules, and the view-weakly correlated rendering calculation tasks are transferred to the server for execution. This avoids the problem of being unable to perform high-fidelity image rendering due to limited client hardware resources. In the case of limited client resources, the server and the client work together to ensure the real-time and smoothness of user interaction and achieve a high-fidelity image rendering effect, thereby improving the performance of the overall rendering system and user experience.

[0077] In a feasible implementation manner, after step S01, step S100 may also be included: Step S100, saving the weakly correlated rendering result, so that when the expected camera control data sent by any target client is within a preset data range with the camera control data, the weakly correlated rendering result is returned to the target client as the weakly correlated rendering result of the expected camera control data.

[0078] It should be noted that the target client refers to the client device requesting the rendering service, which may be a virtual reality user or any user terminal that requires remote rendering services; the expected camera control data refers to the data sent by the target client in the future for controlling the virtual camera viewing angle, which includes parameters such as the camera's position, direction, focal length, etc., which are used to determine the user's viewing angle in the virtual environment; the preset data range refers to a set of parameter thresholds defined in advance, which are used to determine whether two different camera control data sets are similar enough to reuse the same weakly correlated rendering results; the weakly correlated rendering result of the expected camera control data refers to a rendering result selected from the saved weakly correlated rendering results based on the expected camera control data of the target client and matching it.

[0079] It is understandable that in an urban scene, when users concurrently access the scene, they generally do not or rarely interact directly with the urban scene, which causes changes in scene lighting and models. Therefore, step S100 is performed. By synchronously storing the weakly correlated rendering results of the view, when another client needs to render a similar target scene in the future or the current client needs to render the similar target scene in the future, the rendering results that have been calculated at the current moment are reused. This can avoid the problem of unnecessary waste of computing resources caused by re-rendering the same or similar perspectives according to each camera control data request, thereby improving the response speed of the rendering service, reducing the server load, and optimizing the resource utilization of the overall system. In addition, in high-concurrency scenarios, the weakly correlated rendering results of the server-side view can be reused by multiple clients and multiple consecutive frames, thereby realizing cross-terminal and cross-frame amortization of rendering tasks, thereby reducing the overall computing cost of the rendering system.

[0080] Exemplarily, after the server completes the compression and encoding of the light probe index and texture data, it stores these weakly correlated rendering results in a cache database and records the corresponding original camera control data. When receiving the expected camera control data from any target client, the server first compares these data with the camera control data recorded in the cache. If both are within the preset data range (for example, the camera position deviation is less than a certain threshold, the direction difference is less than a certain angle, etc.), the server directly extracts the corresponding weakly correlated rendering results from the cache and sends them back to the target client as a response. In this way, the target client can reuse the stored rendering results to quickly update its view without having to re-perform the complete rendering calculation.

[0081] In this implementation, by caching and reusing weakly correlated rendering results of views, the problem of waste of computing resources and rendering delays caused by frequent recalculation of the same or similar scene renderings is avoided, thereby achieving the effect of improving rendering efficiency, reducing server load, and accelerating client response speed.

[0082] For example, to help understand the implementation process of the collaborative rendering method for city-level digital twins obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 4 , Figure 4 A brief flowchart of a collaborative rendering method for city-level digital twins is provided. Specifically: Figure 4 The two main branches of the rendering process are shown. The left branch represents the operation of the server side, and the right branch represents the operation of the client side. The process starts with the initialization phase. Both the server and the client need to perform some initial settings. Then both the server and the client obtain camera control data from their respective sources, which usually includes the position, direction and other related parameters of the camera. The data source of the client is the user input data, while the data source of the server is the user input data sent by the client. The server will update the scene state based on the camera control data and determine the level of detail of the light probe. This is to determine the accuracy of the light probe so as to more accurately simulate the performance of light in the scene. The determination of the level of detail includes constructing the light probe based on LoD and selecting the LoD light probe based on the view. Next, the server calculates the light probe based on ray tracing, updates the light probe index and texture data index, compresses the light probe index, encodes the light probe texture data, and obtains the intermediate rendering result of the indirect lighting finally calculated by the server, and then returns the result to the client. After the client updates the scene state based on the camera control data input by the user, it performs direct lighting calculations based on the updated scene state, decompresses and decodes the rendering results returned by the server, and completes the remaining indirect lighting calculations. Finally, the client superimposes the direct lighting rendering results and the indirect lighting rendering results to obtain the coordinated rendering results of the target scene.

[0083] For further information, please refer to Figure 5 , the figure shows that improvements are made based on the existing city-level end rendering architecture and cloud rendering architecture, thereby obtaining a city-level high-concurrency end-cloud collaborative rendering architecture proposed in the present application. In this architecture, direct lighting rendering calculation, post-processing and data decoding processing can be performed in any client, while indirect lighting rendering calculation and data encoding processing can be performed on the server, and then the encoded weakly correlated rendering result is returned to the corresponding client, so that the corresponding client and the already calculated strongly correlated rendering result are post-processed to obtain the corresponding collaborative rendering result, wherein the data service connected to the server provides high-precision model data and lighting data, and the data service will synchronize lightweight model data to each client.

[0084] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the collaborative rendering method of the present application for city-level digital twins. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0085] The present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the collaborative rendering method for city-level digital twins in the above-mentioned embodiment one.

[0086] Reference below Figure 6 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0087] like Figure 6As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0088] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0089] The electronic device provided by this application adopts the collaborative rendering method for city-level digital twins in the above-mentioned embodiment, which can solve the technical problem of how to achieve highly realistic picture rendering effects when the client hardware resources are limited. Compared with the prior art, the beneficial effects of the electronic device provided by this application are the same as the beneficial effects of the collaborative rendering method for city-level digital twins provided in the above-mentioned embodiment, and the other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0090] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0091] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0092] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the collaborative rendering method for city-level digital twins in the above-mentioned embodiment.

[0093] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0094] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0095] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device: monitors the input camera control data, and performs view strongly correlated rendering calculation on the target scene based on the camera control data to obtain a strongly correlated rendering result of the target scene, wherein the process of the view strongly correlated rendering calculation is determined by the view strongly correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule; sends the camera control data to the server, so that the server performs view weakly correlated rendering calculation on the target scene based on the camera control data, and returns the weakly correlated rendering result of the target scene, wherein the process of the weakly correlated rendering calculation is determined by the view weakly correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule. Determine; mix the strongly correlated rendering result and the weakly correlated rendering result to obtain a collaborative rendering result of the target scene; and / or receive camera control data sent by the client, and perform view weakly correlated rendering calculation on the target scene based on the camera control data to obtain a weakly correlated rendering result of the target scene, wherein the process of the weakly correlated rendering calculation is determined by the view weakly correlated rendering calculation task obtained by splitting the original rendering calculation task based on a preset correlation division rule; return the weakly correlated rendering result to the client, so that the client can generate a collaborative rendering result of the target scene based on the weakly correlated rendering result and the calculated strongly correlated rendering result, wherein the calculation process of the strongly correlated rendering result is determined by the view strongly correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation division rule.

[0096] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0099] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned collaborative rendering method for city-level digital twins, and can solve the technical problem of how to achieve highly realistic picture rendering effects when the client hardware resources are limited. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the collaborative rendering method for city-level digital twins provided in the above-mentioned embodiments, and will not be repeated here.

[0100] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the collaborative rendering method for city-level digital twins as described above.

[0101] The computer program product provided by this application can solve the technical problem of how to achieve a highly realistic picture rendering effect when the client hardware resources are limited. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the collaborative rendering method for city-level digital twins provided in the above embodiment, which will not be repeated here.

[0102] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A collaborative rendering method for city-level digital twins, characterized in that: Applied to the client, the collaborative rendering method for city-level digital twins includes: Monitor input camera control data, and perform view strong correlation rendering calculation on a target scene based on the camera control data to obtain a strong correlation rendering result of the target scene, wherein the process of the view strong correlation rendering calculation is determined by splitting the original rendering calculation task based on a preset correlation partitioning rule to obtain a view strong correlation rendering calculation task; The camera control data is sent to a server, so that the server performs view weak correlation rendering calculation on a target scene based on the camera control data, and returns a weak correlation rendering result of the target scene, wherein the weak correlation rendering calculation process is determined by the view weak correlation rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation partitioning rule; The strongly correlated rendering results and the weakly correlated rendering results are mixed to obtain a collaborative rendering result of the target scene.

2. The collaborative rendering method for city-level digital twins according to claim 1, characterized in that: The strongly correlated rendering result includes a direct lighting rendering result, the weakly correlated rendering result includes a lighting probe index and texture data, and the weakly correlated rendering result is returned from the server to the client at a synchronization frequency lower than a rendering frame rate of the direct lighting rendering result; The step of mixing the strongly correlated rendering results and the weakly correlated rendering results to obtain the collaborative rendering result of the target scene includes: Performing indirect lighting calculation based on the lighting probe index and the texture data to obtain an indirect lighting rendering result of the target scene; The indirect lighting rendering result is superimposed with the direct lighting rendering result to obtain a collaborative rendering result of the target scene at a rendering frame rate.

3. The collaborative rendering method for city-level digital twins according to claim 2, characterized in that: The illumination probe index in the weakly correlated rendering result is compressed data compressed by the server, and the texture data in the weakly correlated rendering result is encoded data encoded by the server. The step of performing indirect illumination calculation based on the illumination probe index and the texture data includes: Decompressing the compressed data to obtain decompressed data of the light probe index, and decoding the encoded data to obtain decoded data of the texture data; Indirect lighting calculation is performed based on the decompressed data and the decoded data.

4. The collaborative rendering method for city-level digital twins according to claim 2, characterized in that: The indirect lighting rendering result is a target lighting probe index and target texture data whose change amount between each lighting probe index and each texture data at the current moment and each historical lighting probe index and each historical texture data at the previous moment is greater than a preset threshold, and the step of performing indirect lighting calculation based on the lighting probe index and the texture data includes: Based on the target light probe index and the target texture data, the light probe index and the texture data stored at the last moment are updated, and indirect lighting calculation is performed based on the updated light probe index and the texture data.

5. A collaborative rendering method for city-level digital twins, characterized in that: Applied to the server, the collaborative rendering method for city-level digital twins includes: Receive camera control data sent by the client, and perform view weak correlation rendering calculation on the target scene based on the camera control data to obtain a weak correlation rendering result of the target scene, wherein the process of the weak correlation rendering calculation is determined by the view weak correlation rendering calculation task obtained by splitting the original rendering calculation task based on a preset correlation partitioning rule; The weakly correlated rendering result is returned to the client, so that the client generates a collaborative rendering result of the target scene based on the weakly correlated rendering result and the calculated strongly correlated rendering result, wherein the calculation process of the strongly correlated rendering result is determined by the view strongly correlated rendering calculation task obtained by splitting the original rendering calculation task based on the preset correlation partitioning rule.

6. The collaborative rendering method for city-level digital twins according to claim 5, characterized in that: The weakly correlated rendering result includes a light probe index and texture data, and the step of performing view weakly correlated rendering calculation on the target scene based on the camera control data to obtain the weakly correlated rendering result of the target scene includes: Determining scale information corresponding to the target scene in the camera control data, and determining a level of detail of the light probe based on the scale information; According to the detail level, a target lighting probe in each lighting probe in a preset view frustum is activated, and lighting information and depth information of the target lighting probe are calculated to obtain the lighting probe index and the texture data.

7. The collaborative rendering method for city-level digital twins according to claim 6, characterized in that: The step of obtaining the light probe index and the texture data comprises: The light probe index is incrementally encoded and compressed to obtain compressed data of the light probe index after compression, and the texture data is video encoded to obtain encoded data of the texture data after encoding, so as to return the compressed data and the encoded data as the weakly correlated rendering result to the client.

8. The collaborative rendering method for city-level digital twins according to claim 5, characterized in that: The step of obtaining the weakly correlated rendering result of the target scene includes: The weakly correlated rendering result is saved so that when the expected camera control data sent by any target client is within a preset data range with the camera control data, the weakly correlated rendering result is returned to the target client as the weakly correlated rendering result of the expected camera control data.

9. An electronic device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the collaborative rendering method for city-level digital twins as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the collaborative rendering method for city-level digital twins as described in any one of claims 1 to 8 are implemented.

Images